Multimodal error information detection method and device

Abstract

This application discloses a multimodal error information detection method and apparatus. The method includes: constructing a search tree containing a root node and multiple first-level child nodes based on multimodal input content; configuring an extensible toolset, using the toolset as the action space in the search process, and ensuring that the tools in the toolset have a preset adaptation relationship with each subtask; determining the node to be expanded based on a node selection mechanism, and selecting a tool from the toolset that is compatible with the node to be expanded to perform expansion and simulation operations; evaluating the expanded node based on inference path scoring and result confidence; transmitting the evaluation result backward along the search path to the root node, and updating the statistical information of each node on the path; and, after completing a preset number of search loops, determining and deciding on error information based on the generated comprehensive verification result. This method can adapt to verification needs from various error sources, improving the detection effect.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and apparatus for detecting multimodal error information. Background Technology

[0002] In the field of multimodal error detection, early research and datasets focused primarily on single-modality error detection (such as text-only or image-only). With technological advancements, multimodal error detection has become a research trend, but challenges remain, especially for tasks involving cross-modal correlations and mixed-source multimodal error detection. These tasks require verification of content from different modalities and potential error sources (such as image content errors, text description errors, and image-text mismatch errors). Only through comprehensive analysis of multi-source and mixed evidence can a reliable determination of whether the information is erroneous.

[0003] However, current automatic multimodal error detection methods still have significant limitations when dealing with complex error types and diverse scenarios: On the one hand, due to the diverse sources of errors (such as image errors, text description errors, etc.), different detection tools need to be customized for different scenarios. However, existing methods based on large language models (LLM) usually rely on fixed and limited toolsets, lacking flexibility and scalability, and are difficult to adapt to the verification needs of various error sources in the detection of multimodal error information from mixed sources.

[0004] On the other hand, real-world error messages often originate from a mixture of various error types, requiring detection systems to not only deeply search for evidence of each error type but also dynamically explore multiple possible error types. However, most existing methods employ fixed verification processes, making it difficult to strike a balance between deeply searching for evidence of a single error type and dynamically exploring multiple potential error types. This limits their detection performance in complex scenarios such as detecting multimodal error messages from mixed sources. Summary of the Invention

[0005] Therefore, it is necessary to provide a multimodal error information detection method and apparatus to address the above-mentioned technical problems and solve at least one of the problems existing in the prior art.

[0006] Firstly, a multimodal error information detection method is provided, including: Based on the multimodal input content, a search tree containing a root node and multiple first-level child nodes is constructed, where each first-level child node corresponds to a subtask of a different error source of the multimodal input content; Configure an extensible toolset, which serves as the action space during the search process, and the tools in the toolset have a preset adaptation relationship with each subtask; The node to be expanded is determined based on the node selection mechanism, and the tool that is compatible with the node to be expanded is selected from the toolset to perform the expansion and simulation operations. The expanded nodes are evaluated based on inference path scores and result confidence. The evaluation results are passed back to the root node along the search path, and the statistical information of each node on the path is updated. After completing a preset number of search cycles, a comprehensive verification result is generated, and error information is determined and decisions are made based on the comprehensive verification result.

[0007] In one possible implementation, before determining the node to be expanded based on the node selection mechanism, the following steps are included: Each subtask is assigned a probability score based on the multimodal input content. The probability score is used to reflect the relevance and importance of the corresponding subtask in verifying the authenticity of the information. Based on the probability score, the first node to be expanded is determined.

[0008] In one possible implementation, the evaluation of the expanded nodes based on inference path scores and result confidence includes: Obtain the set of all state-action pairs from the initial state to the current state, and analyze them from multiple dimensions to obtain the inference path score; Observations are obtained by calling tools, and conclusions to be verified are formed based on these observations; The reliability of the observed values ​​and the consistency between the conclusion to be verified and the observed values ​​are analyzed to quantify the credibility of the evidence chain and obtain the confidence level of the result. By combining the inference path score and the result confidence level, a comprehensive evaluation value for the expanded node is formed.

[0009] In one possible implementation, the node selection mechanism for determining the nodes to be expanded includes: The value of terminal nodes is determined based on inference path scores and result confidence. Based on the value of the terminal node and the number of visits to the parent and child nodes, an upper confidence bound algorithm is constructed. The selection index of each candidate node is calculated based on the upper confidence bound algorithm. The candidate nodes include nodes to be explored and nodes that have been explored. The selection index of the node to be explored is dynamically updated according to the increase of the number of visits to the corresponding parent node. The node with the highest selection index is selected from the candidate nodes as the node to be expanded.

[0010] In one possible implementation, after selecting a tool from the toolset that is compatible with the node to be extended to perform the extension and simulation operations, the method further includes: Obtain the verification results returned by the node to be expanded; If the confidence level of the verification result is greater than the preset confidence threshold, it is determined that the subtask corresponding to the node to be expanded has been completed, and the expansion of the node to be expanded is stopped.

[0011] In one possible implementation, the determination and decision-making based on the comprehensive verification results includes: Calculate the error probability for each subtask based on the confidence score corresponding to each subtask. Aggregate the error probabilities of each subtask to calculate the overall true probability value of the information content; Based on the error probability of each subtask and the overall true probability, it is determined whether there are errors in the multimodal input content and the comprehensive evaluation result of the errors.

[0012] In one possible implementation, the method further includes: A Gaussian process model is constructed and trained using a training dataset, and the performance of various candidate tool combinations is predicted using the Gaussian process model. The expected improvement criterion is used as the acquisition function, and the candidate tool combinations are evaluated based on the performance prediction results; Based on the expected improvement criteria and evaluation results, the tool combination with the highest adaptability to the sub-task requirements is selected and applied to the actual sub-task, and its real performance data is recorded. The real performance data is added to the training dataset, and the Gaussian process model is iteratively updated to dynamically optimize the fit between the tool and the subtask.

[0013] In one possible implementation, constructing and training the Gaussian process model using a training dataset includes: The training dataset is generated based on initial data containing several tool subsets and their corresponding scores; Based on the training dataset, an initial Gaussian process model is constructed and trained, wherein a squared exponential kernel is selected as the kernel function to characterize the smoothness and correlation of the tool combination performance; The hyperparameters of the kernel function and the variance of the observation noise are determined by maximizing the marginal likelihood. During the training process, the gradient descent method is used to calculate the gradient of the marginal likelihood with respect to the hyperparameters and adjust the hyperparameter values. After multiple iterations until the marginal likelihood converges, the trained Gaussian process model is obtained.

[0014] In one possible implementation, the toolset includes at least one of web search tools, pose detection tools, human keypoint detection tools, expert knowledge tools, image understanding tools, and error detection tools.

[0015] Secondly, a multimodal error information detection device is provided, characterized in that the device comprises: The search tree construction unit is used to construct a search tree containing a root node and multiple first-level child nodes based on multimodal input content, wherein each first-level child node corresponds to a subtask of a different error source of the multimodal input content; Toolset configuration unit, used to configure an extensible toolset, using the toolset as the action space in the search process, and the tools in the toolset have a preset adaptation relationship with each subtask; The node to be expanded determination unit is used to determine the node to be expanded based on the node selection mechanism, and select a tool from the toolset that is compatible with the node to be expanded to perform expansion and simulation operations. The evaluation unit is used to evaluate the expanded nodes based on the inference path score and the result confidence. The backpropagation unit is used to propagate the evaluation results back along the search path to the root node and update the statistical information of each node on the path. The verification unit is used to generate a comprehensive verification result after completing a preset number of search cycles, and to determine and make decisions on error information based on the comprehensive verification result.

[0016] The aforementioned multimodal error information detection method and apparatus include the following steps: First, constructing a search tree containing a root node and multiple first-level child nodes based on multimodal input content, wherein each first-level child node corresponds to a subtask representing a different error source of the multimodal input content. Second, configuring an extensible toolset, using the toolset as the action space during the search process, with each tool in the toolset having a preset adaptation relationship with each subtask. Third, determining the node to be expanded based on a node selection mechanism, and selecting a tool from the toolset that is compatible with the node to be expanded to perform expansion and simulation operations. Fourth, evaluating the expanded node based on inference path scoring and result confidence. Fifth, transmitting the evaluation result backward along the search path to the root node, updating the statistical information of each node on the path. Sixth, after completing a preset number of search loops, generating a comprehensive verification result, and determining and deciding on error information based on the comprehensive verification result. In this embodiment, by introducing an extensible toolset and a dynamic reasoning mechanism based on Monte Carlo Tree Search (MCTS), the system can flexibly adapt and dynamically expand the tools to different error source subtasks, addressing the mixed-source characteristics of multimodal error information. This breaks the limitations of traditional fixed toolsets on diverse scenarios. Simultaneously, leveraging the node selection, expansion, and evaluation logic of MCTS, the system can achieve an intelligent balance between in-depth mining of evidence for a single error type and dynamic exploration of multiple potential error types, avoiding the limitations of fixed verification processes. By backpropagating the evaluation results to optimize node statistics and generating comprehensive verification results through multiple rounds of search, the system not only improves the comprehensive detection capability and reliability of mixed-source errors but also enhances its adaptability to complex scenarios through the extensibility of the toolset and the dynamism of the search mechanism, effectively improving the efficiency and accuracy of multimodal error information detection. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of an application environment for a multimodal error information detection method according to an embodiment of this application; Figure 2 This is a flowchart illustrating a multimodal error information detection method according to an embodiment of this application; Figure 3 This is a schematic diagram of a multimodal error information detection device according to an embodiment of this application; Figure 4 This is a schematic diagram of a computer device according to one embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The multimodal error information detection method provided in this embodiment can be applied to, for example, Figure 1 In this application environment, a multi-source verification mechanism is introduced based on the Multi-source Verification Monte Carlo Tree (MCTS) to address the potential mixed error sources in multimodal error information. A novel evaluation function and a task-adaptive backpropagation strategy are designed, working together to achieve a balance between exploration and exploitation during the search process. A decision-making strategy is proposed, which integrates various types of evidence by coordinating multi-source judgment results to make a final judgment on whether an error has occurred. In addition to the search process, the framework of this invention is equipped with an extensible toolset that supports plug-and-play integration of specialized tools. This allows the system to dynamically select appropriate tools according to task requirements when facing various types of multimodal error information, achieving flexible and efficient information verification.

[0021] Specifically, a search tree is constructed, with the root node representing the overall task and multiple first-level child nodes introduced. Each child node corresponds to a specific subtask, representing different error sources, such as text verification, image verification, and consistency verification, each corresponding to different error sources (text errors, image errors, or cross-modal inconsistencies). Bayesian optimization techniques can be used to select the most relevant subset of tools for different tasks, improving efficiency. Tools in the toolset, such as web search, expert knowledge base, error detection, and image understanding, perform expansion and simulation operations. The process includes planning, action, and evaluation stages, carried out in the retrieval order. Evaluation results are updated with node statistics through backpropagation. After multiple rounds of search iterations, decisions are made based on the answers generated from each path, thereby achieving comprehensive, dynamic, and accurate detection and judgment of multimodal error information. This application improves the scalability of the toolset by constructing a toolset based on standardized template descriptions. By introducing a Bayesian optimization-based tool selection mechanism, flexible adaptation detection for multiple types of errors is achieved. The Monte Carlo search tree is extended into an inference framework supporting multi-source verification, decomposing the multi-source error detection task into multiple subtasks and performing dynamic verification among these subtasks. To guide the search process, a scoring evaluation function was designed. By combining the inference path quality score and the result confidence score, it enables the effective use of high-confidence evidence and the exploration of unknown error sources. This achieves adaptive evidence collection and comprehensive judgment for multiple error sources, improving the robustness and generalization ability of the detection.

[0022] It should be noted that Monte Carlo Tree Search (MCTS) is a heuristic search algorithm that excels at solving complex decision problems. Its core mechanism involves iteratively building a search tree and selecting the optimal action based on the results of numerous simulations.

[0023] In one embodiment, such as Figure 2 As shown, a multimodal error information detection method is provided, including the following steps: In step S110, a search tree containing a root node and multiple first-level child nodes is constructed based on the multimodal input content, wherein each first-level child node corresponds to a subtask of a different error source of the multimodal input content; Optionally, the error sources of the multimodal input content are determined by combining common types of multimodal error information with the specific modal characteristics of the object. Taking multimodal input content combining text and images as an example, it covers not only single-modal errors such as inaccurate text descriptions and logical contradictions, and image content tampering and false generation, but also intermodal consistency errors such as mismatch between text and image meanings. Furthermore, it is further refined and supplemented based on the context of the object (e.g., for news reports, errors related to news elements need to be considered). Based on this, a search tree is constructed using Monte Carlo tree search (Multi-source Verification MCTS). The root node represents the overall error detection task for the entire multimodal input content. From the root node, first-level child nodes are derived, corresponding to different error sources such as text, images, and consistency. Each first-level child node corresponds to a subtask for a different error source in the multimodal input content, such as text description errors, image content errors, and text-image mismatch errors. Subsequently, based on each first-level child node and combined with an extensible toolset (including web search, expert knowledge base, etc.), further expansion operations are performed on each first-level child node to generate the next-level node (e.g., under the "text verification" child node, expand to include child nodes for retrieving text-related information using the "web search" tool and comparing text professional content using the "expert knowledge base" tool), and so on, gradually constructing a search tree containing multiple layers of nodes that can cover different error sources and verification paths, providing structural support for subsequent dynamic reasoning and error detection. It should be noted that the entire reasoning process can use a large visual language model (LVLM) as the controller. At each reasoning node, the LVLM first generates a thought process and corresponding action plan based on the current state and previous reasoning paths. Once an action is selected, the system will call the corresponding tools from its extensible toolset to retrieve external evidence or perform specific analyses.

[0024] It should be noted that, in addition to the Monte Carlo tree search method described above for dynamic verification, reinforcement learning can also be used for dynamic path planning. Specifically, the following elements can be defined first: state (such as current verification information, search tree node position, etc.), action (such as choosing a tool or subtask branch), reward (the effect of the feedback action on error detection), and environment (information to be verified, toolset, etc.). Then, a suitable reinforcement learning algorithm is selected, such as Q-Learning (value-based), policy gradient (policy-based), or Actor-Critic (combining both). After that, the model is trained by initializing and allowing the agent to interact with the environment in a loop, updating the policy or value function. After training, the learned policy is applied to new information to be verified, enabling the agent to dynamically select verification actions based on the current state and efficiently plan the path to detect error information.

[0025] The Large Vision-Language Model (LVLM) is a multimodal processing model trained on a large-scale text-visual pairing dataset using deep learning techniques. It possesses powerful natural language understanding and generation capabilities, enabling it to perform tasks such as question answering, summarizing, translating, and creating content. By extending the ability to perceive visual signals to a large language model, LVLM inherits the cognitive abilities and certain reasoning capabilities of large language models for the open world. For multimodal input content such as images and text, and video-text, LVLM can simultaneously analyze visual signals (e.g., image tampering traces, video frame anomalies) and linguistic information (e.g., textual factual errors, logical contradictions), breaking the limitations of single-modal models. Furthermore, it can combine its own understanding of the open world to initially identify error types (e.g., whether the image shows generation traces, whether the text description matches the image content), providing initial judgment criteria for subsequent search tree construction (e.g., first-level child nodes corresponding to "text errors," "image errors," and "cross-modal consistency errors"). Meanwhile, at each inference node of the search tree, LVLM acts as an "intelligent controller": based on the current verification status (e.g., text verification completed, image authenticity to be verified) and historical inference paths (e.g., online search tools have been used to confirm textual facts), it generates a structured thought process (e.g., "currently, we need to prioritize verifying whether the image has been tampered with, because the textual facts have been preliminarily confirmed but there is a contradiction between the text and the image") and outputs corresponding action plans (e.g., "call image tampering detection tools + expert knowledge base tools to cross-verify image authenticity"), providing decision guidance for tool calls and node expansion. On the one hand, LVLM can automatically plan the verification order based on the general features of multimodal inputs (e.g., news content needs to verify time, location, and other elements) (e.g., "verify textual facts first, then check the consistency between the image and the text"), improving detection efficiency; on the other hand, it can combine manually customized scenario-based rules (e.g., in the medical field, the accuracy of medical terminology needs to be verified first) to correct the automatic planning results, ensuring accurate coverage of industry-specific error types (e.g., medical image annotation errors).

[0026] Furthermore, LLM Agent is an autonomous system built upon large language models or multimodal large models. LVLM's understanding and reasoning capabilities are transformed into end-to-end error detection actions. LVLM provides core capabilities in multimodal understanding, reasoning, and plan generation. LLM Agent transforms LVLM's reasoning results into autonomous actions (environmental perception, tool invocation, and dynamic decision-making). Its goal is not limited to simple language generation, but rather to possess the ability to autonomously perceive, understand, think, plan, make decisions, and execute complex tasks in specific environments. Unlike traditional AI systems that rely on single-modal data or are limited by rule-based logic, the core of LLM Agent lies in significantly enhancing autonomy, enabling it to complete tasks independently, thereby significantly reducing the need for human intervention and related costs. Leveraging LLM's reasoning and generation capabilities, and integrating the advantages of tool use, multi-step planning, and reasoning, LLM Agent can adjust its behavior in real time according to the external environment and preset goals, completing tasks ranging from simple to complex in complex and ever-changing environments, achieving dynamic decision-making and continuous adaptation based on environmental feedback.

[0027] It should be noted that large-scale visual language models can be used to detect errors in the input content, while task rules can be manually customized. For example, error types (such as "factual errors," "logical contradictions," and "image tampering") can be identified, and verification paths (i.e., the order in which to verify what) can be planned. The "automatic planning" of the large model improves efficiency, while the "manual rules" ensure accurate coverage of specific scenarios (such as industry-specific error types).

[0028] In step S120, an extensible toolset is configured, which is used as the action space in the search process, and the tools in the toolset have a preset adaptation relationship with each subtask; Optionally, a configurable extensible toolset refers to a collection of various detection tools that can be flexibly added or removed (such as text verification tools, image source tracing tools, cross-modal comparison tools, etc.). This toolset is set as an action space available during the search process, meaning that when exploring different error source subtasks, the system can select appropriate tools from the toolset to perform verification operations. Simultaneously, there are pre-defined compatibility relationships between each type of tool in the toolset and each subtask. For example, the text verification subtask is compatible with tools such as text semantic analysis and fact database retrieval, while the image verification subtask is compatible with tools such as image tampering detection and metadata parsing. This pre-defined relationship provides a foundation for rapid tool matching and invocation, ensuring the targeted nature of tool selection while also meeting the diverse tool needs in different scenarios through the extensibility of the toolset (supporting the addition and removal of tools), allowing the system to flexibly respond to changes in new error types or verification requirements.

[0029] It's important to note that each tool (such as a text verification tool or an image tracing tool) is uniformly encapsulated as a "tool card." A standardized template clearly defines the core information of each tool, including its functionality (e.g., "detecting whether an image has been tampered with"), input format (e.g., image file path), output format (e.g., "tampering probability: 95%)," and calling method (e.g., API interface parameters). This modular design creates a standardized unit with a consistent structure for all tools. This modular design offers two significant advantages: First, existing tools can be quickly integrated into the system based on the unified structure, eliminating the need for additional adaptation due to format differences. Second, when new tools (e.g., dedicated detection tools for new error types) or existing tools need to be added or replaced, simply defining a new tool card according to the template allows for seamless integration into the system, significantly reducing the complexity of tool expansion and maintenance and enhancing the flexibility and scalability of the toolset.

[0030] The toolset includes at least one of the following: web search tools, posture detection tools, human keypoint detection tools, expert knowledge tools, image understanding tools, and error detection tools. Web search tools can integrate web information retrieval APIs, focusing on acquiring general web page content and encyclopedic knowledge from the internet, providing reliable external knowledge support for information verification, and solving the problem of comparing multimodal input content with known facts. Posture detection tools analyze the overall posture and movement direction of the human body to determine the motion state and anomalies; human keypoint detection tools accurately locate key parts such as joints, providing fine-grained data for behavior recognition and posture analysis. Both can work together to detect human-related errors in images or videos (such as false movements, abnormal postures, etc.). Expert knowledge tools store the experience and rule knowledge of domain experts for logical reasoning and judgment of specific types of information, enhancing the system's ability to identify errors in professional fields and improving the interpretability of detection results through empirical evidence. Image understanding tools enable comprehensive analysis of image content, encompassing object recognition, scene classification, action recognition, and contextual modeling. They provide a semantic-level analytical foundation for detecting image-specific errors (such as object tampering and scene inaccuracies) and cross-modal consistency errors (such as text-image content mismatches). Error detection tools, as a comprehensive module, focus on directly identifying potential errors, contradictions, or inconsistencies from multimodal inputs such as text, images, and videos. They are the core execution tools for multimodal error information detection. These tools cover multiple dimensions, including web knowledge retrieval, unimodal (text, image / video) analysis, and cross-modal consistency verification. Each tool has its own focus while complementing the others, providing targeted verification capabilities for different error source subtasks (such as text errors, image errors, and text-image mismatches), and also laying the foundation for the scalability of the toolset.

[0031] In step S130, the node to be expanded is determined based on the node selection mechanism, and a tool adapted to the node to be expanded is selected from the toolset to perform the expansion and simulation operations; Expansion refers to creating one or more new child nodes if the selected node does not represent a terminated state.

[0032] Simulation refers to starting from a newly expanded node and using a fast strategy to simulate the process until a termination state is reached. This simulation provides an estimate of the potential value of that node.

[0033] Optionally, firstly, a comprehensive analysis can be performed using a large-scale visual language model (LVLM). This analysis, combined with its understanding of text semantics, image content, and cross-modal relationships, can identify potential sources of error (such as questionable text descriptions, abnormal image content, or logical contradictions between text and images). Subsequently, for each possible source of error corresponding to a subtask, LVLM outputs a "probability score." This score quantifies the relevance of the subtask to the authenticity of the verified information, as well as the probability that it contains erroneous information (a higher score indicates a higher likelihood of error in the subtask and a more critical impact on the overall authenticity determination).

[0034] This probability score directly guides the first step of expanding the search tree: the system prioritizes creating first-level child nodes and allocating resources to subtasks with high scores, prioritizing the exploration of directions more likely to contain errors, avoiding wasting computing power on low-value subtasks, thereby improving the efficiency and focus of the initial search phase, ensuring the entire detection process focuses on the core verification target from the outset. Leveraging LVLM's multimodal understanding capabilities, it provides a data-driven prioritization basis for search tree construction, balancing comprehensiveness and efficiency, and ensuring the initial expansion direction more closely matches the actual error distribution.

[0035] Then, the selection of nodes to be expanded refers to recursively selecting child nodes starting from the root node according to a preset strategy (such as the Upper Confidence bound applied to Trees, UCT) until a leaf node or a partially expanded node is reached. This algorithm focuses on unexplored sub-task nodes, introducing a bias term and redefining the calculation method to obtain the value of the terminal node by a weighted combination of the inference path score and confidence score using balancing parameters. Simultaneously, the UCT value of each node is calculated by combining the visit counts of child and parent nodes, as well as hyperparameters used to control the trade-off between exploration and exploitation. During calculation, the relevant visit counts are incremented by 1 to ensure a reasonable UCT value is obtained even if the node has not been visited. Thus, in the early stages of the search, newly expanded sub-task nodes are not over-prioritized simply because they have not been visited, avoiding resource waste and inefficient exploration; as the search progresses and the visit count of parent nodes increases, the UCT value of unvisited child nodes gradually increases, thereby encouraging the exploration of more diverse sub-tasks, thus balancing the exploration of less visited sub-task nodes with the exploitation of nodes with high confidence scores.

[0036] Finally, for the selected node to be expanded (i.e., the subtask of a specific error source), the system will use the Bayesian optimization algorithm scalability tool set to call up a tool combination that is compatible with its preset (for example, if the node to be expanded is the "image tampering detection" subtask, then the "image understanding" tool will be called). The tool will perform specific verification operations (such as analyzing image pixel anomalies, detecting editing traces, etc.) to generate new child nodes or verification results, thus completing the expansion of the node. At the same time, the effectiveness of the current path will be simulated and evaluated based on the results returned by the tool, providing a basis for subsequent node value updates.

[0037] It should be noted that evolutionary algorithms such as genetic algorithms can also be used to dynamically select a combination of tools suitable for the current error type.

[0038] In step S140, the expanded nodes are evaluated based on the inference path score and the result confidence. The reasoning path score is used to quantify the reasoning quality and logical coherence from the root node to the current node, and the result confidence score is used to evaluate the quality and internal consistency of the evidence collected by the current node and serves as a reward signal for the terminal node.

[0039] Optionally, when evaluating the expanded nodes, two key dimensions are considered: inference path score and result confidence. The inference path score focuses on the reasoning process from the root node to the current expanded node, measuring the path's performance in terms of logical coherence and step rationality, reflecting the smoothness and reliability of error information inference and verification along this path. Result confidence, on the other hand, assesses the credibility of the verification results obtained after the expanded node performs operations using the tool, evaluating their authenticity and accuracy, such as whether the tool's conclusion that "the image has not been tampered with" is sufficiently supported. Result confidence can also serve as a reward signal for terminal nodes. By combining these two aspects in the evaluation, the value of the expanded nodes in detecting errors in multimodal input content can be comprehensively determined, providing a reliable basis for subsequent search tree updates and error determination.

[0040] In step S150, the evaluation results are passed back to the root node along the search path, and the statistical information of each node on the path is updated. Optionally, after evaluating the expanded nodes (combining inference path scores and result confidence), the system transmits the evaluated information backward along the search path, from the last expanded node all the way to the root node. During this backward propagation, the statistics for each node on the path (including child nodes and the root node) are updated. These statistics may include the number of times a node is visited (incremented by 1 for each backward propagation), the accumulated inference path score (the current node's inference path score is added to the parent node's total score), and the summary of result confidence (calculating the average or weighted value of the confidence scores of all nodes on the path). Through this backward propagation and statistical updates, each node in the search tree reflects the current exploration status, providing a more accurate basis for subsequent node selection mechanisms. This allows the system to more intelligently balance the exploration and utilization of different nodes in subsequent search loops, thereby improving the overall efficiency and accuracy of error detection.

[0041] Backpropagation refers to transmitting simulation results back up the selected path to the root node, updating the statistics of each node along the path, such as the number of visits and cumulative rewards. The process of updating the value (or cumulative rewards, etc.) of a node during backpropagation can be specifically implemented using the following formula: ; in, Represents child nodes The value (or statistical information such as cumulative rewards) This represents the value of the parent node s. Represents child nodes Number of visits, Represents child nodes Before the update, based on its number of visits The corresponding value (which can be understood as the sum of the accumulated value of the child nodes, since the value is usually accumulated step by step according to the number of visits).

[0042] In step S160, after completing a preset number of search cycles, a comprehensive verification result is generated, and error information is determined and decided based on the comprehensive verification result.

[0043] Optionally, after a pre-set number of search iterations, the system integrates the verification results of each path and node in the search tree to generate a comprehensive verification result. This comprehensive verification result gathers various evaluation information from multiple rounds of search for subtasks with different error sources in the multimodal input content, including the rationality of the inference path for each subtask and the confidence level of the tool's verification results. Subsequently, based on this comprehensive verification result, the system can comprehensively and accurately determine whether there is erroneous information in the multimodal input content, as well as the type and degree of the error, and then make corresponding decisions, such as determining that the information is erroneous and providing a specific reason for the error, or determining that the information is true and reliable.

[0044] This application provides a multimodal error information detection method, comprising: constructing a search tree containing a root node and multiple first-level child nodes based on multimodal input content, wherein each first-level child node corresponds to a subtask of a different error source of the multimodal input content; configuring an extensible toolset, using the toolset as the action space in the search process, and wherein the tools in the toolset have a preset adaptation relationship with each subtask; determining the node to be expanded based on a node selection mechanism, and selecting a tool from the toolset that is adapted to the node to be expanded to perform expansion and simulation operations; evaluating the expanded node based on inference path scoring and result confidence; transmitting the evaluation result backward along the search path to the root node, and updating the statistical information of each node on the path; generating a comprehensive verification result after completing a preset number of search loops, and determining and deciding on error information based on the comprehensive verification result. In this embodiment, by introducing an extensible toolset and a dynamic reasoning mechanism based on Monte Carlo Tree Search (MCTS), the system can flexibly adapt and dynamically expand the tools to different error source subtasks, addressing the mixed-source characteristics of multimodal error information. This breaks the limitations of traditional fixed toolsets on diverse scenarios. Simultaneously, leveraging the node selection, expansion, and evaluation logic of MCTS, the system can achieve an intelligent balance between in-depth mining of evidence for a single error type and dynamic exploration of multiple potential error types, avoiding the limitations of fixed verification processes. By backpropagating the evaluation results to optimize node statistics and generating comprehensive verification results through multiple rounds of search, the system not only improves the comprehensive detection capability and reliability of mixed-source errors but also enhances its adaptability to complex scenarios through the extensibility of the toolset and the dynamism of the search mechanism, effectively improving the efficiency and accuracy of multimodal error information detection.

[0045] In one embodiment of this application, before determining the node to be expanded based on the node selection mechanism, the following steps are included: Each subtask is assigned a probability score based on the multimodal input content. The probability score is used to reflect the relevance and importance of the corresponding subtask in verifying the authenticity of the information. Based on the probability score, the first node to be expanded is determined.

[0046] Optionally, before determining the nodes to be expanded based on the node selection mechanism, a probability score is assigned to each subtask (corresponding to tasks with different error sources in the multimodal input content) based on the input content. This probability score reflects the relevance and importance of the corresponding subtask in verifying the authenticity of the information. A high relevance means the subtask is more closely related to determining whether the information is true, while a high importance means the subtask has a greater impact on the final determination of the information's authenticity. Then, based on these probability scores, the first node to be expanded is determined from all subtasks—the initial node to be expanded—allowing the search to prioritize exploring subtasks more likely to involve erroneous information from the outset, improving the targeting and efficiency of the initial exploration.

[0047] It's important to note that the probability score can be calculated using a Large Visual Language Model (LVLM). LVLM comprehensively analyzes the multimodal input content (such as text and images), combining its understanding of text semantics, image content, and cross-modal relationships to identify potential error sources. Then, for each possible error source and its corresponding subtask, LVLM outputs a quantified probability score. Alternatively, specific factors related to the relevance and importance of the subtask can be determined from the multimodal input content, such as the subtask's matching degree with the core theme of the input content, the necessity of the information dimensions involved in the subtask in the overall validation, and the probability of this type of subtask finding errors in historical data. Then, different weights are assigned based on the influence of each factor, and a quantified score is calculated for each subtask for each of these factors. Finally, the scores of each factor are multiplied by their corresponding weights and summed to obtain the probability score for that subtask.

[0048] In one embodiment of this application, the evaluation of the expanded nodes based on inference path scores and result confidence includes: Obtain the set of all state-action pairs from the initial state to the current state, and analyze them from multiple dimensions to obtain the inference path score; Observations are obtained by calling tools, and conclusions to be verified are formed based on these observations; The reliability of the observed values ​​and the consistency between the conclusion to be verified and the observed values ​​are analyzed to quantify the credibility of the evidence chain and obtain the confidence level of the result. By combining the inference path score and the result confidence level, a comprehensive evaluation value for the expanded node is formed.

[0049] Optionally, the reasoning path score is used to quantify the quality and coherence of the entire reasoning process from the root node to the current node. It serves two purposes: first, to evaluate the quality of the reasoning path during the simulation and to serve as the value of each node; second, to calculate the reasoning path quality of the final answer node and to serve as part of the node reward. It should be noted that the reasoning path score can be calculated using a reasoning path score function, which takes as input all state-action pairs from the initial state to the current state. (That is, the combination of each node's state and the corresponding tool call action), the logic and effectiveness of the current path are evaluated by analyzing this historical information, and the reasoning path score is specifically calculated using the following formula: ; The confidence score assesses the quality and internal consistency of the collected evidence. This score reflects the overall quality of the collected evidence and its support for the final conclusion. In the MCTS framework, the confidence score serves as a reward signal for terminal nodes. The input to the confidence score function includes all evidence (i.e., observations) collected along the current inference path. The function estimates a confidence score by analyzing the alignment and consistency between different pieces of evidence, reflecting the consistency of the entire verification process and thus assessing the credibility of the conclusion. This confidence score can be calculated using the following formula: .

[0050] Finally, the reasoning path score (reflecting the rationality of the process) and the result confidence score (reflecting the reliability of the conclusion) are comprehensively integrated (e.g., through weighted calculation) to form a comprehensive evaluation value for the current expanded node. This evaluation value reflects both the quality of the reasoning process and the credibility of the verification results, providing a comprehensive basis for the reverse update of subsequent nodes and the optimization of the search tree.

[0051] In one embodiment of this application, the determination of the node to be expanded based on the node selection mechanism includes: The value of terminal nodes is determined based on inference path scores and result confidence. Based on the value of the terminal node and the number of visits to the parent and child nodes, an upper confidence bound algorithm is constructed. The selection index of each candidate node is calculated based on the upper confidence bound algorithm. The candidate nodes include nodes to be explored and nodes that have been explored. The selection index of the node to be explored is dynamically updated according to the increase of the number of visits to the corresponding parent node. The node with the highest selection index is selected from the candidate nodes as the node to be expanded.

[0052] Optionally, in Monte Carlo Tree Search (MCTS), the Upper Confidence Bound for Trees (UCT) algorithm plays a crucial role. In each iteration, the node with the highest UCT value is selected for expansion. However, traditional methods often assign arbitrarily large UCT values ​​to unvisited nodes, which can lead to inefficient exploration. Therefore, by using the terminal node value as the core criterion, combined with the number of visits to parent and child nodes (i.e., the number of times the node and its parent nodes have been explored), attention is paid to unexplored sub-task nodes. This addresses the issue by introducing a bias term and redefining the Upper Confidence Bounds (UCT) algorithm to better balance exploring less visited subtask nodes with utilizing nodes with high confidence scores. Unexplored nodes can dynamically update their UCT values ​​as the search tree develops. In the early stages of the search, newly expanded subtask nodes are not over-prioritized simply because they haven't been visited, thus avoiding wasted resources and inefficient exploration. As the search progresses, the number of visits to parent nodes gradually increases, which progressively increases the UCT values ​​of their unvisited child nodes, encouraging the exploration of diverse subtasks. The redefined UCT function is shown below: ; in, Indicates the value of the terminal node. It is a balancing parameter used to weight and combine two scores: This represents the trajectory score. This represents the confidence score. and These represent the number of visits to the child node and the parent node, respectively. It is a hyperparameter used to control the trade-off between exploration and exploitation. Specifically, this invention achieves this by... Add 1 to ensure that The UCT value can still be calculated even under certain circumstances. Expression The design allows unexplored nodes to dynamically update their UCT values ​​as the entire search tree develops.

[0053] The candidate nodes fall into two categories: explored nodes (nodes with existing verification results) and unexplored nodes (nodes that have not yet been thoroughly verified). Based on the reconstructed upper confidence bound algorithm described above, a selection metric is calculated for each candidate node. This metric combines the node's historical value (such as the value of an end node) and the degree to which it has been explored (reflected by the number of visits).

[0054] Among them, the selection criteria for nodes to be explored will be dynamically updated as the number of visits to their corresponding parent nodes increases: the more the parent node is visited, the deeper the system’s overall exploration of the branch will be, and the “exploration priority” of the nodes to be explored will be increased accordingly to encourage the system to explore uncovered verification directions.

[0055] Finally, the node with the highest selection metric from all candidate nodes can be selected as the next node to be expanded. This ensures that the system prioritizes exploring high-value but insufficiently validated nodes, avoiding wasting resources on low-value paths while continuously expanding the breadth and depth of validation, ultimately improving the efficiency and accuracy of error message detection.

[0056] In one embodiment of this application, after selecting a tool adapted to the node to be extended from the toolset to perform the extension and simulation operations, the method further includes: Obtain the verification results returned by the node to be expanded; If the confidence level of the verification result is greater than the preset confidence threshold, it is determined that the subtask corresponding to the node to be expanded has been completed, and the expansion of the node to be expanded is stopped.

[0057] Optionally, a pruning strategy can be introduced. When a subtask node returns a high-confidence result, clearly indicating that the corresponding error source is likely to be real, the system determines that the subtask has been completed and will no longer expand its subsequent sub-nodes. By pruning such confirmed branches, redundant exploration during the search process can be avoided, and limited computing resources can be concentrated on verifying other unconfirmed modalities or subtasks, which can effectively improve the overall reasoning efficiency and accelerate the formation of the final error information judgment.

[0058] In one embodiment of this application, the determination and decision-making regarding error information based on the comprehensive verification results includes: Calculate the error probability for each subtask based on the confidence score corresponding to each subtask. Aggregate the error probabilities of each subtask to calculate the overall true probability value of the information content; Based on the error probability of each subtask and the overall true probability, it is determined whether there are errors in the multimodal input content and the comprehensive evaluation result of the errors.

[0059] Optionally, if Monte Carlo Tree Search (MCTS) determines the content to be erroneous with high confidence at a certain subtask node, the system terminates directly and outputs the result without further aggregation, thereby improving efficiency and ensuring that verification can be quickly terminated when a clear error is detected. If no clear error signal is found: the results of all verified subtask nodes are aggregated, and a heuristic probability analysis strategy is used for comprehensive judgment. First, the probability of each subtask being judged as erroneous is calculated, which can be obtained using the following formula:

[0060] in, This represents the probability that the i-th subtask is judged as incorrect. This represents the confidence score of the i-th subtask. This represents the verification result of the i-th subtask, where fake indicates that the content of the subtask is incorrect, and real indicates that the content of the subtask is true.

[0061] Then, based on the assumption that the evidence among the subtasks is independent, the overall probability that the information content is "true" is calculated. Finally, by comparing the true probability with the error probabilities of each subtask, the final result is obtained. The overall probability that the information content is "true" can be calculated using the following formula: ; Where n represents the number of subtasks that have been verified. This represents the probability that the content corresponding to the i-th subtask is true.

[0062] Finally, from the perspective of "true" probability and the error probability of each subtask The category corresponding to the largest value (true or subtask error) is selected as the final comprehensive evaluation result. This can be calculated using the following formula: ; Here, answer represents the final judgment result, and argmax represents taking the value of the independent variable that maximizes the value of the function in parentheses (in this case, selecting the category with the highest probability).

[0063] For example, the algorithm steps of Algorithm1 multi-source verification Monte Carlo Tree Search, used for reasoning and decision-making in multimodal information verification scenarios, are as follows: The core is to use the Monte Carlo Tree Search (MCTS) framework, combined with a Large Language Model (LVLM) to generate candidate actions, select nodes using the UCB (Upper Confidence Bound) formula, update node values ​​through backpropagation, and finally output the judgment result of the information's authenticity. Its steps include: Input information: Initial state s: The root node state of the search tree, representing the multimodal input to be verified (such as text, image, video, etc.).

[0064] LVLM Model PLVLM: Large Language Model, used to generate candidate "actions" (such as generating "fact-checking direction" for text subtasks, and "tamper detection angle" for image subtasks).

[0065] Number of actions generated n: The number of candidate subtasks / verification directions generated by LVLM each time a node is expanded.

[0066] Depth limit d: The maximum depth of the search tree, controlling the search range.

[0067] Number of simulations K: The total number of Monte Carlo simulations, which affects the sufficiency of the search.

[0068] Explore weight C and hyperparameter α: balance the parameters of "exploring unknown nodes" and "utilizing known superior nodes".

[0069] Specify subtask node and calculated value: The specific subtask to be verified (such as "whether the text contains factual errors" or "whether the image has been tampered with") and the relevant calculation baseline value.

[0070] Initialization and Iteration The algorithm performs K rounds of Monte Carlo simulation, with each round following the following process: Node selection (from root node to child nodes): Starting from the root node s, select child nodes using the UCB formula. The specific formula is as follows: ; in, Represents a node The historical value, N(s) represents the number of visits to parent node S, N( ) represents a child node The number of visits, C is the exploration weight, and the larger C is, the more inclined to explore unknown nodes.

[0071] Extensions and simulations: When node s is not in the "terminated state" (verification completed) and its depth is less than d, if s has unvisited child nodes, use PLVLM to generate n new child nodes (i.e., new verification subtasks / directions) to expand the search tree.

[0072] Evaluate: Combination score: ,in, Score the reasoning path. The confidence score is... This is a hyperparameter used to balance the weights of the two.

[0073] Backpropagation: Updating the value of all nodes on the path from child nodes to root node in reverse can be represented by the following formula: ; in, Represents child nodes The value (or statistical information such as cumulative rewards) This represents the value of the parent node s. Represents child nodes Number of visits, Represents child nodes Before the update, based on its number of visits The corresponding value (which can be understood as the sum of the accumulated value of the child nodes, since the value is usually accumulated step by step according to the number of visits).

[0074] Decision making: After the simulation, the results will be output in two cases: Early termination: If a subtask node has returned a "high confidence error conclusion", terminate directly and output the result (such as "the text contains factual errors").

[0075] When there is no explicit error: aggregate the error probabilities of all subtasks and calculate the overall true probability; Ultimately, the term with the largest value between "true probability" and "error probability of each subtask" is chosen as the final conclusion.

[0076] In this embodiment, multi-source verification Monte Carlo tree search enables the system to explore multiple sub-tasks in a structured manner, evaluate the quality of evidence collected along different reasoning paths, prune verified branches to focus on remaining uncertainties, and integrate the results of each sub-task to arrive at a final judgment. This simulates the iterative and dynamic reasoning process of human experts: flexibly switching between exploring new sources, verifying evidence across modalities, and integrating findings into a final conclusion, thereby achieving more efficient and intelligent multi-modal error information detection.

[0077] In one embodiment of this application, the method further includes: A Gaussian process model is constructed and trained using a training dataset, and the performance of various candidate tool combinations is predicted using the Gaussian process model. The expected improvement criterion is used as the acquisition function, and the candidate tool combinations are evaluated based on the performance prediction results; Based on the expected improvement criteria and evaluation results, the tool combination with the highest adaptability to the sub-task requirements is selected and applied to the actual sub-task, and its real performance data is recorded. The real performance data is added to the training dataset, and the Gaussian process model is iteratively updated to dynamically optimize the fit between the tool and the subtask.

[0078] Optionally, a configurable extensible toolset refers to a collection of various flexibly addable and removeable detection tools (such as text verification tools, image source tracing tools, cross-modal comparison tools, etc.). This toolset is set as an action space available during the search process; that is, when exploring different error source subtasks, the system can select appropriate tools from the toolset to perform verification operations. Furthermore, there are preset adaptation relationships between each type of tool in the toolset and each subtask, and Bayesian optimization techniques can be used to select the most relevant subset of tools for different tasks, improving efficiency.

[0079] Specifically, this fit can be dynamically optimized. First, a Gaussian process model is built and trained using the training dataset, and the performance of each candidate tool combination is predicted using this model. Then, the expected improvement criterion, a collection function, is used to evaluate each candidate tool combination based on the performance prediction results. Next, based on the expected improvement criterion and the evaluation results, the optimizer maximizes the collection function to find the next tool combination most likely to bring performance improvement and applies it to the actual subtask, while recording its actual performance data. Finally, the actual performance data is added to the training dataset, and the Gaussian process model is iteratively updated until the preset maximum number of iterations is reached or the convergence condition is met, thereby dynamically optimizing the fit between the tool and the subtask.

[0080] For example, the specific process of iteratively optimizing and selecting the "tool combination with the highest suitability to the subtask" using a Gaussian process model and an expected improved acquisition function (Algorithm 2) is as follows: Input Requirements: Initial Toolset Evaluation function f(), iteration number I.

[0081] Initialization: Construct observation data D, initially containing The set (i.e., the initial toolset and its evaluation values).

[0082] Model building and training: A Gaussian process (GP) model is built using the selected kernel function and initialized with the observed data. D Train the model.

[0083] Iterative process : Based on the current P model, we define the expected improvement (EI) acquisition function.

[0084] Optimize the EI acquisition function to find the next subset of candidate tools. .

[0085] Calculated using the evaluation function The value will Add observation data D.

[0086] Retrain or update the GP model parameters using the updated D.

[0087] Output: Returns the best subset of tools in the observed data D that maximizes the evaluation function. .

[0088] It should be noted that the Expected Improvement (EI) criterion is used as the sampling function to measure the performance improvement that the candidate tool combination may bring compared to the current optimal solution: ; in, This represents the best-performing combination of tools currently known. This represents the performance evaluation value of the corresponding tool combination.

[0089] Furthermore, specific tools can be selected for specific needs (such as "fact-finding on the web"); it can also be expanded into a multi-expert integrated system, designing multiple specialized detection models for each need (such as "text fact-checking" and "image tampering detection"), and finally judging the results through "voting" or "weighted fusion". Specific tools meet the efficiency of a single need; multi-expert integration improves the robustness of the results (i.e., the ability to resist interference from various complex errors) through the collaboration of multiple models (different models cover different detection angles).

[0090] In one embodiment of this application, constructing and training a Gaussian process model using a training dataset includes: The training dataset is generated based on initial data containing several tool subsets and their corresponding scores; Based on the training dataset, an initial Gaussian process model is constructed and trained, wherein a squared exponential kernel is selected as the kernel function to characterize the smoothness and correlation of the tool combination performance; The hyperparameters of the kernel function and the variance of the observation noise are determined by maximizing the marginal likelihood. During the training process, the gradient descent method is used to calculate the gradient of the marginal likelihood with respect to the hyperparameters and adjust the hyperparameter values. After multiple iterations until the marginal likelihood converges, the trained Gaussian process model is obtained.

[0091] Optionally, a training dataset is constructed based on the initial data, and the Gaussian process model is trained using this dataset to fit the performance distribution of different known tool combinations. This training dataset may include several pairs of... ,in Indicates the first A subset of tools, The score of the subset is calculated using an evaluation function.

[0092] For example, initially we have a dataset like this: .

[0093] Assuming the scores of all tool subsets The data follows a multivariate normal distribution. To fit these observations using a Gaussian process model, we need to learn the hyperparameters of the kernel function and the noise variance. .

[0094] A squared exponential kernel can be chosen as the kernel function: ; in, It is the signal variance, which controls the scale of the output variable. It is the length scale, which determines how quickly the similarity between two input points decreases as the distance between them increases. It assumes, as a priori, that the output follows a Gaussian distribution with zero mean. , Indicates input point and The Euclidean distance between them.

[0095] The training process of a Gaussian process model aims to maximize the marginal likelihood to find the optimal combination of hyperparameters. The marginal likelihood can be calculated analytically. ; in, θ It is the set of hyperparameters of the kernel function (such as...) , ), It is the covariance matrix calculated by the kernel function. It is the variance of the observation noise. µ Represents the prior mean. I Represents the identity matrix. This represents the observation data vector.

[0096] Finally, gradient descent can be used to calculate the gradient of the marginal likelihood with respect to the hyperparameters, and the hyperparameter values ​​can be adjusted accordingly. After several iterations, the hyperparameters will be adjusted to the positions that best represent the observed data, resulting in a well-trained Gaussian process model. Using this model, it is possible to apply it to any new subset of tools. The score is used to predict.

[0097] In this embodiment, by introducing an extensible toolset and a dynamic inference mechanism based on Monte Carlo Tree Search (MCTS), the system can flexibly adapt and dynamically expand tools to subtasks from different error sources, addressing the mixed-source characteristics of multimodal error information. This breaks the limitations of traditional fixed toolsets on diverse scenarios. Leveraging the node selection, expansion, and evaluation logic of MCTS, the detection task is decomposed into multiple subtasks and dynamic path planning is performed. Combined with an evaluation function that includes inference path scores and result confidence, the system guides the agent to explore and utilize resources efficiently, improving detection robustness and accuracy. Simultaneously, a unified template multi-functional tool library (including network retrieval, image understanding, etc.) is constructed, supporting flexible integration of new tools. Through the extensibility of the toolset and the dynamic nature of the search mechanism, the system's adaptability to complex scenarios is enhanced, effectively improving detection efficiency and accuracy.

[0098] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0099] In one embodiment, a multimodal error information detection device is provided, which corresponds one-to-one with the multimodal error information detection method described in the above embodiments. For example... Figure 3 As shown, the multimodal error information detection device includes a search tree construction unit 10, a toolset configuration unit 20, a node to be expanded determination unit 30, an evaluation unit 40, a backpropagation unit 50, and a verification unit 60. Detailed descriptions of each functional module are as follows: The search tree construction unit 10 is used to construct a search tree containing a root node and multiple first-level child nodes based on the multimodal input content, wherein each first-level child node corresponds to a subtask of a different error source of the multimodal input content. Toolset configuration unit 20 is used to configure an extensible toolset, which is used as the action space in the search process, and the tools in the toolset have a preset adaptation relationship with each subtask; The node to be expanded determination unit 30 is used to determine the node to be expanded based on the node selection mechanism, and select a tool from the toolset that is compatible with the node to be expanded to perform expansion and simulation operations. Evaluation unit 40 is used to evaluate the expanded nodes based on inference path scores and result confidence. Backpropagation unit 50 is used to propagate the evaluation results back along the search path to the root node and update the statistical information of each node on the path. The verification unit 60 is used to generate a comprehensive verification result after completing a preset number of search cycles, and to determine and make decisions on error information based on the comprehensive verification result.

[0100] In one embodiment of this application, the device further includes a probability score generation unit, used for: Each subtask is assigned a probability score based on the multimodal input content. The probability score is used to reflect the relevance and importance of the corresponding subtask in verifying the authenticity of the information. Based on the probability score, the first node to be expanded is determined.

[0101] In one embodiment of this application, the evaluation unit 40 is further configured to: Obtain the set of all state-action pairs from the initial state to the current state, and analyze them from multiple dimensions to obtain the inference path score; Observations are obtained by calling tools, and conclusions to be verified are formed based on these observations; The reliability of the observed values ​​and the consistency between the conclusion to be verified and the observed values ​​are analyzed to quantify the credibility of the evidence chain and obtain the confidence level of the result. By combining the inference path score and the result confidence level, a comprehensive evaluation value for the expanded node is formed.

[0102] In one embodiment of this application, the node to be expanded determination unit 30 is further configured to: The value of terminal nodes is determined based on inference path scores and result confidence. Based on the value of the terminal node and the number of visits to the parent and child nodes, an upper confidence bound algorithm is constructed. The selection index of each candidate node is calculated based on the upper confidence bound algorithm. The candidate nodes include nodes to be explored and nodes that have been explored. The selection index of the node to be explored is dynamically updated according to the increase of the number of visits to the corresponding parent node. The node with the highest selection index is selected from the candidate nodes as the node to be expanded.

[0103] In one embodiment of this application, the apparatus further includes a pruning strategy execution unit, used for: Obtain the verification results returned by the node to be expanded; If the confidence level of the verification result is greater than the preset confidence threshold, it is determined that the subtask corresponding to the node to be expanded has been completed, and the expansion of the node to be expanded is stopped.

[0104] In one embodiment of this application, the verification unit 60 is further configured to: Calculate the error probability for each subtask based on the confidence score corresponding to each subtask. Aggregate the error probabilities of each subtask to calculate the overall true probability value of the information content; Based on the error probability of each subtask and the overall true probability, it is determined whether there are errors in the multimodal input content and the comprehensive evaluation result of the errors.

[0105] In one embodiment of this application, the device further includes a tool dynamic optimization unit, used for: A Gaussian process model is constructed and trained using a training dataset, and the performance of various candidate tool combinations is predicted using the Gaussian process model. The expected improvement criterion is used as the acquisition function, and the candidate tool combinations are evaluated based on the performance prediction results; Based on the expected improvement criteria and evaluation results, the tool combination with the highest adaptability to the sub-task requirements is selected and applied to the actual sub-task, and its real performance data is recorded. The real performance data is added to the training dataset, and the Gaussian process model is iteratively updated to dynamically optimize the fit between the tool and the subtask.

[0106] In one embodiment of this application, the tool dynamic optimization unit is further configured to: The training dataset is generated based on initial data containing several tool subsets and their corresponding scores; Based on the training dataset, an initial Gaussian process model is constructed and trained, wherein a squared exponential kernel is selected as the kernel function to characterize the smoothness and correlation of the tool combination performance; The hyperparameters of the kernel function and the variance of the observation noise are determined by maximizing the marginal likelihood. During the training process, the gradient descent method is used to calculate the gradient of the marginal likelihood with respect to the hyperparameters and adjust the hyperparameter values. After multiple iterations until the marginal likelihood converges, the trained Gaussian process model is obtained.

[0107] In this embodiment, by introducing an extensible toolset and a dynamic reasoning mechanism based on Monte Carlo Tree Search (MCTS), the system can flexibly adapt and dynamically expand the tools to different error source subtasks, addressing the mixed-source characteristics of multimodal error information. This breaks the limitations of traditional fixed toolsets on diverse scenarios. Simultaneously, leveraging the node selection, expansion, and evaluation logic of MCTS, the system can achieve an intelligent balance between in-depth mining of evidence for a single error type and dynamic exploration of multiple potential error types, avoiding the limitations of fixed verification processes. By backpropagating the evaluation results to optimize node statistics and generating comprehensive verification results through multiple rounds of search, the system not only improves the comprehensive detection capability and reliability of mixed-source errors but also enhances its adaptability to complex scenarios through the extensibility of the toolset and the dynamism of the search mechanism, effectively improving the efficiency and accuracy of multimodal error information detection.

[0108] Specific limitations regarding the multimodal error information detection device can be found in the limitations of the multimodal error information detection method described above, and will not be repeated here. Each module in the aforementioned multimodal error information detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0109] In one embodiment, a computer device is provided, which may be a terminal device, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, and network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a readable storage medium storing computer-readable instructions. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer-readable instructions implement a multimodal error information detection method. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0110] In this application embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, it implements the steps of the multimodal error information detection method described above.

[0111] In this embodiment of the application, a readable storage medium is provided, which stores computer-readable instructions. When the computer-readable instructions are executed by a processor, they implement the steps of the multimodal error information detection method described above.

[0112] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0113] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0114] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A multimodal error information detection method, characterized in that, The method, applied in the field of computer technology, includes: Based on the text data and image data of the multimodal input content, a search tree containing a root node and multiple first-level child nodes is constructed, wherein each first-level child node corresponds to a subtask of different error sources of the multimodal input content, and the error sources include text errors, image errors or cross-modal inconsistency issues; Configure an extensible toolset, wherein the toolset includes at least one of the following: web search tools, pose detection tools, human keypoint detection tools, expert knowledge tools, image understanding tools, and error detection tools. Each tool is uniformly encapsulated as a tool card, and the core information of the tool can be clearly defined through a standardized template, including the functions that the tool can perform, the input format it receives, the output format it returns, and the calling method, so that all tools form a standardized unit with a consistent structure. The toolset serves as the action space in the search process, and the tools in the toolset have a preset adaptation relationship with each subtask. The adaptation relationship is dynamically updated through a pre-trained Gaussian process model. The node to be expanded is determined based on a node selection mechanism, which includes calculating selection indicators by combining the number of node visits, inference path scores and result confidence, and selecting tools from the toolset that are compatible with the node to be expanded to perform expansion and simulation operations based on the selection indicators. The expanded node is evaluated based on the reasoning path score and the result confidence score, including: obtaining the set of all state-action pairs from the initial state to the current state and analyzing them from multiple dimensions to obtain the reasoning path score; obtaining observations through tool calls and forming a conclusion to be verified based on the observations; analyzing the reliability of the observations and the consistency between the conclusion to be verified and the observations, quantifying the credibility of the evidence chain, and obtaining the result confidence score; and combining the reasoning path score and the result confidence score to form a comprehensive evaluation value for the expanded node. The evaluation results are passed back to the root node along the search path, and the statistical information of each node on the path is updated. After completing a preset number of search cycles, a comprehensive verification result is generated, and error information is judged and decided based on the comprehensive verification result, including: calculating the error probability of each subtask based on the confidence score of each subtask; aggregating the error probabilities of each subtask to calculate the overall true probability value of the information content; and determining whether there are errors in the multimodal input content and the comprehensive evaluation result of the errors based on the error probabilities of each subtask and the overall true probability.

2. The multimodal error information detection method as described in claim 1, characterized in that, Before determining the nodes to be expanded based on the node selection mechanism, the following steps are included: Each subtask is assigned a probability score based on the multimodal input content. The probability score is used to reflect the relevance and importance of the corresponding subtask in verifying the authenticity of the information. Based on the probability score, the first node to be expanded is determined.

3. The multimodal error information detection method as described in claim 1, characterized in that, The node selection mechanism for determining the nodes to be expanded includes: The value of terminal nodes is determined based on inference path scores and result confidence. Based on the value of the terminal node and the number of visits to the parent and child nodes, an upper confidence bound algorithm is constructed. The selection index of each candidate node is calculated based on the upper confidence bound algorithm. The candidate nodes include nodes to be explored and nodes that have been explored. The selection index of the node to be explored is dynamically updated according to the increase of the number of visits to the corresponding parent node. The node with the highest selection index is selected from the candidate nodes as the node to be expanded.

4. The multimodal error information detection method as described in claim 3, characterized in that, After selecting a tool from the toolset that is compatible with the node to be expanded to perform the expansion and simulation operations, the process further includes: Obtain the verification results returned by the node to be expanded; If the confidence level of the verification result is greater than the preset confidence threshold, it is determined that the subtask corresponding to the node to be expanded has been completed, and the expansion of the node to be expanded is stopped.

5. The multimodal error information detection method as described in claim 1, characterized in that, The method further includes: A Gaussian process model is constructed and trained using a training dataset, and the performance of various candidate tool combinations is predicted using the Gaussian process model. The expected improvement criterion is used as the acquisition function, and the candidate tool combinations are evaluated based on the performance prediction results; Based on the expected improvement criteria and evaluation results, the tool combination with the highest adaptability to the sub-task requirements is selected and applied to the actual sub-task, and its real performance data is recorded. The real performance data is added to the training dataset, and the Gaussian process model is iteratively updated to dynamically optimize the fit between the tool and the subtask.

6. The multimodal error information detection method as described in claim 5, characterized in that, The construction and training of the Gaussian process model using the training dataset includes: The training dataset is generated based on initial data containing several tool subsets and their corresponding scores; Based on the training dataset, an initial Gaussian process model is constructed and trained, wherein a squared exponential kernel is selected as the kernel function to characterize the smoothness and correlation of the tool combination performance; The hyperparameters of the kernel function and the variance of observation noise are determined by maximizing the marginal likelihood. During training, gradient descent is used to calculate the gradient of the marginal likelihood with respect to the hyperparameters, and the hyperparameter values ​​are adjusted based on the gradient. After multiple iterations until the marginal likelihood converges, the trained Gaussian process model is obtained.

7. A multimodal error information detection device, characterized in that, The apparatus, applied in the field of computer technology, includes: The search tree construction unit is used to construct a search tree containing a root node and multiple first-level child nodes based on text data and image data of multimodal input content. Each first-level child node corresponds to a subtask of different error sources of the multimodal input content, including text errors, image errors, or cross-modal inconsistency issues. A toolset configuration unit is used to configure an extensible toolset, wherein the toolset includes at least one of web search tools, pose detection tools, human keypoint detection tools, expert knowledge tools, image understanding tools, and error detection tools. Each tool is uniformly encapsulated as a tool card, and the core information of the tool can be clearly defined through a standardized template, including the functions that the tool can perform, the input format it receives, the output format it returns, and the calling method, so that all tools form a standardized unit with a consistent structure. The toolset serves as the action space in the search process, and the tools in the toolset have a preset adaptation relationship with each subtask. The adaptation relationship is dynamically updated through a pre-trained Gaussian process model. The node to be expanded determination unit is used to determine the node to be expanded based on a node selection mechanism. The node selection mechanism includes calculating selection indicators by combining the number of node visits, inference path scores and result confidence, and selecting tools from the toolset that are compatible with the node to be expanded to perform expansion and simulation operations based on the selection indicators. The evaluation unit is used to evaluate the expanded nodes based on the inference path score and the result confidence. The evaluation unit is also used to obtain a set of all state-action pairs from the initial state to the current state, and analyze them from multiple dimensions to obtain the reasoning path score; obtain observation values ​​through tool calls, and form a conclusion to be verified based on the observation values; analyze the reliability of the observation values ​​and the consistency between the conclusion to be verified and the observation values, quantify the credibility of the evidence chain, and obtain the result confidence level; and combine the reasoning path score and the result confidence level to form a comprehensive evaluation value for the expanded node. The backpropagation unit is used to propagate the evaluation results back along the search path to the root node and update the statistical information of each node on the path. The verification unit is used to generate a comprehensive verification result after completing a preset number of search cycles, and to determine and make decisions on error information based on the comprehensive verification result; The verification unit is further configured to calculate the error probability of each subtask based on the confidence score of each subtask; aggregate the error probabilities of each subtask to calculate the overall true probability value of the information content; and determine whether the multimodal input content has errors and the comprehensive evaluation result of the errors based on the error probabilities of each subtask and the overall true probability.

Images