Multi-modal coordination method based on multi-agent uncertainty fusion and consistency self-checking

Abstract

The multimodal collaborative method based on multi-agent uncertainty fusion and consistency self-checking comprises the following steps: 1. Acquire the input image and its corresponding question text, and test it based on a public dataset; 2. Use a full-modal model to parse the image and text information, identify the task intent and type, generate, check and revise the task plan, and obtain the main evidence after execution; 3. Set up multiple agents to generate the main description from different perspectives; calculate the relevance after sentence segmentation, select the top-k sentences for weighting, and combine them with N alternative expression samples for support judgment, and calculate the self-checking score; 4. Calculate the agent uncertainty and assign weights based on the self-checking score to achieve evidence fusion, system uncertainty estimation and answer conflict degree calculation; 5. When the system uncertainty or conflict degree exceeds the threshold, trigger the extraction of dispute focus and targeted supplementary evidence, iterating until convergence; 6. Use the fused evidence as the initial node for graph search reasoning, generate candidate nodes through action expansion, deduplication and bundle search, and output the optimal answer based on the self-checking score.

Description

Technical Field

[0001] This invention addresses the shortcomings of current methods in characterizing the execution process, consistency, and reliability of evidence in cross-modal tasks. It proposes a precise and comprehensive multimodal collaborative method based on multi-agent uncertainty fusion and consistency self-checking. This method generates evidence through multi-agent collaboration, performs gating iteration based on consistency self-checking and uncertainty fusion, and achieves robust decision-making by combining graph search reasoning. It can be applied to scenarios such as visual understanding, OCR recognition, scene analysis, and intelligent assistants. Background Technology

[0002] In recent years, multimodal large models have been widely deployed in image question answering, visual reasoning, OCR recognition, and agent applications. However, they still face challenges in stability and reliability in complex scenarios. Existing methods often use a single model to generate answers end-to-end. While simple to implement, this approach is prone to issues such as illusions, insufficient evidence, or gaps in reasoning despite seeing. Furthermore, when problems involve counting, spatial relationships, text recognition, or identity verification, the model often fails to utilize key clues consistently, leading to significant fluctuations in results. To improve performance, some works introduce multiple sampling, voting, or simple ensembles, but these typically lack quantification mechanisms for evidence consistency and output reliability, making it difficult to distinguish between verifiable facts and the reasonableness of verbal statements. In addition, with the development of multi-agent frameworks, systems are increasingly adopting different roles to handle perception and reasoning. However, existing collaboration methods often rely on fixed weights or empirical rules, lacking adaptive fusion strategies based on uncertainty and conflict. This can easily lead to information redundancy between agents, overlapping contradictions, or amplified erroneous conclusions. On the other hand, while search-based reasoning or self-reflection mechanisms can improve answer quality through iterative rewriting, they are prone to generating invalid loops, excessive costs, or over-reasoning if they lack effective scoring functions and convergence criteria. Therefore, there is an urgent need for a multimodal reasoning framework that is evidence-centric, has quantifiable reliability, and can adaptively iterate under conflict and uncertainty, thereby simultaneously improving output stability, interpretability, and robustness across different types of tasks. Summary of the Invention

[0003] To address the problems of existing technologies, this invention proposes a multimodal reasoning framework based on multi-agent collaboration and consistency self-checking. This method first generates initial evidence from images and questions according to a task plan. Then, agents such as visual observation, question alignment, and textual clues refine the evidence, and the reliability and uncertainty are calculated through Top-k relevant sentence filtering and Softmax weighted self-checking. Based on adaptively assigned agent weights, the evidence is fused to form comprehensive evidence. The uncertainty and conflict rate of the entire system are used as gating mechanisms to trigger the extraction of disputed points and iterative, targeted supplementary evidence iteration until convergence. Finally, the fused evidence serves as the starting point for a graph search reasoning mechanism. Candidate nodes are generated through action expansion, deduplication, and bundle search iterations, and self-checked for scoring. The optimal answer is then output, improving stability and interpretability.

[0004] This invention is achieved through the following method: a multimodal cooperative method based on multi-agent uncertainty fusion and consistency self-checking, the steps of which are as follows: Step 1) Obtain the input image and the corresponding question text. Use a publicly available dataset for testing, with one image corresponding to one question. 1.1) This study uses the open-source dataset MME, which can be downloaded from official channels. This dataset comprehensively evaluates the capabilities of multimodal large models from two dimensions: perception and cognition. These two dimensions complement and support each other. The dataset has a rich and diverse sample composition. Some images are taken from public data sources, but only the original image information is used, without using their annotations. At the same time, a large number of manually captured images have been added, covering handwritten arithmetic problems, real-life images of daily scenes, and images generated by diffusion models, effectively reducing the risk of data leakage and ensuring the reliability and fairness of the evaluation.

[0005] 1.2) The dataset is loaded locally, mainly consisting of a perception component and a cognitive component. The perception task comprises ten sub-tasks, including recognition of basic attributes of common objects, domain-specific professional recognition, and image-text recognition. The cognitive task comprises four tasks. All images for the cognitive tasks are manually captured, and the instruction-answer pairs are manually designed to ensure no data leakage.

[0006] 1.3) The perception and cognition parts in the dataset are each image corresponding to a question, and are input into the full-modal large model in the form of each image corresponding to a question.

[0007] Step 2) Based on the input image and corresponding question, perform task analysis using a full-modal model to determine the intent and task type. Generate a task plan based on the understanding of the task. The full-modal model breaks down the task into several steps and checks for missing steps, correcting any omissions. Invite relevant visual and linguistic experts to execute the task according to the plan to obtain key evidence related to the image. 2.1) The full-modal model enters the perception stage. The full-modal large model performs task parsing based on the input image and question, analyzing the intent and task type, such as visual recognition counting tasks, spatial relationship tasks, and text output tasks, and generating concise semantic representations.

[0008] 2.2) The full-modal model enters the planning phase. Based on the understanding of the task, the full-modal model generates a task plan, transforms the semantic representation into executable task steps, clarifies the modal operations of each step, and arranges the sequence of each step to form a structured task plan, including the selection of the model (visual model, text model) and the selection of tools.

[0009] 2.3) The full-modal model enters the reflection phase, verifying whether the task plan matches the user's intent, identifying missing or redundant steps, such as supplementing the video text description if it is not included in the plan. This triggers plan revision, calls the relevant models to execute the task according to the plan, and obtains the main evidence for the image.

[0010] Step 3) Set up multiple agents to answer questions based on different focuses. Each agent generates a main description from different angles based on the main evidence. The generated main description is divided into sentences. The relevance score between the sentences and the question is calculated using a full-modal model. The top-k relevant sentences are selected and softmax weights are assigned within the top-k. Each agent generates N alternative expression samples based on its own main description and performs support judgment on each of the top-k sentences to obtain the weighted illusion rate and calculate the self-test score. 3.1) Set up multiple agents in the full-modal model from different perspectives, and provide answers to questions based on different focuses. Each agent generates a master description from different perspectives based on the main evidence. The master description generated by each agent is divided into sentences. Using a full-modal model to provide sentences and questions correlation score The calculation formula is as follows:

[0011] in, It is every sentence. It's a problem with the image matching; It is the relevance score of each sentence to the question; the more relevant the sentence is to the question, the higher its relevance. The closer the value is to 1.

[0012] 3.2) Select the Top-k Relevant Sentences: Choose the k most relevant sentences, i.e., the k sentences with the highest relevance scores. Remove irrelevant sentences to reduce their interference with the score. The calculation formula is as follows:

[0013] in, It is every sentence; It is the relevance score of each sentence to the question; It is the set of Top-k sentences.

[0014] Softmax weights are then assigned within the Top-k sentence set, calculated using the following formula:

[0015] in, This represents the weight of a sentence in the Top-k sentence set of a proxy; It is the relevance score of each sentence to the question; It's temperature. The smaller the difference, the more amplified it will be; the most relevant sentence will have a very high weight. The larger the gap, the smoother the transition, and each sentence becomes more even.

[0016] Each agent generates N alternative representation samples based on its own master description and performs a support determination on each of the top-k sentences. The sentence support determination is calculated by the full-modal model using a yes or no approach, and is used to determine whether each of the top-k sentences can be stably supported by different paraphrased versions. The calculation formula is as follows:

[0017] in, The sentence support determination is 1 for support (Yes) and 0 for no support (No); This represents a sentence within the Top-k sentence set; This represents the description of the nth sample.

[0018] 3.3) Calculate the illusion rate of each sentence in the Top-k sentence set based on the support decision of each sentence. The number of "no" statements given by the full modality model based on N alternative expression samples for each sentence is calculated as follows:

[0019] in, This represents a sentence within the Top-k sentence set; This represents the description of the nth sample; This is a sentence support judgment, where 1 indicates support (Yes) and 0 indicates dissatisfaction (No). The more negative the sentence is by the substitute expression samples, the more unstable the sentence is and the higher the illusion rate.

[0020] 3.4) Calculate the weighted illusion rate based on the illusion rate of each sentence, which is the overall illusion rate of the Top-k sentence set generated by each agent. This is achieved by combining the illusion rates of each sentence according to their sentence weights. The calculation formula is as follows:

[0021] in, This represents the illusion rate of a sentence in the Top-k sentence set of an agent; H represents the weight of a sentence in a proxy's Top-k sentence set; H represents the overall illusion rate of each proxy's Top-k sentence set. The larger H is, the higher the overall illusion rate, and the worse the quality of evidence generated by that proxy.

[0022] The agent self-test score is calculated based on the overall illusion rate of the top-k sentence set for each agent, using the following formula:

[0023] Where H represents the overall illusion rate of the Top-k sentence set for each agent; score represents the self-test score of each agent. The higher the score, the lower the overall illusion rate and the better the quality of evidence generated by the agent.

[0024] Step 4) Calculate the uncertainty index of each agent based on the self-inspection score, and obtain the agent weight through weight allocation to realize agent-level evidence fusion and system uncertainty estimation, and give the candidate answer for each agent. At the same time, calculate the conflict metric based on the consistency of the candidate answers. 4.1) Calculate the uncertainty index for each agent based on the self-assessment score. The calculation formula is as follows:

[0025] in, This represents the self-test score for each agent. The higher the value, the lower the uncertainty. This represents the uncertainty index for each agent. The higher the value, the greater the uncertainty.

[0026] 4.2) Calculate the weight of each agent based on the uncertainty index of each agent, which will be used for subsequent calculation of the uncertainty of the entire system. The calculation formula is as follows:

[0027] in, It is the sharpness of the weight. The smaller the difference, the more amplified it will be, and the more reliable the agent will have a very high weight. The larger the gap, the more even it becomes, and the more evenly distributed among each agent. Weights that represent the uncertainty of an agent; This represents the uncertainty metric for each agent; agents that are more reliable... The smaller, The higher the weight.

[0028] 4.3) Calculate the uncertainty of the entire system based on the uncertainty index and weight of each agent, i.e., the uncertainty of all evidence provided by multiple agents. The calculation formula is as follows:

[0029] in, This represents the uncertainty index for each agent; This represents the weight of the uncertainty index corresponding to each agent; This indicates the uncertainty of the entire system. The larger the value, the higher the overall uncertainty, and the worse the quality of evidence in the entire system.

[0030] 4.4) Calculate the conflict value between agents by statistically analyzing the evidence provided by each agent and determining whether the conclusions drawn from the evidence provided by different agents are consistent using the full modality model. The calculation formula is as follows:

[0031] in, The answer with the most votes across all answer categories, representing the number of proxies who agree with the viewpoint or conclusion. The degree of conflict is indicated by the greater the disagreement between the agents.

[0032] Step 5) When the system uncertainty estimate or conflict metric exceeds the threshold, trigger the iterative dispute focus extraction and targeted supplementary evidence mechanism. Self-check by resetting multiple agents, recalculate agent weights and iterate until the convergence condition is met. When the overall uncertainty of the system is very low or the change is very small, the benefit of continuing iteration is very low or the number of iterations reaches the upper limit, stop iterating. 5.1) When the system uncertainty estimate or the change in Debate convergence exceeds the threshold, an iterative mechanism for extracting disputed points and providing targeted supplementary evidence is triggered. This involves resetting multiple proxies for self-checking and recalculating proxies' weights for iteration. The formula for calculating the change in Debate convergence is as follows:

[0033] in, Indicates the first Uncertainty of the entire system during round iteration; Indicates the first Uncertainty of the entire system during round iteration; This represents the convergence change of the Debate, if The value is very small, indicating that the benefits of continuing to iterate are low, and iteration should be stopped.

[0034] 5.2) If iteration is performed, multiple agents need to be reassigned, each providing a new answer based on different emphases. Each agent generates a main description from different perspectives based on key evidence, and the main description is re-segmented. A full-modal model is used to calculate the relevance score between sentences and the question. The top-k relevant sentences are selected, and softmax weights are assigned within the top-k. N alternative expression samples are generated based on each agent's main description, and the supportability of each of the top-k sentences is assessed to obtain a weighted illusion rate and calculate a self-check score. Based on the self-check score, an uncertainty index is calculated for each agent, and agent weights are obtained through weight allocation. This achieves agent-level evidence fusion and system uncertainty estimation, ultimately providing candidate answers for each agent.

[0035] 5.3) Iterate through multiple agents' self-checking arguments until the convergence condition is met. Stop iterating when the overall uncertainty of the system is very low or the change is very small, the benefit of continuing iteration is very low, or the maximum number of iterations is reached. The calculation formula is as follows:

[0036] in, This is the uncertainty threshold that indicates the system is sufficiently certain and debate can be stopped. It can be defined by the user, but a value of 0.15 to 0.25 is usually reasonable, which means stopping when the system's average score is above 0.75 to 0.85. express The convergence threshold, which essentially no longer changes, It is usually set to 0.01~0.03. The smaller the size, the stricter the requirements. Stop when there is basically no change. The larger the size, the looser the fit. Stop if there is no obvious change; Indicates the iteration round; This indicates the maximum number of iterations allowed, typically set to 3-4 rounds to prevent infinite loops. The threshold representing the conflict rate.

[0037] Step 6): Based on steps 3), 4), and 5), the self-check score of the fused evidence is used as the initial node of the graph search reasoning mechanism. Based on the evidence obtained after multi-agent fusion, candidate nodes are generated through action expansion, deduplication search iteration, and self-check scoring. The iteration ends when the highest score exceeds a threshold or the iteration depth exceeds the maximum depth. The answer corresponding to the optimal node is output.

[0038] 6.1) Using the evidence from the multi-agent self-check scores and argument fusion in steps 3), 4), and 5) as the initial nodes of the graph search reasoning mechanism, the calculation formula is as follows:

[0039] in, This represents the primary evidence for the execution of the full-modal model; This means that each agent concatenates the answers according to its own weight; This represents the statement evidence that is ultimately input into the initial node of the graph search reasoning mechanism.

[0040] according to The statement evidence input to the initial node of the graph search inference mechanism is used to generate a statement description input to the graph search inference mechanism by the full-modal model.

[0041] 6.2) Calculate the self-test score of the initial node, which will be used as the value of that node. The calculation formula is as follows:

[0042] in, It's a problem with the image matching; This describes the statements input into the graph search reasoning mechanism. The initial node score represents the graph search reasoning mechanism; If the initial node score is greater than the threshold for the graph search reasoning mechanism score, the graph search reasoning mechanism is not entered, and the answer is output directly. The calculation formula is as follows:

[0043] in, The threshold for the graph search reasoning mechanism score is set to 0.6.

[0044] 6.3) If the initial node score is less than the threshold of the graph search reasoning mechanism score, then the graph search reasoning mechanism is entered. Candidate nodes are generated through action expansion, deduplication, and bundle search iterations, and self-scoring is performed. When the highest score is greater than the threshold or the iteration depth is greater than the maximum depth, the iteration ends, and the answer corresponding to the optimal node is output. The node retention rule calculation formula is as follows:

[0045] in, Indicates to retain the previous text One node; This represents the set of candidate nodes.

[0046] The formula for calculating the optimal node update rule is as follows:

[0047] in, This indicates that the node with the highest score has been visited.

[0048] The formula for calculating the termination of the iterative process in the graph search reasoning mechanism is as follows:

[0049] in, The threshold for the graph search reasoning mechanism score is set to 0.6; This represents the node with the highest self-check score that has been visited.

[0050] The beneficial effects of this invention are as follows: The multimodal reasoning framework proposed in this invention, based on multi-agent collaboration and consistency self-checking, significantly improves the stability, interpretability, and robustness of multimodal tasks. First, by generating initial evidence according to the task plan and refining it through division of labor at the agent level, the system simultaneously covers key information channels such as visual observation, question alignment, and textual clues, avoiding bias and missed detections caused by a single perspective. Second, a consistency self-checking mechanism using Top-k relevant sentence filtering and Softmax weighting quantifies the verifiability of key evidence points, reducing the interference of irrelevant sentences and exaggerated statements on the judgment, and suppressing illusory outputs from the source. Furthermore, based on adaptive weight allocation and conflict rate calculation under uncertainty, it can trigger the extraction of disputed points and targeted supplementary evidence iteration when there are disagreements among agents or high overall uncertainty, promoting gradual convergence of evidence and reducing invalid calculations. Finally, by using fused evidence as the starting point for the graph search reasoning mechanism, more reliable solutions are efficiently explored in the evidence space through action expansion, deduplication, and bundle search. Combined with self-checking scores, early stopping and optimal node selection are achieved, thereby obtaining more consistent and interpretable output results in complex cross-modal reasoning, counting relationships, OCR, and identity verification scenarios, and improving engineering controllability and diagnostic capabilities. Attached Figure Description

[0051] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a comparison chart of experimental data for the method of the present invention; Figure 3 This is a comparison chart of data from different tasks using the method of the present invention; Figure 4 This is a comparison chart of the problems in the data perception of the method of the present invention; Figure 5 This is a comparison diagram of two problems in the data perception method of the present invention; Figure 6 This is a comparison chart of cognitive data sheets in the method of this invention; Figure 7 This is a comparison chart of two problems in cognitive data for the method of this invention. Detailed Implementation

[0052] like Figure 1 As shown, the steps of the multimodal cooperative method based on multi-agent uncertainty fusion and consistency self-check are as follows: Step 1) Obtain the input image and the corresponding question text. Use a publicly available dataset for testing; one image corresponds to one question, such as... Figure 2 , Figure 3 The chart shows a comparison of the total MME score on different models and the task scores in MME-P and MME-C. As can be seen from the chart, the total MME score of the model continuously improves during the improvement process. 1.1) This study uses the open-source dataset MME, which can be downloaded from official channels. This dataset comprehensively evaluates the capabilities of multimodal large models from two dimensions: perception and cognition. These two dimensions complement and support each other. The dataset has a rich and diverse sample composition. Some images are taken from public data sources, but only the original image information is used, without using their annotations. At the same time, a large number of manually captured images have been added, covering handwritten arithmetic problems, real-life images of daily scenes, and images generated by diffusion models, effectively reducing the risk of data leakage and ensuring the reliability and fairness of the evaluation.

[0053] 1.2) The dataset is loaded locally, mainly consisting of a perception component and a cognitive component. The perception task comprises ten sub-tasks, including recognition of basic attributes of common objects, domain-specific professional recognition, and image-text recognition. The cognitive task comprises four tasks. All images for the cognitive tasks are manually captured, and the instruction-answer pairs are manually designed to ensure no data leakage.

[0054] 1.3) The perception and cognition parts in the dataset are each image corresponding to a question, and are input into the full-modal large model in the form of each image corresponding to a question.

[0055] Step 2) Based on the input image and corresponding question, perform task analysis using a full-modal model to determine the intent and task type. Generate a task plan based on the understanding of the task. The full-modal model breaks down the task into several steps and checks for missing steps, correcting any omissions. Invite relevant visual and linguistic experts to execute the task according to the plan to obtain key evidence related to the image. 2.1) The full-modal model enters the perception stage. The full-modal large model performs task parsing based on the input image and question, analyzing the intent and task type, such as visual recognition counting tasks, spatial relationship tasks, and text output tasks, and generating concise semantic representations.

[0056] 2.2) The full-modal model enters the planning phase. Based on the understanding of the task, the full-modal model generates a task plan, transforms the semantic representation into executable task steps, clarifies the modal operations of each step, and arranges the sequence of each step to form a structured task plan, including the selection of the model (visual model, text model) and the selection of tools.

[0057] 2.3) The full-modal model enters the reflection phase, verifying whether the task plan matches the user's intent, identifying missing or redundant steps, such as supplementing the video text description if it is not included in the plan. This triggers plan revision, calls the relevant models to execute the task according to the plan, and obtains the main evidence for the image.

[0058] Step 3) Set up multiple agents, each providing answers based on different focuses. Each agent generates a main description from different angles based on key evidence. The generated main descriptions are then segmented into sentences. A full-modal model is used to calculate the relevance scores between sentences and questions. The top-k relevant sentences are selected, and softmax weights are assigned within the top-k. Each agent generates N alternative expression samples based on its own main description, and each of the top-k sentences is evaluated for support. The weighted illusion rate is then calculated, and a self-test score is determined. Figure 4 , Figure 5 As shown in the green curve of the graph, the scores are higher in perception tasks after adding the hallucination rate and sentence-level self-check. 3.1) Set up multiple agents in the full-modal model from different perspectives, and provide answers to questions based on different focuses. Each agent generates a master description from different perspectives based on the main evidence. The master description generated by each agent is divided into sentences. Using a full-modal model to provide sentences and questions correlation score The calculation formula is as follows:

[0059] in, It is every sentence. It's a problem with the image matching; It is the relevance score of each sentence to the question; the more relevant the sentence is to the question, the higher its relevance. The closer the value is to 1.

[0060] 3.2) Select the Top-k Relevant Sentences: Choose the k most relevant sentences, i.e., the k sentences with the highest relevance scores. Remove irrelevant sentences to reduce their interference with the score. The calculation formula is as follows:

[0061] in, It is every sentence; It is the relevance score of each sentence to the question; It is the set of Top-k sentences.

[0062] Softmax weights are then assigned within the Top-k sentence set, calculated using the following formula:

[0063] in, This represents the weight of a sentence in the Top-k sentence set of a proxy; It is the relevance score of each sentence to the question; It's temperature. The smaller the difference, the more amplified it will be; the most relevant sentence will have a very high weight. The larger the gap, the smoother the transition, and each sentence becomes more even.

[0064] Each agent generates N alternative representation samples based on its own master description and performs a support determination on each of the top-k sentences. The sentence support determination is calculated by the full-modal model using a yes or no approach, and is used to determine whether each of the top-k sentences can be stably supported by different paraphrased versions. The calculation formula is as follows:

[0065] in, The sentence support determination is 1 for support (Yes) and 0 for no support (No); This represents a sentence within the Top-k sentence set; This represents the description of the nth sample.

[0066] 3.3) Calculate the illusion rate of each sentence in the Top-k sentence set based on the support decision of each sentence. The number of "no" statements given by the full modality model based on N alternative expression samples for each sentence is calculated as follows:

[0067] in, This represents a sentence within the Top-k sentence set; This represents the description of the nth sample; This is a sentence support judgment, where 1 indicates support (Yes) and 0 indicates dissatisfaction (No). The more negative the sentence is by the substitute expression samples, the more unstable the sentence is and the higher the illusion rate.

[0068] 3.4) Calculate the weighted illusion rate based on the illusion rate of each sentence, which is the overall illusion rate of the Top-k sentence set generated by each agent. This is achieved by combining the illusion rates of each sentence according to their sentence weights. The calculation formula is as follows:

[0069] in, This represents the illusion rate of a sentence in the Top-k sentence set of an agent; H represents the weight of a sentence in a proxy's Top-k sentence set; H represents the overall illusion rate of each proxy's Top-k sentence set. The larger H is, the higher the overall illusion rate, and the worse the quality of evidence generated by that proxy.

[0070] The agent self-test score is calculated based on the overall illusion rate of the top-k sentence set for each agent, using the following formula:

[0071] Where H represents the overall illusion rate of the Top-k sentence set for each agent; score represents the self-test score of each agent. The higher the score, the lower the overall illusion rate and the better the quality of evidence generated by the agent.

[0072] Step 4) Calculate the uncertainty index of each agent based on the self-inspection score, and obtain the agent weight through weight allocation to realize agent-level evidence fusion and system uncertainty estimation, and give the candidate answer for each agent. At the same time, calculate the conflict metric based on the consistency of the candidate answers. 4.1) Calculate the uncertainty index for each agent based on the self-assessment score. The calculation formula is as follows:

[0073] in, This represents the self-test score for each agent. The higher the value, the lower the uncertainty. This represents the uncertainty index for each agent. The higher the value, the greater the uncertainty.

[0074] 4.2) Calculate the weight of each agent based on the uncertainty index of each agent, which will be used for subsequent calculation of the uncertainty of the entire system. The calculation formula is as follows:

[0075] in, It is the sharpness of the weight. The smaller the difference, the more amplified it will be, and the more reliable the agent will have a very high weight. The larger the gap, the more even it becomes, and the more evenly distributed among each agent. Weights that represent the uncertainty of an agent; This represents the uncertainty metric for each agent; agents that are more reliable... The smaller, The higher the weight.

[0076] 4.3) Calculate the uncertainty of the entire system based on the uncertainty index and weight of each agent, i.e., the uncertainty of all evidence provided by multiple agents. The calculation formula is as follows:

[0077] in, This represents the uncertainty index for each agent; This represents the weight of the uncertainty index corresponding to each agent; This indicates the uncertainty of the entire system. The larger the value, the higher the overall uncertainty, and the worse the quality of evidence in the entire system.

[0078] 4.4) Calculate the conflict value between agents by statistically analyzing the evidence provided by each agent and determining whether the conclusions drawn from the evidence provided by different agents are consistent using the full modality model. The calculation formula is as follows:

[0079] in, The answer with the most votes across all answer categories, representing the number of proxies who agree with the viewpoint or conclusion. The degree of conflict is indicated by the greater the disagreement between the agents.

[0080] Step 5) When the system uncertainty estimate or conflict metric exceeds the threshold, an iterative dispute focus extraction and targeted supplementary evidence mechanism is triggered. This involves resetting multiple proxies for self-checking, recalculating proxies' weights, and iterating until the convergence condition is met. The system's overall uncertainty is very low or the change is very small; further iteration yields little benefit; or the maximum number of iterations is reached. At this point, iteration stops. Figure 6 , Figure 7 As shown in the red line in the figure, the model scores higher on cognitive tasks after adding a multi-agent discussion mechanism to the model. 5.1) When the system uncertainty estimate or the change in Debate convergence exceeds the threshold, an iterative mechanism for extracting disputed points and providing targeted supplementary evidence is triggered. This involves resetting multiple proxies for self-checking and recalculating proxies' weights for iteration. The formula for calculating the change in Debate convergence is as follows:

[0081] in, Indicates the first Uncertainty of the entire system during round iteration; Indicates the first Uncertainty of the entire system during round iteration; This represents the convergence change of the Debate, if The value is very small, indicating that the benefits of continuing to iterate are low, and iteration should be stopped.

[0082] 5.2) If iteration is performed, multiple agents need to be reassigned, each providing a new answer based on different emphases. Each agent generates a main description from different perspectives based on key evidence, and the main description is re-segmented. A full-modal model is used to calculate the relevance score between sentences and the question. The top-k relevant sentences are selected, and softmax weights are assigned within the top-k. N alternative expression samples are generated based on each agent's main description, and the supportability of each of the top-k sentences is assessed to obtain a weighted illusion rate and calculate a self-check score. Based on the self-check score, an uncertainty index is calculated for each agent, and agent weights are obtained through weight allocation. This achieves agent-level evidence fusion and system uncertainty estimation, ultimately providing candidate answers for each agent.

[0083] 5.3) Iterate through multiple agents' self-checking arguments until the convergence condition is met. Stop iterating when the overall uncertainty of the system is very low or the change is very small, the benefit of continuing iteration is very low, or the maximum number of iterations is reached. The calculation formula is as follows:

[0084] in, This is the uncertainty threshold that indicates the system is sufficiently certain and debate can be stopped. It can be defined by the user, but a value of 0.15 to 0.25 is usually reasonable, which means stopping when the system's average score is above 0.75 to 0.85. express The convergence threshold, which essentially no longer changes, It is usually set to 0.01~0.03. The smaller the size, the stricter the requirements. Stop when there is basically no change. The larger the size, the looser the fit. Stop if there is no obvious change; Indicates the iteration round; This indicates the maximum number of iterations allowed, typically set to 3-4 rounds to prevent infinite loops. The threshold representing the conflict rate.

[0085] Step 6): Based on steps 3), 4), and 5), the self-check score of the fused evidence is used as the initial node of the graph search reasoning mechanism. Based on the evidence obtained after multi-agent fusion, candidate nodes are generated through action expansion, deduplication search iteration, and self-check scoring. The iteration ends when the highest score exceeds a threshold or the iteration depth exceeds the maximum depth. The answer corresponding to the optimal node is output.

[0086] 6.1) Using the evidence from the multi-agent self-check scores and argument fusion in steps 3), 4), and 5) as the initial nodes of the graph search reasoning mechanism, the calculation formula is as follows:

[0087] in, This represents the primary evidence for the execution of the full-modal model; This means that each agent concatenates the answers according to its own weight; This represents the statement evidence that is ultimately input into the initial node of the graph search reasoning mechanism.

[0088] according to The statement evidence input to the initial node of the graph search inference mechanism is used to generate a statement description input to the graph search inference mechanism by the full-modal model.

[0089] 6.2) Calculate the self-test score of the initial node, which will be used as the value of that node. The calculation formula is as follows:

[0090] in, It's a problem with the image matching; This describes the statements input into the graph search reasoning mechanism. The initial node score represents the graph search reasoning mechanism; If the initial node score is greater than the threshold for the graph search reasoning mechanism score, the graph search reasoning mechanism is not entered, and the answer is output directly. The calculation formula is as follows:

[0091] in, The threshold for the graph search reasoning mechanism score is set to 0.6; 6.3) If the initial node score is less than the threshold of the graph search reasoning mechanism score, then the graph search reasoning mechanism is entered. Candidate nodes are generated through action expansion, deduplication, and bundle search iterations, and self-scoring is performed. When the highest score is greater than the threshold or the iteration depth is greater than the maximum depth, the iteration ends, and the answer corresponding to the optimal node is output. The node retention rule calculation formula is as follows:

[0092] in, Indicates to retain the previous text One node; This represents the set of candidate nodes.

[0093] The formula for calculating the optimal node update rule is as follows:

[0094] in, This indicates that the node with the highest score has been visited.

[0095] The formula for calculating the termination of the iterative process in the graph search reasoning mechanism is as follows:

[0096] in, The threshold for the graph search reasoning mechanism score is set to 0.6; This represents the node with the highest self-check score that has been visited.

[0097] Example 1

[0098] Example: Input the loaded MME dataset, images, and corresponding questions for the images, and test a multimodal collaborative method based on multi-agent uncertainty fusion and consistency self-checking.

[0099] Step 1: Using the publicly available standardized dataset, the MME dataset is used as input. The input is an image, existence / 000000494427.jpg, and the corresponding question: Is there a laptop in this image? Please answer yes or no.

[0100] Step 2: First, a full-modal model is used for perception, and the input image and question are analyzed to determine the intent and task type. Second, based on the understanding of the task, the full-modal model generates a task plan, transforming the semantic representation into executable task steps. Finally, the full-modal model enters a reflection phase to verify whether the task plan matches the user's intent and to identify missing or redundant steps.

[0101] Step 3: Set up multiple agents (OBS, TEXT, REL) to answer questions and provide answers based on different focuses. Each agent generates a main description from different angles based on the main evidence. The generated main description is segmented into sentences, and a full-modal model is used to calculate the relevance score between the sentences and the question. The top-k relevant sentences are selected and softmax weights are assigned within the top-k. Each agent generates 8 alternative expression samples based on its own main description, and the top-k sentences are selected with k=3 for support judgment sentence by sentence. The weighted illusion rate is obtained and the self-test score is calculated.

[0102] Step 4: Calculate the uncertainty index for each agent based on the self-test score, and obtain the agent weight through weight allocation. OBS: , , TEXT: , , ;REL: , , Achieving agent-level evidence fusion and system uncertainty estimation. It also provides candidate answers for each agent and calculates a conflict metric based on the consistency of the candidate answers. .

[0103] Step 5: Since the system uncertainty estimate or conflict metric has not exceeded the threshold, there is no need to trigger the iterative dispute focus extraction and targeted supplementary evidence mechanism, and the iteration continues.

[0104] Step Six: Finally, the self-test score of the fused evidence is used as the initial node of the graph search reasoning mechanism. The weighted illusion score is calculated by adding the initial node score to the self-test score. , Based on the evidence obtained through multi-agent fusion, candidate nodes are generated through action expansion, deduplication search, and iterative scoring, and a self-check score is performed. The iteration ends when the highest score exceeds a threshold or the iteration depth exceeds the maximum depth. The answer corresponding to the optimal node is output.

Claims

1. A multi-modal collaborative method based on multi-agent uncertainty fusion and consistency self-checking, characterized in that, The steps are as follows: Step 1) Obtain the input image and the corresponding question text. Use a publicly available dataset for testing, with one image corresponding to one question. Step 2) Based on the input image and corresponding question, perform task analysis using a full-modal model to analyze the intent and task type. Generate a task plan based on the understanding of the task. The full-modal model breaks down the task into several steps and checks if any steps are missing. If so, it corrects them and calls upon relevant visual and linguistic experts to execute the task according to the plan, obtaining a key piece of evidence for the image. Step 3) Set up multiple agents to answer questions based on different focuses. Each agent generates a main description from different angles based on the main evidence. The generated main description is divided into sentences. The relevance score between the sentences and the question is calculated using a full-modal model. The top-k relevant sentences are selected and softmax weights are assigned within the top-k. Each agent generates N alternative expression samples based on its own main description and performs support judgment on each of the top-k sentences to obtain the weighted illusion rate and calculate the self-test score. Step 4) Calculate the uncertainty index of each agent based on the self-inspection score, and obtain the agent weight through weight allocation to realize agent-level evidence fusion and system uncertainty estimation, and give the candidate answer for each agent. At the same time, calculate the conflict metric based on the consistency of the candidate answers. Step 5) When the system uncertainty estimate or conflict metric exceeds the threshold, the iterative dispute focus extraction and targeted supplementary evidence mechanism is triggered. The multi-agent self-check is performed by resetting multiple agents, and the agent weights are recalculated for iteration until the convergence condition is met. The overall system uncertainty is set to 0.

45. When the overall system uncertainty is lower than 0.45 or the number of iteration rounds reaches the upper limit of 3 rounds, the iteration stops. Step 6) Based on Steps 3), 4), and 5), the self-check score of the fused evidence is used as the initial node of the graph search reasoning mechanism. Based on the evidence obtained after multi-agent fusion, candidate nodes are generated and self-check scores are obtained through action expansion, deduplication search iteration. The iteration ends when the highest score exceeds the threshold or the iteration depth exceeds the maximum depth. Output the answer corresponding to the optimal node.

2. The multi-agent uncertainty fusion and consensus self-checking based multi-modal collaboration method according to claim 1, characterized in that, In step 1), the specific method is as follows: Step 1.1) Obtain and download the open-source dataset MME from official channels to comprehensively evaluate the capabilities of multimodal large models from the two dimensions of perception and cognition. The dataset has a diverse sample composition. Some images are taken from public data sources, but only the original image information is used, and the annotation content is not used. At the same time, a large number of real-life images have been added, covering handwritten arithmetic problems, real-life images of daily scenes, and images generated by diffusion models. Step 1.2) Load the dataset locally, which includes the perception part and the cognition part; the perception task has ten sub-tasks, including the recognition of basic attributes of common objects, professional recognition of specific fields, and image and text recognition capabilities; the cognition task has four tasks. All images for the cognition tasks are taken manually, and the instruction-answer pairs are designed manually to ensure no data leakage. Step 1.3) The perception and cognition parts of the dataset are each image corresponding to a question, and are input into the full-modal large model in the form of each image corresponding to a question.

3. The multimodal collaborative method based on multi-agent uncertainty fusion and consistency self-checking as described in claim 1, characterized in that, In step 2), the specific method is as follows: Step 2.1) The full-modal model enters the perception stage: The full-modal big model performs task parsing based on the input image and question, parsing the intent and task type, such as visual recognition counting task, spatial relationship task, and text output task, and generates a concise semantic representation; Step 2.2) Full-modal model enters the planning stage: Based on the understanding of the task, the full-modal model generates a task plan, transforms the semantic representation into executable task steps, clarifies the modal operations of each step, and arranges the order of each step to structure the task plan, including the selection of visual models, text models, and tools. Step 2.3) The full-modal model enters the reflection phase to verify whether the task plan matches the user's intent, identify missing or redundant steps, and supplement them if the plan does not include video text descriptions; trigger plan revision, call the relevant models to execute the task according to the plan, and obtain the main evidence for the image.

4. The multimodal collaborative method based on multi-agent uncertainty fusion and consistency self-checking as described in claim 1, characterized in that, In step 3), the specific method is as follows: Step 3.1) Set up multiple agents in the full-modal model from different perspectives, answer questions and provide answers based on different focuses. Each agent generates a master description from different perspectives based on the main evidence. Divide the master description generated by each agent into sentences. Using a full-modal model to provide sentences and questions correlation score The calculation formula is as follows: in, It is every sentence. It's a problem with the image matching; It is the relevance score of each sentence to the question; the more relevant the sentence is to the question, the higher its relevance. The closer the value is to 1; Step 3.2) Select the Top-k relevant sentences: Choose the k most relevant sentences, i.e., the k sentences with the highest relevance scores, and remove irrelevant sentences to reduce the interference of irrelevant sentences on the score. The calculation formula is as follows: in, It is every sentence; It is the relevance score of each sentence to the question; It is a set of Top-k sentences; Softmax weights are then assigned within the Top-k sentence set, calculated using the following formula: in, This represents the weight of a sentence in the Top-k sentence set of a proxy; It is the relevance score of each sentence to the question; It's temperature; the difference will vary. The weight of the most relevant sentence will increase as the weight decreases. The decrease is amplified; Each agent generates N alternative representation samples based on its own master description and performs a support determination on each of the top-k sentences. The sentence support determination is calculated by the full-modal model using a yes or no approach, and is used to determine whether each of the top-k sentences can be stably supported by different paraphrased versions. The calculation formula is as follows: wherein, is a sentence support decision 1 representing support, i.e. Yes, and 0 representing no support, i.e. No; represents a sentence within the Top-k sentence set; represents a description of the n-th sample; Step 3.3) Calculate the illusion rate of each sentence in the Top-k sentence set based on the support decision of each sentence. The number of "no" statements given by the full modality model based on N alternative expression samples for each sentence is calculated as follows: in, This represents a sentence within the Top-k sentence set; This represents the description of the nth sample; This is a sentence support determination, with 1 indicating support and 0 indicating no support; the more negations a sentence receives from the substitute expression samples, the more unstable the sentence is and the higher the illusion rate. Step 3.4) Calculate the weighted illusion rate based on the illusion rate of each sentence, which is the overall illusion rate of the Top-k sentence set generated by each agent. The overall illusion rate is calculated by combining the illusion rates of each sentence according to their sentence weights. The calculation formula is as follows: in, This represents the illusion rate of a sentence in the Top-k sentence set of an agent; H represents the weight of a sentence in a proxy's Top-k sentence set; H represents the overall illusion rate of each proxy's Top-k sentence set. The larger H is, the higher the overall illusion rate, and the worse the quality of evidence generated by the proxy. The agent self-test score is calculated based on the overall illusion rate of the top-k sentence set for each agent, using the following formula: Where H represents the overall illusion rate of the Top-k sentence set for each agent; score represents the self-test score of each agent. The higher the score, the lower the overall illusion rate and the better the quality of evidence generated by the agent.

5. The multimodal collaborative method based on multi-agent uncertainty fusion and consistency self-checking according to claim 1, characterized in that, In step 4), the specific method is as follows: Step 4.1) Calculate the uncertainty index for each agent based on the self-test score. The calculation formula is as follows: in, This represents the self-test score for each agent. The higher the value, the lower the uncertainty. This represents the uncertainty index for each agent. The higher the value, the greater the uncertainty. Step 4.2) Calculate the weight of each agent based on the uncertainty index of each agent, which will be used for subsequent calculation of the uncertainty of the entire system. The calculation formula is as follows: in, It's about weight sharpness, setting... It is 1.6, with The larger the value, the more prominent the weight of the uncertainty agent; Weights that represent the uncertainty of an agent; This represents the uncertainty metric for each agent, and the reliability of the agent is related to... Inversely proportional to, with The weights are proportional; Step 4.3) Calculate the uncertainty of the entire system based on the uncertainty index and weight of each agent, that is, the uncertainty of all evidence given by multiple agents. The calculation formula is as follows: in, This represents the uncertainty index for each agent; This represents the weight of the uncertainty index corresponding to each agent; This indicates the uncertainty of the entire system. The larger the value, the higher the overall uncertainty, and the worse the quality of evidence in the entire system. Step 4.4) Calculate the conflict value between agents. This is done by statistically analyzing the evidence provided by each agent and using the full modality model to determine whether the conclusions drawn from the evidence provided by different agents are consistent. The calculation formula is as follows: in, The answer with the most votes across all answer categories, representing the number of proxies who agree with the viewpoint or conclusion. The degree of conflict is indicated by the greater the disagreement between the agents.

6. The multimodal collaborative method based on multi-agent uncertainty fusion and consistency self-checking according to claim 1, characterized in that, In step 5), the specific method is as follows: Step 5.1) When the system uncertainty estimate or the change in Debate convergence exceeds the threshold, the iterative dispute focus extraction and targeted supplementary evidence mechanism is triggered. This involves resetting multiple proxies for self-checking, recalculating the proxies' weights, and iterating. The formula for calculating the change in Debate convergence is as follows: in, Indicates the first Uncertainty of the entire system during round iteration; Indicates the first Uncertainty of the entire system during round iteration; This represents the change in the convergence of the Debate statement. The convergence threshold is 0.

01. A value less than 0.01 indicates that the benefit of continuing the iteration is low, and the iteration should be stopped. Step 5.2) If iteration is required, multiple agents need to be reassigned to answer the question and provide new answers based on different emphases. Each agent generates a main description from different angles based on the main evidence, and the main description is re-segmented. The relevance score between the sentence and the question is calculated using a full-modal model. The top-k relevant sentences are selected and softmax weights are assigned within the top-k. N alternative expression samples are generated based on the agent's main description, and the support of each of the top-k sentences is evaluated to obtain the weighted illusion rate and calculate the self-check score. The uncertainty index of each agent is calculated based on the self-check score, and the agent weight is obtained through weight assignment. Agent-level evidence fusion and system uncertainty estimation are achieved, and finally, candidate answers for each agent are given again. Step 5.3) The multi-agent self-checking argument iteration continues until the convergence condition is met and the overall uncertainty threshold of 0.45 is reached, or the number of iteration rounds reaches the upper limit of 3 rounds. Then, the iteration stops. The calculation formula is as follows: in, This indicates that the system is sufficiently certain, and the uncertainty threshold for stopping debate has been reached. express The convergence threshold that essentially no longer changes; Indicates the iteration round; This indicates the maximum number of iterations allowed.

7. The multimodal collaborative method based on multi-agent uncertainty fusion and consistency self-checking according to claim 1, characterized in that, In step 6), the specific method is as follows: Step 6.1) Using the evidence from the multi-agent self-check scores and argument fusion in steps 3), 4), and 5) as the initial nodes of the graph search reasoning mechanism, the calculation formula is as follows: in, This represents the primary evidence for the execution of the full-modal model; This means that each agent concatenates the answers according to its own weight; This represents the statement evidence that is ultimately input into the initial node of the graph search reasoning mechanism; according to The statement evidence input to the initial node of the graph search reasoning mechanism is used to generate a statement description input to the graph search reasoning mechanism by the full-modal model; Step 6.2) Calculate the self-test score of the initial node, which will be used as the value of that node. The calculation formula is as follows: in, It's a problem with the image matching; This describes the statements input into the graph search reasoning mechanism. The initial node score represents the graph search reasoning mechanism; If the initial node score is greater than the threshold of the graph search reasoning mechanism score, the graph search reasoning mechanism is not entered, and the answer is output directly; the calculation formula is as follows: in, The threshold for the graph search reasoning mechanism score is set to 0.6; Step 6.3) If the initial node score is less than the threshold of the graph search reasoning mechanism score, then enter the graph search reasoning mechanism; generate candidate nodes through action expansion, deduplication, and bundle search iterations and self-check the score. When the highest score is greater than the threshold or the iteration depth is greater than the maximum depth, the iteration ends and the answer corresponding to the optimal node is output. The node retention rule calculation formula is as follows: in, Indicates to retain the previous text One node; Represents the set of candidate nodes; The formula for calculating the optimal node update rule is as follows: in, This indicates that the node with the highest score was visited; The formula for calculating the termination of the iterative process in the graph search reasoning mechanism is as follows: in, The threshold for the graph search reasoning mechanism score is set to 0.6; This represents the node with the highest self-check score that has been visited.

Images