A multi-modal hallucination correction method for expert task assignment and visual memory rollback

Abstract

The application relates to the field of artificial intelligence and power system monitoring, and provides a multi-modal illusion correction method for expert task allocation and visual memory backtracking, which comprises the following steps: key concept extraction, processing input images and texts through a multi-modal perception system to extract key concepts; question generation and visual verification, generating a series of verification question sets for the key concepts to check whether the text and image information are consistent; expert task allocation and execution, distributing subtasks to the most matched experts for processing through a dynamic expert activation mechanism; visual backtracking, comparing historical visual information with the results generated by the expert module, and correcting the results in real time; multi-dimensional verification correction, weighting and integrating all output results of the expert module according to the roles of the experts in actual tasks; and the application solves the multi-modal illusion problem and greatly improves the intelligent decision-making capability in the fields of power monitoring and equipment fault detection.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and power system monitoring, and more specifically, to a multimodal hallucination correction method involving expert task allocation and visual memory retrieval. Background Technology

[0002] Multimodal large language models commonly suffer from the illusion problem when processing visual and textual information, i.e., generating content descriptions inconsistent with the visual input. This problem is particularly prominent in applications with extremely high accuracy requirements, such as power monitoring. Current solutions to multimodal illusions mostly focus on intervention during the decoding stage. This process can be mainly divided into two categories: intervention during the generation process and post-generation processing. The most prominent intervention method in the generation process is the contrastive decoding scheme. Considering that language priors and statistical biases are considered the main causes of hallucinations, various contrastive decoding strategies attempt to eliminate the influence of model prior knowledge by "subtracting" from the model. Taking the VCD method as an example, its core logic is shown in the formula: , in Representing images with added visual perturbations, VCDs amplify prior linguistic information by adding noise to the original image. This amplified prior linguistic information is then compared to the probability distribution of the original output to arrive at the correct answer to remove the illusion. While this type of method incorporates adaptive constraints to preserve the model's highly confident output, when the image content closely matches prior knowledge, the contrastive decoding method can easily exclude the originally correct answer, leading to the illusion. Additionally, there is the Chain of Thought (CoT) expert module chain reasoning method, which utilizes a single multimodal model to play different expert roles to complete complex reasoning tasks. While this approach simplifies the system architecture, it suffers from fixed expert module designs, difficulty adapting to diverse task requirements, and a tendency for error propagation in the decision chain. Particularly when dealing with abnormal situations in power systems, a single incorrect judgment can cause the entire reasoning chain to fail. Furthermore, this approach does not fully utilize visual information, making it prone to misjudgments in critical tasks such as substation equipment identification and status assessment.

[0003] Post-processing correction methods are a typical multi-stage pipeline-style de-illusion scheme. Their core idea is to reduce or eliminate illusions after the initial output is generated by a multimodal large model, followed by a series of verification and correction steps. These methods typically include five consecutive stages: key concept extraction, question formulation, visual knowledge verification, visual statement generation, and final illusion correction. Specifically, the system first extracts key entities or attributes that may be involved in the illusion from the model-generated text. Then, it automatically generates targeted questions around these concepts and performs multiple rounds of verification on the image content using object detection, image question answering (VQA), or other visual expert models. The verification results are then integrated into a structured visual statement, which is used to revise the original output. However, these schemes rely on multiple external expert models such as object detectors and VQA models, increasing system complexity and computational overhead. Furthermore, the correction process is a passive post-processing approach, unable to intervene in real-time during the generation process. In power monitoring scenarios, this approach lacks accuracy in recognizing technical terms related to substation equipment status and struggles to handle complex power grid topologies and dynamic changes. Summary of the Invention

[0004] The problem addressed by this invention is how to provide an efficient, flexible, and accurate method for correcting multimodal illusions.

[0005] To address the aforementioned problems, this invention provides a multimodal hallucination correction method for expert task allocation and visual memory retrieval, comprising: key concept extraction, which involves processing input images and text through a multimodal perception system to extract key concepts; Question generation and visual verification: Generate a set of verification questions targeting key concepts to check whether the text and image information are consistent; Expert task allocation and execution: The expert module includes multiple intelligent experts. Through a dynamic expert activation mechanism, sub-tasks are assigned to the most suitable expert for processing. If the confidence level of a single expert's judgment is lower than a set threshold, the system automatically calls other relevant experts for cross-validation and performs visual backtracking. Visual retrospection compares historical visual information with the results generated by the expert module and corrects the results in real time. Multi-dimensional verification and correction involves weighting and integrating all outputs from the expert module according to the role each expert played in the actual task, resulting in the final corrected answer. .

[0006] Furthermore, the process of processing the input images and text using a multimodal perception system to extract key concepts includes: A two-layer concept recognition architecture, consisting of a general concept layer and a specialized concept layer, is adopted to process inputs from different domains, extracting a set of key concepts from the input text and images; Based on a set of key concepts, the weights of general concepts and specialized concepts are calculated separately using a normalization function, and a balance coefficient is introduced. To obtain the integrated concept weights that combine general concept understanding with professional concept understanding.

[0007] Furthermore, generating a set of verification questions targeting key concepts to check the consistency between text and image information includes: generating the set of verification questions based on fused concept weights, visually verifying each verification question using object detection, visual question answering, and image segmentation techniques, and obtaining verification results. If the verification result Greater than the set threshold If the answer matches the image, then the expert task is assigned and executed; otherwise, a backtracking mechanism is triggered to re-evaluate the image information.

[0008] Furthermore, the visual verification of each verification question using object detection, visual question answering, and image segmentation techniques includes: using an object detection model to identify and locate key objects in the image; having a visual question answering model answer specific natural language questions based on the image content; and, with the help of the object detection and visual question answering models, matching the device in the text description with the detected objects in the image to further verify the device's attributes.

[0009] Furthermore, the expert module includes four intelligent experts: a device status identification expert, a signal priority evaluation expert, an operation process verification expert, and a system consistency check expert.

[0010] Furthermore, the dynamic expert activation mechanism, which assigns subtasks to the most suitable expert for processing, includes: constructing a task relevance matrix. and expert capability matrix The activation probability of the expert is calculated as follows: Based on the expert activation probability, subtasks are assigned to the matching expert modules for processing.

[0011] Furthermore, the step of comparing historical visual information with the results generated by the expert module and correcting the results in real time includes: A visual backtracking mechanism is designed based on the key-value memory characteristics of the FFN layer, given the current hidden state. Visual features after querying and alignment The visual backtracking formula for the l-th FFN layer is: Where α is the proportion of visual information injected, which is proportional to the image complexity. This indicates the process of visually backtracking visual features using hidden states; The original FFN layer consists of two fully connected layers and a nonlinear activation between them, and its expression is: , The activation function and weight matrix can also be written as: , ,in and These correspond to the key and the value, respectively. The formal formula for the retrieval process that performs visual backtracking at layer l is as follows: .

[0012] Furthermore, the final corrected answer The expression is: ,in, These are the weighting coefficients for the four AI experts. These are the outputs of four AI experts.

[0013] Compared with the prior art, the beneficial effects of the present invention are: First, the expert task allocation mechanism breaks down complex multimodal tasks into multiple sub-tasks and flexibly selects appropriate expert modules for processing based on task type and processing stage. This task allocation method avoids the error propagation problem caused by fixed expert modules in traditional methods, enabling the system to process different types of tasks more efficiently and improving the accuracy and stability of the overall reasoning process. Secondly, this solution innovatively introduces a dynamic expert activation mechanism, which enables the system to dynamically adjust the activation of expert modules and task allocation according to actual task requirements, ensuring the flexibility and efficiency of task execution. Unlike the rigid expert chain in traditional methods, dynamic expert activation enables the system to respond quickly and provide accurate solutions when facing complex and ever-changing tasks. In addition, this solution uses a visual backtracking mechanism to correct the current reasoning results in real time using historical visual information. This mechanism can intervene in the generation process in real time, avoiding the lag and redundant calculations in traditional post-processing methods, and greatly improving the real-time performance and accuracy of the system. Through these technological innovations, this solution not only effectively solves the multimodal illusion problem, but also significantly enhances intelligent decision-making capabilities in fields such as power monitoring and equipment fault detection, providing strong support for real-time monitoring and decision-making in complex environments. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the overall process of an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the visual backtracking principle of an embodiment of the present invention. Figure 3 This is a table showing the POPE test set results of an embodiment of the present invention; Figure 4 This is the MME test set result table of an embodiment of the present invention. Detailed Implementation

[0015] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0016] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0017] In the description of this specification, references to terms such as "embodiment," "one embodiment," and "one implementation" indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or implementation is included in at least one embodiment or illustrative implementation of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or implementation. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or implementations.

[0018] like Figure 1 As shown, the core technical architecture of the multimodal hallucination correction method of expert task allocation and visual memory retrieval of the present invention includes five main modules: key concept extraction module, question generation and visual verification module, expert task allocation and execution module, visual retrieval module, and multidimensional verification and correction module.

[0019] The key concept extraction module is the intelligent entry point for the entire solution. A crucial step involves processing the input images and text using a multimodal perception system to extract key concepts. Considering the method's versatility, this process employs a two-layer concept recognition architecture to handle inputs from different domains. On one hand, the general concept layer uses a pre-trained concept classifier to identify basic concepts such as everyday objects, attributes, and actions. On the other hand, the professional concept layer loads relevant professional knowledge bases based on the application domain, and uses a large language model (LLM) or multimodal large language model (MLLM) to jointly analyze information in text and images, identifying important objects, attributes, and the relationships between them. In the power grid monitoring scenario, the professional concept layer includes a power equipment ontology library, an operational terminology dictionary, and status description specifications, enabling accurate identification of complex equipment numbers such as "10kV Park #1 Capacitor Park 41 Switch" and professional status descriptions such as "Closed in Position." Formally, it is assumed that the input is text. and images We extract a set of key concepts from it. ,in This represents key concepts, such as equipment failure and trip signals. First, the weights of general concepts are calculated. Then, calculate the professional concept weights based on the specific type of task. Ultimately, it can be represented as: , in Represents the balance coefficient. This is a weight matrix, representing the strength of the association between general concepts and extracted key concepts. This is a weight matrix that represents the strength of the association between power industry concepts and extracted key concepts, including prior knowledge of power equipment ontology, operational terminology, and state specifications. By adaptively fusing general concept understanding with domain-specific concept understanding, the model can better handle problems in various scenarios, while automatically identifying domain-related key information, providing accurate concept anchors for subsequent memory retrieval and expert decision-making.

[0020] The question generation and visual verification module needs to generate a series of verification questions based on the extracted concepts, aiming to check whether the text and image information are consistent. For example, for device status information, questions can be generated. Each of the problems The format is: "Does the image show a device {object} malfunction?" or "Are the device status and fault signals consistent?". After generating the questions, the next task is to visually verify these questions. The core of the verification is to confirm the accuracy of the description in the generated text through image information. Visual verification is accomplished through a specialized visual model, mainly using techniques such as object detection, visual question answering (VQA), and image segmentation, combined with the specific content of the questions for judgment. The specific steps are as follows: First, an object detection model (such as YOLO, Faster R-CNN, RetinaNet, etc.) is used to identify and locate key objects in the image; then, the VQA model answers specific natural language questions based on the image content; with the help of object detection and the VQA model, the system can match the device in the text description with the detected objects in the image and further verify the device's attributes (such as status, location, etc.); finally, the answer is judged and consistency is checked. Formally, the verification results of the questions can be... Represented as: , in, Image data, representing the raw visual input images collected by the power grid monitoring robot. For the i-th verification question, if the verification result... Greater than the set threshold If the answer matches the image, the system considers it correct and proceeds to the next step; otherwise, the system will trigger a backtracking mechanism to re-evaluate the image information.

[0021] The expert task allocation and execution module borrows from chain reasoning but has been redesigned to meet the specific needs of power grid monitoring. This module includes four specialized intelligent experts: equipment status identification expert, signal priority assessment expert, operation procedure verification expert, and system consistency check expert. Unlike a fixed expert call chain, this solution employs a dynamic expert activation mechanism based on task type and processing stage. This is achieved through a task relevance matrix. and expert capability matrix The probability of expert activation is calculated as follows: .

[0022] When the monitoring robot handles abnormal signal processing tasks, the system first activates equipment status identification experts to analyze the signal source and equipment status. Then, it calls upon signal priority assessment experts to determine the urgency of the task. Finally, it verifies the consistency of information between the D5000 and the new-generation centralized control system through system consistency checks. This dynamic collaborative mechanism avoids error propagation problems in fixed links. When the confidence level of an expert's judgment is low, the system automatically calls upon other relevant experts for cross-validation. Through this task allocation mechanism, we can ensure that each task is handled by the most suitable module, thereby improving the overall system efficiency and accuracy.

[0023] Nevertheless, in complex and dynamic environments, certain scenarios may lead to uncertainty in the initial inference results. In such cases, the system can use a visual backtracking module to introduce previously stored visual information for correction. The purpose of visual backtracking is not simply to supplement MLLMs with visual information, but rather to ensure the relevance of visual evidence to the current prediction. Previous research has found that the FFN layer can act as a key-value store for storing factual knowledge. Inspired by this viewpoint, "visual backtracking" can be viewed as a simple information re-retrieval process. Specifically, given the current hidden state... Aligned visual features Then, the formal representation of the FFN layer performing visual backtracking in the l-th layer is: , in, The proportion of visual information injected should be determined with reference to the actual situation and is proportional to the complexity of the image itself. This represents the process of visual regression using hidden states to access visual features; the original FFN layer consists of two fully connected layers and a non-linear activation between them, and can be represented as: ; This represents activation functions such as ReLU and SiLU. Additionally, the weight matrix can also be written as... , ,in and These correspond to keys and values, respectively; therefore, FFN can be interpreted as using input... As a query, calculate its relation to The similarity is used to search for matching values; similarly, the retrieval process that performs visual backtracking at the l-th layer can be formalized as follows: .

[0024] From the perspective of FFN, the working principle of visual backtracking is to hide the state. Treating it as a query, combining visual features into Treating these as new key-value entries, the system supplements visually relevant information through a matching retrieval process. This process does not rely on any additional parameters requiring training, allowing the system to correct its original inference results based on the new visual information, thus reducing the occurrence of hallucinations. The overall process flow is as follows: Figure 2 As shown, the blue area on the left side of the figure represents the hierarchical structure of the multimodal large model, corresponding to the core layer of the Transformer encoder or the FFN feedforward neural network layer. The bar chart on the right side of each layer shows the distribution of hidden state features for that layer. The eye icons marked on layers 16 and 24 are the trigger nodes for visual memory retrieval. After retrieval, the feature distribution changes from "fluctuating and deviating from visual facts" to "stable and consistent with the real image," effectively suppressing the influence of language priors and eliminating the risk of hallucination.

[0025] Finally, the outputs of all expert modules will be weighted and integrated according to the role each expert played in the actual task to form the final corrected answer. In a formal sense, there are: ,in, These are the weighting coefficients for the four AI experts. These are the outputs of four intelligent experts; through this series of steps, we ensure that in the complex and ever-changing power monitoring environment, the monitoring bionic robot can process various information in real time and accurately, providing efficient decision support.

[0026] In one embodiment, taking the task of recognizing the opening and closing status of switch #5 in a power grid park as an application scenario, the input consists of a real-world image I+ of switch #5 collected on-site by a robot and equipment text information T (including equipment number and historical closing operation records) transmitted from the power grid backend. The preset visual verification threshold γ=0.8, expert inference confidence threshold β=0.9, visual information injection ratio α=0.7 (for moderately complex equipment images), and balance coefficient λ=0.3 (emphasizing power grid professional concept recognition). The complete execution process is as follows: Step S1: Extraction of key concepts: Multimodal input reception: Acquire high-definition image I of switch #5 collected by the monitoring robot (including switch body, mechanical position, contact status, etc.) + background text T (e.g., switch #5, historical normal closing operation, no fault records, etc.); General concept layer: Extract basic concepts such as switch, capacitor, device, and location through a pre-trained classifier to form a key concept set X1; Professional concept layer: Load the power grid equipment body library and operation terminology dictionary, extract power grid professional concepts such as closing status, opening status, and equipment number, and form a key concept set X2; Integrate the key concepts X [5# switch, closed status, open status, equipment normal], assuming the characteristic values ​​of each concept are [0.98, 0.85, 0.30, 0.90]; Weight calculation and fusion: General concept weight =[ 0.23,0.21,0.18,0.16]; Professional concept weight =[ 0.25,0.24,0.18,0.13]; With λ=0.3, the concept weights for fusion are obtained as C= [0.244,0.231,0.180,0.139]; The output results are integrated with the concept weight C, among which switch #5 (0.244) and closing status (0.231) are core high-weight concepts, providing precise anchor points for subsequent question generation.

[0027] Step S2, Problem Generation and Visual Verification: Based on the fusion weight C, a set of verification questions Q is generated: q1: Is the image of switch #5? q2: Is the switch in a fault-free state? Visual verification execution (YOLOv8 object detection + power grid customized VQA model): Object detection: Locate the switch (circle 41) in the image, match its device ID feature, and confirm the device's identity; VQA model inference: Input image I and a single question qi, calculate the verification result. Verification results: V(q1) = 0.99 and V(q2) = 0.92 are obtained, which are greater than γ = 0.8. Proceed to the next step.

[0028] Step S3, Expert Task Allocation and Execution: Constructing a task matrix: For the task of "switch opening and closing status identification", construct a task correlation matrix. =[1,0.1,0.2,0.3] (corresponding to experts in device status identification, signal priority assessment, operation process verification, and system consistency check, respectively). Expert Capability Matrix: Utilizing a customized expert capability matrix for the power grid (4×4) represents the processing ability of each expert for the switch state identification subtask; Calculate the activation probability: =[0.95,0.08,0.12,0.20], the Device Status Recognition Expert has the highest activation probability (0.95), so it should be activated first; Expert inference execution: Based on the features extracted from the image and combined with the power grid switch status recognition standard, the equipment status recognition expert outputs the preliminary inference result: the switch is in the closed state, with an inference confidence level of 0.62; Confidence level assessment: 0.62 < β = 0.9, indicating uncertainty in the expert's reasoning result, triggering visual memory recall.

[0029] Step S4, Visual Memory Retrieval (Correcting the underlying features of expert reasoning in S3): Extract the current S3 stage model hidden state x2; retrieve the visual feature library again. Using x2 as the query, a new search and matching of visual features was performed to confirm the "visual features of the tripped state". ",45" is the optimal match; Visual backtracking computation (based on the 24th FFN layer, with a higher level of abstraction, catering to the needs of expert professional reasoning): According to the formula The fused features, α=0.7, yielded the corrected hidden state x2′. The activation value of the tripping feature was further increased to 0.95, while the closing feature was reduced to 0.25, perfectly matching the actual visual state of the image. The corrected hidden state x2′ is reflowed to S3, and the expert reasoning is completed again: Based on the corrected hidden state x2′, the expert activation probability is recalculated, and the device state recognition expert (0.96) is still activated first. At the same time, because the reasoning involves state accuracy, the operation process verification expert (0.25) is automatically activated for cross-validation; and the two experts conduct collaborative reasoning. Equipment status identification expert: Based on the high opening characteristics of x2′, combined with the visual characteristics of switch contacts and mechanical positions, it is inferred that "Switch #5 is in the open state", with a confidence level of 0.98; Operational procedure verification expert: Cross-verified the visual characteristics of the switch opening status and the matching of the power grid operation specifications, confirming that "the opening status characteristics are complete and without abnormalities", with a confidence level of 0.97; Confidence determination: The confidence scores of both experts' reasoning are ≥β=0.9, so the reasoning result is determined to be valid, and proceed to S5.

[0030] Step S5: Multi-dimensional verification and correction: Expert weighting coefficients are assigned as follows: Based on the core requirements of this task (primarily status identification, supplemented by process verification), the weights are set as follows: α1=0.7 (equipment status identification expert), α2=0 (signal priority assessment expert, not activated), α3=0.3 (operation process verification expert), and α4=0 (system consistency check expert, not activated). Weighted integration calculation: according to the formula The integrated results are as follows: y1 represents the "opening state, confidence level 0.98", and y3 represents the "opening state matching specification, confidence level 0.97". Generate a final, hallucination-free result: Integrate the conclusions of both experts, eliminate the "closing the switch" illusion caused by the prior knowledge of the original language, and form a unique and accurate decision result.

[0031] To verify the effectiveness of this invention, we conducted manual evaluation on a power grid monitoring robot task dataset to demonstrate its superiority in image removal. Furthermore, to verify the universality of this solution, we performed comparative experiments on two public multimodal datasets, POPE and MME. The experimental results... Figure 3 and 4 As shown; In summary, this invention significantly outperforms commonly used mainstream methods, both on power grid datasets and on open public datasets. This demonstrates that the "expert task allocation and visual memory backtracking" mechanism proposed in this invention has cross-task and cross-modal universality and leading advantages, ensuring that the model maintains higher accuracy and consistency in different application scenarios.

[0032] While the disclosure is as stated above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of this disclosure, and all such changes and modifications will fall within the protection scope of this invention.

Claims

1. A multimodal hallucination correction method based on expert task allocation and visual memory recall, characterized in that, include: Key concept extraction involves processing the input images and text using a multimodal perception system to extract key concepts. Question generation and visual verification: Generate a set of verification questions targeting key concepts to check whether the text and image information are consistent; Expert task allocation and execution: The expert module includes multiple intelligent experts. Through a dynamic expert activation mechanism, sub-tasks are assigned to the most suitable expert for processing. If the confidence level of a single expert's judgment is lower than a set threshold, the system automatically calls other relevant experts for cross-validation and performs visual backtracking. Visual retrospection compares historical visual information with the results generated by the expert module and corrects the results in real time. Multi-dimensional verification and correction involves weighting and integrating all outputs from the expert module according to the role each expert played in the actual task, resulting in the final corrected answer. .

2. The multimodal hallucination correction method based on expert task allocation and visual memory retrieval according to claim 1, characterized in that, The process of processing input images and text using a multimodal perception system to extract key concepts includes: A two-layer concept recognition architecture, consisting of a general concept layer and a specialized concept layer, is adopted to process inputs from different domains, extracting a set of key concepts from the input text and images; Based on a set of key concepts, the weights of general concepts and specialized concepts are calculated separately using a normalization function, and a balance coefficient is introduced. To obtain the integrated concept weights that combine general concept understanding with professional concept understanding.

3. The multimodal hallucination correction method based on expert task allocation and visual memory retrieval according to claim 2, characterized in that, The generation of a set of verification questions targeting key concepts to check the consistency between text and image information includes: A set of verification questions is generated based on fused concept weights. Each verification question is then visually verified using object detection, visual question answering, and image segmentation techniques to obtain the verification results. If the verification result Greater than the set threshold If the answer matches the image, then the expert task is assigned and executed; otherwise, a backtracking mechanism is triggered to re-evaluate the image information.

4. The multimodal hallucination correction method based on expert task allocation and visual memory retrieval according to claim 3, characterized in that, The process of visually verifying each verification question using object detection, visual question answering, and image segmentation techniques includes: using an object detection model to identify and locate key objects in the image; having a visual question answering model answer specific natural language questions based on the image content; and, with the help of the object detection and visual question answering models, matching the device in the text description with the detected objects in the image to verify the device's attributes.

5. The multimodal hallucination correction method for expert task allocation and visual memory retrieval according to claim 4, characterized in that, The expert module includes four intelligent experts: device status identification expert, signal priority evaluation expert, operation process verification expert, and system consistency check expert.

6. The multimodal hallucination correction method based on expert task allocation and visual memory retrieval according to claim 5, characterized in that, The dynamic expert activation mechanism assigns subtasks to the best-matching expert for processing, including: Construct a task relevance matrix and expert capability matrix The activation probability of the expert is calculated as follows: Based on the expert activation probability, subtasks are assigned to the matching expert modules for processing.

7. The multimodal hallucination correction method for expert task allocation and visual memory retrieval according to claim 6, characterized in that, The step of comparing historical visual information with the results generated by the expert module and correcting the results in real time includes: A visual backtracking mechanism is designed based on the characteristics of the FFN layer key-value memory, given the current hidden state. Visual features after querying and alignment The visual backtracking formula for the l-th FFN layer is: Where α is the proportion of visual information injected, which is proportional to the image complexity. This indicates the process of visually backtracking visual features using hidden states; The original FFN layer consists of two fully connected layers and a nonlinear activation between them, and its expression is: , The activation function and weight matrix can also be written as: , ,in and These correspond to the key and the value, respectively. The formal formula for the retrieval process that performs visual backtracking at layer l is as follows: .

8. The multimodal hallucination correction method for expert task allocation and visual memory retrieval according to claim 7, characterized in that, The final corrected answer The expression is: ,in, These are the weighting coefficients for the four AI experts. These are the outputs of four AI experts.

Images