Fine-grained hallucination detection and correction method and device for multi-modal diffusion language model

Abstract

The application discloses a fine-grained hallucination detection and correction method and device of a multi-modal diffusion language model. The method finds a token with the maximum confidence fluctuation value, that is, a token most likely to produce hallucination, and then locks the semantic anchor point of the found token, so as to more easily lock a visual entity, so as to more accurately find a corresponding object image from an original image. The encoded object image, the original image, and an input mask containing a union mask are used as inputs of the trained multi-modal diffusion language model, so that the text segment needing to be repaired is aligned with the object image, the global semantics are maintained, the visual hallucination is locked and corrected, and a more accurate semantic description is obtained.

Description

Technical Field

[0001] This invention belongs to the field of visual image correction technology, specifically relating to a fine-grained method and apparatus for detecting and correcting illusions using a multimodal diffusion language model. Background Technology

[0002] Multimodal large language models have demonstrated powerful capabilities in visual understanding tasks. While existing multimodal large language models primarily employ autoregressive architectures, discrete diffusion language models are gradually emerging as a competitive paradigm. Due to their bidirectional context modeling and parallel decoding capabilities, multimodal diffusion language models (MDLMs) exhibit extremely high generation efficiency and global consistency in visual language understanding.

[0003] Patent application CN121708152A discloses a text-to-image generation method based on thought chain and visual prior guidance, implemented using a diffusion model. The method includes the following steps: obtaining original text prompts and constructing latent noise variables using Gaussian random noise; logically deconstructing the original text prompts using a large language model to generate a multi-dimensional semantic description set S; initializing the current denoising time step t and performing denoising processing to update the latent noise variables; decrementing the current denoising time step t by a step size of 1; determining whether the current denoising time step t is greater than 0, and if so, performing denoising processing again; if t=0, mapping the updated latent noise variables back to the pixel space through an image decoder to generate the final high-quality target image. This invention effectively solves the problems of semantic ambiguity, chaotic spatial layout, and visual fragmentation inherent in existing technologies in complex environments.

[0004] Patent application CN120355611A discloses an unsupervised CT image denoising method and apparatus based on multimodal large model text prompts. The method includes: using a pre-trained encoding / decoding model to initially denoise the original CT image, generating a first denoised CT image; inputting the first denoised CT image and preset text into a multimodal visual language large model to generate quality and detail prompts about the first denoised CT image; using a trained generative diffusion model to further denoise the first denoised CT image based on the quality and detail prompts, generating a second denoised CT image; inputting the second denoised CT image as the initial denoised CT image into a three-domain consistency iterative framework for fidelity preservation; and obtaining the final denoised CT image when the iteration stopping condition is met; wherein the three domains include the image domain, projection domain, and wavelet domain. This invention utilizes the emergent and representational capabilities of the large model, combined with a designed denoising framework, and can be conveniently applied to various noisy CT images.

[0005] Then, the existing MDLM disclosed in the aforementioned patent application encodes visual information as static features during initialization. In the subsequent iterative denoising and generation process, it lacks dynamic interaction with the visual content, causing the generation process to gradually deviate from visual evidence and tend towards linguistic priors, thus producing hallucinations.

[0006] In autoregressive generative paradigms, token confidence has been proven to be an effective measure of generative uncertainty. Existing work uses low-confidence tokens to mark potential hallucination locations, and then employs methods such as retrieval-enhanced generation and self-correction to correct hallucinations. Recent work has enabled vision-centric thinking by dynamically reviewing relevant visual information through tool invocation or insertion of visual tokens. However, there are structural barriers to transferring these strategies to MDLM to alleviate the hallucination problem.

[0007] First, MDLM employs a non-sequential generation method, resulting in discrete, semantically incomplete decoded tokens that hinder the evaluation of the inference process. Second, MDLM uses a fixed generation sequence length, limiting the dynamic insertion of visual tokens. Most critically, experiments show that the final token confidence level is unreliable as a measure of MDLM generation uncertainty, with many hallucination tokens maintaining high confidence levels. Therefore, a method compatible with diffusion architectures is urgently needed to achieve fine-grained detection and correction of visual hallucinations through endogenous signals generated during the denoising process. Summary of the Invention

[0008] This invention provides a fine-grained method for detecting and correcting illusions using a multimodal diffusion language model, which can achieve fine-grained detection and correction of visual illusions.

[0009] This invention provides a fine-grained method for hallucination detection and correction using a multimodal diffusion language model, comprising: S1. Based on the original image, the object image of the original image, the global description of the original image, the object description, and the mask description obtained by masking the object description of the global description, construct training data. Based on multiple training data, train a multimodal diffusion language model through the mask reconstruction loss function. S2. Encode the specified image, corresponding prompt text, and mask description, and input them into the trained multimodal diffusion language model to obtain a denoised object description. Identify the token with the largest confidence fluctuation value in the object description. Lock the semantic anchor point of the identified token through a syntax parsing tool. Map the identified token and semantic anchor point to the corresponding visual entity. Map the visual entity to the bounding box of the specified image through an open vocabulary detector to obtain the corresponding object image. At the same time, mask the semantic anchor point, the identified token, and the tokens between them to obtain a union mask. S3. Encode the specified image, object image, and corresponding prompt text, and concatenate them with the input mask. Input the concatenated encoded data into the trained multimodal diffusion language model to correct the denoised object description and obtain the corrected object description. The input mask includes a union mask.

[0010] Preferably, the confidence fluctuation value is calculated by subtracting the difference in confidence between tokens at adjacent time steps from the difference in confidence between the tokens corresponding to the mask description and the denoised object description.

[0011] Preferably, the identified token is used to lock semantic anchors using a syntax parsing tool, including: If the identified token is located within a predefined noun block, the noun block is locked as a semantic anchor; if the identified token is an isolated token, the syntactic dependency tree is traversed until the noun ancestor is reached, and the noun ancestor is locked as a semantic anchor.

[0012] Preferably, if the identified token is an isolated token, then the syntactic dependency tree is traversed until the noun ancestor is reached, including: If the identified token is a verb, then identify the subject or object related to the action of the verb; if the identified token is a preposition, then identify the object of the preposition; if the identified token is a modifier, then perform a recursive head traversal to process nested attributes, tracing the dependency chain upwards until the noun ancestor is reached.

[0013] Preferably, the corresponding object image detected by the open vocabulary detector also needs to satisfy the requirement that the detector confidence exceeds a preset threshold.

[0014] Preferably, the input mask further includes adjacent background tokens, which are token windows adjacent to each other in the union mask.

[0015] Preferably, steps S1 and S2 are iterated until a set number of iterations is reached, at which point the iteration is stopped and the final object description is obtained.

[0016] Preferably, the method for obtaining training data includes: The process involves labeling objects in the original image with bounding boxes and setting long reference expressions to obtain samples. A global description of the samples is then generated using a multimodal large model. This global description includes a detailed description of the object, i.e., an object description. The detailed description of the object is then masked to obtain a mask description. Finally, the object image is cropped from the original image using bounding boxes.

[0017] Preferably, a multimodal diffusion language model is trained based on multiple training data using a mask reconstruction loss function, including: The object image and the original image are encoded by a visual encoder to obtain the image encoding, and the mask description and text prompts are encoded by a text encoder to obtain the text encoding. The image encoding and text encoding are aligned by a projection layer and then input into a multimodal diffusion language model to obtain the denoised sequence. A mask reconstruction loss function is constructed based on the denoised and clear sequences. The parameters of the projection layer and the multimodal diffusion language model are updated using the mask reconstruction loss function based on multiple training data to obtain the trained multimodal diffusion language model.

[0018] The present invention also provides a fine-grained illusion detection and correction device for a multimodal diffusion language model, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the fine-grained illusion detection and correction method for the multimodal diffusion language model.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention identifies the tokens most likely to cause hallucinations by calculating the confidence fluctuations of tokens in the denoising trajectory of a multimodal diffusion language model. Then, semantic anchors of the tokens are located through syntactic analysis, and the union of these anchors yields a complete visual entity description. Based on this, corresponding local images are accurately located from the original image as visual evidence. The tokens contained in the visual entity description are generated through remasking. The encoded local image, global image, and relevant prompts are used as input to the trained multimodal diffusion language model to re-denoise the generated text sequence, ensuring that the text fragments requiring repair are aligned with the local images while preserving global semantics. This achieves the identification and correction of visual hallucinations, resulting in a more accurate semantic description. Attached Figure Description

[0020] Figure 1 The flowchart illustrates a fine-grained illusion detection and correction method for a multimodal diffusion language model provided in a specific embodiment of the present invention.

[0021] Figure 2 The probability density distribution diagrams of ordinary tokens, illusion tokens, and fine-grained tokens are provided for specific embodiments of the present invention.

[0022] Figure 3 The part-of-speech chart of the tokens with the highest and lowest confidence fluctuation values ​​(CF) provided for specific embodiments of the present invention, wherein, Figure 3 (a) in the figure is a statistical chart of the part-of-speech tags of the token with the highest confidence fluctuation value (CF). Figure 3 (b) in the figure is a part-of-speech chart of the token with the lowest confidence fluctuation value (CF). Detailed Implementation

[0023] This invention provides a fine-grained method for detecting and correcting visual hallucinations using a multimodal diffusion language model. The method aims to detect and correct visual hallucinations by mining the inherent uncertainty signals in the diffusion denoising process. A schematic diagram of the overall framework is shown below. Figure 1 As shown, it includes: S1. Based on the original image, the object image of the original image, and the global description, object description, and mask description obtained by masking the object description of the global description, training data is constructed. Based on multiple training data, a multimodal diffusion language model is trained through a mask reconstruction loss function.

[0024] To enable the model to handle dual-view inputs and correct text based on visual evidence, this invention constructs the VGR-Instruct dataset and performs supervised fine-tuning of the Multimodal Diffusion Language Model (MDLM).

[0025] Data Construction Pipeline: Most existing multimodal datasets are limited to global descriptions and lack a "local-global" correspondence. This invention, in a specific embodiment, constructs approximately 37,000 samples based on the RefCOCO+ dataset. The data construction process includes the following three steps: (1) Instance sampling: For each image, at most two object instances are sampled, and each instance corresponds to the text description of an object in the image and its bounding box coordinates.

[0026] (2) Description synthesis: Use advanced multimodal large models (such as Gemini-2.5-flash) to generate a global description for each sample and prompt it to include a detailed description of the specified object.

[0027] (3) Masking and cropping: A sliding window search matching algorithm is used in the global description. Description of objects in mid-range positioning And replace it with a mask marker to generate At the same time, based on the bounding box from the original image The corresponding area, i.e., the object image, is cropped out. As local visual evidence, each data instance can ultimately be represented as a quintuple. As training data, the training data corresponding to multiple samples are used to construct the training dataset.

[0028] The training strategy provided in this specific embodiment of the invention is as follows: A multimodal diffusion language model is trained based on multiple training data using a mask reconstruction loss function. The specific steps are as follows: by and As input conditions, the multimodal diffusion language model is taught from recover To maintain the model's fundamental capabilities, a small amount of globally generated description data was mixed in during training. The training objective can be modeled as minimizing the reconstruction loss of the masked regions:

[0029] in It is a mask sequence. It is a clear sequence, i.e., a truth value. It is a multimodal context, for global image description generation tasks. Q is a text prompt word; for tasks requiring localized refinement, This hybrid training method enables the model to maintain global semantic consistency while utilizing local visual features for text correction. It is understood that the real-world test cases provided in the specific embodiments of this invention use English examples to more easily represent the same number of tokens.

[0030] The specific embodiments of the present invention provide a method for training a multimodal diffusion language model based on multiple training data using a mask reconstruction loss function, including: Image encoding is obtained by encoding the object image and the original image using a visual encoder, and the mask description is used. The text prompt word Q is encoded using a text encoder to obtain the text encoding, for example... The sequence is "the [MASK][MASK][MASK] on the right side", and Q is "Look at the dual-view images and restore the text." After aligning the image encoding and text encoding through the projection layer, the sequence is input into the multimodal diffusion language model to recover the denoised sequence; for example, "the yellow arrow sign on the right side".

[0031] A mask reconstruction loss function is constructed based on the denoised and clear sequences. The parameters of the projection layer and the multimodal diffusion language model are updated using the mask reconstruction loss function based on multiple training data to obtain the trained multimodal diffusion language model, thus completing the dual-view perception training.

[0032] The MDLM provided by this invention is LLaDA-V.

[0033] S2 and S3: The specified image, the corresponding prompt text Q (e.g., "Describe the image in detail."), and the mask description are encoded and input into the trained multimodal diffusion language model to obtain a denoised object description. The token with the highest confidence fluctuation value in the object description is identified, i.e., the high-confidence fluctuation token, such as "wooden." The identified token is used to lock semantic anchors using a syntax parsing tool. Based on the identified tokens and semantic anchors (e.g., "wooden fence"), they are mapped to corresponding visual entities. An open vocabulary detector (in one embodiment, GroundingDINO) is used to map the visual entities to the bounding boxes of the specified image to obtain the corresponding object image. Simultaneously, the semantic anchors, the identified tokens, and the tokens between them are masked to obtain a union mask. The specified image, the object image (visual evidence), and the corresponding prompt text are encoded and concatenated with the input mask. The concatenated encoded data is input into the trained multimodal diffusion language model to correct the denoised object description, obtaining a corrected object description. The input mask includes a union mask, specifically: Phase 1: Uncertainty Localization. This invention utilizes the CF (Copycat) metric to measure visual uncertainty during the generation process. Since simple thresholding might extract irrelevant tokens such as punctuation marks, this invention proposes a syntax-aware localization strategy: Volatility Filtering: First, calculate the CF score of all tokens and identify the token with the highest volatility. As the top candidate word.

[0034] Semantic anchoring: using parsing tools (such as spaCy) to... Map to the corresponding visual entity. If the token is located within a predefined noun block, lock the entire block as a semantic anchor. If it is an isolated token (such as a verb or adjective), then the syntactic dependency tree is traversed. For verbs, the subject or object related to the action is identified; for prepositions, the object is located; for modifiers such as adjectives, a recursive head traversal is performed to process nested attributes, tracing the dependency chain upwards until the noun ancestor is reached, which is a word that can be easily detected in the image.

[0035] Union Mask: To ensure syntactic coherence, the final mask is defined as the union of the undulation markers and semantic anchors. ,in Indicates the starting position. The values ​​indicate the ending position, min(▪) is the position before the starting position, and max(▪) is the position after the ending position.

[0036] Phase 2: Visual Evidence Extraction. This invention utilizes the open-vocabulary detector Grounding DINO to map uncertain text fragments to bounding boxes in an image. To prevent the introduction of new hallucinations due to unclear local visual evidence, only when the detector confidence exceeds a preset threshold... Refinement is only performed when necessary. Otherwise, the extracted local visual information is considered insufficient, and the refinement stage only refers to the global image. To preserve the necessary environmental context, the gated visual regions are cropped, and regions accounting for less than 5% are expanded by a factor of 1.5. The extracted visual evidence is then encoded by an encoder.

[0037] Phase 3: Dual-view evidence guidance and refinement. In this phase, the text fragment containing the highest CF (Confirmation of Credential Analysis) along with... All adjacent background tokens were remasked together (experiments showed) (For best results). This extension creates a semantic buffer, enabling MDLM to regenerate the region with greater flexibility and coherence. To address the lack of context caused by only local image input, this invention employs a dual-view input strategy: the model is simultaneously input... and The local visual token (i.e., ...) is concatenated using a sequence concatenation method. The encoded local visual token is appended to the global visual token (which is about to be generated). After encoding the global visual token, MDLM leverages its native bidirectional attention mechanism to focus on fine-grained details while maintaining global semantic consistency. To ensure accurate alignment between local visual evidence and the text fragment requiring repair, only one fragment is processed per round, and a maximum number of refinement rounds is set to balance performance and computational efficiency. The refinement loop stops when the maximum number of rounds is reached.

[0038] Experimental observations in specific embodiments of this invention revealed that the confidence level of correctly predicted tokens increases smoothly over time, while tokens that produce hallucinations often exhibit drastic fluctuations during the denoising process. To quantify this phenomenon, this invention introduces Confidence Fluctuation (CF) to measure the cumulative bias of the confidence trajectory.

[0039] Specifically, the first i Confidence fluctuation value of each token for:

[0040] in, is token At time step confidence level The token described by the mask confidence level Tokens for denoising object descriptions The confidence level is calculated as follows. This metric is the difference between the cumulative absolute change and the net increment of the token confidence trajectory during the generation of MDLMs. In a specific embodiment of this invention, tests were conducted using LLaDA-V on the publicly available image description dataset DetailCaps-4870. The confidence change trajectory of each token in the generated text of each data point was preserved, and the normalized CF value was calculated. Gemini-2.5-flash was used to annotate the generated text of all data. Tokens that did not match the image description were labeled as illusion tokens; tokens that correctly described the image but involved fine-grained content were labeled as fine-grained tokens; and the remaining words were labeled as ordinary tokens. Experiments show that illusion tokens exhibit a clear trend of shifting towards higher CF values, such as... Figure 2 As shown, the probability density reflects the distribution frequency of each type of token within different CF value ranges. Within the high CF range, the probability density of illusion tokens significantly exceeds that of ordinary tokens. Furthermore, as... Figure 3 As shown in (a) and (b), the high-confidence (CF) marker group is mainly dominated by content words (nouns / verbs) related to visual semantics, while the low-confidence group is biased towards functional words (function words / punctuation). This indicates that low confidence reflects syntactic uncertainty, while the CF proposed in this invention is a specialized indicator of visual cognitive uncertainty caused by multimodal conflict.

[0041] The MDLMs provided in specific embodiments of this invention model text generation as a forward and backward process. Formally, let... Indicates a by A clear sentence composed of tokens, using and These represent the input image and text prompt, respectively. During the forward pass, from... Medium uniform sampling time step In this context, 0 represents a clear sentence, and 1 represents a completely masked sentence. The process of adding noise is equivalent to the process of masking a certain number of tokens. The time step t is equivalent to the probability of masking a token. Each token in the probability The noise sequence is generated by replacing the special mask marker [MASK]. In the reverse process, MDLMs iteratively denoise the fully masked sequence by predicting the masked tokens. At each time step... Model predicted probability This updates all tags in the entire sequence in parallel. The maximum predictable probability of a token is called the token's confidence level. :

[0042] in This represents the vocabulary, where each masked token has a probability of being a word at the current time. The probability of the word with the highest probability is used as the confidence level of that token. Based on this metric, high-confidence tokens are decoded, while the remaining tokens are re-masked and predicted in subsequent steps.

[0043] On the other hand, specific embodiments of the present invention also provide a fine-grained illusion detection and correction device for a multimodal diffusion language model, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the fine-grained illusion detection and correction method for the multimodal diffusion language model.

[0044] To verify the effectiveness of the method provided in the specific embodiments of the present invention, the specific embodiments of the present invention conducted a comprehensive experimental evaluation on five publicly available benchmark datasets, covering two major categories: fine-grained image description and hallucination evaluation.

[0045] Evaluation benchmark datasets: For fine-grained image captioning, three benchmarks were used: CapMAS, CapArena, and DetailCaps-4870. CapMAS assesses the trade-off between information density and accuracy through coverage and factuality, and uses the CLAIR metric to evaluate overall caption quality. CapArena is a pairwise preference benchmark, using the CapArena-Auto metric to measure caption win rate. DetailCaps-4870 provides high-quality, detailed reference captions, using the CAPTURE metric to evaluate recall for visual elements such as objects, attributes, and relationships. For hallucination evaluation, two benchmarks were used: AMBER-g and MMHal-Bench. AMBER-g uses four metrics: CHAIR measures the frequency of hallucinatory objects in the response; Cover measures the object coverage of the response; Hallucination rate (Hal) indicates the proportion of responses containing hallucinations; and Cog assesses whether hallucinations in a multimodal large model are similar to hallucinations in human cognition. MMHal-Bench evaluates hallucinations in open-ended responses, reporting the average quality score (0-6) and the Hallucination rate (Hal-Rate).

[0046] Baseline Models: The experiment was compared with four existing multimodal diffusion language models, including LLaDA-V, MMaDA, LaViDa, and FUDOKI. Three representative autoregressive models, LLaVA-1.5-7B, InternVL-2.5-7B, and Qwen2.5-VL-7B, were also selected as references.

[0047] Model Architecture and Training Configuration: This specific embodiment of the invention is based on LLaDA-V (8B parameters) and employs a hybrid fine-tuning strategy: the visual projection layer undergoes full parameter fine-tuning, while the language backbone network uses LoRA for low-rank adaptation. Specific hyperparameter configurations include: the visual encoder uses Siglip2-so400m-patch14-384; the target modules for LoRA are the attention layer and the feedforward network layer; the LoRA rank r is set to 64, the scaling factor α is set to 128, and the dropout rate is 0.05. Training is performed on four NVIDIA A100 (80GB) GPUs, with a batch size of 4 per GPU, a gradient accumulation step count of 8, and a learning rate of 1×10⁻⁶. -4 During the inference phase, the maximum generated length is 128 tokens, the denoising steps are 128, and the denoising speed is 1 token per step; the visual gating threshold τ is set to 0.40, and the maximum number of refinement rounds is set to 6; for areas with an area ratio of less than 5%, an expansion ratio of 1.5 is used. In a specific embodiment of this invention, the device LLaDA-VGR provided by this invention is obtained by adjusting the parameters of LLaDA-V using LoRA.

[0048] Main Experiment Results: Table 1 presents the experimental results on all five benchmark datasets. The proposed LLaDA-VGR achieves state-of-the-art performance across all multimodal diffusion language models and significantly narrows the gap with leading autoregressive models.

[0049] In terms of fine-grained image description, LLaDA-VGR achieved an average score of 60.84 on the CapMAS benchmark, exceeding the baseline LLaDA-V by 2.23 points. On the CapArena benchmark, LLaDA-VGR's CapArena-Auto score was -52.17, an improvement of 25.00 points compared to LLaDA-V's -77.17, indicating that its generated descriptions are more favored in terms of richness and accuracy. On the DetailCaps-4870 benchmark, LLaDA-VGR achieved a CAPTURE score of 62.35, surpassing the baseline LLaDA-V by 2.73 points, demonstrating the superior ability of the method of this invention in recalling fine-grained visual elements.

[0050] In terms of hallucination mitigation, LLaDA-VGR demonstrates significant robustness. On the AMBER-g benchmark generation task, the three hallucination metrics CHAIR, Hal, and Cog decreased by 2.2, 7.7, and 2.0 respectively compared to LLaDA-V. Simultaneously, the Cover metric improved from 61.8 to 64.5, indicating that LLaDA-VGR possesses more nuanced perceptual capabilities. On the MMHal-Bench benchmark, LLaDA-VGR achieved the lowest hallucination rate (0.60) and the highest quality score (2.92) in multimodal diffusion language models. These results validate that the present invention can effectively correct generative hallucinations in multimodal diffusion language models through visual evidence anchoring.

[0051] It is worth noting that even compared to strong autoregressive baselines such as Qwen2.5-VL-7B, the method of this invention demonstrates competitive performance on DetailCaps-4870, AMBER, and MMHal-Bench, highlighting the potential of diffusion models in reliable visual reasoning. The experimental data above show that LLaDA-VGR comprehensively outperforms all multimodal diffusion language models, significantly improving not only description quality but also effectively suppressing illusions, achieving the optimal balance between information richness and factual accuracy. Understandably, the test cases above use English for ease of token number correspondence.

[0052] Table 1 presents the experimental results on all five benchmark datasets.

Claims

1. A fine-grained method for hallucination detection and correction based on a multimodal diffusion language model, characterized in that, include: S1. Based on the original image, the object image of the original image, the global description of the original image, the object description, and the mask description obtained by masking the object description of the global description, construct training data. Based on multiple training data, train a multimodal diffusion language model through the mask reconstruction loss function. S2. Encode the specified image, corresponding prompt text, and mask description, and input them into the trained multimodal diffusion language model to obtain a denoised object description. Identify the token with the largest confidence fluctuation value in the object description. Lock the semantic anchor point of the identified token through a syntax parsing tool. Map the identified token and semantic anchor point to the corresponding visual entity. Map the visual entity to the bounding box of the specified image through an open vocabulary detector to obtain the corresponding object image. At the same time, mask the semantic anchor point, the identified token, and the tokens between them to obtain a union mask. S3. Encode the specified image, object image, and corresponding prompt text, and concatenate them with the input mask. Input the concatenated encoded data into the trained multimodal diffusion language model to correct the denoised object description and obtain the corrected object description. The input mask includes a union mask.

2. The fine-grained hallucination detection and correction method for a multimodal diffusion language model according to claim 1, characterized in that, The confidence fluctuation value is calculated by subtracting the difference in confidence between tokens at adjacent time steps from the difference in confidence between the tokens corresponding to the mask description and the denoised object description.

3. The fine-grained hallucination detection and correction method for a multimodal diffusion language model according to claim 1, characterized in that, The identified token is used to lock semantic anchors using a syntax parsing tool, including: If the identified token is located within a predefined noun block, the noun block is locked as a semantic anchor; if the identified token is an isolated token, the syntactic dependency tree is traversed until the noun ancestor is reached, and the noun ancestor is locked as a semantic anchor.

4. The fine-grained hallucination detection and correction method for the multimodal diffusion language model according to claim 3, characterized in that, If the identified token is an isolated token, then traverse the grammatical dependency tree until the noun ancestor is reached, including: If the identified token is a verb, then identify the subject or object related to the action of the verb; if the identified token is a preposition, then identify the object of the preposition; if the identified token is a modifier, then perform a recursive head traversal to process nested attributes, tracing the dependency chain upwards until the noun ancestor is reached.

5. The fine-grained hallucination detection and correction method for a multimodal diffusion language model according to claim 1, characterized in that, The corresponding object image detected by the open vocabulary detector also needs to meet the requirement that the detector confidence exceeds a preset threshold.

6. The fine-grained hallucination detection and correction method for a multimodal diffusion language model according to claim 1, characterized in that, The input mask also includes adjacent background tokens, which are token windows adjacent to each other in the union mask.

7. The fine-grained hallucination detection and correction method for a multimodal diffusion language model according to claim 1, characterized in that, Iterate through steps S1 and S2 until the set number of iterations is reached, then stop iterating to obtain the final object description.

8. The fine-grained hallucination detection and correction method for a multimodal diffusion language model according to claim 1, characterized in that, Methods for obtaining training data include: The process involves labeling objects in the original image with bounding boxes and setting long reference expressions to obtain samples. A global description of the samples is then generated using a multimodal large model. This global description includes a detailed description of the object, i.e., an object description. The detailed description of the object is then masked to obtain a mask description. Finally, the object image is cropped from the original image using bounding boxes.

9. The fine-grained hallucination detection and correction method for a multimodal diffusion language model according to claim 1, characterized in that, A multimodal diffusion language model is trained based on multiple training data using a mask reconstruction loss function, including: The object image and the original image are encoded by a visual encoder to obtain the image encoding, and the mask description and text prompts are encoded by a text encoder to obtain the text encoding. The image encoding and text encoding are aligned by a projection layer and then input into a multimodal diffusion language model to obtain the denoised sequence. A mask reconstruction loss function is constructed based on the denoised and clear sequences. The parameters of the projection layer and the multimodal diffusion language model are updated using the mask reconstruction loss function based on multiple training data to obtain the trained multimodal diffusion language model.

10. A fine-grained illusion detection and correction device for a multimodal diffusion language model, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the fine-grained illusion detection and correction method of the multimodal diffusion language model according to any one of claims 1-9.

Images