A Stability-Aware Multimodal Large Model Hallucination Suppression Method

By estimating the token reliability of the visual encoder and performing attention calibration, the object illusion problem in large visual language models is solved, achieving a significant reduction in illusion rate and preservation of multimodal capabilities.

CN123117201APending Publication Date: 2026-11-17SICHUAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611579183.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-13
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Large visual language models suffer from object illusion during the generation process. Existing methods typically only intervene in one side of the multimodal generation process and cannot explicitly control how the language decoder utilizes refined visual evidence.

Method used

Token-level visual reliability is estimated by using bounded adversarial perturbation probes to suppress perturbation-sensitive tokens, and reliability-aware visual priors are injected into the early-mid layers of the large language model through quality-preserving attention calibration, explicitly routing visual attention.

Benefits of technology

It significantly reduces the object illusion rate, maintains the model's general multimodal capabilities, is applicable to mainstream visual language model architectures, requires no additional training or external detectors, and has wide applicability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123117201A_ABST
    Figure CN123117201A_ABST
Patent Text Reader

Abstract

This invention discloses a stability-aware multimodal large-scale model illusion suppression method, relating to the fields of computer vision and artificial intelligence. Addressing object illusion in large-scale visual language model generation, this invention first estimates the perturbation sensitivity of visual tokens using a bounded adversarial perturbation probe, generating a token-level reliability mask. Second, the reliability mask is injected into a pre-defined intermediate layer of the visual encoder to suppress perturbation-sensitive tokens and extract reliability-aware visual priors. Furthermore, the priors are injected into the early-mid layers of the large language model through quality-preserving attention calibration, redistributing the visual attention distribution while maintaining the overall quality of image token attention. Finally, the model is guided to focus on reliable visual evidence, significantly reducing object illusion. This invention requires no additional training or model-specific supervision, is applicable to multimodal large-scale models based on projection and query alignment, and can be applied to image description, visual question answering, and other fields, showing broad application prospects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to a multimodal large-model illusion suppression method based on stability perception, belonging to the fields of computer vision and artificial intelligence. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Vision-Language Models (LVLMs) have demonstrated outstanding performance in tasks such as image description generation, visual question answering, multimodal dialogue, and visual reasoning by combining visual encoders with large language models. However, these models commonly suffer from object hallucination during the generation process. This means that while the model's generated responses are linguistically fluent and natural, they are inconsistent with the visual evidence in the input image. For example, they might describe objects that are not actually present in the image, or infer the existence of objects based on contextual associations and linguistic priors. Such object descriptions lacking visual evidence directly undermine the credibility of the model's output, limiting its deployment in scenarios requiring faithful visual perception.

[0003] Existing research indicates that object illusion is closely related to two coupled factors: first, unreliable visual evidence. Within the visual encoder, some image patch tokens are highly sensitive to input perturbations (such as small bounded adversarial perturbations), causing their representations to shift significantly under these perturbations. These perturbation-sensitive tokens propagate unstable visual evidence in visual self-attention, thus contaminating higher-level visual representations. Second, misrouting of visual attention. Although the visual pathway (visual encoder or visual alignment module) can provide relatively focused visual priors, the attention of large language models to visual tokens in the early and mid-level layers is often diffuse, easily focusing on irrelevant background areas and influenced by generated context and language priors, leading to the model's failure to correctly utilize reliable visual evidence.

[0004] Currently, training-free suppression methods for object illusion can be mainly divided into three categories: decoding-side methods suppress language priors by comparing the output distributions of original and distorted visual inputs (or different model states) (such as the VCD method); attention-side methods enhance visual grounding by adjusting the attention patterns or image token contributions during the decoding process (such as OPERA, PAI, DeCo, etc.); and vision-side methods improve visual fidelity by suppressing uncertain or perturbation-sensitive visual tokens in the visual encoder. However, the above methods typically only intervene in one side of the multimodal generation process: while language-side methods can improve attention routing, they often assume that the visual priors they use are reliable; while vision-side methods can suppress unstable tokens, they cannot explicitly control how the language decoder subsequently utilizes refined visual evidence.

[0005] Therefore, there is an urgent need for an illusion suppression method that can first obtain reliable visual priors and then explicitly route those priors into a language decoder, so as to solve the problem of object illusion from a unified perspective of "reliability-routing". Summary of the Invention

[0006] To address the object illusion problem caused by unreliable visual evidence and erroneous attention routing during the generation of large visual language models, this invention proposes a stability-aware multimodal large model illusion suppression method, following the principle of "anchoring first, then guiding": First, token-level visual reliability is estimated using a bounded adversarial perturbation probe, and perturbation-sensitive tokens are suppressed to obtain a reliability-aware visual prior; then, this prior is injected into the early-mid layers of the large language model through quality-preserving attention calibration, directing visual attention from irrelevant or unreliable tokens to reliable visual evidence. This invention requires no additional training, no external detectors, and no model-specific supervision, yet it significantly reduces object illusions while maintaining general multimodal capabilities.

[0007] The technical solution of the present invention includes the following steps: (1) Multimodal input processing encodes the input image through a vision encoder (Vision Transformer, ViT) to generate visual tokens, which are then converted into a sequence of visual tokens that match the dimensions of the language model by a projector or query bridging module. These visual tokens are then input into the Large Language Model along with the text tokens for processing.

[0008] (2) Token reliability estimation: Construct bounded adversarial perturbation samples of the input image, calculate the difference between the perturbation samples and the original image in the patch-level representation in the visual encoder, obtain the normalized difference score of each visual token, and generate a token-level reliability mask accordingly.

[0009] (3) Reliability-guided visual coding injects the reliability mask into the preset intermediate layer of the visual encoder to suppress the contribution of the perturbation sensitive token to the visual self-attention output and extract the visual prior of reliability perception from the reliability-guided visual path.

[0010] (4) Quality-preserving attention guidance: The visual prior is injected into the preset early-mid layer of the large language model. Under the condition of keeping the total quality of attention in the image token interval unchanged, the visual attention distribution is redistributed in the image token interval according to the visual prior.

[0011] (5) The hallucination suppression output large language model focuses visual attention on reliable visual evidence during autoregressive decoding, generating a response that is consistent with the image content and significantly reduces object hallucinations.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention adopts a unified "reliability-routing" perspective, coupling visual evidence refinement with language-based attention routing: first, reliable visual evidence is anchored through adversarial sensitivity estimation, and then the reliability-perceived visual prior is explicitly routed into the language decoder. Experiments show that on the LLaVA-1.5-7B model, this invention reduces the sentence-level hallucination rate (CHAIRs) from 47.6% in the original model to 21.6%, and consistently reduces object hallucinations across multiple representative models. 2. This invention employs quality-preserving attention calibration: it only changes the attention distribution within the image token interval without increasing the overall attention quality of the image token, thus maintaining the text-visual attention balance; while reducing illusions, it maintains the model's general multimodal capabilities, and its perception, cognition, and overall scores on the MME comprehensive evaluation benchmark are maintained or better than the original model. 3. This invention requires no additional training, external detectors, or model-specific supervision: it is a plug-and-play, training-free inference method that is applicable to both projection alignment-based (such as LLaVA, Shikra) and query alignment-based (such as MiniGPT-4, Qwen3-VL) mainstream visual language model architectures, and has broad applicability. Attached Figure Description

[0013] Figure 1 This is a block diagram illustrating the principle of the stability-aware multimodal large-model hallucination suppression method of the present invention.

[0014] Figure 2 This is a schematic diagram of the bounded anti-disturbance probe of the present invention.

[0015] Figure 3 This is a schematic diagram illustrating the attention calibration for maintaining quality in this invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0017] First, the general structure and notation of LVLMs are explained. Given an input image... With text instructions LVLM first encodes the image into a visual token, which, along with the text token, is input into the large language model decoder. The response is generated in an autoregressive manner: in the first... One decoding step, next token Sampling according to formula (1): (1) in, Indicates the input image. Indicates text instructions. Indicates the autoregressive decoding step index. Indicates the first The token generated by the decoding steps Indicates the first The tokens generated before the previous decoding step This indicates the frozen model parameters. Indicates that the parameter is The conditional probability distribution determined by the model, This represents the logits on the vocabulary. For the decoder, the... Layer Each attention head, the self-attention map, and the output are calculated according to equation (2): (2) in, The layer index represents the large language model. Indicates the first Attention head index in the layer, , and They represent the first Layer The query matrix, key matrix, and value matrix of each attention head. For the attention head dimension, This represents the corresponding self-attention weight matrix. This represents the corresponding self-attention output.

[0018] like Figure 1 As shown, a multimodal large-model hallucination suppression method based on stability awareness includes the following steps: Step (1) Multimodal input processing: Process the input image via visual encoder The generated visual token sequence is then converted into visual tokens that match the dimensions of the large language model by a projector (typically a two-layer multilayer perceptron, corresponding to projection-based alignment) or a query bridging module (such as Q-Former, corresponding to query-based alignment). These visual tokens, along with the text tokens, are then input into the large language model decoder. This step is completely consistent with the forward pass of existing LVLMs and does not change the model structure or parameters.

[0019] Step (2) Token reliability estimation: such as Figure 2 As shown, given the input image Construct bounded adversarial perturbation samples, as shown in equation (3): (3) in, Indicates the input image. This represents the perturbation sample obtained after applying a bounded perturbation to the input image. Indicates image perturbation. express Maximum permissible perturbation amplitude under norm constraints Represents the infinite norm; The perturbation samples are used only as reliability probes and do not participate in response generation. The perturbation is then solved by maximizing the patch-level feature difference between the visual encoder and the perturbation image using equation (4). : (4) in, For visual encoders, This represents the model parameters of the visual encoder, and the subscript `patch` indicates the patch token representation output by the visual encoder. Let represent the 2-norm. Let be the norm. , They are respectively the first one Patch token in visual encoder The original image representation and the perturbed image representation in the layer, where, Indicates the patch token index. This indicates the index of the visual encoder layer. Then, in the preset reliability estimation layer... On the set, the normalized difference score for each visual token is calculated according to equation (5): (5) in, This represents the preset set of reliability estimation layers. This indicates the number of visual encoder layers included in the reliability estimation layer set. This represents the layer index of the visual encoder. This represents the patch token index of the difference score to be calculated. This represents the index used to iterate through all patch tokens. This represents a numerical stability constant used to avoid a denominator of zero. Indicates the first Normalized difference score of each patch token The larger the value, the stronger the token's perturbation sensitivity and the lower its visual reliability. Then, the difference score is thresholded according to equation (6) to obtain the token-level reliability mask: (6) in, This represents the set of normalized difference scores for all patch tokens. and Let these represent the mean and standard deviation of the normalized set of difference scores, respectively. Indicates the threshold coefficient. Indicates an indicator function, Indicates the first Each visual token is a reliable token. Indicates the first The visual token is a perturbation-sensitive token. This mask measures the token's reliability rather than its salience.

[0020] Step (3) Reliability-guided visual encoding: Inject the reliability mask M into the preset intermediate layer set of the visual encoder. For layers The patch-level self-attention output is token-by-token modulated according to equation (7): (7) in, This represents the token-level reliability mask. Indicates the first The reliability mask value corresponding to each visual token. This represents the set of pre-defined mask injection layers in the visual encoder. This represents the index of the visual encoder layer in the mask injection layer set. , and These represent the visual encoder's first... The query matrix, key matrix, and value matrix in layered visual self-attention. This represents visual self-attention operations. Indicates the first The attention output of each visual token is modulated by a reliability mask. This operation suppresses the contribution of perturbation-sensitive tokens to visual self-attention while preserving the original visual token sequence, unlike token pruning. After completing the reliability-guided visual encoding, a clean forward pass is performed, and the visual prior of reliability perception is extracted along the refined visual path, as shown in Equation (8): (8) in, Indicated by reliability mask Guided visual encoding process, For architecture-specific attention readout functions. For projection-aligned LVLMs (such as LLaVA, Shikra). Corresponding visual encoder Attention to tokens for image patch tokens; for query-aligned LVLMs (such as MiniGPT-4). This is the readout function obtained by the bridging module by aggregating attention across attention heads and image tokens. This indicates a normalization operation. This represents the normalized visual prior for perceived reliability. Indicates the relationship with the first The reliability-aware visual prior weights corresponding to each image token.

[0021] Step (4) Attention guidance for quality maintenance: Let's assume This represents the image token range in the input of the large language model. and These represent the start and end positions of the image token interval, respectively. For the large language model... Layer One point of attention, Indicates the current decoding query position. Indicates the position of the token in the input sequence. Indicates the query position For token position The original attention weights, Indicates the query position Image token range The total quality of original visual attention. The quality of original visual attention is then calculated according to equation (9): (9) Maintaining this attention quality, attention is redistributed within the image token interval based on the visual prior of perceived reliability, as shown in Equation (10): (10) in, Indicates the location to be queried after calibration. For token position Attention weights Indicates the calibration coefficient. Indicates the position within the image token range The corresponding reliability-perceived visual prior weights, where Used to map absolute token positions in the input sequence of a large language model to position indices within image token intervals, main settings The softmax is implemented in logit space before calibration. As can be seen from equation (10), this invention changes the model's attention focus within the visual token, rather than the amount of attention the model pays to the overall image. Main settings The softmax logit space is implemented before calibration. Then the logarithm and exponent of the image token interval logits are calculated according to Equation (11): (11) in, Indicates the first before calibration Layer Search location in attention With token position The original attention logit between Represents the image token range The logarithm and exponential values ​​of the original attention logits. For the attention injection layer. Replace the logits within the image token range according to equation (12): (12) in This represents the pre-defined set of attention injection layers in a large language model. The calibrated attention logit. Because After normalization, the attention quality of the image token interval remains unchanged after softmax, as shown in Equation (13): (13) in To calibrate logits Attention weights are obtained after applying Softmax normalization. Thus, attention guidance directs existing visual attention toward reliable perceived visual evidence while maintaining the original text-visual attention balance.

[0022] like Figure 3 As shown, the model affects the image token interval before and after calibration. Total quality of attention The attention distribution within the interval remains unchanged, but changes from a diffuse state (with some mass falling on the background or unreliable tokens) to a concentrated state on reliable tokens, and the attention received by unreliable tokens is reduced.

[0023] Step (5) Illusion Suppression Output: After the above reliability anchoring and attention guidance, the large language model focuses visual attention on reliable visual evidence during the autoregressive decoding process, generating a response that is consistent with the image content and significantly reduces object illusions.

[0024] Regarding the selection of the layer set: Assume the visual encoder contains a total of... The large language model contains [number] encoding layers. A set of decoding layers. A set of reliability estimation layers. Selected from early to mid-stages of the visual encoder to estimate the perturbation sensitivity of visual tokens before unreliable visual evidence has fully propagated; mask injection layer set. Selected from intermediate stages of the visual encoder to suppress perturbation-sensitive tokens while maintaining subsequent visual semantic modeling capabilities; attention injection layer set Layers are selected from early to mid-stages of a large language model to reassign attention to reliable visual evidence before language priors dominate final token prediction. The sets of layers are pre-determined based on the total number of model layers, illusion suppression performance on the validation set, and computational cost, and remain fixed during inference. Furthermore, calibration coefficients... With threshold coefficient Adjustable according to application scenario: The default value is 1; Control the filtering threshold, The smaller the value, the fewer tokens are retained and the stronger the suppression.

[0025] To verify the effectiveness of the method of this invention, it was validated on four representative visual language models: LLaVA-1.5-7B (based on projection alignment), Shikra-7B (based on projection alignment), MiniGPT-4 (based on query alignment), and Qwen3-VL. Vanilla represents the experimental results of the original model output without additional modifications. Five representative training-free hallucination suppression methods were also selected as comparison methods, specifically: Method 1: VCD (Visual Contrastive Decoding) proposed by Leng et al., reference "Leng, S.; Zhang, H.; Chen, G.; Li, X.; Lu, S.; Miao, C.; onComputer Vision and Pattern Recognition." Paper 2:Wang Introduction to DeCo(Dynamic Correction Decoding)"Wang, C.; Chen, X.; Zhang, N.; Tian, ​​B.; Xu, H.; Deng, S.; and Chen, H. 2025. MLLMCan See? Dynamic Correction Decoding for Hallucination Mitigation In TheThirteenth International Conference on Learning Representations." Figure 3:Liu's personalized inventory of PAI (Paying More Attention to Image), published in"Liu,S.; Zheng, K.; and Chen, W. 2024. Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs In European Conference onComputer Vision." Figure 4: Li Performance of Visual Information Steering (VISTA)"Li, Z.; Shi, H.; Gao, Y.; Liu, D.; Wang, Z.; Chen, Y.; Liu, T.; Zhao, L.; Wang, H.; and Metaxas, DN 2025. The Hidden Life of Tokens: Reducing Hallucination ofLarge Visual-Language Models via Visual Information Steering In Proceedingsof the 42nd International Conference on Machine Learning." Method 5: Epistemic (Visual Token Cognitive Uncertainty Reduction) proposed by Seo et al., reference: "Seo, H.; Kang, DU; Cho, H.; Lee, J.; and Chun, SY 2025. On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Models. Advances in Neural Information Processing Systems, 38." The evaluation benchmarks include: CHAIR, a benchmark for evaluating sentence-level illusion rates CHAIRs and instance-level illusion rates CHAIRi (lower values ​​are better) in image descriptions; POPE, a benchmark for evaluating the accuracy and F1 score of object existence judgments, including three settings: Random, Popular, and Adversarial; MMHal-Bench, an illusion evaluation benchmark covering eight types of questions, with responses scored by a large language model; and MME, a multimodal comprehensive evaluation benchmark covering perception and cognition tasks.

[0026] As shown in Table 1, on the CHAIR benchmark, this invention reduces CHAIRs from 47.6% to 21.6% and CHAIRi from 13.3% to 6.2% on the LLaVA-1.5-7B model; it also achieves the lowest illusion rate on the Shikra-7B and Qwen3-VL models; and it consistently outperforms the original decoding on the MiniGPT-4 model, indicating that this invention can consistently reduce object illusions across architectures.

[0027] Table 1 compares the performance of the methods with other methods on the CHAIR benchmark, with the best results shown in bold.

[0028] As shown in Table 2, on the POPE benchmark, this invention achieved the highest average accuracy and F1 score across all four models, and consistently improved across the random, common, and adversarial settings. This indicates that the invention improves the ability to determine the existence of objects and effectively reduces hallucinations. The data in the table represent the average accuracy and F1 score of each model across the three settings; higher values ​​are better.

[0029] Table 2 compares the performance of other methods on the POPE benchmark, with the best results shown in bold.

[0030] The ablation experiments are shown in Table 3: using only raw visual prior attentional guidance (CHAIRs decreased from 47.6 to 40.8) showed a certain improvement compared to the original decoding (CHAIRs decreased from 47.6 to 40.8); using only reliability anchoring brought a greater improvement (CHAIRs decreased to 28.6); the complete method of this invention achieved the best results (CHAIRs decreased to 21.6), indicating that the reliability anchoring at the visual end and the attentional guidance at the linguistic end complement each other.

[0031] Table 3 shows the ablation experiments of the components on LLaVA-1.5-7B, with the best results shown in bold.

[0032] On the MMHal-Bench benchmark, this invention improved visual grounding and reduced illusions across all eight problem categories, achieving the best overall performance across all evaluation frameworks. On the MME benchmark, this invention achieved the highest perceptual, cognitive, and overall scores, demonstrating that it maintains general multimodal capabilities while reducing object illusions.

[0033] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal large-model hallucination suppression method based on stability perception, characterized in that... Includes the following steps: (1) Multimodal input processing: The input image is encoded by a visual encoder to generate visual tokens, which are then converted into a sequence of visual tokens that match the dimensions of the language model by a projector or query bridging module. These visual tokens are then input into the large language model along with the text tokens for processing. (2) Token reliability estimation: Construct bounded adversarial perturbation samples of the input image, calculate the difference between the perturbation samples and the original image in the patch-level representation in the visual encoder, obtain the normalized difference score of each visual token, and generate a token-level reliability mask accordingly. (3) Reliability-guided visual encoding: The reliability mask is injected into the preset intermediate layer of the visual encoder to suppress the contribution of the perturbation-sensitive visual token to the visual self-attention output and extract the visual prior of reliability perception from the reliability-guided visual path. (4) Quality-preserving attention guidance: The visual prior is injected into the preset early-mid layer of the large language model. Under the condition of keeping the total quality of attention in the image token interval unchanged, the visual attention distribution is redistributed in the image token interval according to the visual prior. (5) Hallucination suppression output: During the autoregressive decoding process, the large language model focuses visual attention on reliable visual evidence, generating a response that is consistent with the image content and significantly reduces object hallucinations.

2. The multimodal large-model hallucination suppression method based on stability perception according to claim 1, characterized in that, The visual encoder in step (1) adopts the Vision Transformer architecture, which segments the input image into fixed-size patches and generates an initial visual representation; the projector is a two-layer multilayer perceptron, which converts the dimension of the visual embedding vector into a feature dimension compatible with the large language model; the alignment between the large language model and the visual encoder includes a projection-based alignment and a query-based alignment.

3. The multimodal large-model hallucination suppression method based on stability perception according to claim 1, characterized in that, The bounded adversarial perturbation samples in step (2) are constructed as follows: Under norm constraints, perturbation samples are constructed according to the following formula: (3) The perturbation is solved by maximizing the patch-level feature difference between the visual encoder and the perturbed image. : (4) in, Indicates the input image. This represents a bounded, adversarial perturbation sample. This represents the image perturbation applied to the input image. express Maximum permissible perturbation amplitude under norm constraints Represents a visual encoder. Indicates the model parameters of the visual encoder, subscript The patch token representation of the visual encoder output. The L2 norm is represented; the perturbation sample is used only as a reliability probe and does not participate in response generation.

4. The multimodal large-model hallucination suppression method based on stability perception according to claim 1, characterized in that, The token-level reliability mask in step (2) is generated as follows: In the pre-defined reliability estimation layer set Above, the normalized difference score for each visual token is calculated according to the following formula. : (5) in, This represents the preset set of reliability estimation layers. This indicates the number of visual encoder layers included in the reliability estimation layer set. This represents the layer index of the visual encoder. and These represent the indices of the visual patch token. and They represent the first The patch token is in the... The original image representation and perturbed image representation in each visual encoder layer and They represent the first The patch token is in the... The original image representation and perturbed image representation in each visual encoder layer This represents the numerical stability constant used to avoid a denominator of zero; and the normalized difference fraction is thresholded according to the following formula to obtain the first... The reliability mask value corresponding to each visual token : (6) in, This represents the set of normalized difference scores for all patch tokens. and Let these represent the mean and standard deviation of the normalized set of difference scores, respectively. Represents the threshold coefficient. Indicates an indicator function, Indicates the first Each visual token is a reliable token. Indicates the first The visual token is a disturbance-sensitive token.

5. The multimodal large-model hallucination suppression method based on stability perception according to claim 1, characterized in that, The reliability-guided visual coding in step (3) includes: The reliability mask Preset intermediate layer set injected into the visual encoder For the preset intermediate layer set The first in The layer multiplies the output of visual self-attention with the reliability mask token by token, the first... The attention output after the visual tokens are modulated by a reliability mask is: (7) in, This represents the token-level reliability mask. Indicates the first The reliability mask value corresponding to each visual token. This represents the set of pre-defined mask injection layers in the visual encoder. This represents the index of the visual encoder layer in the mask injection layer set. , and These represent the visual encoder's first... The query matrix, key matrix, and value matrix in layered visual self-attention. This represents visual self-attention operations. Indicates the first The first in the layer The self-attention output corresponding to each visual token. Indicates the first Attention output after a visual token is modulated with a reliability mask; The above-described token-by-token modulation suppresses the contribution of perturbation-sensitive visual tokens to the visual self-attention output while preserving the complete visual token sequence; subsequently, reliability-perceived visual priors are extracted along the reliability-masked visual pathway: (8) in, Indicated in the reliability mask Guided input image Visual representation obtained by performing visual encoding. This represents the attention readout function associated with the visual language model architecture. This indicates a normalization operation. This represents the reliability-perceived visual prior obtained after normalization.

6. The method for suppressing multimodal large-model hallucinations based on stability perception according to claim 5, characterized in that, When the large language model adopts a projection-based alignment method, the attention readout function For visual encoders Attention to the token-to-image patch token; When the large language model adopts a query-based alignment method, the attention readout function This is the readout function obtained by the bridging module by aggregating attention across attention heads and image tokens.

7. The multimodal large-model hallucination suppression method based on stability perception according to claim 1, characterized in that, The attention guidance for quality maintenance in step (4) includes: set up The image token intervals in the input sequence of the large language model are given, where and These represent the start and end positions of the image token interval, respectively. For the large language model, the... The first layer Each attention head will be used to decode the current query position. Image token location The original attention weights are denoted as The current decoding query position The corresponding raw visual attention quality is: (9) Based on the aforementioned reliability perception visual prior With calibration coefficient Interpolation is performed between the original attention distribution and the visually prior-guided attention distribution: (10) in, Represents the index of the large language model layer. Indicates the attention head index. Indicates the current decoding query position. Indicates the position of the token in the input sequence. Indicates the first Layer Search location in attention For token position The original attention weights, Indicates the query position Image token range The overall quality of original visual attention This represents the calibrated attention weights. Indicates the calibration coefficient. Indicates the location of the image token The corresponding reliability perception visual prior value; when Attention calibration is performed in the logit space before attention normalization. First, the logarithm and exponent of the original attention logits corresponding to the image token interval are calculated: (11) And replace the original attention logit within the image token interval with: (12) in, Indicates the first before calibration Layer Search location in attention With token position The original attention logit between This represents the calibrated attention logit. The logarithm and exponent of the original attention logits within the image token interval are represented; after performing Softmax normalization on the calibrated attention logits, the query position is preserved. Image token range The overall quality of attention remains unchanged.

8. The multimodal large-model hallucination suppression method based on stability perception according to claim 7, characterized in that, The calibration coefficient satisfy The preset early-middle layer refers to the first third to the middle part of the large language model; the visual prior, after normalization, satisfies... and .

9. The multimodal large-model hallucination suppression method based on stability perception according to claim 1, characterized in that, The method requires no additional training, no external detector, and no model-specific supervision. It is applicable to both projection-aligned and query-aligned visual language models. The visual language models include, but are not limited to, LLaVA series models, Shikra models, MiniGPT-4 models, and Qwen3-VL models.