Illusion processing method and device of multi-modal large model, equipment and medium

Abstract

The application discloses a hallucination processing method and device of a multimodal large model, equipment and a medium, relates to the field of artificial intelligence and information processing technology, and is based on a hallucination evaluation framework. According to the differentiated reweighting adjustment of attention weight of a key attention head according to semantic segmentation of a multimodal sequence, the attention weight distribution corresponding to different semantic segments can be reasonably regulated and controlled, the hallucination problem generated in the reasoning process can be effectively relieved from the model internal bottom mechanism level while the normal recognition and reasoning ability of the multimodal large model is maintained, and the scene adaptation ability of the multimodal large model in the hallucination relief aspect is enhanced. The method comprises the following steps: constructing a hallucination evaluation framework; positioning the key attention head causing hallucination in the multimodal large model by using the hallucination evaluation framework; and adjusting the attention weight of the key attention head according to the semantic segmentation of the multimodal sequence, so as to realize the hallucination relief in the reasoning stage of the multimodal large model.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and information processing technology, and in particular to a method, apparatus, device and medium for processing illusions in a multimodal large model. Background Technology

[0002] In recent years, multimodal large models have made significant progress in tasks such as image description, visual question answering, and multimodal reasoning, and are gradually expanding into the field of image understanding. Compared with natural images, images input to multimodal large models are characterized by dense distribution of small objects, large scale variations, and complex spatial relationships. Therefore, multimodal large models are more prone to hallucinations that are inconsistent with visual reality.

[0003] To address the hallucination problem prevalent in large multimodal models, relevant technologies mainly fall into two categories: hallucination evaluation and hallucination mitigation. Hallucination evaluation primarily employs two paradigms: open-ended generative evaluation and closed-ended probing evaluation. A representative evaluation method transforms target hallucination detection into a binary classification problem, verifying the model's authenticity by querying whether a specific object exists in the image. Hallucination mitigation mainly involves optimization during the inference phase. This is achieved through decoding control strategies or attention intervention mechanisms, such as decoding constraints, attention calibration, and internal representation adjustment, to reduce hallucination phenomena without retraining the model. Furthermore, some visual language models already support image question answering and spatial reasoning tasks.

[0004] While the aforementioned methods have made some progress in the evaluation and mitigation of hallucinations in multimodal large models, they still have the following shortcomings: First, most of the hallucination evaluation methods focus on the existence of targets, mainly used to detect whether the model has fabricated an object. They struggle to cover fine-grained hallucination types such as quantity errors, spatial relationship errors, and color attribute errors, and cannot comprehensively reflect the true failure modes in complex visual semantic tasks in remote sensing scenarios. Second, most of the aforementioned hallucination mitigation methods are designed for natural image scenes, failing to adequately consider the characteristics of other scene images, such as dense distribution of small targets, large scale variations, complex backgrounds, and strong spatial relationship dependencies. This limits their relevance and applicability in visual language tasks. Third, the intervention process during the inference stage of these hallucination mitigation methods typically treats textual dependence as a uniform phenomenon, lacking fine-grained differentiation between different semantic segments such as system prompts, visual features, user instructions, and generated responses. This makes it difficult to effectively suppress the model's excessive reliance on object words and attribute words in the question text during question-and-answer processes, easily leading the model to make judgments based on linguistic cues rather than visual evidence.

[0005] In summary, existing multimodal large-scale hallucination processing methods have shortcomings such as difficulty in covering fine-grained hallucination types in hallucination evaluation, insufficient scene adaptability in hallucination relief, and lack of fine-grained differentiation in reasoning intervention. Summary of the Invention

[0006] In view of this, this application provides a method, apparatus, device and medium for processing hallucinations in a multimodal large model. The main purpose is to solve the shortcomings of existing methods for processing hallucinations in multimodal large models, such as difficulty in covering fine-grained hallucination types in hallucination evaluation, insufficient scene adaptability in hallucination relief, and lack of fine-grained distinction in reasoning intervention.

[0007] The first aspect provides a method for processing illusions in a multimodal large model, which includes: A hallucination assessment framework is constructed. The hallucination assessment framework is based on the assessment sample and its real label. According to the preset assessment dimensions and assessment index system, the prediction results obtained by inputting the assessment sample into the multimodal large model are compared with the real label, and the quantitative index of hallucination performance of the multimodal large model in each assessment dimension is calculated. The hallucination assessment framework is used to locate key attention heads that trigger hallucinations in a multimodal large model; For the key attention head, attention weights are adjusted based on the semantic segments of the multimodal sequence to alleviate illusions during the inference stage of the multimodal large model; the semantic segments of the multimodal sequence are obtained by structured parsing of the input sequence of the multimodal large model; the attention weight adjustment process is to adjust the attention weights corresponding to each semantic segment according to the attention weight distribution of the key attention head on each semantic segment using a differentiated reweighting strategy.

[0008] Secondly, a multimodal large-scale illusion processing device is provided, the device comprising: The construction unit is used to construct a hallucination assessment framework. The hallucination assessment framework is based on the assessment sample and its real label. According to the preset assessment dimensions and assessment index system, the prediction results obtained by inputting the assessment sample into the multimodal large model are compared with the real label, and the quantitative index of hallucination performance of the multimodal large model in each assessment dimension is calculated. The localization unit is used to locate the key attention head that triggers hallucinations in the multimodal large model using the hallucination assessment framework. The adjustment unit is used to adjust the attention weights of the key attention head based on the semantic segments of the multimodal sequence, thereby alleviating the illusion during the inference stage of the multimodal large model. The semantic segments of the multimodal sequence are obtained by structured parsing of the input sequence of the multimodal large model. The attention weight adjustment process is to adjust the attention weights corresponding to each semantic segment according to the attention weight distribution of the key attention head on each semantic segment using a differentiated reweighting strategy.

[0009] Thirdly, a multimodal large-scale illusion processing device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the aforementioned multimodal large-scale illusion processing method.

[0010] Fourthly, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the aforementioned multimodal large-scale illusion processing method.

[0011] By employing the above technical solutions, this application provides a method, apparatus, device, and medium for processing hallucinations in a multimodal large model. Compared with existing methods, this application constructs a hallucination assessment framework. Based on assessment samples and their real labels, and according to a preset assessment dimension and assessment index system, the framework compares the prediction results obtained by inputting the assessment samples into the multimodal large model with the real labels to calculate the quantitative indicators of hallucination performance of the multimodal large model in each assessment dimension. The framework is used to locate the key attention heads that trigger hallucinations in the multimodal large model. For these key attention heads, attention weights are adjusted based on the semantic segments of the multimodal sequence to alleviate hallucinations during the inference stage of the multimodal large model. The semantic segments of the multimodal sequence are obtained from the input sequence of the multimodal large model through structured parsing. The attention weight adjustment process involves adjusting the attention weights corresponding to each semantic segment using a differentiated reweighting strategy based on the attention weight distribution of the key attention heads on each semantic segment. The entire process is based on a constructed hallucination assessment framework, enabling fine-grained quantitative evaluation of hallucination performance across different assessment dimensions for a multimodal large model. This allows for accurate identification of key attention heads that induce hallucinations within the multimodal large model, facilitating precise tracing of the causes of hallucinations. Furthermore, based on the semantic segmentation of the multimodal sequence obtained through structured parsing of the input sequence, differential reweighting of attention weights is applied to the key attention heads. This allows for reasonable control of attention weight allocation across different semantic segments, effectively suppressing the hallucination-inducing effect of key attention heads. While maintaining the normal recognition and reasoning capabilities of the multimodal large model, it effectively alleviates hallucination problems arising during reasoning from the underlying mechanisms within the model, enhancing the multimodal large model's scenario adaptability in hallucination mitigation and significantly improving the accuracy and reliability of the multimodal large model's output results.

[0012] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart illustrating a method for processing illusions in a multimodal large model according to an embodiment of this application; Figure 2 yes Figure 1 A flowchart illustrating a specific implementation method of step 101; Figure 3 yes Figure 1 A flowchart illustrating a specific implementation method for step 102; Figure 4 yes Figure 3 A flowchart illustrating a specific implementation method for step 303; Figure 5 yes Figure 1 A flowchart illustrating a specific implementation method for step 103; Figure 6 This is a flowchart illustrating a method for processing illusions in a multimodal large model, as described in another embodiment of this application. Figure 7 This is a flowchart of a method for adjusting the weights of attention heads in semantic segmentation according to an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a multimodal large-scale illusion processing device according to an embodiment of this application; Figure 9 This is a schematic diagram of the device structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0014] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0015] To address the hallucination problem prevalent in large multimodal models, relevant technologies mainly fall into two categories: hallucination evaluation and hallucination mitigation. Hallucination evaluation primarily employs two paradigms: open-ended generative evaluation and closed-ended probing evaluation. A representative evaluation method transforms target hallucination detection into a binary classification problem, verifying the model's authenticity by querying whether a specific object exists in the image. Hallucination mitigation mainly involves optimization during the inference phase. This is achieved through decoding control strategies or attention intervention mechanisms, such as decoding constraints, attention calibration, and internal representation adjustment, to reduce hallucination phenomena without retraining the model. Furthermore, some visual language models already support image question answering and spatial reasoning tasks.

[0016] However, existing hallucination processing methods for multimodal large models have shortcomings such as difficulty in covering fine-grained hallucination types in hallucination evaluation, insufficient adaptability to scenarios in hallucination relief, and lack of fine-grained differentiation in reasoning intervention.

[0017] To address this problem, this embodiment provides a method for processing illusions in a multimodal large model, such as... Figure 1 As shown, it includes the following steps: 101. Construct a framework for assessing hallucinations.

[0018] The hallucination assessment framework is based on the assessment samples and their real labels. According to the preset assessment dimensions and assessment index system, the prediction results obtained by inputting the assessment samples into the multimodal large model are compared with the real labels, and the quantitative indicators of hallucination performance of the multimodal large model in each assessment dimension are calculated.

[0019] In the specific implementation process, the first step is to prepare a test sample set. This set contains multiple representative multimodal input data and their corresponding real labels. Taking image question answering as an example, the test sample may include one thousand images covering different scenarios, each image accompanied by a manually annotated accurate description and multiple questions and their standard answers. The real labels are rigorously reviewed and determined by domain experts to ensure their accuracy and authority. Secondly, the preset test dimensions need to be determined. These dimensions are set based on the application scenarios and common hallucination types of the multimodal large model. In this embodiment, the test dimensions include at least one of the following: existence dimension, quantity dimension, location dimension, and color dimension. Next, a test indicator system needs to be established. This system sets specific quantitative scoring standards for each test dimension. In this embodiment, the system includes multiple indicators, including at least one of accuracy, precision, recall, affirmative ratio, and macro-accuracy. After constructing the hallucination assessment framework, the assessment samples are input one by one into the multimodal large model to be assessed, so as to predict all the assessment samples in the assessment sample set, summarize the scores of each assessment sample on different assessment dimensions, and calculate the average hallucination performance quantitative index of the multimodal large model on each assessment dimension.

[0020] Taking an image description task involving a picture of cats and dogs and the related question "How many animals are in the picture?" as an example, the multimodal large model is input into the picture. The multimodal large model outputs the prediction "There is one cat and two dogs in the picture." Simultaneously, the true label for this evaluation sample is "There is one cat and one dog in the picture." The prediction results of the multimodal large model are systematically compared with the true labels to analyze the differences between them across various evaluation dimensions. The accuracy metric is used to evaluate the hallucination performance in the existence dimension. If the multimodal large model generates completely correct descriptions for 75 out of 100 evaluation samples, the accuracy rate is considered to be 75%. Higher accuracy indicates a lower probability of the multimodal large model generating hallucinations.

[0021] 102. Use the hallucination assessment framework to locate the key attention points that trigger hallucinations in the multimodal large model.

[0022] In this embodiment, the attention head is the basic computational unit of the multi-head attention mechanism in a multimodal large model. Each layer of the multimodal large model typically contains multiple parallel attention heads, each independently learning the relationships between different positions in the input sequence. A key attention head refers to an attention head in the multimodal large model that has a decisive influence on a specific phenomenon or problem. In the hallucination research scenario of this invention, key attention heads specifically refer to those attention heads that play a dominant role in the hallucination generation process; the abnormal behavior of these attention heads directly leads to the model producing hallucination output. The identification and localization of key attention heads are of great significance for understanding and solving the hallucination problem in multimodal large models. Accurately locating key attention heads allows for a deeper understanding of the intrinsic mechanisms of hallucination generation, helping to reveal the information processing processes within the multimodal large model.

[0023] In practical implementation, the key components inducing hallucinations can be systematically located by comparing the differences in the influence of attention heads on incorrect and correct predictions in a multimodal large model. First, a screening sample set is constructed, divided into a set of correctly predicted samples and a set of incorrectly predicted samples. The average influence of each attention head on both sets is statistically analyzed, and the difference between the two is used as the scoring criterion for key attention heads. All attention heads are ranked according to their scores, and the highest-scoring attention heads are selected as key attention heads. Higher scores indicate that the attention head is more inclined to support incorrect predictions rather than correct predictions; this scoring mechanism can effectively identify attention heads that behave abnormally on hallucination samples.

[0024] As an feasible approach, the aforementioned average influence can be achieved using a direct zeroing method. This process sets the attention head output to a zero vector and calculates the change in the multimodal large model prediction results before and after zeroing. This is performed on both the incorrectly predicted sample set and the correctly predicted sample set to obtain the average influence of each attention head.

[0025] As another feasible approach, the aforementioned average influence can be achieved using attention weight analysis. This process extracts the attention weight distribution of the attention head during the reasoning process and calculates the entropy or maximum value of the weight distribution. The lower the entropy or the higher the maximum value, the more concentrated the attention and the greater the influence. The average entropy or average maximum value is then calculated for both types of sample sets.

[0026] 103. For the key attention head, adjust the attention weight according to the semantic segmentation of the multimodal sequence to achieve hallucination relief in the multimodal large model inference stage.

[0027] In this embodiment, a multimodal sequence refers to an ordered set of information containing multiple data modalities, equivalent to the input sequence of a multimodal large model. These modalities include different forms of data such as text content, image regions, audio clips, and video frames. In the multimodal large model, these different modalities of data are uniformly encoded into a sequence form, forming a continuous sequence that integrates multiple information types. The semantic segmentation of the multimodal sequence is obtained from the input sequence of the multimodal large model through structured parsing. Structured parsing decomposes the continuous multimodal sequence into multiple logical units with independent semantics, each unit corresponding to a complete semantic segment in the input content.

[0028] For multimodal sequences of text, natural language processing techniques can be used for paragraph boundary recognition and sentence segmentation. By analyzing features such as punctuation, paragraph markers, and semantic coherence, long texts can be divided into independent sentence or paragraph units. For example, in technical documents, titles, body paragraphs, code blocks, and explanatory text can be identified as different semantic segments.

[0029] For multimodal sequences in an image, computer vision techniques can be used for region segmentation and object detection. Object detection algorithms identify different object regions in the image, and image segmentation techniques divide the image into semantically meaningful regions. For example, in an image containing multiple objects, each object and its background region can be divided into independent semantic segments.

[0030] The attention weight adjustment process involves adjusting the attention weights corresponding to each semantic segment based on the attention weight distribution of the key attention head for each semantic segment using a differentiated reweighting strategy.

[0031] The specific attention weight adjustment process involves adjusting the attention weights of each semantic segment based on the distribution of attention weights for each semantic segment by the key attention head, using a differentiated reweighting strategy. This process first obtains the proportion and distribution characteristics of each semantic segment in the attention weights. The attention weight distribution reflects the degree of attention the key attention head pays to different semantic segments; a more concentrated distribution indicates stronger attention, while a more dispersed distribution indicates more even attention. Next, based on the attention weight distribution characteristics, a differentiated reweighting strategy is used to adjust the attention weights of each semantic segment. For semantic segments with excessively high attention weights and irrelevant to the correct answer, a decay strategy is used to reduce their weights; for semantic segments with excessively low attention weights but relevant to the correct answer, an enhancement strategy is used to increase their weights; for semantic segments with moderate attention weights, their weights are kept unchanged. Finally, the adjusted attention weights are re-injected into the key attention head, completing the illusion mitigation in the reasoning process. The adjusted attention weights guide the key attention head to allocate more attention to semantic segments relevant to the correct answer, reducing attention to irrelevant or incorrect semantic segments, thereby reducing the probability of illusions.

[0032] The above weight adjustment can be calculated using a weighted factor multiplication method. First, an adjustment factor is calculated for each semantic segment. This adjustment factor is determined based on the difference between the segment's attention weight and a preset threshold. For semantic segments whose attention weight exceeds the upper threshold, the adjustment factor is less than 1; for semantic segments whose attention weight is below the lower threshold, the adjustment factor is greater than 1; and for semantic segments whose attention weight is within the normal range, the adjustment factor is equal to 1. Then, the original attention weight is multiplied by the corresponding adjustment factor to obtain the adjusted attention weight.

[0033] Taking an image-text question-answering task as an example, the input is an image containing dogs and cats, and the question "How many dogs are in the image?". Structured parsing divides the image into three semantic segments: a dog region, a cat region, and a background region. The initial key attention head assigned too much attention weight to the cat region, causing the multimodal model to incorrectly answer "There is one cat." Through a differentiated reweighting strategy, the attention weight of the cat region was detected to be exceeding the upper threshold, and its adjustment factor was reduced; simultaneously, the attention weight of the dog region was detected to be below the lower threshold, and its adjustment factor was increased. After adjustment, the key attention head increased its attention to the dog region and decreased its attention to the cat region. At this point, the multimodal model correctly answered "There are two dogs," thus alleviating the illusion.

[0034] The hallucination processing method for multimodal large models provided in this application involves constructing a hallucination assessment framework. This framework is based on assessment samples and their true labels. According to a preset assessment dimension and assessment index system, the prediction results obtained by inputting the assessment samples into the multimodal large model are compared with the true labels to calculate the quantitative indicators of hallucination performance of the multimodal large model in each assessment dimension. The hallucination assessment framework is used to locate the key attention heads that trigger hallucinations in the multimodal large model. For the key attention heads, attention weights are adjusted based on the semantic segments of the multimodal sequence to alleviate hallucinations during the inference stage of the multimodal large model. The semantic segments of the multimodal sequence are obtained from the input sequence of the multimodal large model through structured parsing. The attention weight adjustment process involves adjusting the attention weights corresponding to each semantic segment using a differentiated reweighting strategy based on the attention weight distribution of the key attention heads for each semantic segment. The entire process is based on a constructed hallucination assessment framework, enabling fine-grained quantitative evaluation of hallucination performance across different assessment dimensions for a multimodal large model. This allows for accurate identification of key attention heads that induce hallucinations within the multimodal large model, facilitating precise tracing of the causes of hallucinations. Furthermore, based on the semantic segmentation of the multimodal sequence obtained through structured parsing of the input sequence, differential reweighting of attention weights is applied to the key attention heads. This allows for reasonable control of attention weight allocation across different semantic segments, effectively suppressing the hallucination-inducing effect of key attention heads. While maintaining the normal recognition and reasoning capabilities of the multimodal large model, it effectively alleviates hallucination problems arising during reasoning from the underlying mechanisms within the model, enhancing the multimodal large model's scenario adaptability in hallucination mitigation and significantly improving the accuracy and reliability of the multimodal large model's output results.

[0035] In practical applications, large multimodal models are prone to producing erroneous outputs that contradict the input or violate objective facts when processing cross-modal information; this phenomenon is known as hallucination. Traditional methods for handling hallucinations in large multimodal models primarily focus on general metrics such as accuracy and recall, failing to effectively identify and quantify hallucination phenomena. Therefore, it is necessary to construct a hallucination assessment framework through systematic evaluation samples, multi-dimensional evaluation metrics, and standardized evaluation processes to objectively and accurately evaluate the hallucination performance of large multimodal models. Specifically, for example... Figure 2 As shown, step 101 above includes the following steps: 201. Construct evaluation samples based on scenario data sources.

[0036] 202. Define the evaluation dimensions and evaluation indicator system.

[0037] 203. The prediction results obtained by inputting the evaluation samples into the multimodal large model are compared with the real labels. Based on the comparison results, the hallucination performance of the multimodal large model in different evaluation dimensions is quantified, and the construction of the hallucination evaluation framework is completed.

[0038] In this embodiment, the scene data source refers to the raw multimodal data set generated in a specific application scenario, covering various data formats such as text, images, audio, and video, reflecting the complexity of the real application environment. Taking a remote sensing scenario as an example, the scene data source is remote sensing image data, including but not limited to satellite remote sensing images, aerial remote sensing images, and UAV remote sensing images. Remote sensing image data covers different geographical areas, seasonal changes, weather conditions, and application scenarios.

[0039] Specifically, in the process of constructing evaluation samples based on scene data sources, factual information of each scene image in the scene data source can be obtained. The factual information includes multiple basic attribute types, and each type of attribute serves as the basis for different evaluation dimensions. Based on the factual information, verification statements are constructed for each evaluation dimension. Each set of verification statements includes one positive example statement and at least one negative example statement. The positive example statement is a correct statement that conforms to the facts, and the negative example statement is an incorrect statement that violates the facts. The scene images and verification statements are associated and combined to form evaluation samples.

[0040] Specifically, in the process of constructing verification statements for each evaluation dimension, the original factual information can be converted into structured factual descriptions, which have clear semantic constraints and true labels. Without changing the original factual information, the structured factual descriptions are converted into verification statements in natural language form. The verification statements themselves contain the content to be verified, and the model is required to judge their correctness.

[0041] The construction process of the above evaluation samples can be described in three steps. The first step is structured fact extraction, which represents information such as object categories, quantities, positional relationships, and color attributes in the image as structured descriptions with explicit semantic constraints and answer labels. The second step is to construct verification statements from the structured facts according to the evaluation dimensions. This involves constructing positive example statements using structured facts while designing distracting false facts for different evaluation dimensions. The third step is the construction of binary classification questions, which rewrites the positive and negative example statements into yes / no questions, requiring the multimodal large model to output only affirmative or negative results. This binary classification question design simplifies the output requirements of the multimodal large model, reduces the complexity of the evaluation, and ensures the clarity and comparability of the evaluation results. Each verification statement corresponds to a clear answer label, facilitating subsequent automated evaluation and result statistics.

[0042] Specifically, the verification statement construction process employs a two-stage implementation workflow combining structured factual constraints and language variant generation. In the first stage, image-level factual information extracted from remote sensing images is converted into a structured description document with explicit semantic constraints and answer labels. This document clearly defines the basic elements such as the category attributes, quantity statistics, spatial relationships, and color features of various objects in the image. In the second stage, without introducing any new factual information, the predefined object categories, quantity values, spatial relationship descriptions, and color attributes in the aforementioned structured description are rewritten at the language level to generate a binary classification question expression form that conforms to everyday language expression habits. This rewriting process only optimizes and adjusts the language expression of the question, completely maintaining the accuracy and consistency of the underlying factual labels, thereby significantly improving the natural fluency and linguistic diversity of the question expression while ensuring semantic controllability. To further ensure the reliability of the generated evaluation samples, this invention combines a rule-based filtering mechanism and a manual review mechanism during implementation to conduct strict quality control on the generated evaluation samples, promptly deleting samples with inconsistencies in facts, incorrect labeling, or unclear dimension attribution, ensuring the accuracy and effectiveness of the final evaluation sample set.

[0043] The process of constructing evaluation samples in a remote sensing scenario begins with the analysis of remote sensing images. First, various objective factual information needs to be extracted from the images, including the types of target objects appearing in the images, the specific quantity statistics of each type of object, the spatial distribution relationships between objects, and the visual characteristics of the objects. This extracted factual information is then standardized and organized into a structured descriptive document, serving as the basis for subsequent evaluation sample construction. Then, based on these structured facts, correct descriptions that fully conform to the actual situation are generated as positive examples. Simultaneously, incorrect descriptions that contradict the actual situation are consciously constructed as negative examples. Finally, the remote sensing images are organically combined with these positive and negative examples to form a complete set of evaluation samples. Each sample includes the remote sensing image itself and a corresponding verification question. The question uses a binary classification format (yes or no) to facilitate the quantitative evaluation of the judgment ability of multimodal large models. The entire process ensures that the evaluation samples originate from real remote sensing scenarios and possess clear factual basis and controllable interference design.

[0044] In the specific implementation process, the illusion assessment framework is constructed based on the annotation of two typical remote sensing data images and the remote sensing image text matching dataset, forming remote sensing image subsets and remote sensing image text matching subsets with different assessment dimensions. This method of construction based on real remote sensing data sources ensures that the assessment samples are representative and practical, and can effectively reflect the actual application needs of remote sensing scenarios.

[0045] For each remote sensing image, factual information is first systematically extracted. After obtaining complete factual information, verification statements are constructed for different evaluation dimensions based on this information. Finally, the remote sensing image and the corresponding verification statements are associated and combined to form a complete evaluation sample. Thus, each evaluation sample contains a remote sensing image, one or more verification statements, and corresponding answer labels. These evaluation samples are classified and organized according to different evaluation dimensions to form a systematic evaluation benchmark.

[0046] It should be noted that the evaluation samples constructed using the above method are highly objective and targeted, effectively covering various common hallucination problems in remote sensing scenarios. However, this method is not only applicable to remote sensing scenarios but can also be extended to other application areas that require multi-dimensional hallucination evaluation.

[0047] Specifically, in remote sensing image analysis, multimodal large models not only need to accurately identify the existence of target objects, but also need to precisely determine the quantity, spatial location, and attribute characteristics of target objects. Common types of illusions include quantity errors (misjudging the number of targets by the multimodal large model); spatial relationship errors (misjudging the location and interrelationships of targets by the multimodal large model); and attribute binding errors (misjudging the attribute characteristics of targets by the multimodal large model). These fine-grained illusion problems seriously affect the reliability of remote sensing multimodal models in important fields.

[0048] To enhance the illusion assessment of remote sensing multimodal large models, this embodiment includes at least one of the following assessment dimensions: existence, quantity, location, and color. Correspondingly, as the basis for constructing the assessment samples, the factual information includes at least the following: object category, i.e., the classification of target objects existing in the image; number of object instances, i.e., the specific statistical count of each type of object in the image; relative spatial relationships or regional distribution relationships between objects, i.e., the location distribution and interrelationships of target objects in the image; and identifiable color attributes, i.e., the color characteristics exhibited by the target objects. This factual information forms the basis for constructing subsequent verification statements.

[0049] Specifically, in the existence dimension, positive example statements are constructed around the judgment of whether a certain object exists in the image; negative example statements are constructed through at least one of the following methods: randomly selecting missing categories, selecting categories that frequently co-occur but do not actually exist, or constructing adversarial missing categories based on co-occurrence relationships; negative example statements are used to detect whether the model has a tendency to fabricate targets. For example, for the existence dimension, construct the question "Does a building exist in the remote sensing image?". Positive examples state that a building actually exists in the remote sensing image, while negative examples state that a building does not exist in the remote sensing image, or that a scene co-occurs with a building in the remote sensing image.

[0050] Specifically, in the quantity dimension, positive example statements are constructed around the judgment of whether the quantity of a certain object in the image is consistent with a given value; negative example statements are constructed by at least one of the following methods: fabricating a quantity for a non-existent object, assigning an incorrect quantity to an existing object, or misbinding quantities across categories; negative example statements are used to test the model's counting ability and the ability to correctly bind objects to quantities. For example, for the quantity dimension, construct the question "How many cars are in the remote sensing image?". A positive example statement is that there are actually 5 cars in the remote sensing image, and the given value in the question is 5. A negative example statement is that there are actually no cars in the remote sensing image, or there are actually 3 cars in the remote sensing image, or there are 5 trucks in the remote sensing image.

[0051] Specifically, in the location dimension, positive example statements are constructed around the judgment of whether a given relative positional relationship or regional distribution relationship exists between objects; negative example statements are constructed through at least one of the following methods: constructing a relationship involving missing objects, reversing or replacing the true relationship, or giving an incorrect regional distribution description; negative example statements are used to test the model's ability to perceive spatial layout and reason about relative relationships. For example, for the location dimension, construct the question "Is the building in the remote sensing image located on the east side of the road?". A positive example statement is that the building in the remote sensing image is actually located on the east side of the road, and the relative positional relationship posed by the question is correct. A negative example statement is that there is no road in the remote sensing image, or that the building in the remote sensing image is actually located on the west side of the road, or that the building in the remote sensing image is distributed on both sides of the road.

[0052] Specifically in the color dimension, positive example statements are constructed around the judgment of whether the color attribute of an object is correct; negative example statements are constructed through at least one of the following methods: assigning a color to a missing object, assigning an incorrect color to an existing object, constructing a color description that conflicts with common sense, or performing cross-object color misbinding; negative example statements are used to test the model's ability to recognize color attributes and the ability to correctly bind objects to color attributes.

[0053] For example, for the color dimension, construct the question "What color is the roof of the building in the remote sensing image?". A positive example statement would be that the roof of the building in the remote sensing image is actually red, and the color attribute queried in the question is correct. A negative example statement would be that there is no building in the remote sensing image, or that the roof of the building in the remote sensing image is actually blue.

[0054] In terms of the evaluation index system, this embodiment of the invention unifies the evaluation samples into a closed-loop binary question-answering task, meaning the multimodal large model only outputs "yes" or "no". For the free text output of the generative large model, a rule-based post-processing method is used to map it into affirmative or negative labels. During evaluation, a unified index is used to compare different dimensions and different models. Specific indices include at least one of accuracy, precision, recall, affirmative ratio, and macro-accuracy. The affirmative ratio characterizes the model's affirmative bias, and macro-accuracy reflects the overall evaluation results across the four evaluation dimensions. This approach reveals fine-grained illusion patterns of the multimodal large model in terms of target existence, quantity, positional relationships, and color attributes within a unified framework.

[0055] The constructed evaluation samples are then batch-input into the remote sensing multimodal large-scale model to be evaluated, obtaining the model's prediction results for each sample. Complete model output information is recorded, including judgment results, confidence scores, and inference processes. The prediction results of the multimodal large-scale model are compared one by one with the true labels, and the model's performance data across the four evaluation dimensions are statistically analyzed. Specific index values ​​for each dimension are calculated, generating detailed dimension score reports. Simultaneously, a comprehensive hallucination index is calculated to provide a comprehensive evaluation of the multimodal large-scale model's overall hallucination performance.

[0056] In practical applications, the attention mechanism within multimodal large models plays a crucial role in processing complex tasks such as remote sensing images. Different attention heads are responsible for capturing feature information at different levels and types. However, some attention heads may tend to produce incorrect attention distributions due to training data bias, model architecture defects, or imperfect optimization objectives, resulting in illusory content in the model output that does not match reality. By accurately locating these key attention heads that trigger illusions, we can not only reveal the underlying mechanisms of illusion generation but also provide clear guidance for subsequent model optimization, attention intervention, and targeted training. Specifically, for example... Figure 3 As shown, step 102 above includes the following steps: 301. Without any intervention, the predicted results obtained by inputting the screened samples into the multimodal large model are compared with the true labels. Based on the comparison results, the screened samples are divided into a set of correctly predicted samples and a set of incorrectly predicted samples.

[0057] 302. Extract the activation features of the correctly predicted sample set and the incorrectly predicted sample set on each layer of the multimodal large model, and calculate the activation difference of each attention head on the two sample sets.

[0058] 303. Based on the average influence of each attention head on the two types of sample sets, quantify the contribution of each attention head to the generation of multimodal large model illusion.

[0059] 304. Sort the attention heads from highest to lowest according to their contribution, and select the top-ranked attention heads as the key attention heads that trigger the hallucination.

[0060] In the specific implementation process, a screening sample set is first constructed. This set contains representative multimodal input data and their corresponding ground truth labels. Then, each screening sample in the set is sequentially input into the multimodal large-scale model to be evaluated. Under natural reasoning conditions, the multimodal model outputs a binary classification prediction result for each question, i.e., a yes or no answer. Simultaneously, the ground truth labels for each screening sample are obtained from a pre-constructed labeled database. The prediction results output by the multimodal large-scale model are compared with the ground truth labels. Screening samples whose prediction results are completely consistent with the ground truth labels are classified into the correctly predicted sample set; these samples reflect the model's normal reasoning ability on that question. Samples whose prediction results are inconsistent with the ground truth labels are classified into the incorrectly predicted sample set; these samples represent specific instances where the model has misjudged the model. Through this comparison and classification mechanism, two mutually exclusive and complete sample sets are ultimately formed.

[0061] In the specific implementation process, for the processing of the correctly predicted sample set, each sample in the correctly predicted sample set is sequentially input into the multimodal large model. During the layer-by-layer forward propagation, the multimodal large model captures the intermediate calculation results generated by each attention head in each layer in real time. These intermediate calculation results include the attention weight matrix, query vector, key vector, value vector, and output features of the attention head, which together constitute the activation feature set of the attention head when processing the correct samples. The above process is repeated for all samples in the correctly predicted sample set, and the arithmetic mean of these activation features is calculated as the average influence of the attention head on the correctly predicted sample set. Similarly, the incorrectly predicted sample set is input into the multimodal large model, using the same feature extraction process and parameter settings as when processing the correctly predicted sample set. All intermediate calculation results generated by each attention head in each layer when processing incorrect samples are recorded synchronously, forming the activation feature set of each attention head in the hallucination state. The arithmetic mean of these activation features is calculated as the average influence of the attention head on the incorrectly predicted sample set. To ensure the comparability of feature extraction processes for the two types of sample sets, it is necessary to maintain consistency in external factors such as batch size of input samples, computing equipment, and random seeds, and to eliminate the influence of non-model factors on activated features.

[0062] After calculating the average influence on both types of sample sets, a score is calculated for each attention head. The difference between the average influence of the attention head on the incorrectly predicted sample set and its average influence on the correctly predicted sample set is the contribution of that attention head to the multimodal large model illusion. This contribution reflects the attention head's preference for incorrect predictions; a higher contribution indicates that the attention head is more active when processing incorrect samples than when processing correct samples, and is more inclined to support the incorrect reasoning process.

[0063] Finally, all attention heads in the multimodal large model are sorted in descending order of contribution, forming an attention head contribution sequence. The top N attention heads with the highest scores from this contribution sequence are considered the key attention heads that cause the model to produce illusions. The value of N can be flexibly set according to actual needs, and is usually selected as 5% to 20% of the total number of attention heads as the candidate range. For example, if the multimodal large model contains a total of 120 attention heads, and the preset selection quantity is 10%, then the 12 attention heads with the highest contribution at the beginning of the sorted sequence are selected as key attention heads.

[0064] In practice, by statistically analyzing the activation differences of each attention head on the correctly predicted sample set and the incorrectly predicted sample set, attention heads that exhibit significantly different performance on the two types of samples can be quickly identified. These attention heads are more likely to be associated with hallucinations, and using them as candidate targets can significantly reduce the computational load of subsequent experiments while ensuring that important attention heads are not overlooked. Specifically, such as... Figure 4 As shown, step 303 above includes the following steps: 401. Select attention heads whose average influence exceeds a preset threshold as candidate attention heads, and perform mean ablation operation on each candidate attention head.

[0065] 402. The change in illusion rate and the change in accuracy are weighted and fused to obtain the contribution of each attention point to the generation of multimodal large model illusion.

[0066] Understandably, large multimodal models typically contain a large number of attention heads. While contribution can reflect the activity of attention heads on erroneous samples, this correlation is not equivalent to causation. To ensure the accuracy of key attention head selection, mean ablation is used to accurately determine whether the attention heads initially selected can truly reduce model illusions after intervention.

[0067] In this embodiment, mean ablation is an effective method for causal inference. By replacing the output of a candidate attention head with a specific value, the changes in the performance of a large multimodal model after the attention head is masked can be observed, thereby inferring its actual impact on model behavior. This method avoids the model structure destruction caused by directly deleting attention heads, maintaining the controllability and reproducibility of the experiment. The replacement operation only changes the output of the target attention head and does not affect the normal operation of other parts of the model, ensuring the accuracy of the experimental results.

[0068] The specific mean ablation operation includes the following steps: replacing the output of the candidate attention head with the average activation value or zero vector at the corresponding position on the correctly predicted sample set, keeping other parameters of the multimodal large model unchanged, re-inputting the correctly predicted sample set and the incorrectly predicted sample set into the ablation-processed multimodal large model for inference, and calculating the change in illusion rate of the multimodal large model on the incorrectly predicted sample set and the change in accuracy on the correctly predicted sample set before and after the ablation operation.

[0069] Understandably, correctly predicted samples represent the normal working state of a multimodal large model, and the activation pattern of its attention head reflects the correct information processing method. Therefore, the average activation value at the corresponding position on the correctly predicted sample set is chosen as the replacement value. Replacing the original output of the candidate attention head with this normal state activation value is equivalent to forcing the attention head to the correct working mode, and observing the effect of this adjustment on suppressing hallucinations. Simultaneously providing a zero vector as a replacement option allows verification of the impact when the attention head output completely disappears. The two replacement methods complement each other, providing a more comprehensive analytical perspective.

[0070] Calculating the changes in hallucination rate and accuracy is to assess the impact of the attention head from two dimensions. The change in hallucination rate directly reflects the effectiveness of the ablation operation in suppressing hallucinations and is an indicator of the correlation between the attention head and hallucinations. The change in accuracy is used to assess the impact of the ablation operation on the normal functioning of a large multimodal model, avoiding excessive impairment of the model's correct reasoning ability in order to suppress hallucinations. Using both indicators together allows for a comprehensive evaluation of the attention head's role.

[0071] Further weighting and fusing the changes in hallucination rate and accuracy, with a higher weight given to the change in hallucination rate, reflects a trade-off based on practical application requirements. In multimodal large-scale model applications, suppressing hallucinations typically takes precedence over maintaining absolute accuracy. A slight decrease in accuracy is acceptable, but severe hallucination problems must be addressed first. By adjusting the weight ratio, an optimal balance can be found between suppressing hallucinations and maintaining performance, ensuring the accuracy of the final contribution metric.

[0072] In practice, the first step is to prepare a set of correctly predicted samples and a set of incorrectly predicted samples, containing both true correct samples from model inference and false samples from model inference that produce hallucinations. For each attention head, the average of its output activation values ​​at corresponding positions across all samples in both sets is calculated, and the absolute value of the difference between the two averages is taken as the average influence of that attention head. A preset threshold is set, and attention heads whose average influence exceeds the threshold are selected as candidate attention heads. For each candidate attention head, its output is replaced with the average activation value or zero vector at the corresponding position in the correctly predicted sample set, while keeping other model parameters unchanged. The two sets of samples are then re-inputted into the processed multimodal large model for inference, and the hallucination rate and accuracy before and after ablation are statistically analyzed, with corresponding changes calculated. Finally, the changes in hallucination rate and accuracy are weighted and summed according to preset weight ratios to obtain a quantified result of the contribution of each attention head to the hallucination.

[0073] In a specific embodiment, the semantic segmentation method for multimodal sequences is implemented as follows. First, a multimodal input sequence containing four semantic segments is constructed: system prompt segment, visual feature segment, user command segment, and generated response segment. The boundaries of each semantic segment are dynamically determined using a template-aware approach.

[0074] The boundaries between the system prompt segment and the user command segment are located by special markers in a predefined dialog template. Specific separators, such as start and end markers, are set in the predefined dialog template. By scanning the multimodal sequence, the positions of these special markers are located, thus accurately defining the start and end boundaries of the system prompt segment and the user command segment.

[0075] The boundaries of visual feature segments are dynamically determined based on the number and position of visual feature markers. In a multimodal sequence, each visual feature corresponds to a unique visual marker. The total number of visual markers in the multimodal sequence is counted, and the index position of each visual marker is recorded. The starting boundary of a visual feature segment is the position of the first visual marker, and the ending boundary is the position of the last visual marker, thus completely covering all visual feature information.

[0076] The boundary of the generated response segment is determined by the length of the historical response content generated in the current decoding step. During model inference, the number of generated response tags is tracked in real time. The starting boundary of the generated response segment is the first position after the end of the user instruction segment, and the ending boundary is the position of the last generated response tag. As the decoding step progresses, the boundary of the generated response segment dynamically expands, reflecting the model's generation progress in real time.

[0077] This template-aware dynamic boundary determination method can accurately identify the range of each semantic segment in a multimodal sequence, providing clear structured input for subsequent attention head analysis and hallucination detection.

[0078] In a specific embodiment, the root cause of hallucinations in the inference phase of a multimodal large model lies in the imbalance of attention mechanism distribution. When key attention heads overemphasize text semantic segments while neglecting visual feature segments, the multimodal large model tends to rely on linguistic priors for mental completion, thus generating descriptions inconsistent with the image content. Therefore, it is necessary to dynamically adjust the distribution of attention weights to enable the multimodal large model to pay more balanced attention to visual evidence and textual information during the inference phase, thereby effectively alleviating hallucinations. Specifically, as... Figure 5 As shown, step 103 above includes the following steps: 501. Obtain the original attention logic values ​​of the key attention head at each key position during the multimodal large model decoding process.

[0079] 502. Based on the semantic segment to which each key position belongs, scale the original attention logic value using the scaling factor corresponding to each semantic segment to obtain the scaled attention logic value.

[0080] 503. Normalize the scaled attention logic values ​​to obtain the adjusted attention weight distribution of each semantic segment.

[0081] 504. The adjusted attention distribution is used for subsequent label generation in the multimodal large model decoding process to achieve hallucination relief in the multimodal large model inference stage.

[0082] In the decoding process of the multimodal large model, this embodiment only performs reweighting on candidate attention heads, while other attention heads in the multimodal large model remain in their original state. This selective intervention strategy ensures the stability of the overall model architecture while achieving precise control over key attention paths.

[0083] In practice, for a given candidate attention head, the first step in the current decoding process is to obtain the original attention logic values ​​corresponding to each key position. These logic values ​​are intermediate results calculated using scaled dot products but not yet normalized, preserving the original numerical characteristics of the attention calculation. In this embodiment, based on pre-determined semantic segmentation boundary information, all key positions are divided into four semantic segments: system prompt segment, visual feature segment, user command segment, and generated response segment.

[0084] Subsequently, based on the semantic segment to which each key position belongs, it is multiplied by the corresponding semantic segment scaling factor. The scaling factor for the system prompt segment is used to moderately reduce the focus on system prompt information. The scaling factor for the visual feature segment is set to significantly enhance the focus on visual content, ensuring that the output of the multimodal large model relies more on visual evidence. The scaling factor for the user instruction segment is used to effectively suppress the model's abnormal reliance on object words or attribute words in the question text. The scaling factor for the generated response segment is used to maintain normal focus on historically generated content.

[0085] It is important to note that the scaling factor for the user instruction segment can be set to 0. This does not mean that the multimodal large model's access to the question text is globally removed. This scaling factor only moderately suppresses the attention of a small number of candidate attention heads in the user instruction segment during local decoding steps that meet the triggering conditions. Since the vast majority of attention heads in the multimodal large model remain unchanged, this method is a local and controllable inference-time intervention strategy.

[0086] After scaling, all adjusted attention logic values ​​are normalized to obtain a new attention weight distribution. The sum of the normalized attention weights is one, ensuring the validity and numerical stability of the probability distribution. The adjusted attention weight distribution is then used in the subsequent label generation process of the current decoding step to guide the multimodal large model in generating the next output label.

[0087] This embodiment achieves differentiated adjustment of different semantic segments through this segmented attention reweighting mechanism. It prioritizes suppressing the multimodal large model's abnormal dependence on user-instruction segments, effectively reducing the risk of hallucinations caused by excessive focus on object or attribute words in the question text. Simultaneously, by enhancing attention to visual feature segments, it ensures that the multimodal large model's output relies more on visual evidence, improving the consistency between generated content and image content. This refined control strategy suppresses hallucinations while maintaining the normal reasoning ability and generation quality of the multimodal large model.

[0088] In a specific embodiment, the attention distribution requirements differ across decoding steps during the inference process of a multimodal large model. Applying reweighting intervention to all decoding steps could lead to the multimodal large model over-relying on visual information and neglecting necessary textual context, affecting the coherence and accuracy of the model's output. Therefore, an intelligent triggering mechanism is needed to perform attention adjustments only in decoding steps where intervention is truly necessary. That is, this embodiment does not apply weighting intervention to all key attention heads simultaneously. Instead, it determines whether to perform segmented attention reweighting on the current key attention head. Specifically, as... Figure 6 As shown, before step 103 above, the method further includes the following steps: 601. Calculate the attention quality of the key attention head for each semantic segment during the decoding process of the multimodal large model.

[0089] 602. Calculate the text attention percentage based on the attention quality.

[0090] Accordingly, 603, when the text attention ratio exceeds a preset threshold, the attention weight adjustment based on semantic segmentation of the multimodal sequence is triggered; 604. When the text attention ratio does not exceed a preset threshold, maintain the attention weight distribution of each semantic segment in the multimodal sequence.

[0091] In the specific implementation process, the attention quality of the current candidate attention head for the four semantic segments is first calculated. The complete attention weight matrix of this attention head in the current decoding step is obtained; the row dimension of this matrix corresponds to the current query position, and the column dimension corresponds to all key positions. Based on pre-determined semantic segmentation boundary information, the column dimension of the attention weight matrix is ​​divided into four continuous regions, corresponding to the system prompt segment, visual feature segment, user command segment, and generated response segment, respectively. All attention weight values ​​within each region are summed to obtain the attention quality value of the attention head for the corresponding semantic segment.

[0092] Then, the text attention percentage is calculated. The text attention percentage is obtained by taking the sum of the attention quality of the text semantic segments as a percentage of the sum of the attention quality of all semantic segments. The text semantic segments include system prompt segments and user instruction segments. The attention quality of the system prompt segments and the attention quality of the user instruction segments are added together to obtain the total attention quality of the text semantic segments.

[0093] First, the attention quality of the system prompt segment is added to the attention quality of the user command segment to obtain the total attention quality of the text semantic segment. Then, the attention quality of the system prompt segment, visual feature segment, user command segment, and generated response segment are all added together to obtain the total attention quality of the four semantic segments. Finally, the total attention quality of the text semantic segment is divided by the total attention quality of the four semantic segments to obtain the text attention ratio, which reflects the degree of dependence of the current decoding step on the text information.

[0094] A preset threshold for the text attention percentage is further defined. This threshold is determined through a grid search on the validation set, achieving an optimal balance between hallucination suppression and maintaining generation quality. For example, setting the threshold to 65% means that if the calculated text attention percentage exceeds 65%, the system determines that the current decoding step is overly reliant on the text, posing a high risk of hallucination, and automatically triggers an attention weight adjustment. If the text attention percentage does not exceed 65%, the system determines that the attention distribution of the current decoding step is within a reasonable range, requiring no intervention and maintaining the original attention weight distribution.

[0095] In practical applications, the process of adjusting the weights of attention heads for semantic segmentation can be found in [link to relevant documentation]. Figure 7 As shown, its overall execution process follows Figure 7 The three-stage progressive structure shown consists of key attention head selection, multimodal sequence semantic segmentation, and segmented attention head reweighting. The entire process runs in real time during the model inference stage without involving any parameter updates or network structure adjustments.

[0096] The first stage is the selection of key attention heads. First, a selection sample is constructed, containing several representative remote sensing multimodal samples, including both correctly generated samples and erroneous samples exhibiting hallucination phenomena. Then, baseline inference is performed, i.e., running the large multimodal model without any intervention to obtain the raw output behavior of each attention head. Based on the inference results, the samples are divided into correct and erroneous samples, and the text attention ratio of each attention head in both categories is statistically analyzed. The attention shift of each candidate attention head relative to correct samples in erroneous samples is calculated; a higher score indicates that the head is more likely to induce hallucinations. The top few attention heads are selected based on their scores to form candidate attention heads. Then, a mean ablation validation strategy is used to adjust the candidate attention heads, which will serve as the target objects for subsequent interventions.

[0097] The second stage is multimodal sequence semantic segmentation. A complete multimodal input sequence is received, which is sequentially concatenated from system prompts, visual feature encodings, user instructions, and existing response history. Using a template-aware segmentation module, the boundaries of each semantic segment are automatically identified according to predefined sequence structure rules, dividing the input sequence into four distinct semantic regions: the system prompt segment carries model initialization instructions, the visual feature segment corresponds to the embedded representation after image encoding, the user instruction segment contains key query content from the question text, and the response history segment records generated text fragments. This segmentation process is based on a fixed template, requiring no additional training, ensuring stable and reproducible segmentation results.

[0098] The third stage is segmented attention head reweighting. When the multimodal large model enters a certain time step in the decoding stage, it first determines whether the currently used attention head belongs to the set of key attention heads. If it does, it further checks whether the text attention ratio of that attention head in the current time step exceeds a preset threshold. Once the trigger condition is met, the segmented attention head reweighting mechanism is activated: the original attention logic values ​​of all key positions corresponding to that attention head at the current query position are extracted, and multiplied by the corresponding scaling factor according to the semantic segment to which each key belongs. After scaling, all adjusted logic values ​​are normalized so that the sum of the weights is one, forming a new attention distribution. This distribution is directly used for the context vector calculation of the current time step, thereby driving the next prediction. The entire process only applies to a small number of key attention heads, accurately suppressing abnormal dependence on user command segments and strengthening reasonable attention to visual feature segments while ensuring the stability of the model's main behavior, thus effectively mitigating the generation of hallucinations.

[0099] Furthermore, as a specific implementation of the above method, embodiments of this application provide a multimodal large-scale illusion processing device, such as... Figure 8 As shown, the device includes: a construction unit 71, a positioning unit 72, and an adjustment unit 73.

[0100] The construction unit 71 is used to construct a hallucination assessment framework. The hallucination assessment framework is based on the assessment sample and its real label. According to the preset assessment dimensions and assessment index system, the prediction results obtained by inputting the assessment sample into the multimodal large model are compared with the real label, and the quantitative index of hallucination performance of the multimodal large model in each assessment dimension is calculated. The localization unit 72 is used to locate the key attention head that triggers hallucinations in the multimodal large model using the hallucination assessment framework; The adjustment unit 73 is used to adjust the attention weights of the key attention head based on the semantic segments of the multimodal sequence to alleviate the illusion during the inference stage of the multimodal large model; the semantic segments of the multimodal sequence are obtained by structured parsing of the input sequence of the multimodal large model; the attention weight adjustment process is to adjust the attention weights corresponding to each semantic segment according to the attention weight distribution of the key attention head on each semantic segment using a differentiated reweighting strategy.

[0101] The hallucination processing device for a multimodal large model provided in this invention constructs a hallucination assessment framework. Based on assessment samples and their real labels, and according to a preset assessment dimension and assessment index system, the framework compares the prediction results obtained by inputting the assessment samples into the multimodal large model with the real labels to calculate the quantitative indicators of hallucination performance of the multimodal large model in each assessment dimension. The framework is used to locate the key attention heads that trigger hallucinations in the multimodal large model. For these key attention heads, attention weights are adjusted based on the semantic segments of the multimodal sequence to alleviate hallucinations during the inference stage of the multimodal large model. The semantic segments of the multimodal sequence are obtained from the input sequence of the multimodal large model through structured parsing. The attention weight adjustment process involves adjusting the attention weights corresponding to each semantic segment using a differentiated reweighting strategy based on the attention weight distribution of the key attention heads on each semantic segment. The entire process is based on a constructed hallucination assessment framework, enabling fine-grained quantitative evaluation of hallucination performance across different assessment dimensions for a multimodal large model. This allows for accurate identification of key attention heads that induce hallucinations within the multimodal large model, facilitating precise tracing of the causes of hallucinations. Furthermore, based on the semantic segmentation of the multimodal sequence obtained through structured parsing of the input sequence, differential reweighting of attention weights is applied to the key attention heads. This allows for reasonable control of attention weight allocation across different semantic segments, effectively suppressing the hallucination-inducing effect of key attention heads. While maintaining the normal recognition and reasoning capabilities of the multimodal large model, it effectively alleviates hallucination problems arising during reasoning from the underlying mechanisms within the model, enhancing the multimodal large model's scenario adaptability in hallucination mitigation and significantly improving the accuracy and reliability of the multimodal large model's output results.

[0102] In specific application scenarios, the building unit includes: The construction module is used to construct evaluation samples based on the scenario data source. The evaluation samples adopt a true / false judgment form and are labeled with real labels. The real labels are used to indicate the correctness of the verification statement. The definition module is used to define the evaluation dimensions and evaluation indicator system; The comparison module is used to compare the prediction results obtained by inputting the evaluation samples into the multimodal large model with the real labels, and quantify the hallucination performance of the multimodal large model in different evaluation dimensions based on the comparison results, thereby completing the construction of the hallucination evaluation framework.

[0103] In specific application scenarios, the construction module is specifically used for: For each scene image in the scene data source, factual information of the scene image is obtained. The factual information includes multiple basic attribute types, and each type of attribute serves as the basis for different evaluation dimensions. Based on the factual information, verification statements are constructed for each evaluation dimension. Each set of verification statements includes one positive example statement and at least one negative example statement. The positive example statement is a correct statement that conforms to the facts, and the negative example statement is an incorrect statement that contradicts the facts. The scene image is associated with the verification statement and combined to form an evaluation sample; The construction module is further used for: The original factual information is converted into a structured factual description, which has explicit semantic constraints and real labels; Without altering the original factual information, the structured factual description is converted into a verification statement in natural language form. The verification statement itself contains the content to be verified, requiring the model to determine its correctness.

[0104] In specific application scenarios, the positioning unit is specifically used for: Without any intervention, the predicted results obtained by inputting the screened samples into the multimodal large model are compared with the true labels. Based on the comparison results, the screened samples are divided into a set of correctly predicted samples and a set of incorrectly predicted samples. The activation features of the correctly predicted sample set and the incorrectly predicted sample set are extracted from the attention heads of each layer of the multimodal large model, and the average influence of each attention head on the two sample sets is calculated. Based on the average influence of each attention head on the two types of sample sets, the contribution of each attention head to the generation of multimodal large model illusion is quantified; The attention heads are sorted from highest to lowest according to their contribution, and a preset number of attention heads with the highest contribution are selected as the key attention heads that trigger hallucinations.

[0105] In specific application scenarios, the positioning unit is further used for: Attention heads whose average influence exceeds a preset threshold are selected as candidate attention heads. A mean ablation operation is performed on each candidate attention head. The mean ablation operation includes the following steps: replacing the output of the candidate attention head with the average activation value or zero vector at the corresponding position on the correctly predicted sample set, keeping other parameters of the multimodal large model unchanged, re-inputting the correctly predicted sample set and the incorrectly predicted sample set into the multimodal large model after ablation for inference, and calculating the change in illusion rate of the multimodal large model on the incorrectly predicted sample set and the change in accuracy on the correctly predicted sample set before and after the ablation operation. The changes in illusion rate and accuracy are weighted and fused to obtain the contribution of each attention point to the generation of multimodal large model illusion, wherein the weight of the change in illusion rate is higher than the weight of the change in accuracy.

[0106] In specific application scenarios, the semantic segmentation of the multimodal sequence includes a system prompt segment, a visual feature segment, a user instruction segment, and a generated response segment; the boundaries of each semantic segment are dynamically determined using a template-aware method. Specifically, the boundaries of the system prompt segment and the user instruction segment are located by special markers in a predefined dialogue template; the boundaries of the visual feature segment are determined by the number and position of the visual feature markers; and the boundaries of the generated response segment are determined by the length of the historical response content generated in the current decoding step. The adjustment unit is specifically used for: Obtain the original attention logic values ​​of the key attention head at each key position during the multimodal large model decoding process; Based on the semantic segment to which each key position belongs, the original attention logic value is scaled using the scaling factor corresponding to each semantic segment to obtain the scaled attention logic value. The scaled attention logic values ​​are normalized to obtain the adjusted attention weight distribution for each semantic segment. The adjusted attention distribution is used for subsequent label generation in the multimodal large model decoding process to achieve hallucination relief in the multimodal large model inference stage; The device further includes: The first computing unit is used to calculate the attention quality of the key attention head to each semantic segment during the decoding process of the multimodal large model before adjusting the attention weights for the key attention head according to the semantic segmentation of the multimodal sequence to achieve hallucination relief in the multimodal large model inference stage. The second calculation unit is used to calculate the text attention ratio based on the attention quality. The text attention ratio is obtained by the proportion of the sum of the attention quality of the text semantic segments to the sum of the attention quality of all semantic segments. The text semantic segments include system prompt segments and user instruction segments. Correspondingly, the adjustment unit is also used to trigger the attention weight adjustment based on the semantic segmentation of the multimodal sequence when the text attention ratio exceeds a preset threshold; The holding unit is used to maintain the attention weight distribution of each semantic segment in the multimodal sequence when the text attention ratio does not exceed a preset threshold.

[0107] Based on the above-described multimodal large model illusion processing method, this application embodiment also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described multimodal large model illusion processing method.

[0108] Based on this understanding, the technical solution of this application can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive), and includes several instructions to cause a computer device (such as a personal computer, server, or network device) to execute the methods described in the various implementation scenarios of this application.

[0109] Based on the above-described multimodal large-scale illusion processing method and corresponding virtual device embodiments, in order to achieve the above objectives, this application embodiment also provides a physical device for multimodal large-scale illusion processing, which may be a computer, smartphone, tablet computer, smartwatch, server, or network device, etc. The physical device includes a storage medium and a processor; the storage medium is used to store computer programs; the processor is used to execute the computer programs to implement the above-described multimodal large-scale illusion processing method.

[0110] Optionally, the physical device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0111] In an exemplary embodiment, see Figure 9 The aforementioned physical device includes a communication bus, a processor, a memory, and a communication interface. It may also include an input / output interface and a display device. The various functional units can communicate with each other via the bus. The memory stores a computer program, and the processor executes the program stored in the memory to perform the multimodal large-scale illusion processing method described in the above embodiments.

[0112] Those skilled in the art will understand that the physical device structure for multimodal large-scale illusion processing provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0113] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the physical device for the aforementioned multimodal large-scale illusion processing, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware platforms, or it can be implemented in hardware. By applying the technical solution of this application, compared with the existing methods, this application, based on the constructed hallucination assessment framework, can achieve fine-grained quantitative evaluation of the hallucination performance of a multimodal large model across different assessment dimensions, accurately locating the key attention heads that induce hallucinations within the multimodal large model, and completing the precise tracing of the causes of hallucinations. On this basis, based on the semantic segmentation of the multimodal sequence obtained through structured parsing of the input sequence, the attention weights of the key attention heads are differentially reweighted, which can reasonably regulate the attention weight allocation corresponding to different semantic segments, effectively suppress the hallucination-inducing effect generated by the key attention heads, and while maintaining the normal recognition and reasoning ability of the multimodal large model, effectively alleviate the hallucination problem generated during the reasoning process from the underlying mechanism level of the model, enhance the scene adaptability of the multimodal large model in hallucination relief, and significantly improve the accuracy and reliability of the output results of the multimodal large model.

[0115] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application. Those skilled in the art will understand that the modules in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or can be modified to be located in one or more apparatuses different from this embodiment. The modules of the above-described embodiment can be combined into one module, or further divided into multiple sub-modules.

[0116] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of any particular implementation scenario. The above disclosures are merely a few specific implementation scenarios of this application; however, this application is not limited thereto, and any variations conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A method for processing hallucinations in a multimodal large model, characterized in that, include: A hallucination assessment framework is constructed. The hallucination assessment framework is based on the assessment sample and its real label. According to the preset assessment dimensions and assessment index system, the prediction results obtained by inputting the assessment sample into the multimodal large model are compared with the real label, and the quantitative index of hallucination performance of the multimodal large model in each assessment dimension is calculated. The hallucination assessment framework is used to locate key attention heads that trigger hallucinations in a multimodal large model; For the key attention head, attention weights are adjusted based on the semantic segmentation of the multimodal sequence to alleviate hallucinations during the inference stage of the multimodal large model; the semantic segmentation of the multimodal sequence is obtained by structured parsing of the input sequence of the multimodal large model; The attention weight adjustment process involves adjusting the attention weights corresponding to each semantic segment based on the attention weight distribution of the key attention heads for each semantic segment using a differentiated reweighting strategy. The semantic segmentation of the multimodal sequence includes a system prompt segment, a visual feature segment, a user command segment, and a generated response segment. The boundaries of each semantic segment are dynamically determined using a template-aware method. Specifically, the boundaries of the system prompt segment and the user command segment are located by special markers in a predefined dialogue template; the boundaries of the visual feature segment are determined by the number and position of visual feature markers; and the boundaries of the generated response segment are determined by the length of the historical response content generated in the current decoding step. The adjustment of attention weights for the key attention head based on the semantic segmentation of the multimodal sequence, to alleviate illusions during the multimodal large-scale model inference stage, includes: obtaining the original attention logic values ​​of the key attention head corresponding to each key position during the multimodal large-scale model decoding process; scaling the original attention logic values ​​according to the semantic segment to which each key position belongs, using the scaling factor corresponding to each semantic segment, to obtain scaled attention logic values; normalizing the scaled attention logic values ​​to obtain the adjusted attention weight distribution of each semantic segment; and using the adjusted attention weight distribution for subsequent marker generation during the multimodal large-scale model decoding process to alleviate illusions during the multimodal large-scale model inference stage.

2. The method according to claim 1, characterized in that, The proposed framework for assessing hallucinations includes: The evaluation samples are constructed based on the scenario data source. The evaluation samples adopt the form of true or false judgment and are labeled with real labels. The real labels are used to indicate the correctness of the verification statement. Define the assessment dimensions and assessment indicator system; The prediction results obtained by inputting the evaluation samples into the multimodal large model are compared with the real labels. Based on the comparison results, the hallucination performance of the multimodal large model in different evaluation dimensions is quantified, thus completing the construction of the hallucination evaluation framework.

3. The method according to claim 2, characterized in that, The construction of evaluation samples based on scenario data sources includes: For each scene image in the scene data source, the factual information of the scene image is obtained. The factual information includes multiple basic attribute types, and each type of attribute serves as the basis for different evaluation dimensions. Based on the factual information, verification statements are constructed for each evaluation dimension. Each set of verification statements includes one positive example statement and at least one negative example statement. The positive example statement is a correct statement that conforms to the facts, and the negative example statement is an incorrect statement that contradicts the facts. The scene image is associated with the verification statement and combined to form an evaluation sample; Based on the aforementioned factual information, verification statements are constructed for each assessment dimension, including: The raw factual information is converted into a structured factual description, which has explicit semantic constraints and real labels; Without altering the original factual information, the structured factual description is converted into a verification statement in natural language form. The verification statement itself contains the content to be verified, and the model is required to determine its correctness.

4. The method according to any one of claims 1-3, characterized in that, The method of using the hallucination assessment framework to locate the key attention heads that trigger hallucinations in a multimodal large model includes: Without any intervention, the predicted results obtained by inputting the screened samples into the multimodal large model are compared with the true labels. Based on the comparison results, the screened samples are divided into a set of correctly predicted samples and a set of incorrectly predicted samples. The activation features of the correctly predicted sample set and the incorrectly predicted sample set are extracted from the attention heads of each layer of the multimodal large model, and the average influence of each attention head on the two sample sets is calculated. Based on the average influence of each attention head on the two types of sample sets, the contribution of each attention head to the generation of multimodal large model illusion is quantified; The attention heads are sorted from highest to lowest according to their contribution, and a preset number of attention heads with the highest contribution are selected as the key attention heads that trigger hallucinations.

5. The method according to claim 4, characterized in that, The contribution of each attention head to the generation of multimodal large model illusion is quantified based on the average influence of each attention head on the two types of sample sets, including: Attention heads whose average influence exceeds a preset threshold are selected as candidate attention heads. A mean ablation operation is performed on each candidate attention head. The mean ablation operation includes the following steps: replacing the output of the candidate attention head with the average activation value or zero vector at the corresponding position on the correctly predicted sample set, keeping other parameters of the multimodal large model unchanged, re-inputting the correctly predicted sample set and the incorrectly predicted sample set into the multimodal large model after ablation for inference, and calculating the change in illusion rate of the multimodal large model on the incorrectly predicted sample set and the change in accuracy on the correctly predicted sample set before and after the ablation operation. The changes in illusion rate and accuracy are weighted and fused to obtain the contribution of each attention point to the generation of illusions in the multimodal large model, wherein the weight of the change in illusion rate is higher than the weight of the change in accuracy.

6. The method according to claim 1, characterized in that, Before adjusting the attention weights for the key attention head based on the semantic segmentation of the multimodal sequence to alleviate hallucinations during the multimodal large-model inference stage, the method further includes: The attention quality of the key attention head for each semantic segment is calculated during the decoding process of the multimodal large model; The text attention percentage is calculated based on the attention quality. The text attention percentage is obtained by the ratio of the sum of the attention quality of the text semantic segments to the sum of the attention quality of all semantic segments. The text semantic segments include system prompt segments and user instruction segments. Correspondingly, when the text attention ratio exceeds a preset threshold, the attention weight adjustment based on semantic segmentation of the multimodal sequence is triggered; When the text attention ratio does not exceed a preset threshold, the attention weight distribution of each semantic segment in the multimodal sequence is maintained.

7. A multimodal large-scale illusion processing device, characterized in that, include: The construction unit is used to construct a hallucination assessment framework. The hallucination assessment framework is based on the assessment sample and its real label. According to the preset assessment dimensions and assessment index system, the prediction results obtained by inputting the assessment sample into the multimodal large model are compared with the real label, and the quantitative index of hallucination performance of the multimodal large model in each assessment dimension is calculated. The localization unit is used to locate the key attention head that triggers hallucinations in the multimodal large model using the hallucination assessment framework. The adjustment unit is used to adjust the attention weights of the key attention heads based on the semantic segmentation of the multimodal sequence, thereby alleviating hallucinations during the inference stage of the multimodal large model; the semantic segmentation of the multimodal sequence is obtained by structured parsing of the input sequence of the multimodal large model. The attention weight adjustment process involves adjusting the attention weights corresponding to each semantic segment based on the attention weight distribution of the key attention heads for each semantic segment using a differentiated reweighting strategy. The semantic segmentation of the multimodal sequence includes a system prompt segment, a visual feature segment, a user command segment, and a generated response segment. The boundaries of each semantic segment are dynamically determined using a template-aware method. Specifically, the boundaries of the system prompt segment and the user command segment are located by special markers in a predefined dialogue template, the boundaries of the visual feature segment are determined by the number and position of the visual feature markers, and the boundaries of the generated response segment are determined by the length of the historical response content generated in the current decoding step. The adjustment unit is specifically used to: obtain the original attention logic values ​​of the key attention head corresponding to each key position during the multimodal large model decoding process; and scale the original attention logic values ​​according to the semantic segment to which each key position belongs, using the scaling factor corresponding to each semantic segment, to obtain scaled attention logic values. The scaled attention logic values ​​are normalized to obtain the adjusted attention weight distribution of each semantic segment; the adjusted attention weight distribution is used for subsequent tag generation in the multimodal large model decoding process to achieve hallucination relief in the multimodal large model inference stage.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images