Method and system for reducing illusion generated by large visual language model
By intervening in causal graphs to identify and block hallucination triggers during the generation of large visual language models, and by using image, text and embedding intervention methods, the problem of hallucinations in the model was solved, and an efficient and safe generation process was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- INVENTEC PUDONG TECH CORPOARTION
- Filing Date
- 2024-11-18
- Publication Date
- 2026-05-19
AI Technical Summary
Large visual language models are prone to generating illusions when generating content, which can affect the reliability and security of downstream applications. Existing methods are costly or increase computational burden.
By intervening in causal graphs, the computing device identifies hallucination triggers before the generation process and blocks the influence of hallucinations by replacing parts of the input, including image intervention, text intervention and embedding intervention, to directly intervene in hallucination triggers to reduce the detection and generation of hallucinatory objects.
It effectively reduces illusions in large visual language models without increasing inference time, improves the reliability and security of the generation process, and avoids the need for model retraining and iterative inference.
Smart Images
Figure CN122067066A_ABST
Abstract
Description
Technical Field
[0001] This application relates to large visual language models, and in particular to a method and system for reducing the illusions produced by large visual language models. Background Technology
[0002] Large visual language models possess a powerful ability to understand multimodal data and respond to human commands. With advancements in network architecture, much research has focused on improving response accuracy and reducing deviations from human instructions. Despite these efforts, modern large visual language models still face real-world challenges due to the well-known illusion problem, which can jeopardize the reliability and security of downstream applications.
[0003] When the generated content is inconsistent with the provided visual cues, or contains irrelevant or erroneous text, it indicates that the large visual language model is misleading. Fine-tuning the large visual language model using human preferences is a feasible approach, but it is costly and requires extensive manual annotation. Another approach is to have the large visual language model progressively answer multiple validation questions, but this would significantly increase the computational burden. Summary of the Invention
[0004] In view of this, this application proposes a method and system for reducing the hallucinations produced by large visual language models.
[0005] A method for reducing illusions generated by a large visual language model according to an embodiment of this application includes multiple steps performed by a computing device, the multiple steps including: acquiring a test image; inputting the test image and a prompt to a large visual language model to generate a test embedding; providing a prompt to instruct the large visual language model to describe the test image; identifying a candidate embedding that is closest to the test embedding among multiple reference embeddings; replacing the test embedding with data in a saliency dimension of the candidate embedding; and generating a test result based on the replaced test embedding of the large visual language model.
[0006] A system for reducing hallucinations caused by large visual language models according to an embodiment of this application includes a storage device and a computing device. The storage device stores a test image, a large visual language model, and multiple reference embeddings. The computing device is electrically connected to the storage device. The computing device is used to input the test image and prompts to the large visual language model to generate a test embedding, wherein the prompts are used to instruct the large visual language model to describe the test image. The computing device is also used to find a candidate embedding among the multiple reference embeddings that is closest to the test embedding, replace the data of the test embedding in the saliency dimension with the data of the candidate embedding in the saliency dimension, and generate a test result based on the replaced test embedding using the large visual language model.
[0007] In summary, this application proposes a method and system aimed at efficiently reducing hallucinations in large visual language models without requiring model retraining and iterative inference. This application blocks the influence of hallucination triggers by intervening in the causal graph. This intervention is implemented by replacing a portion of the input, adding almost no increase to inference time. The proposed method and system directly intervene in the identified hallucination triggers, thereby reducing the detection of hallucinatory objects and the repeated generation of hallucinatory objects. Compared to previous methods that only focused on eliminating generated hallucinatory objects, this application captures and modifies the sources of influence that may lead to hallucinations before the generation process.
[0008] The foregoing description of the disclosure and the following description of the embodiments are intended to demonstrate and explain the spirit and principles of this application, and to further explain the scope of this patent application. Attached Figure Description
[0009] Figure 1 This is a block architecture diagram of a system for reducing illusions caused by large visual language models according to an embodiment of this application;
[0010] Figure 2 This is a flowchart of a method for reducing the illusions produced by large visual language models according to an embodiment of this application;
[0011] Figure 3 This is a flowchart illustrating the generation of a reference embedding according to an embodiment of this application;
[0012] Figure 4 This is a schematic diagram of the ideal output of a large visual language model;
[0013] Figure 5 It is a causal graph model for large visual language models when outputting hallucinations; and
[0014] Figures 6 to 8 These correspond to the three intervention implementation examples proposed in this application.
[0015] Component designation explanation
[0016] S1-S5, T1-T6: Steps
[0017] X: Input
[0018] I: Test Image
[0019] Q: Hint
[0020] A: Test Results
[0021] Zo: Latent variables of the target object
[0022] Zc: Background Factor
[0023] S: Mediating variable Detailed Implementation
[0024] The following detailed description of the features and advantages of this application in the embodiments is sufficient to enable anyone skilled in the art to understand the technical content of this application and implement it accordingly. Furthermore, based on the content disclosed in this specification, the scope of the patent application, and the drawings, anyone skilled in the art can easily understand the related objectives and advantages of this application. The following embodiments further illustrate the viewpoints of this application in detail, but are not intended to limit the scope of this application in any way.
[0025] Figure 1 This is a system for reducing the hallucinations caused by large visual language models, according to an embodiment of this application. For example... Figure 1 As shown, this system includes a storage device 1 and a computing device 3.
[0026] Storage device 1 is used to store test images, a large visual language model, and multiple reference embeddings.
[0027] In one embodiment, the storage device 1 may be implemented using at least one hardware device from the following categories: flash memory, hard disk (HDD), solid-state drive (SSD), dynamic random access memory (DRAM), static random access memory (SRAM), or other non-volatile memory. However, this application is not limited to the above categories.
[0028] The test image can be any image, and this application is not limited thereto. In one embodiment, the architecture of the large visual language model is an autoregressive Transformer, such as Instructblip (Towards general purpose vision-language models with instruction tuning) and / or mPLUG-Owl2 (Revolutionizing multi-modal large language model with modality collaboration).
[0029] The computing device 3 is electrically connected to the storage device 1. The computing device 3 is used to input a test image and a prompt into a large visual language model to generate a test embedding, wherein the prompt is text used to instruct the large visual language model to describe the test image. The computing device 3 is used to find the candidate embedding that is closest to the test embedding among the plurality of reference embeddings, replace the test embedding's data in the saliency dimension with the data of the candidate embedding in the saliency dimension, and generate a test result based on the replaced test embedding using the large visual language model.
[0030] In one embodiment, the computing device 3 may be implemented using at least one of the following hardware devices: a personal computer, a network server, a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller (MCU), an application processor (AP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system-on-a-chip (SOC), a deep learning accelerator, or any electronic device with similar functionality. This application does not limit the hardware type of the computing device 3.
[0031] Figure 2 This is a flowchart of a method for reducing the illusions produced by large visual language models according to an embodiment of this application, including steps S1 to S5.
[0032] In step S1, the computing device 3 acquires the test image. In one embodiment, the computing device 3 can acquire the test image from the storage device 1 on the local end of the system, or it can acquire the test image in real time from outside the system when the computing device 3 is running a large visual language model. This application does not limit this.
[0033] In step S2, the computing device 3 inputs the test image and prompts to the large visual language model to generate a test embedding, wherein the test embedding is an intermediate output of the large visual language model.
[0034] In step S3, the computing device 3 finds the candidate embedding that is closest to the test embedding among multiple reference embeddings. In one embodiment, the K-nearest neighbor method of L2 distance is used: the L2 distance between each reference embedding and the test embedding is calculated, the K corresponding reference embeddings with the smallest L2 distances are selected from multiple L2 distances, and the average embedding of these K reference embeddings is calculated as the candidate embedding. Figure 3 This is a flowchart of generating a reference embedding according to an embodiment of the present application, which is executed before step S3 and includes steps T1 to T6.
[0035] In step T1, the computing device 3 acquires multiple reference images and multiple standard answers corresponding to these reference images. Each reference image includes a subject, and each standard answer describes the subject. In one embodiment, the images and standard answers come from the AMBER (An LLM-free multi-dimensional benchmark for MLLMs hallucinationevaluation) dataset, which is a benchmark dataset for evaluating hallucinations in large visual language models and contains manually labeled actual objects and potential hallucination objects.
[0036] In step T2, the computing device 3 inputs the plurality of reference images and cues to a large visual language model to generate a plurality of embeddings and a plurality of output texts. The cues are used to instruct the large visual language model to describe each reference image, each embedding is an intermediate output of the large visual language model based on each reference image and cues, and each output text is a description of the reference image obtained by the large visual language model based on the embeddings.
[0037] In step T3, the computing device 3 compares the output text of the reference image with the standard answer for each reference image. If they match, proceed to step T4; otherwise, proceed to step T5.
[0038] In step T4, if the output text matches the standard answer, it means the output text does not mention objects that are not present in the reference image, equivalent to the large visual language model not generating an illusion. Therefore, the processing unit 3 classifies the embeddings of the current image mapping into the non-illusion group. The reference embeddings mentioned in step S3 are all embeddings in the non-illusion group. On the other hand, if the output text does not match the standard answer, it means the output text mentions objects that are not present in the reference image. This is equivalent to the large visual language model generating an illusion. Therefore, as shown in step T5, the processing unit 3 classifies the embeddings of the current image mapping into the illusion group.
[0039] In step T6, the computing device 3 identifies the dimension with the greatest difference between the embeddings of the non-hallucination group and the embeddings of the hallucination group as the saliency dimension. In one embodiment, the computing device 3 examines each dimension between the embeddings of the two groups using statistical methods, such as the Student's t-test, and selects the dimension with a p-value less than 0.001 to derive a saliency map. This map displays at least one saliency dimension in the dataset that distinguishes the hallucination group from the non-hallucination group.
[0040] Please return Figure 2 In step S4, the computing device 3 replaces the test embedding in one saliency dimension with the candidate embedding. In step S5, the large visual language model generates a test result based on the replaced test embedding.
[0041] In one embodiment, the embedded editing is performed as follows:
[0042] E′ q = (1-ρ)·E q +ρ·M·E K
[0043] Where E q Represents test embedding, Represents a saliency map, E K E′ represents the candidate embedding, and ρ is a hyperparameter that determines the editing intensity. q This represents the replacement of the completed test embedding, used to decode and generate test results.
[0044] This application focuses on reducing the illusion of large visual-language models through causal intervention. The specific steps of the "embedded intervention" embodiment have been described above; embodiments of "image intervention" and "text intervention" are also included. Before describing the specific steps of these two embodiments, please refer to... Figures 4 to 8 .
[0045] The Directed Acyclic Graph (DAG) in the diagram represents a causal graphical model, including the test image I, cue Q, latent variable Zo of the target object, context factor Zc, and the final test result A. Directed edges in the graph represent the direct causal influence of parent nodes on child nodes. This application distinguishes and abstracts the variables Zo and Zc at the cognitive level. Zo serves as the ideal semantic representation of the target object (e.g., the concept of a car), while Zc, as a confounding factor, represents a background pattern that diversifies the understanding of the car.
[0046] Figure 4 This is a schematic diagram of the ideal output of a large visual language model, where A is independent of Zc. However, as... Figure 5 As shown, the inherent bias of the training data introduces Zc into the causal graph model of a large visual language model, leading to the undesirable causal effect Zc→Zo. Therefore, this application proposes three embodiments—"image intervention," "text intervention," and / or "embedding intervention"—to prevent the Zc→Zo path, with the schematic diagrams corresponding to... Figures 6 to 8 It must be noted that these three embodiments can be operated in combination or independently, and this application does not limit them in this regard.
[0047] The "image intervention" implementation includes two methods: attaching small objects to the background of a test image, and removing hallucinogenic objects from the test image. The first method involves the following steps: before inputting the test image and cues into a large visual language model to generate a test embedding, a small image is overlaid on the edge of the test image. This small image is smaller than the test image, its overlay position is offset from the subject, and its content is semantically unrelated to the subject. For example, a small image containing a single object is attached to the upper left corner of the test image. The object in the small image is approximately one-sixth the size of the shortest side of the test image, ensuring the object is identifiable and in the background, implicitly influencing Zo. The second method involves removing the most hallucinogenic subjects from the test image based on statistical data; for example, "car" might cause the hallucination of "road." In one embodiment, the computing device 3 uses two techniques, Grounding dino (Marrying dino with grounded pre-training for open-set object detection) and IA (Inpaint anything: Segment anything meets image inpainting), to detect and segment objects, and then uses a patching technique to fill in the masked area.
[0048] An example of "text intervention" is as follows: Before inputting the test image and cues into a large visual language model to generate the test embedding, the cues include commands for the large visual language model to separately describe the foreground and background of the test image. Text intervention comprises two steps, generating cues for the foreground (FG) and background (BG) respectively, by introducing... Figure 7 The mediating variable S is used to perform these two steps of prompting. First, the large visual language model is asked to describe the main subject of the foreground, and this description is used as a prompt. Then, other details of the background are further described. Specifically, the foreground description Af is obtained using the prompt "Describe the foreground, ignoring the background in the test image". The prompt is then changed to "Describe other content in the background given the foreground is [Af]".
[0049] This application aims to reduce hallucinations in large visual language models efficiently without requiring model retraining and iterative inference. Specifically, this application proposes to systematically observe causal relationships in images and block the influence of hallucination triggers by intervening in the causal graph. Figures 2 to 6This provides an overview of the method described above. This intervention is implemented by replacing a portion of the input, adding almost no increase to inference time. Our framework directly intervenes in the identified hallucination triggers, thereby reducing the detection of hallucination objects and the generation of multiple repetitions. Compared to previous methods that focused solely on eliminating generated hallucination objects, our framework captures and modifies potential sources of hallucination before the generation process begins.
[0050] In summary, this application proposes a method and system aimed at efficiently reducing hallucinations in large visual language models without requiring model retraining and iterative inference. This application blocks the influence of hallucination triggers by intervening in the causal graph. This intervention is implemented by replacing a portion of the input, adding almost no increase to inference time. The proposed method and system directly intervene in the identified hallucination triggers, thereby reducing the detection of hallucinatory objects and the repeated generation of hallucinatory objects. Compared to previous methods that only focused on eliminating generated hallucinatory objects, this application captures and modifies the sources of influence that may lead to hallucinations before the generation process.
[0051] While this application discloses the above-described embodiments, it is not intended to limit the scope of this application. Any modifications and refinements made without departing from the spirit and scope of this application are within the scope of patent protection of this application. For the scope of protection defined in this application, please refer to the appended patent claims.
Claims
1. A method for reducing the hallucinations produced by large visual language models, characterized in that, The method includes multiple steps executed by a computing device, the multiple steps including: Obtain a test image; Input the test image and a cue into a large visual language model to generate a test embedding, wherein the cue is used to instruct the large visual language model to describe the test image; Find the candidate embedding that is closest to the test embedding among multiple reference embeddings; The test embeddings in the saliency dimension are replaced with the candidate embeddings in the saliency dimension; and A test result is generated based on the large visual language model and the completed test embedding.
2. The method for reducing hallucinations caused by large visual language models according to claim 1, characterized in that, Before replacing the test embeddings in the saliency dimension with the candidate embeddings, the method further includes: Obtain multiple reference images and multiple standard answers corresponding to the multiple reference images, wherein each of the multiple reference images includes a subject, and each of the multiple standard answers is used to describe the subject; The plurality of reference images and the prompts are input into the large visual language model to generate a plurality of embeddings and a plurality of output texts, wherein the prompts are used to instruct the large language visual model to describe each of the plurality of reference images; The multiple output texts were compared based on the multiple standard answers; When one of the multiple output texts matches one of the multiple standard answers, the corresponding embedding is classified into the non-illusion group, wherein the multiple reference embeddings are the multiple embeddings in the non-illusion group; When one of the multiple output texts does not match one of the multiple standard answers, the corresponding embedding is classified into the illusion group; and The dimension with the greatest difference among the multiple embeddings of the non-hallucination group and the multiple embeddings of the hallucination group is identified as the saliency dimension.
3. The method for reducing hallucinations caused by large visual language models according to claim 2, characterized in that, The method includes using the Stulton t test to identify the saliency dimension that differs most significantly among the multiple embeddings of the non-hallucination group and the multiple embeddings of the hallucination group.
4. The method for reducing hallucinations caused by large visual language models according to claim 2, characterized in that, Before inputting the test image and the prompt into the large visual language model to generate the test embedding, the method further includes: overlaying a small image on the edge of the test image, wherein the size of the small image is smaller than the size of the test image, the position of the small image is offset from the position of the subject, and the content of the small image is semantically unrelated to the subject.
5. The method for reducing hallucinations caused by large visual language models according to claim 1, characterized in that, Before the test image and the prompt are input into the large visual language model to generate the test embedding, the prompt commands the large visual language model to describe the foreground and background of the test image separately.
6. A system for reducing hallucinations caused by large visual language models, characterized in that, The system includes: A storage device for storing a test image, a large visual language model, and multiple reference embeddings; A computing device electrically connected to the storage device is configured to input the test image and a prompt to the large visual language model to generate a test embedding, wherein the prompt is configured to instruct the large visual language model to describe the test image. The computing device is also configured to find a candidate embedding among a plurality of reference embeddings that is closest to the test embedding, replace the test embedding with the data of the candidate embedding in the saliency dimension, and generate a test result based on the replaced test embedding by the large visual language model.
7. The system for reducing hallucinations caused by large visual language models according to claim 6, characterized in that, Before replacing the test embeddings in the saliency dimension with the candidate embeddings, the computing device is further configured to: Obtain multiple reference images and multiple standard answers corresponding to the multiple reference images, wherein each of the multiple reference images includes a subject, and each of the multiple standard answers is used to describe the subject; The plurality of reference images and the prompts are input into the large visual language model to generate a plurality of embeddings and a plurality of output texts, wherein the prompts are used to instruct the large language visual model to describe each of the plurality of reference images; The multiple output texts were compared based on the multiple standard answers; When one of the multiple output texts matches one of the multiple standard answers, the corresponding embedding is classified into the non-illusion group, wherein the multiple reference embeddings are the multiple embeddings in the non-illusion group; When one of the multiple output texts does not match one of the multiple standard answers, the corresponding embedding is classified into the illusion group. as well as The dimension with the greatest difference among the multiple embeddings of the non-hallucination group and the multiple embeddings of the hallucination group is identified as the saliency dimension.
8. The system for reducing hallucinations caused by large visual language models according to claim 7, characterized in that, The computing device is used to perform the Stuttgart t-test to find the saliency dimension that differs most among the plurality of embeddings in the non-illusion group and the plurality of embeddings in the illusion group.
9. The system for reducing hallucinations caused by large visual language models according to claim 7, characterized in that, Before inputting the test image and the prompt into the large visual language model to generate the test embedding, the computing device is further configured to overlay a small image on the edge of the test image, wherein the size of the small image is smaller than the size of the test image, the overlay position of the small image is offset from the subject, and the content of the small image is semantically unrelated to the subject.
10. The system for reducing hallucinations caused by large visual language models according to claim 6, characterized in that, Before inputting the test image and the prompt into the large visual language model to generate the test embedding, the computing device is further configured to instruct the large visual language model to separately describe the foreground and background of the test image by means of the prompt.