Multi-modal visual language model illusion relieving method based on illusion induction optimization

Abstract

The invention discloses a multi-modal visual language model illusion relieving method based on illusion induction optimization, and relates to the field of artificial intelligence and computer vision. According to the method, when a generative large language model generates new word marks, firstly, the probability of each word mark in a vocabulary is generated according to an input mark sequence, the contrast ratio of the current input mark sequence is calculated through a contrast model, the contrast ratio is mapped into a preset vector space, and the corrected logarithmic probability is obtained. And selecting a plurality of illusion tokens with the maximum contrast from the illusion tokens, and adjusting the probability of the illusion tokens according to a preset optimization strategy. If the illusion token belongs to the selected illusion token, amplifying the probability of the illusion token according to preset optimization intensity; otherwise, the current probability is the corrected probability. And selecting a new word mark according to the corrected probability, and circulating the process to generate a final text. According to the method, by introducing the illusion induction optimization strategy, the illusion phenomenon generated when the large visual language model generates the text is remarkably reduced.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and computer vision, and in particular to a multimodal visual language model hallucination alleviation method based on hallucination induction optimization. Background Art

[0002] Currently, Large Vision-Language Models (LVLMs) have made remarkable achievements in the field of multimodal data processing and understanding. These models have performed well in tasks such as image description, visual question answering, and multimodal reasoning by integrating visual and language information. However, these models often suffer from hallucinations when generating text descriptions, that is, the generated text content is inconsistent with the actual content of the image, including imagined objects, fabricated scenes, incorrect spatial relationships, and misidentified categories. These hallucinations not only affect the performance of the model, but may also mislead users and have a negative impact.

[0003] In recent years, researchers have proposed a variety of methods to alleviate the hallucination problem in LVLMs, including post-processing correction methods, self-correction frameworks, and various decoding strategies. Most of these methods rely on introducing visual uncertainty to amplify the contrast gap between hallucinated and target markers. However, due to the uncontrollability of global visual uncertainty, these methods are limited in accurately inducing hallucinated markers, which limits their effectiveness and may even produce new / different hallucinations.

[0004] Existing methods have trade-offs in alleviating hallucinations, especially when high semantic consistency needs to be maintained, where the accuracy and efficiency of these methods are often difficult to strike a balance. To better address this issue, an optimization strategy is needed that can effectively reduce hallucinations during the generation process, while maintaining text quality without sacrificing speed or increasing the possibility of false detection. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a multimodal visual language model hallucination mitigation method based on hallucination-induced optimization (Hallucination-Induced Optimization, HIO), which reduces the hallucination phenomenon when the model generates text by enhancing the contrast between hallucination and target tokens, thereby improving the accuracy and consistency of text description.

[0006] In order to achieve the above-mentioned object of the invention, the multimodal visual language model hallucination alleviation method based on hallucination induction optimization of the present invention comprises the following steps:

[0007] Step 1, initialize the generative large language model, visual input processor, vocabulary and comparison model;

[0008] Among them, the generative large language model is used to generate a probability distribution based on the input text token sequence, and the probability distribution includes the logarithmic probability corresponding to each token in the vocabulary;

[0009] A visual input processor for generating visual context information of an input image;

[0010] Vocabulary, used to store word tokens;

[0011] A contrast model that computes the contrast between a sequence of textual tokens and visual context information and generates a modified probability distribution;

[0012] Step 2, initialize input text and input image;

[0013] A visual input processor is used to obtain visual context information of an input image;

[0014] Use a generative large language model to segment the input text and obtain the corresponding tags for each word, and then form an initial text tag sequence based on the tags of each word;

[0015] Step 3: Based on the current text tag sequence and visual context information input, the generative large language model generates the initial log-probability of each word tag in the vocabulary;

[0016] Step 4, determining the hallucination token and the target token in the current text token sequence based on the initial logarithmic probability, and adjusting the logarithmic probabilities of the hallucination token and the target token to enhance the contrast between the hallucination token and the target token;

[0017] Step 5, based on the contrast between the current enhanced strong hallucination token and the target token, reversely calculate the logarithmic probability of the hallucination token and the target token through the comparison model to obtain the corrected logarithmic probability of each word token of the current text token sequence;

[0018] Step 6, detecting whether the contrast between the current enhanced strong hallucination token and the target token meets the preset condition, if so, executing step 7, otherwise, adjusting the current text tag sequence and returning to step 3 for adjustment;

[0019] Step 7, based on the set sampling method, sample new word tags from the vocabulary according to the corrected logarithmic probability, and add them to the current text tag sequence;

[0020] Step 8, determine whether the preset number of new words is reached, if so, proceed to step 9, otherwise, return to step 3;

[0021] Step 9: The generative large language model uses the reverse processing process to transform the current text tag sequence to obtain the final text output.

[0022] Furthermore, in step 2, the input text is initialized as a description of a question about the input image, and then the answer text of the image question can be obtained based on the final text output of step 9.

[0023] Further, in step 4, determining the hallucination token and the target token in the current text tag sequence based on the initial logarithmic probability specifically includes: considering the tag word corresponding to the minimum initial logarithmic probability in the text tag sequence as the hallucination token, and considering the remaining tag words as the target token.

[0024] Furthermore, in step 4, the contrast between the hallucination token and the target token is enhanced by the inverse Bradley-Terry model (CBTM).

[0025] Furthermore, in step 4, the tag words whose initial logarithmic probability in the text tag sequence is less than or equal to a preset threshold are regarded as potential hallucination tokens, and the remaining tag words are regarded as target tokens; and the differences between multiple potential hallucination tokens are enhanced through a multi-target hallucination amplification method.

[0026] Furthermore, the present invention also includes performing hallucination detection on the final text obtained in step 9, specifically including:

[0027] (1) Using a generative large language model to process (segment and tag) the text to be hallucinated, a test tag sequence is obtained;

[0028] (2) Initialize the contrast model and generate visual context information through the visual input processor;

[0029] (3) using a contrast model to calculate the contrast between the test tag sequence and the currently generated visual context information;

[0030] (4) comparing the contrast of the contrast model output with a preset contrast threshold; if the contrast is less than or equal to the contrast threshold, it indicates that a hallucination exists; otherwise, it indicates that there is no hallucination;

[0031] (5) According to the hallucination detection results, the classification loss of the generative large language model during training is adjusted, and a contrast term is added to the classification loss to reduce the generation of hallucinations.

[0032] Furthermore, the classification loss of the generative large language model during training is set to:

[0033]

[0034] in, represents the classification loss, π θ represents the probability distribution function of the generative large language model under parameter θ, π refrepresents the probability distribution function of the reference model, D represents the sampled data set, represents the sampling expectation, m represents the number of times the generative large language model is input, x represents the input text, y l represents the hallucination token, y w represents the target token, k t The token representing each word, It represents the sampling result according to the probability distribution of its vocabulary when the input image v, input text x, and output y are at time t. It means the expected result when inputting image v, inputting text x, and outputting y at time t, π θ (y li |v,x) represents the initial log probability of the hallucinated token under the current model parameters θ, π ref (y li ∣v,x) represents the initial log probability of the hallucinated token under the reference model, π θ (y w |v,x) represents the initial log probability of the target token under the current model parameters θ, π ref (y w |v,x) represents the initial log probability of the target token under the reference model, and β is the coefficient controlling the contrast amplification.

[0035] The technical solution provided by the present invention brings at least the following beneficial effects:

[0036] When the generative large language model generates a new word tag, the method of the present invention first generates the probability of each word tag in the vocabulary according to the input tag sequence, and uses the contrast model to calculate the contrast of the current input tag sequence, and maps it to a preset vector space to obtain a corrected logarithmic probability. A number of hallucination tokens with the largest contrast are selected from them, and their probabilities are adjusted according to a preset optimization strategy. If it belongs to the selected hallucination token, its probability is amplified according to the preset optimization strength; otherwise, its current probability is the corrected probability. A new word tag is selected according to the corrected probability, and this process is repeated to generate the final text.

[0037] This paper proposes an effective method for alleviating hallucinations. By introducing a hallucination-inducing optimization strategy, it significantly reduces the hallucination phenomenon when large-scale visual language models generate text. Through rigorous testing of different data sets and models, this paper proves that it can effectively reduce hallucinations in text while maintaining the language quality and diversity of the text, and improves the accuracy and consistency of model-generated text. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 It is a processing flow chart of a multimodal visual language model hallucination mitigation method based on hallucination induced optimization (hereinafter referred to as hallucination induced optimization method, HIO) provided in an embodiment of the present invention;

[0040] Figure 2 It is a flow chart summarizing the method for hallucination induction optimization provided in an embodiment of the present invention.

[0041] Figure 3 Schematic diagram of the difference between the illusion mark and the target mark in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] To make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in detail and completely in conjunction with the drawings in the implementation of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present invention.

[0043] Although large visual language models have demonstrated excellent capabilities in understanding multimodal data, the generated text is often inconsistent with the image content, resulting in hallucinations. In response to the hallucination phenomenon in large visual language models, an embodiment of the present invention provides a multimodal visual language model hallucination mitigation method based on hallucination-induced optimization. When generating a new tag, it first generates the logarithmic probability of each tag according to the input tag sequence, uses a contrast decoding model to obtain the logarithmic probability of the current input tag sequence, and corrects it based on a preset theoretical preference model. By amplifying the contrast gap between the hallucination tag and the target tag, the generation of hallucination tags can be reduced. The present invention effectively alleviates the hallucination problem in large visual language models by introducing a new optimization strategy, namely hallucination-induced optimization.

[0044] In one embodiment, see Figure 1 The method for alleviating hallucinations in a multimodal visual language model based on hallucination induction optimization provided by an embodiment of the present invention comprises the following steps:

[0045] S1, system initialization:

[0046] First, initialize the generative large language model, visual input processor, vocabulary and comparison model. The generative large language model is responsible for processing text data to generate text tags; the visual input processor is responsible for extracting image features to generate visual context information; the vocabulary is used to store the set of words that the system can use; and the comparison model is used to evaluate the degree of match between text tags and image content.

[0047] S2, data preprocessing:

[0048] At this step, the input data is officially processed. The visual input processor extracts features from the image and generates visual context information to assist in text generation. At the same time, the generative large language model performs word segmentation on the input text and generates a corresponding text tag sequence. Then, based on these tag sequences and visual context information, the generative large language model predicts the logarithmic probability distribution of each word, indicating the possibility of the word appearing in the current context. During the model generation process, some word tags may not match the actual image content. These tags are called hallucination tokens (that is, words "imagined" by the model), and those that correctly reflect the image content are called target tokens. The initially generated probability distribution provides basic data for subsequent comparative calculations.

[0049] S3, Hallucination Induction:

[0050] In this step, the Contrary Bradley-Terry Model (CBTM) is used to enhance the contrast between the hallucination token and the target token. The Bradley-Terry model is a statistical model that is often used for relative win rate estimation of pairwise comparisons. In this scenario, the Contrary Bradley-Terry model adjusts the logarithmic probabilities of the hallucination token and the target token to amplify the contrast between the two, helping the model to more easily distinguish which words are hallucination tokens. Specifically, based on the text tag sequence and visual context information generated in S2, the generative large language model calculates the logarithmic probability of each word. On this basis, the Contrary Bradley-Terry model amplifies the probability of the hallucination token and compresses the probability of the target token, further widening the probability gap between the hallucination token and the target token. This operation improves the recognition ability of the hallucination token and is ready to provide more accurate basic data for the subsequent hallucination mitigation step.

[0051] S4, contrast calculation:

[0052] In this step, the adjusted log-probability is fed into the contrast model to calculate the contrast between the hallucination token and the target token. The contrast model amplifies the difference in probability between the two types of tokens, ensuring that the probability of the hallucination token is reduced while the probability of the target token remains high. The corrected probability distribution will be used in subsequent steps to ensure that the generation process avoids the selection of hallucination tokens as much as possible.

[0053] S5, Hallucination relief:

[0054] According to the corrected logarithmic probability, it is determined whether the contrast between the illusion token and the target token meets the preset condition. If the condition is met, the process proceeds to step S6, otherwise it returns to step S3 for adjustment.

[0055] S6, generate new tags:

[0056] New word tokens are selected from the vocabulary using the modified probability distribution and added to the current text token sequence. The generated new tokens are words with a higher probability based on the target token, ensuring that the generated results are consistent with the image content.

[0057] S7, iterative process:

[0058] Determine whether the preset number of newly added words has been reached. If yes, proceed to step S8, otherwise return to step S3 to continue generating new tags.

[0059] S8, output text:

[0060] When the text token sequence reaches a predetermined length, the generative large language model converts the final text token sequence into a complete text description and returns it as the output result. At this point, the text has been optimized according to the modified probability distribution to avoid the appearance of hallucination tokens as much as possible.

[0061] In one embodiment, the specific implementation steps of the multimodal visual language model hallucination relief method based on hallucination induction optimization provided in this embodiment include:

[0062] S101: Setting up the generative large language model, visual input processor, vocabulary, and contrast model;

[0063] Among them, the generative language model is used to generate a probability distribution based on the input text token sequence, and the probability distribution includes the logarithmic probability corresponding to each token in the vocabulary;

[0064] A visual input processor, used to process input images and generate visual context information;

[0065] The vocabulary is used to store word tokens, which are generated by the generative large language model based on a preset word set;

[0066] Contrast model, which is used to calculate the contrast between text and visual information and generate the modified probability distribution;

[0067] S102:

[0068] Initialize the input text and input image, use the visual input processor to process the input image to obtain visual context information; use the generative large language model to segment the input text and obtain the corresponding tag for each word, and then form an initial text tag sequence based on the tag of each word, that is, the initialized input text tag sequence Where T is the number of words, t is the word tag index, and y t Mark the tth word;

[0069] S103: Based on the input token sequence and visual context information, the generative large language model generates the initial logarithmic probability logπ of each word token in the vocabulary θ (y|v,x), where π θ represents the probability distribution function of the generative large language model under the parameter θ, y represents the target word tag that the model tries to predict, v represents the visual context information, and x represents the input text.

[0070] S104: Enhancing Hallucination Tokens via the Counter-Bradley-Terry Model (CBTM) l and the target token y w The contrast between the two is:

[0071]

[0072] Where σ(·) is defined as the Sigmoid function; p(y l >y w |x) means that under the condition of given input x, the hallucination token y l Than target token y w The probability of being more likely to be selected; exp(r(x,y)) is the scoring function r(x,y) used to represent the label y. The larger its value, the more likely the label is to be selected by the model; Represents the hallucination token y l The ratio of the log probability under the current model parameters θ to the log probability under the reference model, π θ (y w |v,x) The same applies to the target token y w ; β is the coefficient that controls contrast amplification.

[0073] Among them, the illusion token y l is the minimum initial logarithmic probability logπ in the text tag sequence θThe tag word corresponding to (y|v,x) and the remaining tag words in the text tag sequence are regarded as the target token y w .

[0074] S105: The initial logarithmic probability πθ(y|v,x) is input into the contrast model. The contrast model reversely calculates the corrected logarithmic probability πθ′(y|v,x) based on the contrast between the 4-enhanced hallucination token and the target token currently output by the CBTM model. That is, at this time, the contrast model uses the initial logarithmic probability πθ(y|v,x) and p(y l >y w |x) as input and output the modified log probability πθ′(y|v,x). This modification makes the probability of the hallucinated token lower while the probability of the target token remains high.

[0075] S106: Based on the corrected logarithmic probability, determine whether the contrast between the hallucination token and the target token meets the preset conditions. If the contrast is large enough, it means that the model has successfully distinguished the hallucination token from the target token, and then enter S107; if not, return to step S103 to further adjust the contrast.

[0076] S107: When the contrast condition is met, use the modified logarithmic probability to sample from the vocabulary, select a new word tag, and add it to the current text tag sequence to obtain an updated text tag sequence to continue generating the next word.

[0077] S108: Determine whether the preset number of newly added words has been reached. If yes, proceed to step S109, otherwise return to step S103 to continue generating new tags.

[0078] S109: The generative large language model uses reverse processing to convert the updated complete tag sequence into the final text output to complete the generation process.

[0079] In one embodiment, the multimodal visual language model hallucination mitigation method based on hallucination induction optimization provided by the present embodiment further includes step S10, performing hallucination detection on the generated text (converting the updated complete tag sequence into the final text) finally output by the generative large language model, specifically including:

[0080] S10.1: Use a generative large language model to process the generated text to be tested to obtain a test token sequence;

[0081] S10.2: Initialize the contrast model and generate visual context information through the visual input processor;

[0082] S10.3: Use a contrast model to calculate the contrast between the test tag sequence and the currently generated visual context information;

[0083] S10.4: comparing the contrast output by the contrast model with a preset threshold to determine whether an illusion exists, that is, if there is a contrast less than or equal to the preset threshold, it indicates that an illusion exists;

[0084] S10.5: Adjust the generative large language model based on the detection results to reduce the generation of hallucinations. For example, use contrast as one of the loss functions when training the generative large language model.

[0085] In one embodiment, the difference between multiple potential hallucination tokens can be enhanced during contrast decoding by using the Amplification of Hallucination by Multiple Targets (AMTH) method, with the following goals:

[0086]

[0087] Among them, {y li}, i∈{1,2,…,k} represents multiple potential hallucination tokens, and k is the number of potential hallucination tokens.

[0088] Among them, the potential hallucination token y li Refers to: the initial logarithmic probability logπ in the current text tag sequence θ Those word tokens whose (y|v,x) is less than or equal to the preset threshold. Correspondingly, the non-potential hallucination token y in the current text token sequence li The word token y is considered as the target token w .

[0089] In one embodiment, the limitation of existing classification losses in amplifying the contrast between hallucinations and target tokens can also be overcome by an advanced induced constraint (ACI) method, with the goal of:

[0090]

[0091] in, represents the classification loss, and D represents the loss from sampling, Indicates, m indicates the number of times the generative large language model is input, {k t} represents the tag of each word. It represents the sampling result according to the probability distribution of its vocabulary when inputting picture v, text x and outputting y at time t. It indicates the expected result when inputting image v, text x and outputting y at time t.

[0092] Figure 2 is a flow chart summarizing the optimization method for hallucination induction in this embodiment. Figure 2As shown, the HIO method proposed in this embodiment includes two stages: training stage and inference decoding stage. The specific steps include:

[0093] S201: In the training phase, the input image, query and manual annotation correction, the LVLM generates multiple hallucinated content instances, and the "malicious" LVLM is trained through the HIO method.

[0094] S202: In the inference decoding stage, the logarithmic probability generated by the trained “malicious” LVLM is compared with the logarithmic probability generated by the original LVLM, thereby effectively reducing the occurrence of hallucinations.

[0095] Figure 3 is a schematic diagram of the difference between the illusion mark and the target mark in this embodiment. Figure 3 As shown, the horizontal axis represents the progress of the training steps, and the vertical axis quantifies the log-probability difference between the hallucination mark and the target mark. Through the advanced induced constraint method (ACI), the difference between the hallucination mark and the target mark is significantly increased, making contrast decoding more efficient and reducing the generation of hallucinations. Advanced induced constraints are a technology specifically used to enhance contrast. By dynamically adjusting model parameters during training, the contrast between the hallucination mark and the target mark is amplified. This method can effectively suppress hallucination marks during the model generation process, thereby improving the accuracy of generated text. The specific steps include:

[0096] S301: Calculate the logarithmic probability of the hallucination mark and the target mark, and gradually amplify the logarithmic probability difference between the hallucination mark and the target mark through training steps.

[0097] S302: Apply advanced induced constraints to further enhance the contrast between the hallucination marker and the target marker and reduce the appearance of hallucinations.

[0098] In order to better illustrate the technical effect of the present invention, a specific example is used to experimentally verify the present invention.

[0099] This example uses the MSCOCO and CHAIR datasets for experimental analysis, and the visual language model uses the LLaVA1.5 and VCD models. The visual language model is a model that can process images and texts simultaneously. It generates descriptions or answers to multimodal tasks by combining visual and language information. In order to measure the effect of hallucination reduction, the evaluation indicators use CHAIRs (CHAIR strict score) and CHAIRi (CHAIR integrated score).

[0100] To verify the effectiveness of the HIO method, the experimental steps are as follows:

[0101] (1) Using the MSCOCO and CHAIR datasets, LLaVA1.5 and VCD models are used as base models.

[0102] (2) During the training phase, multiple instances of hallucinated content are generated through the original model, and the HIO method is used to train a “malicious” model. The “malicious model” is a model that intentionally amplifies the hallucinated content in order to help the model better learn and distinguish between hallucinations and target content.

[0103] (3) In the inference decoding stage, the trained “malicious model” is used to generate log-probability and compared with the log-probability of the original model to evaluate the effect of hallucination reduction.

[0104] In order to quantify the effect of HIO method, CHAIRs and CHAIRi indicators are used for evaluation. CHAIRs and CHAIRi measure the frequency and severity of hallucinations when the model generates image descriptions, respectively. The experimental results are shown in Table 1.

[0105] Table 1

[0106]

[0107]

[0108] As shown in Table 1, the HIO method of the embodiment of the present invention significantly reduces the CHAIRs and CHAIRi scores on the MSCOCO dataset, indicating that the present invention can effectively reduce the hallucination problem in large visual language models, that is, the method of the present invention outperforms the existing advanced methods on multiple benchmark tests, proving the effectiveness and superiority of the present invention in practical applications.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0110] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A multimodal visual language model hallucination mitigation method based on hallucination induction optimization, characterized in that: The following steps are involved: Step 1, initialize the generative large language model, visual input processor, vocabulary and comparison model; Among them, the generative large language model is used to generate a probability distribution based on the input text token sequence, and the probability distribution includes the logarithmic probability corresponding to each token in the vocabulary; A visual input processor for generating visual context information of an input image; Vocabulary, used to store word tokens; A contrast model that computes the contrast between a sequence of textual tokens and visual context information and generates a modified probability distribution; Step 2, initialize input text and input image; A visual input processor is used to obtain visual context information of an input image; Use a generative large language model to segment the input text and obtain the corresponding tags for each word, and then form an initial text tag sequence based on the tags of each word; Step 3: Based on the current text tag sequence and visual context information input, the generative large language model generates the initial log-probability of each word tag in the vocabulary; Step 4, determining the hallucination token and the target token in the current text token sequence based on the initial logarithmic probability, and adjusting the logarithmic probabilities of the hallucination token and the target token to enhance the contrast between the hallucination token and the target token; Step 5, based on the contrast between the current enhanced strong hallucination token and the target token, reversely calculate the logarithmic probability of the hallucination token and the target token through the contrast model to obtain the corrected logarithmic probability of each word token of the current text token sequence; Step 6, detecting whether the contrast between the current enhanced strong hallucination token and the target token meets the preset condition, if so, executing step 7, otherwise, adjusting the current text tag sequence and returning to step 3 for adjustment; Step 7, based on the set sampling method, sample new word tags from the vocabulary according to the corrected logarithmic probability, and add them to the current text tag sequence; Step 8, determine whether the preset number of new words is reached, if so, proceed to step 9, otherwise, return to step 3; Step 9: The generative large language model uses the reverse processing process to transform the current text tag sequence to obtain the final text output.

2. The method according to claim 1, characterized in that In step 2, the input text is initialized to be a description of a question about the input image, and then the answer text of the image question can be obtained based on the final text output of step 9.

3. The method according to claim 1, characterized in that In step 4, determining the hallucination token and the target token in the current text tag sequence based on the initial logarithmic probability specifically includes: considering the tag word corresponding to the minimum initial logarithmic probability in the text tag sequence as the hallucination token, and considering the remaining tag words as the target token.

4. The method according to claim 1, characterized in that In step 4, the contrast between the hallucinated token and the target token is enhanced by the inverse Bradley-Terry model.

5. The method according to claim 1, characterized in that In step 4, the tag words whose initial logarithmic probability in the text tag sequence is less than or equal to the preset threshold are regarded as potential hallucination tokens, and the remaining tag words are regarded as target tokens; and the differences between multiple potential hallucination tokens are enhanced through the multi-target hallucination amplification method.

6. The method according to claim 1, characterized in that It also includes hallucination detection on the final text obtained in step 9, specifically including: (1) Using a generative large language model to process the text to be hallucinated, a test tag sequence is obtained; (2) Initialize the contrast model and generate visual context information through the visual input processor; (3) using a contrast model to calculate the contrast between the test tag sequence and the currently generated visual context information; (4) comparing the contrast of the contrast model output with a preset contrast threshold; if the contrast is less than or equal to the contrast threshold, it indicates that a hallucination exists; otherwise, it indicates that there is no hallucination; (5) According to the hallucination detection results, the classification loss of the generative large language model during training is adjusted, and a term related to contrast is added to the classification loss.

7. The method according to claim 1, characterized in that The classification loss of the generative large language model during training is set to: in, represents the classification loss, π θ represents the probability distribution function of the generative large language model under parameter θ, π ref represents the probability distribution function of the reference model, D represents the sampled data set, represents the sampling expectation, m represents the number of times the generative large language model is input, x represents the input text, and y l represents the hallucination token, y w represents the target token, k t The token representing each word, It represents the sampling result according to the probability distribution of its vocabulary when input image v, input text x, and output y at time t. It means the expected result when inputting image v, inputting text x, and outputting y at time t, π θ (y li |v,x) represents the initial log probability of the hallucinated token under the current model parameters θ, π ref (y li ∣v,x) represents the initial log probability of the hallucinated token under the reference model, π θ (y w |v,x) represents the initial log probability of the target token under the current model parameters θ, π ref (y w |v,x) represents the initial log probability of the target token under the reference model, and β is the coefficient controlling the contrast amplification.

Images