Vision vocabulary guided multimodal large model hallucination optimization method and storage medium

Abstract

The present application relates to the technical field of large model training, and discloses a visual vocabulary guided multimodal large model hallucination optimization method and a storage medium, training samples including images, text prompts, preferred answers and rejected answers are constructed, a visual target detection model is used to perform image detection on the images according to the preferred answers, relevant indexes consistent with visual words are extracted, and the visual weights of each visual word are calculated accordingly, different types of words are differentially weighted according to the visual weights, a dynamic adjustment coefficient is defined according to the proportion of the corrected segment, and the dynamic adjustment coefficient is combined with the probability difference and the differential weighting to construct a DPO loss function, so as to guide the multimodal large model to generate answers consistent with images and texts. The differential optimization weight is dynamically assigned, so that the multimodal large model pays attention to the words highly related to the image semantics, thereby generating information description more consistent with visual facts, and improving the consistency and authenticity of text generation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large model training, in particular to a visual vocabulary guided multimodal large model hallucination optimization method and a storage medium. BACKGROUND

[0002] Since the advent of artificial intelligence technology and its rapid development, multimodal large language models (MLLMs) have made significant progress in image question answering, image-text generation, text understanding, content description, and other tasks, and are widely used in human-computer interaction, auxiliary annotation, scientific education, intelligent medical care, social security, and many other fields, and are deeply integrated into people's daily life. Such multimodal large language models usually combine visual encoders and language generators to achieve the ability to understand image semantic content and the fusion of natural language expression.

[0003] Although existing MLLMs perform well in image-text content understanding, coherent generation, and language fluency, they still have an unavoidable "large model hallucination" problem. Large model hallucination refers to the phenomenon that a multimodal large language model outputs information that does not match the actual content of an image when processing image question answering or description generation tasks, such as Figure 5 、 6 、7, which can be divided into three categories: (1) hallucination of fictitious entities, i.e., the model generates objects that do not exist in the image, such as "a cat lying on the sofa" in an image with no cat; (2) hallucination of attribute misjudgment, i.e., the model identifies real objects in the image but describes their attributes (such as color, number, position, etc.) incorrectly, for example, describing "a black sofa" as "a red sofa"; (3) hallucination of missing entities, i.e., some important elements in the image are not recognized or reflected by the model, such as ignoring objects on the edge of the image and generating statements like "there is no obvious content in the image." The above three types of hallucination phenomena are more likely to occur in multi-target images, complex backgrounds, or non-central regions, which not only reduces the consistency of the model's image-text, but also seriously affects its practical value in high-reliability application scenarios, especially in fields such as medical image analysis and autonomous driving scene perception, which can pose a significant risk.

[0004] To alleviate the hallucination problem of large models, existing researches mostly use reinforcement learning based on human feedback (RLHF) or direct preference optimization (DPO) methods to guide the model to generate more in line with human expectations by introducing user preference signals. Although these methods perform well in pure language tasks, they still have obvious limitations in multi-modal scenarios involving image-text collaboration, mainly in three aspects: first, there is a lack of effective modeling of image and text semantic consistency, making it difficult to identify key words in the text that need to be accurately aligned with image content; second, it is unable to measure the importance of each word in the visual semantic, and all words are treated equally in training, making it difficult for the model to focus on key visual elements; third, for incorrect answers containing hallucinations, existing methods lack a dynamically adjustable punishment mechanism for hallucinations, limiting the improvement of the model's correction ability. SUMMARY

[0005] In view of the deficiencies in the prior art, the purpose of the present application is to provide a visual vocabulary guided multi-modal large model hallucination optimization method.

[0006] In order to achieve the above purpose, the present application provides the following technical solutions:

[0007] The visual vocabulary guided multi-modal large model hallucination optimization method is an optimization method with word-level visual recognition ability, fine-grained preference adjustment mechanism and dynamic hallucination content punishment, which enhances the multi-modal large model's understanding of real image content in the preference learning framework by introducing a visual guidance mechanism and a weighted preference optimization strategy, effectively reducing hallucination output and improving the accuracy and reliability of multi-modal large model generation results.

[0008] The multi-modal large model hallucination optimization method specifically includes the following steps:

[0009] Training data construction: construct training samples including images, text prompts, preferred answers and rejected answers, wherein the preferred answers are highly corresponding to the images and the rejected answers contain hallucination components as a reference for comparison;

[0010] Visual information extraction and word weight calculation: use a visual object detection model to detect images based on preferred answers, extract relevant indicators consistent with visual words, and calculate the visual weight of each visual word accordingly;

[0011] Visual guided multi-modal large model modeling: differentially weight different types of words according to the visual weight to improve the multi-modal large model's attention to image-text alignment words during modeling;

[0012] Dynamic hallucination punishment adjustment: dynamically adjust according to the proportion of corrected segments in the rejected answers, so that answers containing more incorrect segments receive stronger constraint signals in training;

[0013] The visual guidance preference loss function is constructed: the probability difference of the preferred answer and the rejected answer is fused, and a differentiated weighting and dynamic adjustment mechanism is combined to construct the DPO loss function of the multi-modal large model, so as to guide the model to generate answers consistent with the image semantics.

[0014] In the present application, preferably, in each of the training samples, a quadruple (I, x, y, y') is constructed, ), wherein I represents an input image, x is a text prompt, is a preferred answer, is a rejected answer.

[0015] In the present application, preferably, the open target detection model performs image detection on the image according to the noun in the preferred answer, and when the target region corresponding to the noun is found, the noun entity is a visual word, and the visual weight of the visual word is calculated.

[0016] In the present application, preferably, the correlation index includes:

[0017] Detection confidence : the confidence degree of the target detection model to the correspondence between the visual word and the entity in the image;

[0018] Target area ratio : representing the area ratio of the detection region in the whole image, the larger the area ratio, the more significant the area ratio;

[0019] Center deviation : measuring the position of the target corresponding to the visual word in the image, the closer to the center, the stronger the visual saliency.

[0020] In the present application, preferably, the calculation of the visual weight of each visual word is specifically:

[0021]

[0022] wherein is the detection confidence, is the target area ratio, is the center deviation. The final visual weight will be used to guide the optimization intensity of the word in subsequent language modeling, ensuring that the model focuses on the core entity words that actually exist in the image, thereby significantly reducing the generation probability of entity hallucination.

[0023] In the present application, preferably, a multi-modal large model modeling objective function corresponding to the differentiated weighting is constructed:

[0024]

[0025] wherein​ is the visual weight of the visual word, is the overall weighting factor of the attribute word, N is the total number of words in the sequence, and is used as a normalization factor, is the segment identified by GroundingDINO, is the noun, adjective or quantifier not identified, is the rest of the unrecognized words; the attribute word weighting needs to be attached to the visual target word: no weighting for unbound visual words, and independent weighting for multiple visual words, is the text sequence generated by the multi-modal large model relative to the multi-modal input log-likelihood value, is the conditional probability, is the sequence element, and is the text sequence the word in the text sequence.

[0026] In the present application, a dynamic adjustment coefficient is introduced to scale the probability difference between the preferred answer and the rejected answer, wherein the is defined as:

[0027]

[0028] wherein is the length of the corrected segment, specifically the token length corresponding to the segment that is inconsistent with the preferred answer in the rejected answer and needs to be corrected; is the total length of the answer, specifically the total number of tokens obtained by the rejected answer or the preferred answer; is the basic scaling coefficient. During training, will be used in the DPO loss function: can be directly replaced by in the loss calculation of each sample , or take the average of all in a batch as of this iteration. The length of the corrected segment is calculated in token granularity, and the determination adopts an automatic contrast combination mode, which aligns the text of the preferred and rejected answers through the model and performs differential analysis, and if necessary, manual verification is supplemented to ensure accurate determination.

[0029] In the present application, preferably, the DPO loss function of the multi-modal large model is constructed as follows:

[0030] ,

[0031] wherein, represents the multi-modal model being trained, is a reference model, whose parameters remain frozen and are not updated during the training process, serving as a contrastive benchmark, providing a stable reference distribution, to ensure the effectiveness of the loss function optimization, the reference model is constructed based on the same training data as the current model; is a regularization coefficient, used to control the scaling magnitude of the probability ratio difference; is a Sigmoid function, used to normalize the output and ensure numerical stability, is a set of training samples expected calculation, is a multi-modal input, specifically an image and a text prompt.

[0032] Compared with the prior art, the beneficial effects of the present application are:

[0033] The method of the present application dynamically assigns differentiated optimization weights to the image-guided word-level importance mechanism by introducing it, making the multi-modal large model pay more attention to the words highly related to the image semantics, thereby generating information descriptions that are more consistent with visual facts, improving the consistency and authenticity of text generation; improving hallucination recognition ability, significantly reducing hallucination rate in multiple standard evaluation tasks; the hallucination recognition ability of the multi-modal large model is improved, and the hallucination rate of the generated text description is significantly reduced. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a flowchart of the visual vocabulary-guided multi-modal large model hallucination optimization method of the present application.

[0035] Figure 2 is an effect diagram of a specific embodiment in the present application.

[0036] Figure 3 is an effect diagram of another specific embodiment in the present application.

[0037] Figure 4 is an effect diagram of another specific embodiment in the present application.

[0038] Figure 5 is an existing diagram of a large model hallucination.

[0039] Figure 6 is another existing diagram of a large model hallucination.

[0040] Figure 7 is another existing diagram of a large model hallucination. DETAILED DESCRIPTION

[0041] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0043] Please refer to Figure 1 A preferred embodiment of the present application provides a visual vocabulary guided multimodal large model hallucination optimization method. On the basis of a traditional DPO framework, image perception information is introduced, visual feature keywords are used as clues to dynamically adjust the sensitivity of a multimodal large model to hallucination content, and a differentiated loss function is constructed to guide the multimodal large model to more accurately generate text descriptions consistent with image content.

[0044] Specific steps are as follows:

[0045] S1, training data construction: constructing training samples including images, text prompts, preferred answers and rejected answers, wherein the preferred answers are highly corresponding to the images, and the rejected answers contain hallucination components as contrast references;

[0046] S2, visual information extraction and word weight calculation: using a visual target detection model to perform image detection on the images according to the preferred answers, extracting related indexes consistent with visual words, and calculating the visual weights of each visual word according to the related indexes;

[0047] S3, visual guided multimodal large model modeling: differentially weighting different types of words according to the visual weights, and improving the attention ability of the multimodal large model to the image-text alignment words in the modeling process;

[0048] S4, dynamic hallucination penalty adjustment: dynamically adjusting according to the proportion of the corrected segments in the rejected answers, so that the answers containing more error segments receive stronger constraint signals in the training;

[0049] S5, visual guided preference loss function construction: fusing the probability difference of the preferred answers and the rejected answers, combining the differentiated weighting and the regularization coefficient, and constructing a DPO loss function of the multimodal large model to guide the model to generate answers consistent with the image semantics.

[0050] In step S1, in each of the training samples, a quadruple (I, x, ..., ...) is constructed. , ), where I represents the input image and x represents the text prompt. For the best answer, This answer will be rejected. (Preferred answer) It is a text description that is highly consistent with the image content and has clear and natural semantics; while the answer was rejected. This includes at least one piece of information that contradicts the image content (i.e., "illusion words"), such as a target that is described but does not exist in the image, incorrect color attributes, or logical contradictions. This sample structure allows the training process to not only learn "how to generate the correct answer" but also explicitly learn "which answers are unacceptable," thereby establishing clear boundaries for image-text matching.

[0051] In step S2, traditional language models cannot combine images to determine which keywords should be focused on; instead, they distribute attention evenly, which easily leads to the generation of words unrelated to the image, i.e., "illusion words." To solve this problem, this step aims to identify core words that require image semantic support, i.e., "visual words," in the generated text and evaluate the semantic importance of these words in the image, forming a word-level visual perception mechanism.

[0052] Specifically, advanced visual object detection models (such as GroundingDINO) are first used to select the best answer. The noun entities in the image are detected word by word. When the visual object detection model successfully finds the target region corresponding to the noun in the image, the noun entity is considered a "visual word," meaning that the word plays an irreplaceable core role in image-text consistency modeling. When no match is found, the word is not assigned additional weight in the preferred answer; instead, the predefined model weights for each word are used for calculation, with a default weight of 1.0 in the specific implementation. The visual word is quantified from three dimensions, i.e., the visual weight of the visual word is calculated. These three dimensions include:

[0053] Detection confidence The confidence level of an object detection model is the degree of confidence it has in the correspondence between visual words and entities in an image. After performing detection using a visual object detection model, it returns a confidence score for each detectable object; this confidence score is the detection confidence score. ;

[0054] Target area percentage This indicates the proportion of the detected region in the entire image; a larger value indicates greater significance. It is generated by the visual object detection model, which uses a rectangular box to outline the detected object. The area of ​​this rectangle is proportional to the area of ​​the entire image;

[0055] center deviation : measure the position of the target corresponding to the visual word in the image, the visual target detection model generates a rectangular frame for the detected object, and calculates the deviation value of the center position of the object rectangular frame and the center position of the whole picture. The deviation value is the center deviation , the closer to the center, the stronger the visual saliency. The three dimensions are the corresponding related indicators.

[0056] The calculation of the visual weight of each visual word is as follows:

[0057]

[0058] wherein is the detection confidence, is the target area area ratio, is the center deviation. The final visual weight will be used to guide the optimization strength of the visual word in the subsequent multi-modal large model modeling, ensuring that the multi-modal large model focuses on the core entity words that actually exist in the image, thereby significantly reducing the generation probability of entity class hallucination.

[0059] In step S3, after obtaining the visual weight of each word, the application further constructs a differentiated language training mechanism, so that the multi-modal large model can focus on optimization in the learning process. "words that actually exist in the image" and maintain basic grammar modeling for "words without image correspondence" structure words, and give appropriate strengthening to "modification attribute words" (such as color, quantity), thereby proposing a visual guided language modeling mechanism, introducing a differentiated optimization strategy from the type of vocabulary.

[0060] First, the preferred answer corresponding to the answer text y contains various nouns and adjectives, etc. The words in the answer text y are divided into three categories:

[0061] visual target word : including the noun entity fragment recognized by GroundingDINO, which is the key carrier of image-text consistency;

[0062] attribute word : descriptive words of target words, such as color, quantity, orientation;

[0063] language structure word : such as prepositions, conjunctions, etc., mainly serving as a grammatical structure, and has no direct dependence on images.

[0064] According to this classification, a differentiated weighted multi-modal large model modeling objective function is constructed:

[0065]

[0066] wherein is the visual weight of the visual word, is the overall weighting factor of the attribute word, and , >1, N is the total number of words in the sequence, used as a normalization factor, is the segment recognized by GroundingDINO, is the noun, adjective or quantifier not recognized, is the rest of the unrecognized words. The weighting of the attribute word depends on the visual target word: if the object modified by the attribute word is not recognized as a visual word, no weighting is applied; if an attribute word modifies multiple visual target words, it is bound to each visual target word separately and the weighting is applied independently. Embedding this objective function as a weighted estimation of log π(y|x) into the DPO loss calculation ensures that the multimodal large model can automatically strengthen the semantic core (such as entities and attributes) in the image during the generation process, while keeping the basic modeling of the language structure components, thereby maximizing the optimization ability of the model on the image-text consistency goal.

[0067] In step S4, although the "focus on the real target word" mechanism has been established in the training stage through the visual word weight and the attribute weighting , effectively guiding the multimodal large model to focus on the visible semantic content in the image. However, in multimodal tasks, simply optimizing the preferred answer is not enough to prevent the model from producing hallucinations. Especially in the reject sample , the error degree of the hallucination content is dynamic, some only exist slight deviation, some completely describe the object that does not exist in the image.

[0068] If the uniform intensity constraint is still applied to all reject answers, not only can it not effectively suppress answers containing a large number of error segments, but it can also lead to problems such as gradient instability or training insensitivity. Therefore, a dynamic adjustment mechanism based on the proportion of corrected segments is proposed, which flexibly adjusts the contrast intensity according to the proportion of error segments in each reject answer, thereby improving the sensitivity and suppression ability of the model to abnormal content.

[0069] In an embodiment of the present application, to realize the consistency of dynamic adjustment of reject answers and gradient update, a dynamic adjustment coefficient is introduced to scale the probability difference between the preferred answer and the reject answer. The dynamic adjustment coefficient is defined as:

[0070]

[0071] wherein, is the basic scaling coefficient, denotes the length of the corrected segment in the answer, denotes the total length of the answer. The length of the corrected segment is calculated in token granularity, that is, based on the tokenization result of the multi-modal large model, the difference area is mapped to the corresponding token number. In addition, the determination of the corrected segment adopts an automatic comparison combined mode, and the model is used for text alignment and differential analysis of the preferred and rejected answers, and manual checking is supplemented if necessary, so as to ensure the accuracy of the determination. During training, will be used in the DPO loss function: can be directly used in the loss calculation of each sample instead of , or take the average of all in a batch as of this iteration. Through this definition, the higher the proportion of the corrected segment contained in the answer, the larger the dynamic adjustment coefficient , so as to impose stronger punishment on the rejected answer in the training process, and realize the adaptive inhibition of hallucination content. This mechanism not only enhances the ability of the model to distinguish difficult examples, but also ensures the stability and convergence of the training process.

[0072] In step S5, through the cooperative design of the first four steps, the application has established a structured control mechanism in the aspects of training data construction, visual semantic recognition, language generation optimization and hallucination degree recognition, respectively from the input end, word-level modeling, loss construction and punishment adjustment. Four dimensions, comprehensively enhance the perception and response ability of the multi-modal large model to the consistency of text and image. In order to realize the unified deployment and gradient cooperative update of this series of mechanisms, a unified optimization framework integrating the above information is constructed at the training target level, and a visual guided DPO loss function is proposed as the convergence expression of the key technology. This optimization target not only retains the high efficiency of the DPO framework in preference comparison, but also combines the differentiated weight regulation of visual words and attribute words, realizes adaptive optimization under the condition of no manual annotation, makes the model more inclined to generate preferred answers consistent with the image semantics, and reduces the occurrence of inconsistent content.

[0073] DPO loss function of multi-modal large model:

[0074] ,

[0075] wherein, denotes the multi-modal model being trained, is a reference model, the parameters of which remain frozen and are not updated during the training process, and is used as a comparison benchmark to provide a stable reference distribution. In order to ensure the effectiveness of the loss function optimization, the reference model π_ref and the current model are based on the same training data. is a regularization coefficient for controlling the scaling amplitude of the probability ratio difference; Sigmoid function for normalizing the output and ensuring numerical stability. This optimization objective makes the model more inclined to choose preferred answers consistent with the image semantics through a preference comparison mechanism, reducing the risk of generating inconsistent or unreasonable content.

[0076] As shown in Figure 2 , 3 , 4, the visual guidance vocabulary preference optimization method proposed by the application effectively improves the consistency and authenticity of the multi-modal large language model in the text generation process. By introducing an image-guided word-level importance calculation mechanism, the correlation degree of different words and visual content can be distinguished during training, and differential optimization weights are dynamically assigned. In the training of the multi-modal large model, the generated data is not reprocessed before being input into the model, but is directly input into the training process in the form of image-text pairs; the model extracts image and word features through visual and language encoders respectively in the forward propagation stage, and calculates the correlation between words and visual semantics through an attention mechanism to obtain a word-level importance distribution; this distribution does not change the input data, but dynamically assigns different weights to different words in the loss function layer, so that the gradient of words highly related to the image is amplified and the gradient of weakly related words is weakened, thereby guiding the model to more accurately align the image and text semantics in parameter updating and reducing the problem of image-text inconsistency. The multi-modal large model pays more attention to words highly related to image semantics, thereby generating information descriptions that are more consistent with visual facts, significantly improving the problems of "image-text disconnection" and "semantic drift" in traditional methods.

[0077] At the same time, it has the advantages of strengthening the multi-modal large model hallucination recognition and suppression ability. By constructing a word-level comparison mechanism, further combining the differences between high importance words in positive and negative samples, and dynamically adjusting the weight of negative samples in the training loss, the multi-modal large model becomes more sensitive to inconsistent expressions and effectively distinguishes them during the learning process, thereby reducing typical hallucination phenomena such as "out of thin air" and attribute errors. In multiple standard evaluation tasks, this method can significantly reduce the proportion of hallucinations in the generated content of the model without introducing additional supervision, improving the overall text quality.

[0078] While maintaining the language understanding and generation capabilities of the multi-modal large model, the stability and reliability of the overall training are improved. Through the joint regulation of word weights and preference optimization processes, the multi-modal large model can maintain more balanced gradient updates when facing complex image-text combination data, avoiding performance fluctuations caused by unstable optimization strategies. At the same time, the introduction of dynamic weight mechanism effectively adapts to the difficulty differences between different samples, improving the training efficiency and consistency performance of the final model.

[0079] In another preferred embodiment of the present application, a computer readable storage medium is provided, which stores a computer program. The computer program, when executed by a processor, causes the processor to perform the steps of the method as described in the above embodiments.

[0080] The functions described above can be implemented in software or computer program code, and the software or computer program code can be stored in one or more computer readable storage media. In light of this understanding, the technical solutions of the present application or the parts of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0081] The above description is a detailed description of the preferred embodiments of the present application, but the embodiments are not intended to limit the scope of the patent application of the present application. Any equivalent changes or modifications made under the technical spirit of the present application should be included in the scope of the patent application of the present application.

Claims

1. A method for visual vocabulary guided multi-modal large model hallucination optimization, characterized in that, The method comprises the steps of: constructing a training sample comprising an image, a text prompt, a preferred answer and a rejected answer, wherein the preferred answer corresponds to the image, and the rejected answer contains an illusion component as a contrast reference; using a visual target detection model to perform image detection on the image according to the preferred answer, extracting a related index consistent with a visual word, and calculating a visual weight of each visual word according to the related index; differentially weighting different types of words according to the visual weight, and improving the attention ability of a multi-modal large model to word alignment between an image and a text during modeling; introducing a dynamic adjustment coefficient based on a proportion of corrected segments, and constructing a DPO loss function of the multi-modal large model together with a probability difference between the preferred answer and the rejected answer and the differential weighting, to guide the multi-modal large model to generate an image-text consistent answer; The multi-modal large model further introduces a dynamic adjustment coefficient when constructing a loss function , for scaling the probability difference between the preferred answer and the rejected answer, will be used in the DPO loss function, wherein the dynamic adjustment coefficient is defined as: wherein is the length of the segment to be corrected, specifically the token length of the segment corresponding to the rejected answer that is inconsistent with the preferred answer and needs to be corrected; is the total length of the answer, specifically the total number of tokens of the rejected answer or the preferred answer; is the base scaling factor; constructing a DPO loss function of the multi-modal large model: ， in, This indicates the multimodal model currently being trained. As a reference model, This is the regularization coefficient, specifically, it is directly used in the loss calculation for each sample. Alternative Or take all of them in a batch. The mean of the values ​​is used as the value for the current iteration. ; For the Sigmoid function, For the training sample set Expectation calculation, For text prompts, For the best answer, This is to refuse to answer.

2. The visual vocabulary guided multi-modal large model hallucination optimization method of claim 1, wherein, The visual target detection model performs image detection on the image according to a noun in the preferred answer, and when a target region corresponding to the noun is found, the noun is a visual word, and a visual weight of the visual word is calculated.

3. The visual vocabulary guided multi-modal large model hallucination optimization method of claim 2, wherein, The related index comprises: detection confidence : degree of confidence of the target detection model in the correspondence between the visual word and the entity in the image Target area ratio : represents the area ratio of the detection region in the whole image; centering deviation : measures the position of the target corresponding to a visual word in the image.

4. The visual vocabulary guided multi-modal large model hallucination optimization method of claim 3, wherein, The calculation of the visual weight of each visual word is specifically: wherein is a detection confidence, is a target area area ratio, is a center deviation.

5. The visual vocabulary guided multi-modal large model hallucination optimization method of claim 4, wherein, constructing a multi-modal large model modeling target function corresponding to the differential weighting: wherein is a visual weight of a visual word, is an overall weighting factor for attribute words, N is the total number of words in the sequence, used as a normalization factor, is a segment recognized by GroundingDINO, is a noun, adjective, or quantifier that is not recognized, is the remaining unrecognized words, is a text sequence generated by a multi-modal large model with respect to a multi-modal input is a log-likelihood value, is a conditional probability, is a sequence element, and is a text sequence is the word in the text sequence.

6. A storage medium, characterized by The computer program is stored, and the computer program is executed by a processor to enable the processor to perform the steps of the visual word guided multi-modal large model illusion optimization method according to any one of claims 1-5.

Images