Large visual language model decoding method and device, equipment and medium
By extracting attention scores from a large visual language model to intervene and generate hallucination prediction scores, and then comparing and decoding these scores with the original prediction scores, the problem of insufficient anti-hallucination ability of existing models is solved, achieving a more efficient and robust hallucination suppression effect.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-16
- Publication Date
- 2026-05-01
AI Technical Summary
Existing large-scale visual language models have low anti-hallucination capabilities in the fields of healthcare and fintech. Existing contrastive decoding methods are computationally expensive, have blind perturbation methods with severe noise, rely on complex post-processing constraints, and have limited generalization capabilities.
By extracting the attention scores of the self-attention layer of a large visual language model, a hallucination prediction score is generated and compared with the original prediction score for decoding, thereby suppressing the generation of hallucination content and improving anti-hallucination ability and computational efficiency.
It effectively suppresses hallucinatory content generated by the model that does not match the input, improves anti-hallucination ability and computational efficiency, reduces computational overhead, and has plug-and-play characteristics and robustness.
Smart Images
Figure CN121960587A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology and can be applied to the fields of fintech and healthcare. In particular, it relates to a method, apparatus, device, and medium for decoding large-scale visual language models. Background Technology
[0002] In recent years, Large Vision-Language Models (LVLMs) have demonstrated outstanding performance in multimodal tasks such as image understanding, visual question answering, and content generation. These models typically extract image features through a visual encoder, then map them to the embedding space of a language model via a projection module. Finally, a Large Vision-Language Model (LLM) based on the Transformer architecture performs cross-modal inference and text generation. Despite their remarkable performance, the "hallucination" problem—that is, the model generates text that does not match the input image content, is fabricated, or contains errors—remains a key bottleneck hindering the practical application of LVLMs in high-reliability scenarios such as healthcare (e.g., medical diagnosis), fintech (e.g., intelligent fraud prevention and security protection), autonomous driving, and intelligent assistants. To alleviate this problem, researchers have proposed a series of methods based on "contrastive decoding" (CD). Currently, the two most representative methods are: Visual Contrastive Decoding (VCD), which weakens the model's dependence on real visual information by applying a holistic perturbation to the original visual input, thereby amplifying its linguistic priors and inducing the generation of illusory text; and Instruction Contrastive Decoding (ICD), which uses artificially designed negative instructions to interfere with the model's inference logic, guiding it to produce erroneous or irrelevant outputs. However, existing contrastive decoding methods suffer from the following problems: 1. High computational cost: Both VCD and ICD require performing a complete forward inference process on both the original input and the perturbated input, doubling the model's inference latency and significantly increasing computational cost, making it difficult to meet the efficiency requirements of practical applications; 2. Blind perturbation methods and severe noise introduction: Existing methods typically employ a "visual-text agnostic" global perturbation strategy, lacking awareness of the current contextual semantics. This coarse-grained interference not only fails to accurately induce illusory concepts relevant to the generation context, but may also introduce irrelevant noise, interfering with the model's use of real visual cues and even incorrectly suppressing the generation probability of the true answer; 3. Heavy reliance on post-processing constraints: Without filtering "reasonableness constraints" on low-probability words, the performance of VCD and ICD will drop sharply, even worse than conventional decoding methods. This indicates that the quality of the constructed contrast distribution is low, mixed with a large number of irrelevant assumptions, and a strong post-processing mechanism is needed to compensate for this deficiency; 4. Complex design and limited generalization ability: Especially for the ICD method, its effect is highly dependent on manually designed negative instructions, requiring a lot of trial and error and optimization for different tasks and model structures, lacking universality and adaptability, which limits the promotion and use of the method. Summary of the Invention
[0003] This invention provides a method, apparatus, computer device, and medium for decoding large visual language models, in order to solve the technical problem of low anti-hallucination capability of existing large visual language models.
[0004] Firstly, a method for decoding large-scale visual language models is provided, including: Acquire an input image and input text, and process the input image and input text to obtain a visual lexical sequence and a text word embedding vector; The raw prediction score for predicting the next lexical is obtained by using a large visual language model based on the visual lexical sequence and the text word embedding vector. The attention score of the attention head is extracted from the self-attention layer of the large visual language model, and the large visual language model is intervened based on the attention score to generate an illusion prediction score; The illusion prediction score and the original prediction score are compared and decoded to generate the next word. Then, the process returns to the step of obtaining the original prediction score for predicting the next word through a large visual language model based on the visual word sequence and the text word embedding vector, until a complete output sequence is generated.
[0005] Secondly, a large-scale visual language model decoding device is provided, comprising: The processing unit is used to acquire an input image and input text, and process the input image and input text to obtain a visual word sequence and a text word embedding vector; The output unit is used to obtain the raw prediction score for predicting the next lexical unit by means of the visual lexical sequence and the text word embedding vector through a large visual language model. An intervention extraction unit is used to extract the attention score of the attention head from the self-attention layer of the large visual language model, and to intervene in the large visual language model based on the attention score to generate an illusion prediction score; The decoding generation unit is used to compare and decode the illusion prediction score and the original prediction score to generate the next word, and then return to the step of obtaining the original prediction score for predicting the next word through a large visual language model based on the visual word sequence and the text word embedding vector, until a complete output sequence is generated.
[0006] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described large visual language model decoding method.
[0007] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described large visual language model decoding method.
[0008] The aforementioned large-scale visual language model decoding method, apparatus, computer equipment, and storage medium can acquire input images and input text, process the input images and input text to obtain visual lexical sequences and text word embedding vectors; obtain the original prediction score for predicting the next lexical through the large-scale visual language model based on the visual lexical sequence and the text word embedding vectors; extract the attention score of the attention head from the self-attention layer of the large-scale visual language model, intervene in the large-scale visual language model based on the attention score to generate a hallucination prediction score; compare and decode the hallucination prediction score and the original prediction score to generate the next lexical, and return to execute the step of obtaining the original prediction score for predicting the next lexical through the large-scale visual language model based on the visual lexical sequence and the text word embedding vectors, until a complete output sequence is generated. In this invention, the hallucination prediction score is generated by intervening in the self-attention mechanism of the large-scale visual language model based on the attention score, and then compared and decoded with the original prediction score. This effectively suppresses the generation of hallucination content that does not match the input by the large-scale visual language model, which not only improves the anti-hallucination ability of the large-scale visual language model, but also improves the computational efficiency. Attached Figure Description
[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0010] Figure 1 This is a flowchart illustrating a method for decoding a large visual language model according to an embodiment of the present invention; Figure 2 yes Figure 1 A schematic diagram of a specific implementation of step S120; Figure 3 yes Figure 1 A schematic diagram of a specific implementation of step S130; Figure 4 yes Figure 1 A schematic diagram of a specific implementation of step S140; Figure 5 This is a schematic block diagram of a large visual language model decoding device according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 7 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation
[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0012] The large visual language model decoding method provided in this invention can be applied to either a client or a server. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. Currently, in the fields of fintech and healthcare, existing large visual language models have low anti-hallucination capabilities. To address the above problems, this invention proposes a large visual language model decoding method. This method generates a hallucination prediction score based on attention score intervention in the self-attention mechanism of the large visual language model, and compares it with the original prediction score for decoding. This effectively suppresses the generation of hallucination content by the large visual language model that does not match the input, thus improving not only the anti-hallucination capability of the large visual language model but also its computational efficiency. The invention will be described in detail below through specific embodiments.
[0013] Please see Figure 1 As shown, Figure 1 A flowchart of a large visual language model decoding method provided in an embodiment of the present invention includes the following steps: S110-S140.
[0014] S110. Obtain the input image and input text, and process the input image and input text to obtain a visual lexical sequence and a text word embedding vector.
[0015] Specifically, the process involves acquiring an input image and input text. These can be images and text from the fintech field, such as a screenshot of financial news and specific investment questions; or images and text from the health field, such as a patient or primary care physician uploading an eye photo and symptoms described in text form, like "recent blurred vision and floaters." The input image is processed by a visual encoder to generate visual features; these features are mapped to a visual word sequence using a projection module; the input text is processed by a word segmenter to generate a text word index sequence; and the text word index sequence is converted into a text word embedding vector using an embedding module. More specifically, in the multimodal information processing flow of a large-scale visual language model, feature extraction and representation alignment are first performed on the input image and input text. Specifically, for the input image, a visual encoder (such as a ViT model based on the Transformer architecture or a deep convolutional neural network) is used for hierarchical feature extraction to generate high-dimensional, dense visual features that preserve the spatial structure and semantic information of the image. Subsequently, the projection module receives the visual features as input and transforms them from the visual semantic space to the language model embedding space through a mapping network composed of fully connected layers or small multilayer perceptrons. The output is a sequence of visual lexical units consistent with the text word embedding dimension, thus achieving alignment between the visual and linguistic modalities. For the input text, preprocessing is first performed by a word segmenter: the original text string is cut into discrete sub-words or lexical units according to a preset segmentation strategy, and a unique integer index is assigned to each unit, generating a text lexical index sequence. Next, the embedding module receives this index sequence and, by querying a trainable high-dimensional embedding matrix, converts each discrete lexical index into a continuous, dense text word embedding vector. This process not only achieves a numerical representation of the text but also encodes semantic information into geometric relationships in the vector space, providing a foundation for subsequent steps.
[0016] S120. Based on the visual lexical sequence and the text word embedding vector, obtain the original prediction score output for predicting the next lexical through a large visual language model.
[0017] Specifically, after obtaining the visual lexical sequence and the text word embedding vector, as follows: Figure 2As shown, step S120 includes steps S121-S122: S121, concatenating the visual word sequence and the text word embedding vector to obtain a joint input sequence; S122, inputting the joint input sequence into the large visual language model to obtain an original prediction score output for predicting the next word, wherein the original prediction score is the unnormalized prediction score calculated by the large visual language model for all candidate words in the vocabulary; the vocabulary is a predefined set containing basic text units that the large visual language model can recognize; when generating each new word, each basic text unit in the vocabulary is a candidate word, and the large visual language model calculates a set of unnormalized original prediction scores for all candidate words to represent its original preference for each word as the next word. More specifically, in the multimodal information fusion stage, a sequence concatenation operation is first performed: the visual word sequence output by the projection module and the text word embedding vector generated by the embedding module are aligned and concatenated to form a joint input sequence. The concatenation process must maintain a strict sequence order—typically, visual lexical units are placed at the beginning of the sequence, followed by textual lexical units, to ensure that positional encoding and attention mechanisms can correctly distinguish information sources from different modalities. The concatenated joint sequence is fully dimensionally compatible, forming a complete contextual representation that integrates visual and textual information. Next, the joint input sequence is fed into a large-scale visual language model for forward computation. The large-scale visual language model uses its multi-layer self-attention mechanism and feedforward network to perform deep encoding and interaction modeling of the joint sequence, capturing fine-grained associations between visual elements and textual concepts. In the decoding phase, based on all the processed contextual information, the large-scale visual language model generates raw prediction scores (Raw Logits) for all candidate lexical units in the vocabulary through the top output layer. These scores are unnormalized real values with an unbounded range; positive and negative values represent preference and suppression of specific lexical units, respectively.
[0018] S130. Extract the attention score of the attention head from the self-attention layer of the large visual language model, and intervene in the large visual language model based on the attention score to generate an illusion prediction score.
[0019] Specifically, such as Figure 3As shown, step S130 includes the following steps: S131-S134: S131, read the multi-head attention weight tensor generated in one or more of the self-attention layers inside the large visual language model; S132, extract the attention score of the last position of all visual word pairs in the attention head from the multi-head attention weight tensor based on the preset visual word position index and the current decoding position; S133, construct a perturbation state based on the attention score to suppress the contribution weight of visual words in attention calculation; S134, integrate the perturbation state into the forward computation graph of the large visual language model, obtain the contrast prediction score output by the large visual language model in the visual context suppressed state through conditional forward propagation, and use the contrast prediction score as the hallucination prediction score. It's important to note that introspective importance assessment is a process where a large visual language model, based on the input visual word sequence and text word embedding vectors, predicts and outputs words one by one through autoregression to gradually form a complete text response. This process analyzes its internal attention mechanism to identify and quantify the contribution of different visual words to the current generation decision. By analyzing attention scores, it filters out and suppresses visual elements with lower importance, thereby reducing the possibility of the large visual language model generating illusions. In essence, the "introspection" mechanism dynamically assesses the importance of visual words by analyzing attention scores, masks the least important visual words, and performs lightweight forward propagation based on this, inducing the generation of high-quality illusion prediction scores. Illusion words are then suppressed based on these scores. The "introspection" mechanism reduces the additional computational overhead of contrastive decoding and significantly improves factual accuracy. During introspective importance assessment, the model's internal state monitoring is first performed: the multi-head attention weight tensors generated by one or more self-attention layers within the large visual language model are accessed and read in real-time in the forward computation graph. This tensor has a four-dimensional structure, corresponding to the batch size, the number of attention heads, the sequence query position, and the sequence key position, respectively, and fully records the correlation strength between all word pairs in different attention heads. This operation ensures that the internal attention allocation pattern of the model when processing multimodal information is accurately captured without interrupting the normal forward computation of the model. Subsequently, as shown in formula (1), based on the preset visual word position index set vision_indices and the current decoding position (sequence end index - 1), the attention score extraction operation is performed. Through tensor indexing technology, the attention scores of the last position of all visual word pairs in the specified attention head j are accurately located and extracted from the multi-head attention weight tensor. This score set quantifies the relative contribution of each visual word in the current decoding step, providing a data basis for subsequent intervention.
[0020] (1) In formula (1), v is the input image, and h is the number of attention heads. Let represent the attention weight of all visual words in the j-th attention head to the last word in the sequence (index -1), and let i represent the self-attention matrix Ai obtained after forward computation in the i-th layer.
[0021] Further, the step of constructing a perturbation state based on the attention score includes: ranking all visual words by importance according to the attention score; selecting the visual words with the lowest attention score at a preset proportion as an index set; and performing a masking operation on the hidden states based on the index set, retaining only the hidden states corresponding to the visual words, setting the hidden states of the remaining visual words to zero, and outputting the perturbation state with limited visual information. Specifically, to accurately generate the contrast prediction distribution representing hallucinations, the following structured operations need to be performed: First, based on the attention weights calculated by the large visual language model, the importance of each visual word to the generation of the current word is evaluated to obtain an attention score. The higher the attention score, the more critical the visual information is to the current decision; the lower the attention score, the weaker its correlation with the current language context. After ranking, a subset of visual words with the lowest attention score is selected according to a preset proportion threshold (such as the lowest k%), generating an index set of words to be retained. The index set corresponds to those visual elements judged as "least important" by the model. Subsequently, a masking operation is performed on the hidden state tensor based on this index set: only the hidden states of visual words corresponding to the index are retained, while the hidden states of all other visual words are set to zero. The hidden states of text words are completely preserved and unaffected, thus generating a perturbed state with limited visual information. This perturbed state refers to the intermediate representation formed in the decoder's intermediate layer after removing the hidden states of most important visual words through masking. This state has incomplete visual information, forcing large visual language models to rely primarily on flawed language priors for subsequent computations, thereby inducing hallucination distributions related to the current context. Finally, this perturbed state is integrated into the model's forward computation graph, replacing the original hidden states for conditional forward propagation. In this state, the model continues to reason based on the disrupted visual context, and its output relies more on language priors than visual evidence, thus generating a contrastive prediction score that may contain hallucinations. This score, as a hallucination prediction score, is compared and fused with the original prediction score to suppress the generation probability of hallucination words in the final decoding, improving the consistency between the output and the visual content. The entire process utilizes a perturbation-contrast mechanism to achieve precise intervention in the visual dependencies within the model, without requiring additional forward computation or external perturbation, significantly improving the efficiency and accuracy of anti-hallucination decoding.
[0022] S140. Compare and decode the illusion prediction score and the original prediction score to generate the next word, and return to the step of obtaining the original prediction score for predicting the next word through a large visual language model based on the visual word sequence and the text word embedding vector, until a complete output sequence is generated.
[0023] Specifically, after obtaining the hallucination score, as Figure 4 As shown, step S140 includes the following steps: S141-S143: S141, weighted and fused the hallucination prediction score and the original prediction score to obtain a corrected prediction score; S142, normalized the corrected prediction score to obtain the probability distribution of the next word; S143, generated the next word according to the probability distribution.
[0024] In the final stage of introspective contrast decoding, score fusion and lexical generation operations are performed. First, the hallucination prediction score logitscontrast and the original prediction score logitsoriginal are weighted and linearly fused, and the calculation formula is shown in (2): logitscorrected=logitsoriginal−β⋅logitscontrast(2) Here, β is an adjustable suppression coefficient, and logitscontrast is the corrected prediction score. This fusion operation corrects the original distribution by penalizing candidate words favored by the model when visual context is missing (i.e., illusion words). Subsequently, the corrected prediction score logitscorrected is Softmax normalized to convert it into a probability distribution. This probability distribution satisfies that the sum of the probabilities of all candidate words is 1, and the values are in the range [0,1], intuitively representing the generation confidence of each word as the next element in the sequence. Finally, based on this probability distribution, a preset strategy (such as greedy sampling) is used to generate the next word, completing the current decoding step. This process is repeated until a complete output sequence is generated, continuously suppressing illusions through an iterative contrast correction mechanism to ensure a high degree of consistency between the output text and the visual content.
[0025] The large-scale visual language model decoding method in this invention generates illusion prediction scores based on attention score intervention in the self-attention mechanism of the large-scale visual language model, and compares these scores with the original prediction scores for decoding. This effectively suppresses the generation of illusion content that does not match the input by the large-scale visual language model, improving not only the anti-illusion ability of the large-scale visual language model but also computational efficiency. Specifically, it demonstrates significant performance in effectively reducing illusions and enhancing factual accuracy: by accurately identifying and suppressing fine-grained illusion content related to context, it significantly reduces various types of illusions, including object misjudgment, attribute errors, and position confusion. It also achieves a substantial improvement in computational efficiency: by implementing lightweight operations in the decoder's intermediate layer and fully utilizing the computational results of previous stages, it reduces the additional computational overhead of comparative decoding, making it more suitable for deployment and application in real-world scenarios. Furthermore, it has advantages in maintaining the model's general capabilities and text quality: due to the precise and controllable perturbation mechanism, this invention effectively suppresses illusions while not compromising the model's original multimodal understanding and reasoning capabilities, and maintaining the fluency and information integrity of the generated text. Furthermore, this invention features plug-and-play functionality, requiring no retraining: as a decoding strategy independent of the training phase, it can be directly applied to various pre-trained large-scale visual language models, exhibiting high versatility and ease of use. Finally, the robustness of this invention is significantly enhanced: compared to existing contrastive decoding methods, this invention has lower sensitivity to hyperparameter changes, demonstrating stable performance under different settings, reducing the difficulty of hyperparameter tuning and the deployment threshold in practical applications.
[0026] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0027] The software tools or components not belonging to our company that appear in the embodiments of this application are merely examples and do not represent actual use.
[0028] In one embodiment, a large visual language model decoding device 200 is provided, which corresponds one-to-one with the large visual language model decoding method in the above embodiments. For example... Figure 5 As shown, the large-scale visual language model decoding device includes an acquisition and processing unit 201, an output unit 202, an extraction and intervention unit 203, and a decoding and generation unit 204. Detailed descriptions of each functional module are as follows: The acquisition and processing unit 201 is used to acquire an input image and input text, and process the input image and input text to obtain a visual word sequence and a text word embedding vector; Output unit 202 is used to obtain the original prediction score for predicting the next word by using a large visual language model based on the visual word sequence and the text word embedding vector; The intervention unit 203 is used to extract the attention score of the attention head from the self-attention layer of the large visual language model, and to intervene in the large visual language model based on the attention score to generate an illusion prediction score. The decoding generation unit 204 is used to compare and decode the illusion prediction score and the original prediction score to generate the next word, and then return to the step of obtaining the original prediction score for predicting the next word through a large visual language model based on the visual word sequence and the text word embedding vector, until a complete output sequence is generated.
[0029] In one embodiment, the acquisition processing unit 201 is specifically used for: The input image is processed by a visual encoder to generate visual features; The visual features are mapped into a sequence of visual lexical units using a projection module. The input text is processed by a word segmenter to generate a text word index sequence; The text lexical index sequence is converted into the text word embedding vector through the embedding module.
[0030] In one embodiment, the output unit 202 is specifically used for: The visual word sequence and the text word embedding vector are concatenated to obtain a joint input sequence; The joint input sequence is fed into the large visual language model to obtain the raw prediction score output for predicting the next lexical unit, wherein the raw prediction score is the unnormalized prediction score calculated by the large visual language model for all candidate lexical units in the vocabulary.
[0031] In one embodiment, the extraction intervention unit 203 is specifically used for: Read the multi-head attention weight tensors generated in one or more of the self-attention layers within the large visual language model; Based on the preset visual word position index and the current decoding position, the attention score of the last position of all visual word pairs in the sequence within the attention head is extracted from the multi-head attention weight tensor. Based on the attention score, a perturbation state is constructed to suppress the contribution weight of visual lexical units in attention calculation; The perturbation state is integrated into the forward computation graph of the large visual language model, and the contrast prediction score output by the large visual language model in the state of suppressed visual context is obtained through conditional forward propagation. The contrast prediction score is used as the hallucination prediction score.
[0032] In one embodiment, the extraction intervention unit 203 is further configured to: All visual lexical units are ranked by importance based on the attention scores. Select the visual words with the lowest preset proportion of attention scores as the index set; The hidden states are masked based on the index set, retaining only the hidden states corresponding to the visual words, while setting the hidden states of the remaining visual words to zero, and outputting the perturbation state with limited visual information.
[0033] In one embodiment, the decoding generation unit 204 is specifically used for: The corrected prediction score is obtained by weighted fusion of the hallucination prediction score and the original prediction score. The probability distribution of the next term is obtained by normalizing the corrected prediction score. The next term is generated based on the probability distribution.
[0034] The large visual language model decoding device of this invention generates a hallucination prediction score based on the attention score intervention of the self-attention mechanism of the large visual language model, and compares and decodes it with the original prediction score. This effectively suppresses the generation of hallucination content that does not match the input by the large visual language model, which not only improves the anti-hallucination ability of the large visual language model, but also improves the computational efficiency.
[0035] Specific limitations regarding the large-scale visual language model decoding device can be found in the limitations of the large-scale visual language model decoding method described above, and will not be repeated here. Each unit in the aforementioned large-scale visual language model decoding device can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.
[0036] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side method for decoding a large-scale visual language model.
[0037] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the client-side functions or steps of a large-scale visual language model decoding method.
[0038] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described large visual language model decoding method.
[0039] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described large visual language model decoding method.
[0040] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.
[0041] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0042] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0043] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A method for decoding large-scale visual language models, characterized in that, include: Acquire an input image and input text, and process the input image and input text to obtain a visual lexical sequence and a text word embedding vector; The raw prediction score for predicting the next lexical is obtained by using a large visual language model based on the visual lexical sequence and the text word embedding vector. The attention score of the attention head is extracted from the self-attention layer of the large visual language model, and the large visual language model is intervened based on the attention score to generate an illusion prediction score; The illusion prediction score and the original prediction score are compared and decoded to generate the next word. Then, the process returns to the step of obtaining the original prediction score for predicting the next word through a large visual language model based on the visual word sequence and the text word embedding vector, until a complete output sequence is generated.
2. The large-scale visual language model decoding method as described in claim 1, characterized in that, The step of processing the input image and the input text to obtain a visual lexical sequence and a text word embedding vector includes: The input image is processed by a visual encoder to generate visual features; The visual features are mapped into a sequence of visual lexical units using a projection module. The input text is processed by a word segmenter to generate a text word index sequence; The text lexical index sequence is converted into the text word embedding vector through the embedding module.
3. The large-scale visual language model decoding method as described in claim 1, characterized in that, The step of obtaining the original prediction score for predicting the next lexical unit from the visual lexical sequence and the text word embedding vector through a large visual language model includes: The visual word sequence and the text word embedding vector are concatenated to obtain a joint input sequence; The joint input sequence is fed into the large visual language model to obtain the raw prediction score output for predicting the next lexical unit, wherein the raw prediction score is the unnormalized prediction score calculated by the large visual language model for all candidate lexical units in the vocabulary.
4. The large-scale visual language model decoding method as described in claim 1, characterized in that, The step of extracting the attention score of the attention head from the self-attention layer of the large visual language model includes: Read the multi-head attention weight tensors generated in one or more of the self-attention layers within the large visual language model; Based on the preset visual word position index and the current decoding position, the attention score of the last position of all visual word pairs in the sequence within the attention head is extracted from the multi-head attention weight tensor.
5. The large-scale visual language model decoding method as described in claim 1, characterized in that, The step of intervening in the large visual language model based on the attention score to generate an illusion prediction score includes: Based on the attention score, a perturbation state is constructed to suppress the contribution weight of visual lexical units in attention calculation; The perturbation state is integrated into the forward computation graph of the large visual language model, and the contrast prediction score output by the large visual language model in the state of suppressed visual context is obtained through conditional forward propagation. The contrast prediction score is used as the hallucination prediction score.
6. The large-scale visual language model decoding method as described in claim 5, characterized in that, The step of constructing the perturbation state based on the attention score includes: All visual lexical units are ranked by importance based on the attention scores. Select the visual words with the lowest preset proportion of attention scores as the index set; The hidden states are masked based on the index set, retaining only the hidden states corresponding to the visual words, while setting the hidden states of the remaining visual words to zero, and outputting the perturbation state with limited visual information.
7. The large-scale visual language model decoding method as described in claim 1, characterized in that, The step of comparing and decoding the hallucination prediction score and the original prediction score to generate the next word includes: The corrected prediction score is obtained by weighted fusion of the hallucination prediction score and the original prediction score. The probability distribution of the next term is obtained by normalizing the corrected prediction score. The next term is generated based on the probability distribution.
8. A large-scale visual language model decoding device, characterized in that, include: The processing unit is used to acquire an input image and input text, and process the input image and input text to obtain a visual word sequence and a text word embedding vector; The output unit is used to obtain the raw prediction score for predicting the next lexical unit by means of the visual lexical sequence and the text word embedding vector through a large visual language model. An intervention extraction unit is used to extract the attention score of the attention head from the self-attention layer of the large visual language model, and to intervene in the large visual language model based on the attention score to generate an illusion prediction score; The decoding generation unit is used to compare and decode the illusion prediction score and the original prediction score to generate the next word, and then return to the step of obtaining the original prediction score for predicting the next word through a large visual language model based on the visual word sequence and the text word embedding vector, until a complete output sequence is generated.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the large visual language model decoding method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the large visual language model decoding method as described in any one of claims 1 to 7.