Real-time hallucination detection and processing method and apparatus for multi-modal large language models

Abstract

The present disclosure relates to a method and device for real-time hallucination detection and processing of a multi-modal large language model. The method comprises: using a multi-modal large language model and a preset decoding strategy to perform text generation on visual information to be processed, to obtain a plurality of current candidate word token sequences corresponding to a current time step; based on a preset sliding window, a plurality of word token sequence windows are obtained by respectively intercepting the plurality of current candidate word token sequences; according to the visual attention weight sequence corresponding to each word token sequence window extracted from the multi-modal large language model, a hallucination detection result corresponding to each word token sequence window is determined; and a current candidate word token sequence corresponding to a word token sequence window representing the lowest hallucination probability in the hallucination detection result is selected as the current generated text corresponding to the current time step. The technical solution of the present disclosure can realize real-time hallucination detection and mitigation during the text generation process of the multi-modal large language model, avoid the accumulation of hallucination content, and thus improve the accuracy of text generation.

Description

Technical Field

[0001] This disclosure relates to the field of natural language processing technology, and in particular to a method and apparatus for real-time hallucination detection and processing of a multimodal large language model. Background Technology

[0002] With the rapid development of MLLMs (Multimodal Large Language Models), they have been widely applied in various fields such as autonomous driving, robotics, and healthcare, enabling cross-modal interaction between images and text and completing complex tasks such as image description and visual question answering. However, MLLMs commonly suffer from the illusion problem, where the generated text content does not match the corresponding visual information. This not only reduces the reliability of the model output but also poses potential risks in critical areas such as healthcare and autonomous driving.

[0003] In related technologies, visual uncertainty is introduced through image distortion operations, and contrastive decoding is used to weaken visual information and amplify the "linguistic inertia" of MLLMs, thereby alleviating the hallucination problem caused by language priors. Alternatively, multi-head attention integration can be used to optimize the visual attention of MLLMs, thus alleviating the hallucination problem. However, the former can only alleviate part of the hallucination caused by language priors, and the effect is limited; the latter depends on the complexity of multi-head attention integration, and the effect is not good. Furthermore, none of the above-mentioned hallucination mitigation methods can achieve real-time hallucination detection during the text generation process of MLLMs, resulting in the inability to detect and alleviate hallucinations in real time when they occur. They also cannot be adapted to multimodal large language models with different architectures, resulting in poor adaptability and poor performance in hallucination detection and processing. Summary of the Invention

[0004] This disclosure provides a method and apparatus for real-time hallucination detection and processing using a multimodal large language model, to at least solve the problem in related technologies of how to achieve real-time hallucination detection and mitigation during the text generation process of a multimodal large language model. The technical solution of this disclosure is as follows.

[0005] According to a first aspect of the present disclosure, a real-time hallucination detection and processing method for a multimodal large language model is provided, comprising: Based on the visual information to be processed, the prompt text, and at least one historical candidate word sequence corresponding to the previous time step, the multimodal large language model and the preset decoding strategy are used to generate text, and multiple current candidate word sequences corresponding to the current time step are obtained. Multiple candidate word sequences are extracted based on a preset sliding window to obtain multiple word sequence windows; each word sequence window consists of multiple candidate words corresponding to multiple time steps arranged in order, and the multiple time steps include the current time step; From the multimodal large language model, extract the visual attention weight sequence corresponding to each word sequence window; the visual attention weight sequence is composed of the visual attention weight matrix corresponding to each of the multiple current candidate words arranged in order in the corresponding word sequence window; Based on the visual attention weight sequence corresponding to each word sequence window, determine the hallucination detection result corresponding to each word sequence window; The current candidate word sequence corresponding to the word sequence window that represents the lowest probability of hallucination in the hallucination detection results is selected as the current generated text corresponding to the current time step.

[0006] In one possible implementation, the step of extracting the multiple current candidate word sequences based on a preset sliding window to obtain multiple word sequence windows includes: Under the condition of sliding truncation configured by the preset sliding window, the plurality of current candidate word sequences are truncated based on the preset sliding window to obtain the plurality of word sequence windows; The sliding capture conditions corresponding to the preset sliding window include: the current time step reaches the number of time steps configured for the preset sliding window; or, the current time step is greater than the number of time steps configured for the preset sliding window and satisfies the sliding step length corresponding to the preset sliding window.

[0007] In one possible implementation, the method further includes: If the sliding truncation conditions of the preset sliding window configuration are not met, the multiple current candidate word sequences are used as the at least one historical candidate word sequence, and the text generation step is returned to continue the text generation process.

[0008] In one possible implementation, extracting the visual attention weight sequence corresponding to each word sequence window from the multimodal large language model includes: Identify multiple target network layers in the multimodal large language model; From the multiple target network layers in the multimodal large language model and all attention heads of each target network layer, extract multiple visual attention weights corresponding to each current candidate word in each word sequence window, and each visual attention weight corresponds to an attention head in a network layer; Construct a visual attention weight matrix for each current candidate word element by using multiple visual attention weights corresponding to each current candidate word element. The visual attention weight matrix corresponding to each current candidate word in each word sequence window is arranged in order according to the time step corresponding to each current candidate word, thus obtaining the visual attention weight sequence corresponding to each word sequence window.

[0009] In one possible implementation, determining the multiple target network layers in the multimodal large language model includes: All network layers in the multimodal large language model are identified as the multiple target network layers; Alternatively, the multiple network layers located in the middle of the multimodal large language model can be identified as the multiple target network layers.

[0010] In one possible implementation, determining the hallucination detection result corresponding to each word sequence window based on the visual attention weight sequence corresponding to each word sequence window includes: The visual attention weight sequence is input into the real-time hallucination detection model to obtain the hallucination detection result corresponding to each word sequence window.

[0011] In one possible implementation, the method further includes: Acquire visual information and sample prompt text from multiple samples; The visual information of each sample and the sample prompt text are input into the multimodal large language model to generate text, thereby obtaining the sample generated text corresponding to the visual information of each sample; The generated text of the sample is evaluated for textual illusion, and the sample word units in the generated text that contain illusions are located. Based on the preset sliding window, the sample generated text is extracted to obtain multiple sample text windows; Based on whether each sample text window contains the sample word character indicating hallucination, a hallucination evaluation label is assigned to each sample text window; sample text windows containing the sample word character indicating hallucination are labeled as hallucination evaluation labels, and sample text windows not containing the sample word character indicating hallucination are labeled as non-hallucination evaluation labels. From the multimodal large language model, extract the sample visual attention weight sequence corresponding to each sample text window; each sample visual attention weight sequence includes the sample visual attention weight matrix corresponding to each sample word in the sample text window; Based on the illusion evaluation labels annotated in each sample text window, the visual attention weight sequence corresponding to each sample text window is labeled with illusion evaluation labels, resulting in multiple training sample pairs of sample visual attention weight sequences and illusion evaluation labels; The binary classification network model is trained based on the multiple training sample pairs until the training iteration ends. The binary classification network model that meets the training iteration end condition is used as the real-time hallucination detection model.

[0012] In one possible implementation, the step of performing text illusion evaluation on the sample-generated text and locating sample words in the sample-generated text that exhibit illusion includes: The generated text of the sample is evaluated for textual hallucination based on the hallucination assessment tool, and the sample word units in the generated text that contain hallucinations are located.

[0013] In one possible implementation, the binary classification network model includes a multilayer perceptron, a convolutional neural network, or a Transformer encoder.

[0014] In one possible implementation, the visual attention weight sequence corresponding to each sample text window is extracted from the multimodal large language model, including: Identify multiple target network layers in the multimodal large language model; From the multiple target network layers in the multimodal large language model and all attention heads of each target network layer, extract multiple sample visual attention weights corresponding to each sample word in each sample text window, and each sample visual attention weight corresponds to an attention head in a network layer; Construct a sample visual attention weight matrix for each sample word element by using multiple sample visual attention weights corresponding to multiple sample visual attention weights. The visual attention weight matrix corresponding to each sample word in each sample text window is arranged in order according to the sample time step corresponding to each sample word, thus obtaining the visual attention weight sequence corresponding to each sample text window.

[0015] In one possible implementation, the preset decoding strategy includes: a beam search decoding strategy, or a decoding strategy adopted by the core.

[0016] In one possible implementation, the method further includes: If the text generation termination condition is met at the current time step, the currently generated text will be used as the target generated text. If the text generation termination condition is not met at the current time step, the current candidate word sequence corresponding to the currently generated text is updated to the at least one historical candidate word sequence, the text generation step is returned, and the text generation continues until the text generation termination condition is met. The generated text corresponding to the time when the text generation termination condition is met is taken as the target generated text.

[0017] According to a second aspect of the present disclosure, a real-time hallucination detection and processing apparatus for a multimodal large language model is provided, comprising: The text generation module is configured to perform text generation based on the visual information to be processed, the prompt text, and at least one historical candidate word sequence corresponding to the previous time step, using the multimodal large language model and a preset decoding strategy to obtain multiple current candidate word sequences corresponding to the current time step. The current candidate word sequence acquisition module is configured to extract the multiple current candidate word sequences based on a preset sliding window to obtain multiple word sequence windows; each word sequence window consists of multiple current candidate words corresponding to multiple time steps arranged in order, and the multiple time steps include the current time step; The visual attention weight sequence extraction module is configured to extract the visual attention weight sequence corresponding to each word sequence window from the multimodal large language model; the visual attention weight sequence consists of the visual attention weight matrices corresponding to multiple current candidate words arranged in order in the corresponding word sequence window; The hallucination detection result determination module is configured to determine the hallucination detection result corresponding to each word sequence window based on the visual attention weight sequence corresponding to each word sequence window; The hallucination processing module is configured to perform a process of selecting the current candidate word sequence corresponding to the word sequence window that represents the lowest hallucination probability in the hallucination detection results, and using it as the current generated text corresponding to the current time step.

[0018] In one possible implementation, the current candidate lexical sequence acquisition module includes: The current candidate word sequence acquisition unit is configured to perform sliding truncation based on the preset sliding window to extract the multiple current candidate word sequences respectively, under the condition that the preset sliding window configuration is satisfied, to obtain the multiple word sequence windows; The sliding capture conditions corresponding to the preset sliding window include: the current time step reaches the number of time steps configured for the preset sliding window; or, the current time step is greater than the number of time steps configured for the preset sliding window and satisfies the sliding step length corresponding to the preset sliding window.

[0019] In one possible implementation, the device further includes: The text generation iteration module is configured to perform the following steps when, if the sliding truncation conditions of the preset sliding window configuration are not met, to treat the multiple current candidate word sequences as the at least one historical candidate word sequence and return to the text generation step, and continue the text generation process.

[0020] In one possible implementation, the visual attention weight sequence extraction module includes: The target network layer determination unit is configured to determine multiple target network layers in the multimodal large language model; The visual attention weight extraction unit is configured to extract multiple visual attention weights corresponding to each current candidate word in each word sequence window from the multiple target network layers and all attention heads of each target network layer in the multimodal large language model. Each visual attention weight corresponds to an attention head in a network layer. The visual attention weight matrix construction unit is configured to construct a visual attention weight matrix for each current candidate word element by constructing multiple visual attention weights corresponding to each current candidate word element. The visual attention weight sequence acquisition unit is configured to perform the following: arranging the visual attention weight matrix corresponding to each current candidate word in each word sequence window in order according to the time step corresponding to each current candidate word, thereby obtaining the visual attention weight sequence corresponding to each word sequence window.

[0021] In one possible implementation, the target network layer determination unit includes: The first target network layer determination subunit is configured to perform the task of determining all network layers in the multimodal large language model as the multiple target network layers; Alternatively, the second target network layer determining subunit is configured to perform the determination of multiple network layers located in the middle of the multimodal large language model as the multiple target network layers.

[0022] In one possible implementation, the hallucination detection result determination module includes: The hallucination detection result determination unit is configured to input the visual attention weight sequence into the real-time hallucination detection model to obtain the hallucination detection result corresponding to each word sequence window.

[0023] In one possible implementation, the device further includes: The sample data acquisition module is configured to acquire visual information of multiple samples and sample prompt text; The sample text acquisition module is configured to input the visual information of each sample and the sample prompt text into the multimodal large language model to generate text, thereby obtaining the sample generated text corresponding to the visual information of each sample. The hallucination sample word localization module is configured to perform a text hallucination evaluation on the sample generated text and locate the sample words in the sample generated text that contain hallucinations. The sample text window acquisition module is configured to extract the sample generated text based on the preset sliding window to obtain multiple sample text windows. The first hallucination assessment labeling module is configured to perform a hallucination assessment labeling for each sample text window based on whether each sample text window contains the sample word character indicating the presence of hallucination; sample text windows containing the sample word character indicating the presence of hallucination are labeled as hallucination assessment labels, and sample text windows not containing the sample word character indicating the presence of hallucination are labeled as non-hallucination assessment labels. The sample visual attention weight sequence extraction module is configured to extract the sample visual attention weight sequence corresponding to each sample text window from the multimodal large language model; each sample visual attention weight sequence includes the sample visual attention weight matrix corresponding to each sample word in the sample text window; The second illusion evaluation label annotation module is configured to perform illusion evaluation label annotation based on each sample text window, and to annotate the visual attention weight sequence of the sample corresponding to each sample text window with illusion evaluation labels, thereby obtaining multiple training sample pairs of sample visual attention weight sequences and illusion evaluation labels. The training iteration module is configured to train a binary classification network model based on the multiple training sample pairs until the training iteration termination condition is met, and the binary classification network model corresponding to the training iteration termination condition is used as the real-time hallucination detection model.

[0024] In one possible implementation, the hallucination sample lexical localization module includes: The hallucination sample word localization unit is configured to perform a text hallucination assessment on the sample-generated text based on a hallucination assessment tool, and to locate sample words in the sample-generated text that contain hallucinations.

[0025] In one possible implementation, the binary classification network model includes a multilayer perceptron, a convolutional neural network, or a Transformer encoder.

[0026] In one possible implementation, the sample visual attention weight sequence extraction module includes: The target network layer determination unit is configured to determine multiple target network layers in the multimodal large language model; The sample visual attention weight extraction unit is configured to extract multiple sample visual attention weights corresponding to each sample word in each sample text window from the multiple target network layers and all attention heads of each target network layer in the multimodal large language model. Each sample visual attention weight corresponds to an attention head in a network layer. The sample visual attention weight matrix construction unit is configured to construct a sample visual attention weight matrix corresponding to each sample word by performing multiple sample visual attention weights corresponding to each sample word. The sample visual attention weight sequence acquisition unit is configured to perform the following: arranging the sample visual attention weight matrix corresponding to each sample word in each sample text window in order according to the sample time step corresponding to each sample word, thereby obtaining the sample visual attention weight sequence corresponding to each sample text window.

[0027] In one possible implementation, the preset decoding strategy includes: a beam search decoding strategy, or a decoding strategy adopted by the core.

[0028] In one possible implementation, the device further includes: The first target text generation determination module is configured to use the currently generated text as the target text if the text generation termination condition is met at the current time step. The second target text generation determination module is configured to, if the text generation termination condition is not met at the current time step, update the current candidate word sequence corresponding to the currently generated text to the at least one historical candidate word sequence, return to the text generation step, and continue the text generation until the text generation termination condition is met, and take the generated text corresponding to the text generation termination condition as the target generated text.

[0029] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method as described in any one of the first aspects above.

[0030] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided such that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform any of the methods described in the first aspect of the present disclosure. According to a fifth aspect of the present disclosure, a computer program product is provided, including computer instructions that, when executed by a processor, cause a computer to perform the method described in any one of the first aspects of the present disclosure.

[0031] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects: Multiple candidate word sequence windows are obtained by extracting multiple candidate word sequences corresponding to the current time step through a preset sliding window. Each word sequence window consists of multiple candidate words corresponding to the current time step arranged in order. The multiple time steps include the current time step. The visual attention weight sequence corresponding to each word sequence window is extracted from the multimodal large language model. The visual attention weight sequence consists of the visual attention weight matrix corresponding to the multiple candidate words arranged in order in the corresponding word sequence window. The hallucination detection result corresponding to each word sequence window is determined according to the visual attention weight sequence corresponding to each word sequence window. The current candidate word sequence corresponding to the word sequence window with the lowest hallucination probability represented by the hallucination detection result is selected as the current generated text corresponding to the current time step. By utilizing the visual attention weight sequence corresponding to each lexical sequence window at the current time step—that is, the visual attention weight distribution—hallucination detection is performed. This enables real-time hallucination detection and processing at the current time step, selecting the candidate lexical sequence with the lowest hallucination probability at the current time step as the basis for subsequent text generation. This effectively avoids the transmission and accumulation of hallucination content, achieving both real-time hallucination detection and real-time hallucination mitigation during MLLMs text generation, preventing the transmission and accumulation of hallucination lexical units, and effectively improving the accuracy of MLLMs text generation. Furthermore, the real-time hallucination detection method using visual attention weight distribution is adaptable to hallucination detection and processing in multimodal large language models with different architectures, resulting in good performance and adaptability in hallucination detection and processing.

[0032] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0034] Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment.

[0035] Figure 2 This is a flowchart illustrating a real-time hallucination detection and processing method for a multimodal large language model according to an exemplary embodiment.

[0036] Figure 3 This is a schematic diagram illustrating a time step, a historical candidate word sequence, a current candidate word sequence, and a preset sliding window according to an exemplary embodiment.

[0037] Figure 4This is a schematic flowchart illustrating a method for real-time hallucination detection and processing using a multimodal large language model, according to an exemplary embodiment.

[0038] Figure 5 This is a schematic diagram of a process architecture for hallucination detection based on MLP, according to an exemplary embodiment.

[0039] Figure 6 This is a flowchart illustrating the training process of a real-time hallucination detection model according to an exemplary embodiment.

[0040] Figure 7 This is a schematic diagram illustrating a text illusion assessment according to an exemplary embodiment, which generates text by extracting samples based on a preset sliding window, resulting in multiple sample text windows, and multiple training sample pairs of sample visual attention weight sequences and illusion assessment labels.

[0041] Figure 8 This is a block diagram of a real-time illusion detection and processing apparatus for a multimodal large language model, according to an exemplary embodiment.

[0042] Figure 9 This is a block diagram illustrating an electronic device for real-time illusion detection and processing of a multimodal large language model, according to an exemplary embodiment.

[0043] Figure 10 This is a block diagram illustrating an electronic device for real-time illusion detection and processing of a multimodal large language model, based on an exemplary embodiment. Detailed Implementation

[0044] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0045] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0046] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, in the specific embodiments of this disclosure, when user-related data is involved, user permission or consent is required when the following embodiments of this disclosure are applied to specific products or technologies, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0047] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. AI software technology mainly includes computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0048] In recent years, with the research and progress of artificial intelligence technology, artificial intelligence technology has been widely used in many fields. The solutions provided in the embodiments of this application involve technologies such as computer vision technology and natural language processing, which are specifically illustrated through the following embodiments.

[0049] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment, such as... Figure 1 As shown, the application environment may include server 01 and terminal 02.

[0050] In an optional embodiment, server 01 can be used for real-time illusion detection and processing of multimodal large language models. Specifically, server 01 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0051] In an optional embodiment, terminal 02 can provide any user with services such as triggering text generation tasks based on a multimodal large language model for visual information and displaying the generated text. Specifically, terminal 02 can be, but is not limited to, electronic devices such as smartphones, desktop computers, tablets, laptops, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, and smart wearable devices. Optionally, the operating system running on the electronic device can be, but is not limited to, Android, iOS, Linux, and Windows.

[0052] In one specific embodiment, the text generation task of visual information can refer to the task of generating text based on visual information, that is, the scenario in which MLLMs are required to generate text consistent with the visual information. For example, it may include, but is not limited to, image description tasks (e.g., tasks to generate descriptive text for images), visual question answering tasks, or text generation tasks in autonomous driving scenarios, etc. This disclosure does not limit it.

[0053] In addition, it should be noted that, Figure 1 The example shown is merely one application environment of the real-time hallucination detection and processing method based on the multimodal large language model provided in this disclosure.

[0054] In the embodiments described in this specification, the server 01 and the terminal 02 can be directly or indirectly connected through wired or wireless communication, and this application does not impose any restrictions on this.

[0055] Figure 2 This is a flowchart illustrating a real-time hallucination detection and processing method based on a multimodal large language model, according to an exemplary embodiment. Figure 2 As shown, the method may include the following steps.

[0056] In step S201, based on the visual information to be processed, the prompt text, and at least one historical candidate word sequence corresponding to the previous time step, a multimodal large language model and a preset decoding strategy are used to generate text to obtain multiple current candidate word sequences corresponding to the current time step. In one specific embodiment, the visual information to be processed may refer to the visual information of the object to be processed. The object to be processed may be matched with the text generation task of the visual information, such as an image description task, where the object to be processed may refer to the image to be described.

[0057] In one example, features can be extracted from the object to be processed (e.g., the image to be described) based on a visual feature extraction model (e.g., a visual encoder) to obtain visual information of the object to be processed, such as visual information of the image to be described, like image feature vectors. This disclosure does not limit the visual feature extraction model.

[0058] In one specific embodiment, the prompt text may be matched with a text generation task for visual information, such as an image description task, and the prompt text may be "Help me describe this image," but this disclosure does not limit it in this way.

[0059] In one specific embodiment, when performing the task of generating text from visual information, the multimodal large language model generates text token by token, that is, one token is generated at each time step of the iteration. In this way, as the number of time steps increases, the final token sequence is gradually generated, that is, the text is generated, such as image description text.

[0060] For example, in the autoregressive generation mode of the multimodal large language model, one time step can correspond to the generation process of one word, that is, one iteration process. For instance, at time step t, the multimodal large language model can predict the probability distribution of each word in the vocabulary at time step t based on the visual information to be processed, the prompt text, and at least one historical candidate word sequence retained by the preset decoding strategy from the previous time step (t-1 time step). Thus, based on the preset decoding strategy, the model can perform cumulative probability calculation and filtering based on the probability distribution to obtain multiple current candidate word sequences corresponding to multiple current time steps.

[0061] In one possible implementation, the aforementioned preset decoding strategy may refer to a decoding strategy with multiple candidate lexical sequences. For example, the preset decoding strategy may include a beam search decoding strategy or a kernel-based decoding strategy. This disclosure does not limit this. By setting a decoding strategy with multiple subsequent lexical sequences, a foundation is provided for real-time illusion detection in multimodal large language models.

[0062] Reference Figure 3 In a specific embodiment, a preset decoding strategy is used as the beam search decoding strategy, with a beamwidth of n (n word sequences are retained at each time step). Taking n=2 as an example, assume the current time step is the 8th time step: t8, and correspondingly, the previous time step is the 7th time step: t7. Assume the historical candidate word sequence corresponding to t7 is as follows: Historical candidate word sequence 1: [X1-1, X1-2, X1-3, X1-4, X1-5, X1-6, X1-7]; cumulative score 0.6; Historical candidate word sequence 2: [X2-1, X2-2, X2-3, X2-4, X2-5, X2-6, X2-7]; cumulative score 0.3.

[0063] Among them, X1-1, X1-2, X1-3, X1-4, X1-5, X1-6, and X1-7 in the historical candidate lexical sequence 1 can be used to represent 7 lexical units. The other historical candidate lexical sequences and the current candidate lexical sequences are the same.

[0064] During the t8 process, the probability of the next word can be predicted based on the two sequences (i.e., the historical candidate word sequences) retained in t7.

[0065] The probabilities of predicting the next word from the historical candidate word sequence 1, from high to low, are: X1-8 has a probability of 0.8, "cat" has a probability of 0.2, ...; A new candidate sequence is obtained from the historical candidate lexical sequence 1: [X1-1, X1-2, X1-3, X1-4, X1-5, X1-6, X1-7, X1-8]; Cumulative score: 0.6 * 0.8 = 0.48; [X1-1, X1-2, X1-3, X1-4, X1-5, X1-6, X1-7, "Cat"]; Cumulative score: 0.6 * 0.2 = 0.12; The probabilities of predicting the next word from the historical candidate word sequence 2, from highest to lowest, are: X2-8 has a probability of 0.7, "sofa" has a probability of 0.3, ... A new candidate sequence is obtained from historical candidate lexical sequence 2: [X2-1, X2-2, X2-3, X2-4, X2-5, X2-6, X2-7, X2-8]; Cumulative score: 0.3 * 0.7 = 0.21; [X2-1, X2-2, X2-3, X2-4, X2-5, X2-6, X2-7, "Sofa"]; Cumulative score: 0.3 * 0.3 = 0.09; After filtering the above 4 new candidate sequences, the 2 candidate sequences with the highest scores are retained as the 2 current candidate word sequences corresponding to the current time step t8, as follows.

[0066] [X1-1, X1-2, X1-3, X1-4, X1-5, X1-6, X1-7, X1-8]; [X2-1, X2-2, X2-3, X2-4, X2-5, X2-6, X2-7, X2-8].

[0067] It should be noted that without illusion detection, the two current candidate lexical sequences at the current time step t8 will serve as the two historical candidate lexical sequences for the next time step t9, with t8 being the previous time step relative to t9. With illusion detection, as described below, the current time step t8 will retain one current candidate lexical sequence based on the illusion detection result as the initial text for the next time step t9 (i.e., one historical candidate lexical sequence). Therefore, the number of at least one historical candidate lexical sequence corresponding to the previous time step can be one or n, where n can be the beamwidth of the beam search.

[0068] In step S203, multiple candidate word sequences are extracted based on a preset sliding window to obtain multiple word sequence windows.

[0069] In one specific embodiment, each lexical sequence window consists of multiple time steps arranged in order, each corresponding to a current candidate lexical. These multiple time steps include the current time step.

[0070] In one specific embodiment, the preset sliding window can be a sliding window with a pre-set window length and sliding step size. The preset sliding window can slide along the time step direction.

[0071] In one example, the preset sliding window length can be N, meaning it can include N time steps, and the sliding step size can be less than N but greater than 1. For example, the sliding step size can be half of N, i.e., N / 2.

[0072] Reference Figure 3 In one example, the window length N of the preset sliding window is 8 and the sliding step is 4, but this disclosure does not limit this.

[0073] Understandably, the preset sliding window has a window length of N. Based on this, starting from the first time step, it is necessary to accumulate N time steps before the first truncation of the preset sliding window can be performed. After this first truncation, the window can slide according to the sliding step size. For example, if the window length N=8, the first truncation is for time steps t1 to t8. Afterwards, if the sliding step size is 4, the truncation is for time steps t5 to t12.

[0074] In one possible implementation, the above-mentioned method of extracting multiple current candidate word sequences based on a preset sliding window to obtain multiple word sequence windows may include: extracting the multiple current candidate word sequences based on a preset sliding window to obtain multiple word sequence windows when the sliding extraction conditions configured by the preset sliding window are met.

[0075] The aforementioned conditions for capturing the sliding window corresponding to the preset sliding window may include: the current time step reaching the number of time steps configured for the preset sliding window, such as the case of the first capture mentioned above; or, the current time step being greater than the number of time steps configured for the preset sliding window and satisfying the sliding step length corresponding to the preset sliding window, such as the case of capturing the sliding window according to the sliding step length after the first capture mentioned above.

[0076] By setting a preset sliding window, real-time hallucination detection of multimodal large language models can be achieved. Based on the window length of the preset sliding window, multiple hallucination detection time periods are divided, which can effectively avoid the accumulation of hallucination content during the text generation task of the multimodal large language model, realize multi-stage hallucination relief, and improve the efficiency of text generation task.

[0077] Reference Figure 4 In an optional implementation, the method may further include: if the sliding truncation conditions of the preset sliding window configuration are not met, the plurality of current candidate lexical sequences may be used as the at least one historical candidate lexical sequence, and the process may return to the text generation step to continue text generation. By selecting to continue the iterative text generation process when the sliding truncation conditions of the preset sliding window configuration are not met, a balance can be achieved between illusion detection and text generation efficiency.

[0078] In step S205, the visual attention weight sequence corresponding to each word sequence window is extracted from the multimodal large language model.

[0079] In a specific embodiment, the above-mentioned visual attention weight sequence can be composed of visual attention weight matrices corresponding to multiple current candidate words arranged in order in the corresponding word sequence window.

[0080] In one possible implementation, the extraction of the visual attention weight sequence corresponding to each word sequence window from the multimodal large language model can include: Identify multiple target network layers in a multimodal large language model. For example, the multiple target network layers can be all network layers in the multimodal large language model or multiple intermediate network layers in the multimodal large language model.

[0081] Accordingly, multiple visual attention weights corresponding to each current candidate word in each word sequence window can be extracted from the multiple target network layers and all attention heads of each target network layer in the multimodal large language model. Each visual attention weight corresponds to an attention head in a network layer.

[0082] In one example, during the word generation process at each time step, the attention weights of each attention head of each network layer can be cached, so that the visual attention weights corresponding to each current candidate word can be extracted from the cache of the multiple target network layers and all attention heads of each target network layer.

[0083] Furthermore, the multiple visual attention weights corresponding to each current candidate word can be used to construct a visual attention weight matrix for each current candidate word. For example, by treating each target network layer as a row and the attention head of each network layer as a column, a visual attention weight matrix corresponding to each current candidate word can be obtained.

[0084] This allows us to arrange the visual attention weight matrix corresponding to each current candidate word in each word sequence window in order according to the time step corresponding to each current candidate word, thus obtaining the visual attention weight sequence corresponding to each word sequence window.

[0085] In one example, taking the word sequence window: [X1-1, X1-2, X1-3, X1-4, X1-5, X1-6, X1-7, X1-8] as an example, the corresponding visual attention weight sequence (using...) (This can be represented as follows.)

[0086]

[0087] in, It refers to the visual attention weight matrix corresponding to each current candidate word in the word sequence window. Let be the visual attention weight matrix corresponding to the current candidate word at the ti-th time step in the word sequence window, where i is 1~8, corresponding to t1~t8.

[0088]

[0089] in, For multiple target network layers, 1~ Layer; h is the number of attention heads for each target network layer, 1~h; Both h and h are integers greater than 1. It refers to the current candidate word at the ti-th time step, in the ti-th time step. The visual attention weights corresponding to the layer and the h-th attention head. It can be expressed by the following formula.

[0090]

[0091] in, It refers to the current candidate word element (e.g., one of X1-1 to X1-8 above) at the ti-th time step, in the ti-th time step. The query vector corresponding to the layer and the h-th attention head; It refers to the current candidate word element (e.g., one of X1-1 to X1-8 above) at the ti-th time step, in the ti-th time step. The key vector corresponding to the h-th attention head in the layer. M represents the attention mask, d is the feature dimension, and H is the total number of attention heads.

[0092] By extracting multiple visual attention weights corresponding to each current candidate word in each word sequence window from the multiple target network layers and all attention heads of each target network layer in the above multimodal large language model, and each visual attention weight corresponding to an attention head in a network layer, the visual attention weight sequence corresponding to each word sequence window is combined to achieve the purpose of hallucination detection based on visual attention weight distribution, thereby realizing real-time detection of hallucinations. This can alleviate hallucinations during the text generation process, avoid iterative accumulation of hallucination content, and make text generation more accurate.

[0093] In one possible implementation, determining the multiple target network layers in the multimodal large language model can include: identifying all network layers in the multimodal large language model as the multiple target network layers; or, identifying multiple intermediate network layers in the multimodal large language model (for example, out of a total of 32 network layers, the intermediate network layers could be layers 18-24) as the multiple target network layers. By identifying all network layers in the multimodal large language model as the multiple target network layers, the visual attention weight sequence becomes more comprehensive and accurate; by identifying multiple intermediate network layers in the multimodal large language model as the multiple target network layers, the visual attention weight sequence becomes more accurate and the extraction is more efficient because the intermediate layers can extract key visual information.

[0094] In step S207, the hallucination detection result corresponding to each word sequence window is determined based on the visual attention weight sequence corresponding to each word sequence window.

[0095] This specification's embodiments reveal a correlation between the distribution of visual attention weights and the presence of hallucinations. Specifically, during hallucinations, the distribution of visual attention weights in MLLMs becomes more dispersed, indicating visual attention defocusing. Furthermore, it was found that when generating adjacent word units, visual attention concentrates on the same image region, demonstrating temporal consistency in visual attention. Based on these findings, a method is implemented to determine the hallucination detection result for each word sequence window based on the distribution of visual attention weights in the visual attention weight sequence corresponding to each word sequence window.

[0096] In one possible implementation, the distribution information of attention weights in the visual attention weight sequence corresponding to each word sequence window can be determined. For example, the eigenvalues ​​of the covariance matrix of the attention weights in the visual attention weight sequence corresponding to each word sequence window can be calculated as distribution information. The hallucination detection result corresponding to each word sequence window can then be determined by characterizing the degree of visual attention dispersion through the eigenvalues ​​of the covariance matrix. For example, a range of values ​​for the degree of visual attention dispersion can be set, mapping the degree of visual attention dispersion to this range. For instance, a higher degree of visual attention dispersion corresponds to a larger value, indicating a higher probability of hallucination. This disclosure does not limit this approach.

[0097] In one alternative implementation, the relationship between visual attention weight sequences and hallucinations can be learned through a model, thereby using the model to predict hallucination detection results. Based on this, the above-mentioned determination of the hallucination detection result corresponding to each word sequence window based on the visual attention weight sequence corresponding to each word sequence window can include: inputting the visual attention weight sequence into a real-time hallucination detection model to obtain the hallucination detection result corresponding to each word sequence window.

[0098] In one specific embodiment, the real-time hallucination detection model can be obtained by training a binary classification network model based on multiple training sample pairs of sample visual attention weight sequences and hallucination evaluation labels. For details, please refer to the following detailed description, which will not be repeated here.

[0099] In one specific embodiment, the binary classification network model may include a multilayer perceptron, a convolutional neural network (CNN) (e.g., a 1-layer CNN), or a Transformer encoder. Correspondingly, the real-time hallucination detection model can be a pre-trained multilayer perceptron, convolutional neural network, or Transformer encoder. By setting up a simple binary classification network model, a lightweight design is achieved, integrating it into the decoding process with almost no additional latency, while also being adaptable to hallucination detection using different architectures of MLLMs, thus reducing application costs.

[0100] In one specific embodiment, the hallucination detection result can be a correctness score (negatively correlated with the probability of hallucination, i.e., the higher the correctness score, the lower the probability of hallucination) or a hallucination score (positively correlated with the probability of hallucination, i.e., the higher the hallucination score, the higher the probability of hallucination). Taking the correctness score as an example, it can be represented by the following formula.

[0101]

[0102] in, The sequence represents the visual attention weights; F represents the MLP classifier; y=1 indicates the word sequence window is the correct window, y=0 indicates the word sequence window is a hallucination window; and P is the classification probability (e.g., the correctness score). For example, P can be compared to 0.5; a value greater than 0.5 indicates a correct window, while a value less than or equal to 0.5 indicates a hallucination window. In the embodiments of this specification, binary label determination may be omitted, and instead, the classification probability P can be used, i.e., the hallucination detection result is the classification probability P (e.g., a value between 0 and 1).

[0103] In step S209, the current candidate word sequence corresponding to the word sequence window that represents the lowest probability of hallucination in the above hallucination detection results is selected as the current generated text corresponding to the current time step.

[0104] In one specific embodiment, when using the above-mentioned correctness score with the hallucination detection result, the higher the classification probability P, the lower the probability of representing a hallucination. Therefore, the current candidate word sequence corresponding to the highest classification probability P can be selected as the currently generated text for the current time step. Alternatively, when using the above-mentioned hallucination score with the hallucination detection result, the higher the classification probability P, the higher the probability of representing a hallucination. Therefore, the current candidate word sequence corresponding to the lowest classification probability P can be selected as the currently generated text for the current time step. This disclosure does not limit this aspect.

[0105] In one specific embodiment, if the number of current candidate word sequences corresponding to the word sequence window representing the lowest hallucination probability in the hallucination detection result is one, this single current candidate word sequence can be used as the currently generated text corresponding to the current time step. If the number of current candidate word sequences corresponding to the word sequence window representing the lowest hallucination probability in the hallucination detection result is multiple, the current candidate word sequence with the highest cumulative score among these multiple current candidate word sequences can be used as the currently generated text corresponding to the current time step.

[0106] For example, taking the above example, the current candidate word sequence 1: [X1-1, X1-2, X1-3, X1-4, X1-5, X1-6, X1-7, X1-8]; cumulative score 0.6*0.8=0.48; the current candidate word sequence 2: [X2-1, X2-2, X2-3, X2-4, X2-5, X2-6, X2-7, X2-8]; cumulative score 0.3*0.7=0.21.

[0107] If the hallucination detection result of the current candidate word sequence 1 is a correctness score of 0.45, and the hallucination detection result of the current candidate word sequence 2 is a correctness score of 0.9, then the current candidate word sequence 2 can be selected as the current generated text corresponding to the current time step t8, that is, the text composed of words in [X2-1, X2-2, X2-3, X2-4, X2-5, X2-6, X2-7, X2-8], such as a sentence. This avoids the problem of hallucination accumulation caused by selecting the current candidate word sequence 1 based on cumulative score, allowing hallucinations to be corrected and filtered in real time, preventing the transmission and accumulation of hallucination content. If the correctness scores are both 0.7, then the current candidate word sequence 1 with the highest cumulative score can be selected as the current generated text corresponding to the current time step t8.

[0108] As an example, refer to Figure 5 Taking beam search decoding strategies and MLP real-time hallucination detection models as examples, hallucination detection and processing yield the currently generated text corresponding to the current time step, which can be done as follows: Figure 5 The process shown enables real-time detection and processing of hallucinations at the current time step of text generation in MLLMs, effectively improving the accuracy of text generation.

[0109] Multiple candidate word sequence windows are obtained by extracting multiple candidate word sequences corresponding to the current time step through a preset sliding window. Each word sequence window consists of multiple candidate words corresponding to the current time step arranged in order. The multiple time steps include the current time step. The visual attention weight sequence corresponding to each word sequence window is extracted from the multimodal large language model. The visual attention weight sequence consists of the visual attention weight matrix corresponding to the multiple candidate words arranged in order in the corresponding word sequence window. The hallucination detection result corresponding to each word sequence window is determined according to the visual attention weight sequence corresponding to each word sequence window. The current candidate word sequence corresponding to the word sequence window with the lowest hallucination probability represented by the hallucination detection result is selected as the current generated text corresponding to the current time step. By utilizing the visual attention weight sequence corresponding to each lexical sequence window at the current time step—that is, the visual attention weight distribution—hallucination detection is performed. This enables real-time hallucination detection and processing at the current time step, selecting the candidate lexical sequence with the lowest hallucination probability at the current time step as the basis for subsequent text generation. This effectively avoids the transmission and accumulation of hallucination content, achieving both real-time hallucination detection and real-time hallucination mitigation during MLLMs text generation, preventing the transmission and accumulation of hallucination lexical units, and effectively improving the accuracy of MLLMs text generation. Furthermore, the real-time hallucination detection method using visual attention weight distribution is adaptable to hallucination detection and processing in multimodal large language models with different architectures, resulting in good performance and adaptability in hallucination detection and processing.

[0110] Reference Figure 4 In one alternative implementation, the method may further include: If the text generation termination condition is met at the current time step, the currently generated text will be used as the target generated text. If the text generation termination condition is not met at the current time step, the current candidate word sequence corresponding to the currently generated text is updated to at least one of the above-mentioned historical candidate word sequences, the text generation step is returned, and the text generation continues until the text generation termination condition is met. The generated text corresponding to the time when the text generation termination condition is met is taken as the target generated text.

[0111] For example, the text generation termination condition may include, but is not limited to, reaching the end-of-sequence symbol or reaching the maximum word length, etc., and this disclosure does not limit it.

[0112] By setting a text generation termination condition, iterative text generation can continue. Furthermore, when continuing text generation, the correctly generated text after filtering out the hallucinations can be used as the initial text for the next round, thereby avoiding the transmission of hallucination word windows and improving the accuracy of text generation.

[0113] Reference Figure 6 In one possible implementation, the above method may further include: In step S601, visual information of multiple samples and sample prompt text are acquired.

[0114] In one specific embodiment, sample visual information may refer to the visual information of a sample object, which may be matched with a text generation task of visual information, such as an image description task, and the sample object may refer to a sample image.

[0115] In one specific embodiment, the sample prompt text can be matched with a text generation task for visual information, such as an image description task. The sample prompt text could be "Help me describe this image," but this disclosure does not limit it in this way.

[0116] In step S603, the visual information of each sample and the sample prompt text are input into a multimodal large language model to generate text, thereby obtaining the sample generated text corresponding to the visual information of each sample.

[0117] In a specific embodiment, the output of the multimodal large language model can be decoded using a greedy decoding strategy or the aforementioned preset decoding strategy to obtain sample-generated text corresponding to the visual information of each sample.

[0118] In step S605, a text illusion evaluation is performed on the sample generated text to locate sample words in the sample generated text that exhibit illusion.

[0119] In one specific embodiment, text illusion assessment can be performed manually or using comparison algorithms. For example, the generated text of the sample can be compared with the visual information of the sample. If they match, it can be determined that there are no illusionary word units in the sample; if they do not match, it can be determined that the inconsistent sample words are located, i.e., the sample word units that are illusionary. Figure 7 The "cat" in the text.

[0120] In one specific embodiment, the above-described text hallucination assessment of the generated sample text, locating the hallucinatory sample words in the generated sample text, may include: performing text hallucination assessment on the generated sample text based on a hallucination assessment tool to locate the hallucinatory sample words in the generated sample text. Exemplarily, the hallucination assessment tool may include, but is not limited to, high-precision hallucination assessment tools such as GPT-4o, GPT-4 Turbo, and Claude 3. Locating the hallucinatory sample words in the generated sample text using a hallucination assessment tool can improve the efficiency of text hallucination assessment.

[0121] In step S607, sample text is generated based on a preset sliding window to obtain multiple sample text windows.

[0122] Reference Figure 7 For example, the sample generated text is "A cat is curled up on the sofa and sleeping". Assuming the preset sliding window length is 8, two sample text windows can be obtained: "A cat is curled up on the sofa" and "Sleeping on the sofa".

[0123] In step S609, each sample text window is labeled with a hallucination assessment tag based on whether it contains the sample word character that indicates the presence of hallucination.

[0124] For example, a sample text window containing the sample word for "hallucination" can be labeled as a "hallucination evaluation" label, and a sample text window not containing the sample word for "hallucination" can be labeled as a "non-hallucination evaluation" label.

[0125] Reference Figure 7 The phrase “a cat curled up on the sofa” includes the hallucination-related word “cat” and can be labeled as a hallucination assessment label (e.g., 0). The phrase “curled up on the sofa and sleeping” does not contain the hallucination-related word “cat” and can be labeled as a non-hallucination assessment label (e.g., 1).

[0126] In step S611, the visual attention weight sequence corresponding to each sample text window is extracted from the multimodal large language model.

[0127] For example, each sample visual attention weight sequence includes a sample visual attention weight matrix corresponding to each sample word in the sample text window.

[0128] In a specific embodiment, extracting the sample visual attention weight sequence corresponding to each sample text window from the multimodal large language model may include: determining multiple target network layers in the multimodal large language model; extracting multiple sample visual attention weights corresponding to each sample word in each sample text window from the multiple target network layers and all attention heads of each target network layer in the multimodal large language model, where each sample visual attention weight corresponds to an attention head in a network layer; constructing a sample visual attention weight matrix corresponding to each sample word from the multiple sample visual attention weights corresponding to each sample word; and arranging the sample visual attention weight matrix corresponding to each sample word in each sample text window in order according to the sample time step corresponding to each sample word to obtain the sample visual attention weight sequence corresponding to each sample text window. Specifically, the steps for extracting the visual attention weight sequence corresponding to each word sequence window from the multimodal large language model described above can be referred to, and will not be repeated in detail here.

[0129] In step S613, based on the illusion evaluation label labeled for each sample text window, the visual attention weight sequence corresponding to each sample text window is labeled with an illusion evaluation label, thereby obtaining multiple training sample pairs of sample visual attention weight sequence and illusion evaluation label.

[0130] For example, a training sample pair includes a sequence of sample visual attention weights and a labeled illusion evaluation label.

[0131] In step S615, a binary classification network model is trained based on the above-mentioned multiple training sample pairs until the training iteration termination condition is met. The binary classification network model that meets the training iteration termination condition is used as the above-mentioned real-time hallucination detection model.

[0132] In one specific embodiment, training the binary classification network model based on the aforementioned multiple training samples can be a supervised training method. The loss between the output hallucination prediction probability and the hallucination evaluation label can be calculated, and gradient backpropagation can be performed based on the loss to adjust the model parameters of the binary classification network model until the training iteration termination condition is met. Exemplarily, the loss can be calculated using the cross-entropy loss function, but this disclosure is not limited thereto.

[0133] For example, the conditions for ending a training iteration may include, but are not limited to, falling below a loss threshold or reaching the required number of training iterations.

[0134] By constructing multiple training sample pairs of visual attention weight sequences and hallucination evaluation labels, a binary classification network model is trained. This enables the real-time hallucination detection model to efficiently and accurately perform hallucination detection based on word sequence windows using visual attention weight sequences, thereby improving the accuracy of text generated by MLLMs.

[0135] The hallucination detection and processing methods described in this specification reveal the core causes of hallucinations in MLLMs: the phenomenon of visual attention defocusing breaks through the limitations of existing technologies that only address the language domain; it achieves integrated hallucination detection and mitigation: by deeply integrating MLP detection and decoding strategies, it realizes real-time detection and dynamic mitigation of hallucinations, solving the problem of disconnect between detection and mitigation in existing technologies; it is applicable to MLLMs with different architectures, requiring no large-scale modification of the model, only the extraction of visual attention weights, thus reducing application costs; it uses a lightweight MLP classifier, which is integrated into the decoding process with almost no additional latency, and the time overhead is only about 1.2 times that of the baseline, meeting the real-time requirements of practical applications. Furthermore, during the training phase, it employs text hallucination evaluation methods such as GPT-4o combined with sliding windows, providing high-quality training data for the detection model training, further improving the detection and mitigation effects.

[0136] Specifically, regarding hallucination detection performance, taking the LLaVA-1.5 7B model as an example, this scheme improves hallucination detection by 54%, compared to a 22% improvement over hidden state-based methods. Hallucination mitigation is significantly reduced, with the hallucination rate decreasing from 37.5% to 32.9%. Furthermore, this scheme demonstrates stable generalization performance in hallucination detection and processing, exhibiting broad applicability.

[0137] Figure 8 This is a block diagram of a real-time hallucination detection and processing device based on a multimodal large language model, according to an exemplary embodiment. (Refer to...) Figure 8 The device may include: The text generation module 801 is configured to perform text generation based on the visual information to be processed, the prompt text, and at least one historical candidate word sequence corresponding to the previous time step, using the multimodal large language model and a preset decoding strategy to obtain multiple current candidate word sequences corresponding to the current time step. The current candidate word sequence acquisition module 803 is configured to extract the multiple current candidate word sequences based on a preset sliding window to obtain multiple word sequence windows; each word sequence window consists of multiple current candidate words corresponding to multiple time steps arranged in order, and the multiple time steps include the current time step; The visual attention weight sequence extraction module 805 is configured to extract the visual attention weight sequence corresponding to each word sequence window from the multimodal large language model; the visual attention weight sequence is composed of the visual attention weight matrices corresponding to multiple current candidate words arranged in order in the corresponding word sequence window; The hallucination detection result determination module 807 is configured to determine the hallucination detection result corresponding to each word sequence window based on the visual attention weight sequence corresponding to each word sequence window; The hallucination processing module 809 is configured to perform the filtering of the current candidate word sequence corresponding to the word sequence window that represents the lowest hallucination probability in the hallucination detection result, and use it as the current generated text corresponding to the current time step.

[0138] Multiple candidate word sequence windows are obtained by extracting multiple candidate word sequences corresponding to the current time step through a preset sliding window. Each word sequence window consists of multiple candidate words corresponding to the current time step arranged in order. The multiple time steps include the current time step. The visual attention weight sequence corresponding to each word sequence window is extracted from the multimodal large language model. The visual attention weight sequence consists of the visual attention weight matrix corresponding to the multiple candidate words arranged in order in the corresponding word sequence window. The hallucination detection result corresponding to each word sequence window is determined according to the visual attention weight sequence corresponding to each word sequence window. The current candidate word sequence corresponding to the word sequence window with the lowest hallucination probability represented by the hallucination detection result is selected as the current generated text corresponding to the current time step. By utilizing the visual attention weight sequence corresponding to each lexical sequence window at the current time step—that is, the visual attention weight distribution—hallucination detection is performed. This enables real-time hallucination detection and processing at the current time step, selecting the candidate lexical sequence with the lowest hallucination probability at the current time step as the basis for subsequent text generation. This effectively avoids the transmission and accumulation of hallucination content, achieving both real-time hallucination detection and real-time hallucination mitigation during MLLMs text generation, preventing the transmission and accumulation of hallucination lexical units, and effectively improving the accuracy of MLLMs text generation. Furthermore, the real-time hallucination detection method using visual attention weight distribution is adaptable to hallucination detection and processing in multimodal large language models with different architectures, resulting in good performance and adaptability in hallucination detection and processing.

[0139] In one possible implementation, the current candidate word sequence acquisition module 803 may include: The current candidate word sequence acquisition unit is configured to perform sliding truncation based on the preset sliding window to extract the multiple current candidate word sequences respectively, under the condition that the preset sliding window configuration is satisfied, to obtain the multiple word sequence windows; The sliding capture conditions corresponding to the preset sliding window include: the current time step reaches the number of time steps configured for the preset sliding window; or, the current time step is greater than the number of time steps configured for the preset sliding window and satisfies the sliding step length corresponding to the preset sliding window.

[0140] In one possible implementation, the device may further include: The text generation iteration module is configured to perform the following steps when, if the sliding truncation conditions of the preset sliding window configuration are not met, to treat the multiple current candidate word sequences as the at least one historical candidate word sequence and return to the text generation step, and continue the text generation process.

[0141] In one possible implementation, the visual attention weight sequence extraction module 805 may include: The target network layer determination unit is configured to determine multiple target network layers in the multimodal large language model; The visual attention weight extraction unit is configured to extract multiple visual attention weights corresponding to each current candidate word in each word sequence window from the multiple target network layers and all attention heads of each target network layer in the multimodal large language model. Each visual attention weight corresponds to an attention head in a network layer. The visual attention weight matrix construction unit is configured to construct a visual attention weight matrix for each current candidate word element by constructing multiple visual attention weights corresponding to each current candidate word element. The visual attention weight sequence acquisition unit is configured to perform the following: arranging the visual attention weight matrix corresponding to each current candidate word in each word sequence window in order according to the time step corresponding to each current candidate word, thereby obtaining the visual attention weight sequence corresponding to each word sequence window.

[0142] In one possible implementation, the target network layer determination unit may include: The first target network layer determination subunit is configured to perform the task of determining all network layers in the multimodal large language model as the multiple target network layers; Alternatively, the second target network layer determining subunit is configured to perform the determination of multiple network layers located in the middle of the multimodal large language model as the multiple target network layers.

[0143] In one possible implementation, the hallucination detection result determination module 807 may include: The hallucination detection result determination unit is configured to input the visual attention weight sequence into the real-time hallucination detection model to obtain the hallucination detection result corresponding to each word sequence window.

[0144] In one possible implementation, the device may further include: The sample data acquisition module is configured to acquire visual information of multiple samples and sample prompt text; The sample text acquisition module is configured to input the visual information of each sample and the sample prompt text into the multimodal large language model to generate text, thereby obtaining the sample generated text corresponding to the visual information of each sample. The hallucination sample word localization module is configured to perform a text hallucination evaluation on the sample generated text and locate the sample words in the sample generated text that contain hallucinations. The sample text window acquisition module is configured to extract the sample generated text based on the preset sliding window to obtain multiple sample text windows. The first hallucination assessment labeling module is configured to perform a hallucination assessment labeling for each sample text window based on whether each sample text window contains the sample word character indicating the presence of hallucination; sample text windows containing the sample word character indicating the presence of hallucination are labeled as hallucination assessment labels, and sample text windows not containing the sample word character indicating the presence of hallucination are labeled as non-hallucination assessment labels. The sample visual attention weight sequence extraction module is configured to extract the sample visual attention weight sequence corresponding to each sample text window from the multimodal large language model; each sample visual attention weight sequence includes the sample visual attention weight matrix corresponding to each sample word in the sample text window; The second illusion evaluation label annotation module is configured to perform illusion evaluation label annotation based on each sample text window, and to annotate the visual attention weight sequence of the sample corresponding to each sample text window with illusion evaluation labels, thereby obtaining multiple training sample pairs of sample visual attention weight sequences and illusion evaluation labels. The training iteration module is configured to train a binary classification network model based on the multiple training sample pairs until the training iteration termination condition is met, and the binary classification network model corresponding to the training iteration termination condition is used as the real-time hallucination detection model.

[0145] In one possible implementation, the hallucination sample lexical localization module may include: The hallucination sample word localization unit is configured to perform a text hallucination assessment on the sample-generated text based on a hallucination assessment tool, and to locate sample words in the sample-generated text that contain hallucinations.

[0146] In one possible implementation, the binary classification network model includes a multilayer perceptron, a convolutional neural network, or a Transformer encoder.

[0147] In one possible implementation, the sample visual attention weight sequence extraction module may include: The target network layer determination unit is configured to determine multiple target network layers in the multimodal large language model; The sample visual attention weight extraction unit is configured to extract multiple sample visual attention weights corresponding to each sample word in each sample text window from the multiple target network layers and all attention heads of each target network layer in the multimodal large language model. Each sample visual attention weight corresponds to an attention head in a network layer. The sample visual attention weight matrix construction unit is configured to construct a sample visual attention weight matrix corresponding to each sample word by performing multiple sample visual attention weights corresponding to each sample word. The sample visual attention weight sequence acquisition unit is configured to perform the following: arranging the sample visual attention weight matrix corresponding to each sample word in each sample text window in order according to the sample time step corresponding to each sample word, thereby obtaining the sample visual attention weight sequence corresponding to each sample text window.

[0148] In one possible implementation, the preset decoding strategy may include: a beam search decoding strategy, or a decoding strategy adopted by the core.

[0149] In one possible implementation, the device may further include: The first target text generation determination module is configured to use the currently generated text as the target text if the text generation termination condition is met at the current time step. The second target text generation determination module is configured to, if the text generation termination condition is not met at the current time step, update the current candidate word sequence corresponding to the currently generated text to the at least one historical candidate word sequence, return to the text generation step, and continue the text generation until the text generation termination condition is met, and take the generated text corresponding to the text generation termination condition as the target generated text.

[0150] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0151] Figure 9This is a block diagram illustrating an electronic device for real-time hallucination detection and processing of a multimodal large language model, according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, the electronic device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a method for real-time illusion detection and processing of a multimodal large language model. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse. Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0152] Figure 10 This is a block diagram of an electronic device for real-time hallucination detection and processing of a multimodal large language model, based on an exemplary embodiment. The electronic device can be a server, and its internal structure diagram can be as follows: Figure 10 As shown, the electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a method for real-time hallucination detection and processing using a multimodal large language model. Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements. In an exemplary embodiment, an electronic device is also provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a real-time hallucination detection and processing method for a multimodal large language model as described in the embodiments of this disclosure.

[0153] In an exemplary embodiment, a computer-readable storage medium is also provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the real-time hallucination detection and processing method for a multimodal large language model according to embodiments of this disclosure. The computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0154] In an exemplary embodiment, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform a method for real-time hallucination detection and processing of a multimodal large language model as described in this disclosure.

[0155] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0156] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0157] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for real-time hallucination detection and handling of a multi-modal large language model, the method comprising: include: Based on the visual information to be processed, the prompt text, and at least one historical candidate word sequence corresponding to the previous time step, the multimodal large language model and the preset decoding strategy are used to generate text, and multiple current candidate word sequences corresponding to the current time step are obtained. Multiple candidate word sequences are extracted based on a preset sliding window to obtain multiple word sequence windows; each word sequence window consists of multiple candidate words corresponding to multiple time steps arranged in order, and the multiple time steps include the current time step; From the multimodal large language model, extract the visual attention weight sequence corresponding to each word sequence window; the visual attention weight sequence is composed of the visual attention weight matrix corresponding to each of the multiple current candidate words arranged in order in the corresponding word sequence window; Based on the visual attention weight sequence corresponding to each word sequence window, determine the hallucination detection result corresponding to each word sequence window; The current candidate word sequence corresponding to the word sequence window that represents the lowest probability of hallucination in the hallucination detection results is selected as the current generated text corresponding to the current time step.

2. The method of claim 1, wherein, The process of extracting multiple current candidate word sequences based on a preset sliding window to obtain multiple word sequence windows includes: Under the condition of sliding truncation configured by the preset sliding window, the plurality of current candidate word sequences are truncated based on the preset sliding window to obtain the plurality of word sequence windows; The sliding capture conditions corresponding to the preset sliding window include: the current time step reaches the number of time steps configured for the preset sliding window; or, the current time step is greater than the number of time steps configured for the preset sliding window and satisfies the sliding step length corresponding to the preset sliding window.

3. The method of claim 2, wherein, The method further includes: If the sliding truncation conditions of the preset sliding window configuration are not met, the multiple current candidate word sequences are used as the at least one historical candidate word sequence, and the text generation step is returned to continue the text generation process.

4. The method according to claim 1, characterized in that, The step of extracting the visual attention weight sequence corresponding to each word sequence window from the multimodal large language model includes: Identify multiple target network layers in the multimodal large language model; From the multiple target network layers in the multimodal large language model and all attention heads of each target network layer, extract multiple visual attention weights corresponding to each current candidate word in each word sequence window, and each visual attention weight corresponds to an attention head in a network layer; Construct a visual attention weight matrix for each current candidate word element by using multiple visual attention weights corresponding to each current candidate word element. The visual attention weight matrix corresponding to each current candidate word in each word sequence window is arranged in order according to the time step corresponding to each current candidate word, thus obtaining the visual attention weight sequence corresponding to each word sequence window.

5. The method according to claim 4, characterized in that, The determination of multiple target network layers in the multimodal large language model includes: All network layers in the multimodal large language model are identified as the multiple target network layers; Alternatively, the multiple network layers located in the middle of the multimodal large language model can be identified as the multiple target network layers.

6. The method according to claim 1, characterized in that, The step of determining the hallucination detection result corresponding to each word sequence window based on the visual attention weight sequence corresponding to each word sequence window includes: The visual attention weight sequence is input into the real-time hallucination detection model to obtain the hallucination detection result corresponding to each word sequence window.

7. The method according to claim 6, characterized in that, The method further includes: Acquire visual information and sample prompt text from multiple samples; The visual information of each sample and the sample prompt text are input into the multimodal large language model to generate text, thereby obtaining the sample generated text corresponding to the visual information of each sample; The generated text of the sample is evaluated for textual illusion, and the sample word units in the generated text that contain illusions are located. Based on the preset sliding window, the sample generated text is extracted to obtain multiple sample text windows; Based on whether each sample text window contains the sample word character indicating hallucination, a hallucination evaluation label is assigned to each sample text window; sample text windows containing the sample word character indicating hallucination are labeled as hallucination evaluation labels, and sample text windows not containing the sample word character indicating hallucination are labeled as non-hallucination evaluation labels. From the multimodal large language model, extract the sample visual attention weight sequence corresponding to each sample text window; each sample visual attention weight sequence includes the sample visual attention weight matrix corresponding to each sample word in the sample text window; Based on the illusion evaluation labels annotated in each sample text window, the visual attention weight sequence corresponding to each sample text window is labeled with illusion evaluation labels, resulting in multiple training sample pairs of sample visual attention weight sequences and illusion evaluation labels; The binary classification network model is trained based on the multiple training sample pairs until the training iteration ends. The binary classification network model that meets the training iteration end condition is used as the real-time hallucination detection model.

8. The method according to claim 7, characterized in that, The step of evaluating the text illusion of the generated sample text and locating the sample word units in the generated sample text that exhibit illusion includes: The generated text of the sample is evaluated for textual hallucination based on the hallucination assessment tool, and the sample word units in the generated text that contain hallucinations are located.

9. The method according to claim 7, characterized in that, The binary classification network model includes: a multilayer perceptron, a convolutional neural network, or a Transformer encoder.

10. The method according to claim 7, characterized in that, From the multimodal large language model, extract the sample visual attention weight sequence corresponding to each sample text window, including: Identify multiple target network layers in the multimodal large language model; From the multiple target network layers in the multimodal large language model and all attention heads of each target network layer, extract multiple sample visual attention weights corresponding to each sample word in each sample text window, and each sample visual attention weight corresponds to an attention head in a network layer; Construct a sample visual attention weight matrix for each sample word element by using multiple sample visual attention weights corresponding to multiple sample visual attention weights. The visual attention weight matrix corresponding to each sample word in each sample text window is arranged in order according to the sample time step corresponding to each sample word, thus obtaining the visual attention weight sequence corresponding to each sample text window.

11. The method according to claim 1, characterized in that, The preset decoding strategy includes: a beam search decoding strategy, or a decoding strategy adopted by the core.

12. The method according to claim 1, characterized in that, The method further includes: If the text generation termination condition is met at the current time step, the currently generated text will be used as the target generated text. If the text generation termination condition is not met at the current time step, the current candidate word sequence corresponding to the currently generated text is updated to the at least one historical candidate word sequence, the text generation step is returned, and the text generation continues until the text generation termination condition is met. The generated text corresponding to the time when the text generation termination condition is met is taken as the target generated text.

13. A real-time hallucination detection and processing device for a multimodal large language model, characterized in that, include: The text generation module is configured to perform text generation based on the visual information to be processed, the prompt text, and at least one historical candidate word sequence corresponding to the previous time step, using the multimodal large language model and a preset decoding strategy to obtain multiple current candidate word sequences corresponding to the current time step. The current candidate word sequence acquisition module is configured to extract the multiple current candidate word sequences based on a preset sliding window to obtain multiple word sequence windows; each word sequence window consists of multiple current candidate words corresponding to multiple time steps arranged in order, and the multiple time steps include the current time step; The visual attention weight sequence extraction module is configured to extract the visual attention weight sequence corresponding to each word sequence window from the multimodal large language model; the visual attention weight sequence consists of the visual attention weight matrices corresponding to multiple current candidate words arranged in order in the corresponding word sequence window; The hallucination detection result determination module is configured to determine the hallucination detection result corresponding to each word sequence window based on the visual attention weight sequence corresponding to each word sequence window; The hallucination processing module is configured to perform a process of selecting the current candidate word sequence corresponding to the word sequence window that represents the lowest hallucination probability in the hallucination detection results, and using it as the current generated text corresponding to the current time step.

14. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the real-time hallucination detection and processing method for a multimodal large language model as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the real-time hallucination detection and processing method of the multimodal large language model as described in any one of claims 1 to 12.

16. A computer program product, characterized in that, The method includes computer instructions that, when executed by a processor, cause a computer to perform a real-time hallucination detection and processing method for a multimodal large language model as described in any one of claims 1 to 12.

Images