Model hallucination detection method, device, equipment, storage medium and program product

Abstract

The present application relates to a method, device, equipment, storage medium and program product for detecting model hallucinations. The method includes: obtaining the cross-modal attention of a visual language model to a visual text pair, dividing the cross-modal attention into sample attention when there is hallucination or sample attention when there is no hallucination; inputting the sample attention into a first-stage hallucination detector to obtain a first hallucination detection result, and updating the first-stage hallucination detector according to the first hallucination detection result and the attention category of the sample attention; continuing to input the sample attention when the attention category is hallucination into a second-stage hallucination detector to obtain a second hallucination detection result, and updating the second-stage hallucination detector according to the second hallucination detection result and the sample category of the sample attention; cascading the first-stage hallucination detector and the second-stage hallucination detector to obtain a hallucination detection model, which is used to detect whether a hallucination occurs in the visual language model when performing model inference.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer device, computer-readable storage medium and computer program product for detecting model hallucinations. Background Art

[0002] The Visual Language Model (VLM) is a technology that combines image and natural language processing. By combining a pre-trained visual encoder with a language model, the VLM has the ability to process visual information and natural language understanding. It can be used to understand and interpret the relationship between images and text, and generate accurate and vivid natural language descriptions based on images, giving computers a visual understanding ability that is closer to that of humans. The VLM is widely used in tasks such as image description generation and visual question answering. It can automatically generate text that describes the content of an image, or be used to answer questions about an image.

[0003] Existing studies have found that visual language models have hallucination problems, which significantly challenges the practical application of visual language models. Model hallucination means that the model occasionally generates content that is inconsistent with user input, contradicts previously generated context, or is inconsistent with established world knowledge.

[0004] In the related art, model hallucinations are alleviated by visual contrastive decoding (VCD, which compares the output distributions derived from the original image and the distorted image). However, this method requires reasoning about the original image and the distorted image, which increases the cost of reasoning. Moreover, this method alleviates model hallucinations by reducing the excessive reliance of the visual language model on the prior knowledge of the pre-trained language model. The effect is only significant enough when there is a large difference between the prior knowledge and the input data distribution, and the practicality is not good enough. Therefore, how to quickly and accurately detect model hallucinations is a problem that needs to be solved urgently. Summary of the invention

[0005] Based on this, it is necessary to provide a model hallucination detection method, device, computer equipment, computer readable storage medium and computer program product to address the above technical problems, which can quickly and accurately detect the occurrence of hallucinations and have good practicality.

[0006] In a first aspect, the present application provides a method for detecting model hallucinations, comprising:

[0007] Obtaining cross-modal attention of a visual token of a visual text pair when a visual language model performs model reasoning on the visual text pair, and dividing the cross-modal attention into sample attention when there is hallucination or sample attention when there is no hallucination according to the reasoning result of the visual language model on the visual text pair;

[0008] Inputting the sample attention into a first-stage hallucination detector, obtaining a first hallucination detection result of the first-stage hallucination detector on the sample attention, and updating the first-stage hallucination detector according to the first hallucination detection result and the attention category of the sample attention;

[0009] From the sample attention input to the first-stage hallucination detector, filter out the sample attention when the attention category is hallucination, continue to input into the second-stage hallucination detector, obtain the second-stage hallucination detection result of the sample attention by the second-stage hallucination detector, and update the second-stage hallucination detector according to the second hallucination detection result and the sample category of the sample attention;

[0010] The first-stage hallucination detector obtained after training is cascaded with the second-stage hallucination detector to obtain a hallucination detection model, and the hallucination detection model is used to detect whether hallucination occurs when the visual language model performs model inference.

[0011] In a second aspect, the present application also provides a device for detecting model hallucinations, comprising:

[0012] an acquisition module, configured to acquire cross-modal attention of visual tokens of a visual text pair when a visual language model performs model inference on the visual text pair, and divide the cross-modal attention into sample attention when there is hallucination or sample attention when there is no hallucination according to the inference result of the visual language model on the visual text pair;

[0013] a detector updating module, configured to input the sample attention into a first-stage hallucination detector, obtain a first hallucination detection result of the first-stage hallucination detector on the sample attention, and update the first-stage hallucination detector according to the first hallucination detection result and the attention category of the sample attention;

[0014] The detector updating module is further used to filter out sample attentions with an attention category of hallucination from the sample attentions input to the first-stage hallucination detector, continue to input the sample attentions to the second-stage hallucination detector, obtain a second hallucination detection result of the second-stage hallucination detector on the sample attention, and update the second-stage hallucination detector according to the second hallucination detection result and the sample category of the sample attention;

[0015] The cascade module is used to cascade the first-stage hallucination detector obtained after training with the second-stage hallucination detector to obtain a hallucination detection model, and the hallucination detection model is used to detect whether hallucination occurs when the visual language model performs model inference.

[0016] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0017] Obtaining cross-modal attention of a visual token of a visual text pair when a visual language model performs model reasoning on the visual text pair, and dividing the cross-modal attention into sample attention when there is hallucination or sample attention when there is no hallucination according to the reasoning result of the visual language model on the visual text pair;

[0018] Inputting the sample attention into a first-stage hallucination detector, obtaining a first hallucination detection result of the first-stage hallucination detector on the sample attention, and updating the first-stage hallucination detector according to the first hallucination detection result and the attention category of the sample attention;

[0019] From the sample attention input to the first-stage hallucination detector, filter out the sample attention when the attention category is hallucination, continue to input into the second-stage hallucination detector, obtain the second-stage hallucination detection result of the sample attention by the second-stage hallucination detector, and update the second-stage hallucination detector according to the second hallucination detection result and the sample category of the sample attention;

[0020] The first-stage hallucination detector obtained after training is cascaded with the second-stage hallucination detector to obtain a hallucination detection model, and the hallucination detection model is used to detect whether hallucination occurs when the visual language model performs model inference.

[0021] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0022] Obtaining cross-modal attention of a visual token of a visual text pair when a visual language model performs model reasoning on the visual text pair, and dividing the cross-modal attention into sample attention when there is hallucination or sample attention when there is no hallucination according to the reasoning result of the visual language model on the visual text pair;

[0023] Inputting the sample attention into a first-stage hallucination detector, obtaining a first hallucination detection result of the first-stage hallucination detector on the sample attention, and updating the first-stage hallucination detector according to the first hallucination detection result and the attention category of the sample attention;

[0024] From the sample attention input to the first-stage hallucination detector, filter out the sample attention when the attention category is hallucination, continue to input into the second-stage hallucination detector, obtain the second-stage hallucination detection result of the sample attention by the second-stage hallucination detector, and update the second-stage hallucination detector according to the second hallucination detection result and the sample category of the sample attention;

[0025] The first-stage hallucination detector obtained after training is cascaded with the second-stage hallucination detector to obtain a hallucination detection model, and the hallucination detection model is used to detect whether hallucination occurs when the visual language model performs model inference.

[0026] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0027] Obtaining cross-modal attention of a visual token of a visual text pair when a visual language model performs model reasoning on the visual text pair, and dividing the cross-modal attention into sample attention when there is hallucination or sample attention when there is no hallucination according to the reasoning result of the visual language model on the visual text pair;

[0028] Inputting the sample attention into a first-stage hallucination detector, obtaining a first hallucination detection result of the first-stage hallucination detector on the sample attention, and updating the first-stage hallucination detector according to the first hallucination detection result and the attention category of the sample attention;

[0029] From the sample attention input to the first-stage hallucination detector, filter out the sample attention when the attention category is hallucination, continue to input into the second-stage hallucination detector, obtain the second-stage hallucination detection result of the sample attention by the second-stage hallucination detector, and update the second-stage hallucination detector according to the second hallucination detection result and the sample category of the sample attention;

[0030] The first-stage hallucination detector obtained after training is cascaded with the second-stage hallucination detector to obtain a hallucination detection model, and the hallucination detection model is used to detect whether hallucination occurs when the visual language model performs model inference.

[0031] The above-mentioned detection method, apparatus, computer device, computer-readable storage medium and computer program product of model hallucination pay attention to the difference in cross-modal attention between the visual language model when hallucination occurs and when hallucination does not occur. To this end, first, by obtaining the cross-modal attention of the visual token of the visual text pair when the visual language model performs model reasoning on the visual text pair, the cross-modal attention is divided into sample attention when there is hallucination or sample attention when there is no hallucination according to the reasoning result of the visual language model on the visual text pair, so as to construct a training sample for training the detector. Then, by inputting the sample attention into the first-stage hallucination detector, the first-stage hallucination detector obtains the first hallucination detection result of the sample attention, and according to the first hallucination detection result and the attention category of the sample attention, the first-stage hallucination detector is updated to achieve a high recall rate of attention to the hallucinated samples during the first-stage training process. Next, from the sample attention input to the first-stage hallucination detector, the sample attention with the attention category of hallucination is screened out, and the attention is continuously input to the second-stage hallucination detector for more detailed classification. By obtaining the second hallucination detection result of the sample attention by the second-stage hallucination detector, the second-stage hallucination detector is updated according to the second hallucination detection result and the sample category of the sample attention, so as to achieve a high detection rate of hallucinated samples during the training process of the second stage. Then, the first-stage hallucination detector obtained after the training is cascaded with the second-stage hallucination detector to obtain a hallucination detection model, which is used to detect whether hallucination occurs in the visual language model during model inference, and can ensure the accuracy of detecting whether hallucination occurs in the text language model. At the same time, compared with the existing technology, the hallucination detection model can directly detect whether hallucination occurs, with high detection efficiency and strong practicality.

[0032] In a sixth aspect, the present application provides a method for detecting model hallucinations, comprising:

[0033] Obtaining cross-modal attention of a visual language model to visual tokens of a target visual text pair when performing model reasoning on the target visual text pair;

[0034] Inputting the cross-modal attention into a hallucination detection model, wherein the hallucination detection model is obtained by cascading a first-stage hallucination detector obtained after training and a second-stage hallucination detector, and outputting a first hallucination detection result through the first-stage hallucination detector;

[0035] In a case where the first hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning, inputting the cross-modal attention into the second-stage hallucination detector to obtain a second hallucination detection result;

[0036] When the second hallucination detection result indicates that the visual language model hallucinates when performing model reasoning, it is determined that the visual language model hallucinates when performing model reasoning on the target visual-text pair.

[0037] In a seventh aspect, the present application also provides a device for detecting model hallucinations, comprising:

[0038] An acquisition module, used for acquiring cross-modal attention of a visual token of a target visual text pair when a visual language model performs model reasoning on the target visual text pair;

[0039] A detection module, configured to input the cross-modal attention into an illusion detection model, wherein the illusion detection model is obtained by cascading a first-stage illusion detector obtained after training and a second-stage illusion detector, and outputting a first illusion detection result through the first-stage illusion detector;

[0040] The detection module is further configured to input the cross-modal attention into the second-stage hallucination detector to obtain a second hallucination detection result when the first hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning;

[0041] A detection result determination module is used to determine that the visual language model hallucinates when performing model reasoning on the target visual text pair when the second hallucination detection result indicates that the visual language model hallucinates when performing model reasoning.

[0042] In an eighth aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0043] Obtaining cross-modal attention of a visual language model to visual tokens of a target visual text pair when performing model reasoning on the target visual text pair;

[0044] Inputting the cross-modal attention into a hallucination detection model, wherein the hallucination detection model is obtained by cascading a first-stage hallucination detector obtained after training and a second-stage hallucination detector, and outputting a first hallucination detection result through the first-stage hallucination detector;

[0045] In a case where the first hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning, inputting the cross-modal attention into the second-stage hallucination detector to obtain a second hallucination detection result;

[0046] When the second hallucination detection result indicates that the visual language model hallucinates when performing model reasoning, it is determined that the visual language model hallucinates when performing model reasoning on the target visual-text pair.

[0047] In a ninth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0048] Obtaining cross-modal attention of a visual language model to visual tokens of a target visual text pair when performing model reasoning on the target visual text pair;

[0049] Inputting the cross-modal attention into a hallucination detection model, wherein the hallucination detection model is obtained by cascading a first-stage hallucination detector obtained after training and a second-stage hallucination detector, and outputting a first hallucination detection result through the first-stage hallucination detector;

[0050] In a case where the first hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning, inputting the cross-modal attention into the second-stage hallucination detector to obtain a second hallucination detection result;

[0051] When the second hallucination detection result indicates that the visual language model hallucinates when performing model reasoning, it is determined that the visual language model hallucinates when performing model reasoning on the target visual-text pair.

[0052] In a tenth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0053] Obtaining cross-modal attention of a visual language model to visual tokens of a target visual text pair when performing model reasoning on the target visual text pair;

[0054] Inputting the cross-modal attention into a hallucination detection model, wherein the hallucination detection model is obtained by cascading a first-stage hallucination detector obtained after training and a second-stage hallucination detector, and outputting a first hallucination detection result through the first-stage hallucination detector;

[0055] In a case where the first hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning, inputting the cross-modal attention into the second-stage hallucination detector to obtain a second hallucination detection result;

[0056] When the second hallucination detection result indicates that the visual language model hallucinates when performing model reasoning, it is determined that the visual language model hallucinates when performing model reasoning on the target visual-text pair.

[0057] The above-mentioned model hallucination detection method, apparatus, computer device, computer-readable storage medium and computer program product pay attention to the difference in cross-modal attention between the visual language model when hallucination occurs and when hallucination does not occur. To this end, by obtaining the cross-modal attention of the visual language model to the visual token of the target visual text pair when performing model reasoning on the target visual text pair, the cross-modal attention is input into the hallucination detection model, and the hallucination detection model is obtained by cascading the first-stage hallucination detector and the second-stage hallucination detector obtained after the training is completed. The first-stage hallucination detector outputs a first hallucination detection result. When the first hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning, the cross-modal attention is further input into the second-stage hallucination detector to obtain a second hallucination detection result. When the second hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning, it can be determined that the visual language model has hallucinations when performing model reasoning on the target visual text pair. In this way, efficient and accurate detection of whether the visual language model has hallucinations when performing model reasoning based on cross-modal attention can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 Schematic diagram of the model's hallucination;

[0060] Figure 2 is a schematic diagram of a visual language model in one embodiment;

[0061] Figure 3 Schematic diagram of the obtained cross-modal attention;

[0062] Figure 4 A diagram showing an application environment of a method for detecting model hallucination in one embodiment;

[0063] Figure 5 is a flow chart of a method for detecting model hallucination in one embodiment;

[0064] Figure 6 is a schematic diagram of a first stage hallucination detector in one embodiment;

[0065] Figure 7 is a schematic diagram of a second stage hallucination detector in one embodiment;

[0066] Figure 8 A schematic diagram of a process of using a hallucination detection model to detect whether a hallucination occurs in one embodiment;

[0067] Fig. 9 A schematic diagram of using a hallucination detection model to perform hallucination detection and mitigation in one embodiment;

[0068] Fig.10 A schematic diagram of a process for obtaining cross-modal attention of visual tokens in one embodiment;

[0069] Fig.11 A schematic diagram of a process of constructing a visual-text pair in one embodiment;

[0070] Fig.12 is a schematic diagram of a first stage hallucination detector in another embodiment;

[0071] Fig.13 is a schematic diagram of a first type second stage hallucination detector in one embodiment;

[0072] Fig.14 is a schematic diagram of a second type second stage hallucination detector in one embodiment;

[0073] Fig.15 is a flow chart of a method for detecting model hallucination in another embodiment;

[0074] Fig.16 is a flow chart of a method for detecting model hallucination in yet another embodiment;

[0075] Fig.17 A schematic diagram of using a hallucination detection model to detect and mitigate hallucinations in another embodiment;

[0076] Fig.18 A schematic diagram of the relevant statistics for the hallucination detection model;

[0077] Fig.19 Here is another diagram of the relevant statistics for the hallucination detection model;

[0078] Fig. 20 is a structural block diagram of a device for detecting model hallucination in one embodiment;

[0079] Fig.21 is a structural block diagram of a device for detecting model hallucination in another embodiment;

[0080] Fig. 22 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0082] The Visual Language Model (VLM) is a technology that combines image and natural language processing. By using a pre-trained visual encoder combined with a language model, the Visual Language Model has the ability to process visual information and natural language understanding. It can be used to understand and interpret the relationship between images and text, and generate accurate and vivid natural language descriptions based on images, giving computers a visual understanding ability that is closer to that of humans. Among them, the Large Visual Language Model (LVLM) has good zero-shot capabilities, good generalization capabilities, and can process a variety of image types including documents, web pages, etc. Visual language models have a wide range of applications, including image-based chat, image recognition based on instructions, visual question and answer, document understanding, image description, etc.

[0083] Existing studies have found that visual language models have hallucinations, which significantly challenges their practical applications. Model hallucinations mean that the model occasionally generates content that is inconsistent with user input, contradicts previously generated context, or is inconsistent with established world knowledge. For example, when asked whether an object exists in a picture, the visual language model may be blind to the existence of the object in the picture, or may even think that there is an object in the picture out of thin air. For example, Figure 1 The following is a schematic diagram of the model's hallucination. Figure 1 ,for Figure 1 For the picture in the upper middle part, for the question “is there a chair in the image?”, the visual language model mistakenly believes that there is a chair in the picture. Figure 1 In the lower middle part of the picture, for the question "is there a toothbrush in the image?", the visual language model incorrectly believes that there is no toothbrush in the picture. Obviously, this phenomenon poses a major challenge to the accuracy and reliability of visual language models in real-world scenarios.

[0084] In the related art, the use of visual contrast decoding to alleviate hallucinations has the problems of high reasoning cost and poor practicality. The embodiment of the present application provides a completely different method, which can efficiently and accurately detect whether the model is hallucinating. In a simple binary answer application scenario (the model only needs to answer yes {Yes} or no {No}), the model's answer can also be flipped when it is detected that the model is hallucinating. This operation can alleviate the model's hallucinations. Specifically, it has been found that the visual language model exhibits different patterns in the cross-modal attention of visual tokens when hallucinations occur and when no hallucinations occur. Therefore, the cross-modal attention of the visual token when the visual language model performs model reasoning is used as the detection basis, so that it is possible to detect whether hallucinations occur when the visual language model performs model reasoning.

[0085] The cross-modal attention to visual tokens during model inference in the visual language model is explained as follows.

[0086] like Figure 2 FIG. 1 is a schematic diagram of a visual language model in one embodiment. The visual language model is a multimodal model that can learn from images and text. Figure 2 The visual language model includes an image encoding network (visual encoder), a modality alignment network (vision-language alignment module) and a large language model LLM (Large Language Module), which includes a text encoding network (text encoder) and a text decoding network (text decoder). It can be understood that Figure 2 What is shown is only the basic architecture of the visual language model. The visual language model can be any generative model that can understand images and texts, specifically LLaVA-1.5, InstructBLIP or Qwen-VL, etc. The embodiment of the present application does not limit the specific network used by the visual language model. For example, the image encoding network can be implemented based on any visual encoder, and the text encoding network can be implemented based on any text encoder.

[0087] Among them, the image encoding network is used to encode the image to obtain image encoding features. The text encoding network is used to embed the text (embedding encoder) to obtain text tokens. The modal alignment network is used to align the image representation and the text representation to convert the image encoding features into a format compatible with the large language model, namely visual tokens, which has the same dimension as the word embedding space in the large language model. The text decoding network is used to decode the text based on visual tokens and text tokens, complete specific image and text tasks, and obtain reasoning results. The text decoding network integrates the visual tokens into each layer of the language model with a cross-attention mechanism. The attention mechanism can help the model focus on key visual features, so as to understand and generate text more accurately, that is, when the text is decoded by the text decoding network in the large language model, the model can obtain the cross-modal attention of the visual token. The cross-modal attention in the embodiment of the present application refers to the attention of the text decoding network to the visual token in each layer when the visual language model decodes the text based on the visual text pair and the starting symbol to generate the output word (for example, it can be the first word). For example, one can randomly select a layer of the text decoding network of a large language model and take the maximum value on the multi-head attention dimension as the cross-modal attention for the visual token.

[0088] The visual language model shows different patterns in cross-modal attention of visual tokens when hallucination occurs and when hallucination does not occur. This phenomenon is observed in the following way. In order to obtain cross-modal attention, visual text pairs need to be prepared. First, a set of sample images is obtained, and a large number of visual text pairs are constructed based on the annotated objects of the sample image set and questions of the form "is there a / an{object} in the image?", that is, a sample image and a question constitute a visual text pair. The constructed visual text pairs are input into the visual language model respectively, and the reasoning results obtained by the visual language model for each visual text pair are obtained, that is, the model answer is {Yes} or {No}. At the same time, the cross-modal attention of the visual tokens of the visual text pair when performing model reasoning is obtained, which can be recorded as A(v, t), where v represents vision and t represents text. It can be understood that if the answer of the visual language model is accurate, it means that the model has not hallucinated, and if the answer of the visual language model is wrong, it means that the model has hallucinated. To this end, an association is established between the pattern of cross-modal attention A(v, t) and whether the model has hallucinations, so as to detect whether the model has hallucinations based on cross-modal attention.

[0089] Specifically, the cross-modal attention when the visual language model answers accurately is recorded as A, that is, the cross-modal attention when the model does not hallucinate is recorded as A, and the cross-modal attention when the visual language model answers incorrectly is recorded as AH, that is, the cross-modal attention when the model hallucinates is recorded as AH. In order to further explore the pattern differences of cross-modal attention when the model answers {Yes} and {No}, the cross-modal attention can be further subdivided according to the type of reasoning results of the model: the cross-modal attention when the visual language model answers {Yes} and the answer is accurate (no hallucination) is recorded as AY, the cross-modal attention when the visual language model answers {No} and the answer is accurate (no hallucination) is recorded as AN, the cross-modal attention when the visual language model answers {Yes} and the answer is incorrect (hallucination) is recorded as AYH, and the cross-modal attention when the visual language model answers {No} and the answer is incorrect (hallucination) is recorded as ANH.

[0090] like Figure 3 Shown is a schematic diagram of the obtained cross-modal attention. Figure 3 In the example, the size of the cross-modal attention A(v, t) is "32×32×32", which respectively represents 32 visual tokens, 32 linear layers, and 32 multi-head attention layers in the text decoding network of the large language model. That is, the number of visual tokens is 32 in total. Different visual language models may have different numbers of visual tokens, linear layers, and multi-head attention. Therefore, the size of the cross-modal attention A(v, t) depends on the model. Figure 3 As shown, the horizontal axis represents the 1st visual token to the 32nd visual token from left to right, and the vertical axis is 4000 visual-text pairs. A linear layer is randomly selected and the maximum value of the cross-modal attention dimension is taken for observation. Different colors and grayscale depths represent different values ​​of cross-modal attention. Figure 3 Part (a) is the cross-modal attention AY when the visual language model answers {Yes} to the input visual-text pair and the answer is accurate. Figure 3 Part (b) is the cross-modal attention AYH of the visual language model when the input visual-text pair is answered {Yes} and the answer is wrong. Figure 3 Part (c) of the figure is the cross-modal attention ANH when the visual language model answers {No} to the input visual text pair and the answer is wrong. In parts (a), (b) and (c), the images in the visual text pairs at the same ordinate position are the same.

[0091] Comparing parts (a) and (b), it can be clearly observed that there is a significant difference in cross-modal attention when hallucination occurs and when hallucination does not occur. For example, based on the 27th visual token, the difference in cross-modal attention of the 4,000 images as a whole is significant.

[0092] Comparing parts (b) and (c), we can clearly observe that when the model hallucinates, the cross-modal attention patterns are different when answering {Yes} and {No}, that is, the cross-modal attention patterns corresponding to different types of reasoning results are also different. For example, looking at the 32nd visual token, the overall difference in cross-modal attention for these 4,000 images is significant.

[0093] Based on the conclusions observed above, the embodiment of the present application constructs training samples for training detectors based on the categories of cross-modal attention, trains a first-stage hallucination detector, and during this stage of training, focuses on the high recall rate of hallucinated samples. Then, a second-stage hallucination detector is further trained for the hallucinated samples, and during this stage of training, focuses on the high accuracy of the hallucinated samples. The first-stage hallucination detector obtained after the training is cascaded with the second-stage hallucination detector to obtain a hallucination detection model, which is used to detect whether hallucinations occur in the visual language model when performing model inference. The accuracy of detecting whether hallucinations occur in the text language model can be guaranteed. At the same time, compared with the prior art, the hallucination detection model can directly detect whether hallucinations occur, with high detection efficiency and strong practicality.

[0094] The model hallucination detection method provided in the embodiment of the present application can be applied to Figure 4 In the application environment shown in the figure, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process, such as a set of sample images, visual text pairs generated based on the annotated objects of the sample images, cross-modal attention of different categories, etc. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0095] Reference Figure 4First, the server 104 may perform step 1: obtaining sample attention when there is hallucination or sample attention when there is no hallucination. Specifically, by obtaining the cross-modal attention of the visual token of the visual text pair when the visual language model performs model reasoning on the visual text pair, the cross-modal attention is divided into sample attention when there is hallucination or sample attention when there is no hallucination according to the reasoning result of the visual language model on the visual text pair. Next, the server 104 may perform step 2: training the hallucination detection model based on the sample attention. Specifically, the sample attention is input into the first-stage hallucination detector to obtain the first hallucination detection result of the first-stage hallucination detector on the sample attention, and the first-stage hallucination detector is updated according to the first hallucination detection result and the attention category of the sample attention; from the sample attention input into the first-stage hallucination detector, the sample attention with the attention category of hallucination is screened out, and the sample attention is continuously input into the second-stage hallucination detector to obtain the second hallucination detection result of the second-stage hallucination detector on the sample attention, and the second-stage hallucination detector is updated according to the second hallucination detection result and the sample category of the sample attention. Finally, the server performs step 3: obtaining the hallucination detection model. Specifically, the first-stage hallucination detector obtained after training is cascaded with the second-stage hallucination detector to obtain a hallucination detection model, which is used to detect whether hallucination occurs in the visual language model when performing model inference.

[0096] The obtained hallucination detection model can be deployed on the terminal 102 or the server 104 to detect the model hallucination. For example, the visual language model and the hallucination detection model are both deployed on the server 104, and the server 104 can be a background server that supports image-based chat, image recognition according to instructions, visual question and answer, document understanding, and image description. Figure 4 , the terminal 102 obtains the text and image input by the user, and sends the input text and image to the server 104. The server 104 calls the visual language model to perform model reasoning and obtains the reasoning result. At the same time, the server 104 obtains the cross-modal attention when performing model reasoning, and inputs the cross-modal attention into the above-mentioned hallucination detection model to obtain the hallucination detection result to determine whether the hallucination occurs when performing model reasoning. Based on the hallucination detection result, the server 104 can make a decision on the reasoning result. Specifically, if it is detected that no hallucination occurs, it means that the model reasoning is accurate. The server 104 can return the reasoning result directly to the terminal 102. If a hallucination is detected, in a simple binary question-answering scenario, the server 104 can flip the reasoning result and return the flipped result as the final answer to the terminal 102. It should be noted that the terminal 102 can also execute the model hallucination detection method provided in the embodiment of the present application.

[0097] There are many application scenarios for visual language models. The model hallucination detection method provided in the embodiments of the present application can be applied to these application scenarios to realize hallucination detection of visual language models.

[0098] For example, in an application scenario where a visual language model is applied to visual binary question answering, assume that a user inputs a visual text pair through a terminal. The visual text pair includes a picture and a question, and the question is "is there a cat in the image?" The server that receives the visual text pair can call the visual language model to perform model reasoning based on the visual text pair to obtain a reasoning result. Assuming that the reasoning result of the visual language model is {Yes}, the server obtains the cross-modal attention A(v, t) of the visual token during model reasoning, inputs the cross-modal attention into the hallucination detection model, and obtains the hallucination detection result. When the detection result indicates that the model has not hallucinated, the reasoning result {Yes} can be directly returned to the terminal used by the user. When the detection result indicates that the model has hallucinated, the reasoning result is flipped to {No} and returned to the terminal used by the user to alleviate the model hallucination.

[0099] To give another example, in an application scenario where a visual language model is applied to image description generation, assume that a user inputs a visual text pair through a terminal. The visual text pair includes a picture and a text instruction, and the text instruction is "Start with 'In the picture we can see...' to describe the input image". The server that receives the visual text pair can call the visual language model to perform model inference based on the visual text pair to obtain an inference result. At the same time, the server obtains the cross-modal attention A(v, t) of the visual token when performing model inference, and inputs the cross-modal attention into the hallucination detection model to obtain an hallucination detection result. When the detection result indicates that the model has not hallucinated, the inference result can be directly returned to the terminal used by the user. When the detection result indicates that the model has hallucinated, the inference result and a prompt message such as "The generated content may not be accurate and is for reference only" can be returned to alleviate the model hallucination.

[0100] In one embodiment, the server 104 inputs the cross-modal attention into the above-mentioned hallucination detection model, and the specific steps of obtaining the hallucination detection result include: inputting the cross-modal attention into the hallucination detection model, and outputting the first hallucination detection result through the first-stage hallucination detector; when the first hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning, inputting the cross-modal attention into the second-stage hallucination detector to obtain the second hallucination detection result; when the second hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning, it is determined that the visual language model has hallucinations when performing model reasoning on the target visual text pair. When the first hallucination detection result indicates that the visual language model has not hallucinated when performing model reasoning, or when the second hallucination detection result indicates that the visual language model has not hallucinated when performing model reasoning, it is determined that the visual language model has not hallucinated when performing model reasoning on the target visual text pair. In this way, it is possible to achieve efficient and accurate detection of whether the visual language model has hallucinations when performing model reasoning based on cross-modal attention.

[0101] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0102] In an exemplary embodiment, Figure 5 As shown, a method for detecting model hallucination is provided, and the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps 502 to 508. Among them:

[0103] Step 502, obtaining the cross-modal attention of the visual tokens of the visual text pair when the visual language model performs model inference on the visual text pair, and dividing the cross-modal attention into sample attention when there is hallucination or sample attention when there is no hallucination according to the inference result of the visual language model on the visual text pair.

[0104] In the embodiment of the present application, a visual-text pair refers to the input of a visual language model, including an image and a text. In different application scenarios, the visual-text pair has different forms or contents. For example, in the application scenario of visual answering, the visual-text pair includes an image and a question in the form of text, such as "is there a cat in the input image?". In the application scenario of image recognition based on instructions, the visual-text pair includes an image and an instruction in the form of text, such as "how many animals are in the garden?".

[0105] As mentioned earlier, the visual language model has the ability to process visual information and natural language understanding by combining a pre-trained visual encoder with a language model. It can be used to understand and interpret the relationship between images and text, and generate accurate and vivid natural language descriptions based on images. In a nutshell, the visual language model can perform model reasoning based on the input visual text pairs to obtain reasoning results. During the model reasoning process, the visual language model fuses the visual tokens of the visual text pairs into the layers of the language model with a cross-attention mechanism. This attention mechanism helps the model focus on key visual features, thereby more accurately understanding and generating text. The server can obtain the cross-modal attention of the visual language model to the visual tokens during this model reasoning process.

[0106] Specifically, the server can obtain a large number of visual-text pairs. In the process of inputting each visual-text pair into the visual language model for model inference to obtain the inference result, the server can obtain the cross-modal attention of the visual language model to the visual token. These cross-modal attentions will be used to construct the training samples of the hallucination detection model. Each visual-text pair obtained has its corresponding annotation information (which can be used to determine the standard result). According to the annotation information, it can be determined whether the obtained inference result is correct, that is, whether hallucination occurs during the model inference process. Therefore, according to the inference result, the cross-modal attention can be divided into sample attention when there is hallucination or sample attention when there is no hallucination.

[0107] Step 504: input the sample attention into the first-stage hallucination detector to obtain a first hallucination detection result of the first-stage hallucination detector on the sample attention, and update the first-stage hallucination detector according to the first hallucination detection result and the attention category of the sample attention.

[0108] After obtaining cross-modal attention based on visual-text pairs and dividing the cross-modal attention into sample attention with hallucination or sample attention without hallucination according to whether hallucination occurs during model inference with the visual language model, these sample attentions can be used as training samples of known attention categories to train the hallucination detection model.

[0109] Considering that the accuracy of the visual language model is guaranteed to a certain extent, the cases where hallucinations occur are rare compared to the cases where hallucinations do not occur, and the sample attention used as training samples may have data imbalance. Therefore, in order to take into account both the recall rate and accuracy of hallucination detection, in the embodiment of the present application, the designed hallucination detection model includes a first-stage hallucination detector and a second-stage hallucination detector. The first-stage hallucination detector is used to preliminarily classify the sample attention to recall the sample attention when there is a hallucination, and the second-stage hallucination detector is used to further re-detect the sample attention when there is a hallucination to ensure the accuracy of classifying the sample attention when there is a hallucination. Both the first-stage hallucination detector and the second-stage hallucination detector can be implemented based on neural networks, for example, both can be based on MLP (Multi-Layer Perceptron).

[0110] Specifically, for each sample attention obtained in the aforementioned steps, knowing its attention category (sample attention with hallucination or sample attention without hallucination), the server inputs these sample attentions into the first-stage hallucination detector respectively, and obtains the first hallucination detection result of the first-stage hallucination detector for each sample attention, and updates the first-stage hallucination detector according to the difference between the first hallucination detection result and the attention category of the sample attention. Updating refers to adjusting the model parameters according to new data and feedback information during the training process to improve the model performance. The embodiment of the present application does not limit the model updating strategy adopted. For example, iterative optimization algorithms such as gradient descent can be used.

[0111] The hallucination detection result refers to the detection result obtained by the hallucination detector for the input sample attention. Its essence is the category prediction result obtained by the hallucination detector for the input sample attention. The category prediction result may include the probability that the sample attention belongs to each attention category, which is used to obtain the predicted attention category of the sample attention. For example, if the probability that the sample attention belongs to the sample attention with hallucination and the sample attention without hallucination are 0.9 and 0.1 respectively, then the attention category predicted by the first stage hallucination detector for the sample attention is the sample attention with hallucination. For another example, if the probability that the sample attention belongs to the sample attention with hallucination and the sample attention without hallucination are 0.15 and 0.85 respectively, then the attention category predicted by the first stage hallucination detector for the sample attention is the sample attention without hallucination.

[0112] like Figure 6 FIG. 1 is a schematic diagram of a first-stage hallucination detector in one embodiment. Figure 6 , Figure 6The input of the first-stage hallucination detector Stage-1 MLP (model parameter is denoted as c1) can be the sample attention when there is hallucination, denoted as AH, or the sample attention when there is no hallucination, denoted as A. For the input AH, the corresponding hallucination detection result may be True AH or False AH, and for the input A, the corresponding hallucination detection result may be True A or False A. In order to improve the recall rate of the first-stage hallucination detector, the loss function can be calculated according to the difference between the hallucination detection result and the attention category of the sample attention, and the first-stage hallucination detector is updated according to the loss function.

[0113] Step 506, filter out the sample attention whose attention category is hallucination from the sample attention input to the first-stage hallucination detector, continue to input the sample attention to the second-stage hallucination detector, obtain the second-stage hallucination detector's second hallucination detection result for the sample attention, and update the second-stage hallucination detector based on the second hallucination detection result and the sample category of the sample attention.

[0114] In the second stage, attention is paid to the accurate detection of sample attention when there is hallucination. To this end, the server screens out sample attention whose attention category is hallucination from the sample attention input into the first stage hallucination detector in step 506. For the screened sample attention, the server continues to input the second stage hallucination detector to obtain the second hallucination detection result of the second stage hallucination detector for each sample attention, and updates the second stage hallucination detector according to the difference between the second hallucination detection result and the attention category of the sample attention.

[0115] like Figure 7 FIG. 1 is a schematic diagram of a second stage hallucination detector in one embodiment. Figure 7 , Figure 7 The input of the second-stage hallucination detector Stage-2 MLP (model parameter is denoted as c2) is the sample attention when there is hallucination, denoted as AH. For the input AH, the corresponding hallucination detection result may be True AH or False AH. In order to improve the accuracy of the second-stage hallucination detector, the loss function can be calculated based on the difference between the hallucination detection result and the attention category of the sample attention, and the second-stage hallucination detector can be updated according to the loss function.

[0116] Step 508, cascading the first stage hallucination detector obtained after training with the second stage hallucination detector to obtain a hallucination detection model, which is used to detect whether hallucination occurs when the visual language model performs model inference.

[0117] Specifically, through the aforementioned steps, a large number of cross-modal attentions of known attention categories are used to update the first-stage hallucination detector and the second-stage hallucination detector respectively, until the training is completed, the obtained first-stage hallucination detector and the second-stage hallucination detector are cascaded to obtain a hallucination detection model, which can detect whether the visual language model has hallucinations when performing model reasoning. The condition for the completion of training can be when the loss function drops to a relatively stable value and no longer fluctuates significantly, it can be considered that the training has converged and the training can be stopped, or it can be that the prediction results of the hallucination detector do not change significantly in the attention of several consecutive samples, it can be considered that the training has converged and the training can be stopped.

[0118] In one example, Figure 8 As shown, the step of using the trained hallucination detection model to detect whether hallucination occurs in the visual language model during the model reasoning process specifically includes the following steps 802 to 808:

[0119] Step 802, obtaining a target inference result obtained by performing model inference on a target visual text pair by a visual language model, where the target visual text pair includes a target image and a target text, and the target text is used to inquire whether a target image includes an object.

[0120] When the visual language model is applied in different scenarios, a variety of visual-text pairs can be obtained, all of which can be the target visual-text pairs mentioned here. The target visual-text pairs are provided as input to the visual language model to obtain the target reasoning results.

[0121] Step 804, obtain the cross-modal attention of the visual language model to the visual token of the target visual text pair, and input the cross-modal attention into the hallucination detection model. The hallucination detection model is obtained by cascading the first-stage hallucination detector and the second-stage hallucination detector obtained after the training is completed, and the first-stage hallucination detector outputs the first hallucination detection result.

[0122] In order to avoid hallucinations and inaccurate reasoning results when the visual language model performs model reasoning, the server can extract the cross-modal attention of the visual language model to the visual tokens of the target visual text pair, input the cross-modal attention into the trained hallucination detection model, and output the first hallucination detection result through the first-stage hallucination detector in the hallucination language model.

[0123] Step 806, when the first hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning, the cross-modal attention is input into the second-stage hallucination detector to obtain a second hallucination detection result.

[0124] Since the accuracy of the visual language model is guaranteed to a certain extent, when the first hallucination detection result indicates that the visual language model does not hallucinate when performing model reasoning, to a large extent, no hallucination has occurred. Only when the first hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning, it is necessary to continue to input cross-modal attention into the second-stage hallucination detector.

[0125] In one of the examples, the hallucination detection step further includes: when the first hallucination detection result indicates that the visual language model does not hallucinate when performing model reasoning on the target visual text pair, determining that the visual language model does not hallucinate when performing model reasoning on the target visual text pair.

[0126] Step 808 , when the second hallucination detection result indicates that the visual language model hallucinates when performing model reasoning, determine that the visual language model hallucinates when performing model reasoning on the target visual-text pair.

[0127] In one example, Figure 8 The hallucination detection step shown also includes: when the second hallucination detection result indicates that the visual language model does not hallucinate when performing model reasoning on the target visual text pair, determining that the visual language model does not hallucinate when performing model reasoning on the target visual text pair.

[0128] In this embodiment, a first hallucination detection result is outputted by a first-stage hallucination detector. When the first hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning, the cross-modal attention is further inputted into the second-stage hallucination detector to obtain a second hallucination detection result. When the second hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning, it can be determined that the visual language model has hallucinations when performing model reasoning on the target visual text pair. In this way, efficient and accurate detection of whether the visual language model has hallucinations when performing model reasoning can be achieved based on cross-modal attention.

[0129] In one example, the method further includes:

[0130] When it is determined that the visual language model has hallucinations when performing model reasoning on the target visual text pair, the target reasoning result is flipped, and the flipped reasoning result is used as the reasoning result corresponding to the target visual text.

[0131] Specifically, since it is determined that the visual language model hallucinates when performing model reasoning on the target visual text pair, it means that the target reasoning result obtained by the visual language model for model reasoning on the target visual text pair is inaccurate. In this case, the server can flip the target reasoning result and use the flipped reasoning result as the reasoning result of the corresponding target visual text. For example, in a simple binary answer application scenario (the model only needs to answer yes {Yes} or no {No}), if it is detected that the visual language model hallucinates, then if the reasoning result is {Yes}, it can be flipped to {No}, and if the reasoning result is {No}, it can be flipped to {Yes}. This operation can alleviate the model hallucination.

[0132] like Fig. 9 FIG. 1 is a schematic diagram of using a hallucination detection model to detect and mitigate hallucinations in one embodiment. Fig. 9 , the input is the cross-modal attention A(v, t) extracted by the visual language model when performing model reasoning on the target visual text pair, which is detected by the first-stage hallucination detector to obtain the first hallucination detection result, and the first hallucination detection result indicates that the attention category of the cross-modal attention A(v, t) is AH or A. If it is AH, the extracted cross-modal attention A(v, t) is further detected by the second-stage hallucination detector to obtain the second hallucination detection result. If it is A, it is determined that the visual language model does not hallucinate when performing model reasoning on the target visual text pair, and the final classification result is A. The second hallucination detection result indicates that the attention category of the cross-modal attention A(v, t) is AH or A. If it is AH, the final classification result is AH. If it is A, it is determined that the visual language model does not hallucinate when performing model reasoning on the target visual text pair, and the final classification result is A. In the binary question answering scenario, when the final classification result is AH, the obtained target reasoning result can be flipped. In other application scenarios, when the final classification result is AH, the target inference result can be prompted, and prompt information such as "the generated content may not be accurate and is for reference only" can be displayed to alleviate model hallucinations.

[0133] In the above-mentioned hallucination detection method, since it is noted that the visual language model shows differences in cross-modal attention when hallucination occurs and when hallucination does not occur, firstly, by obtaining the cross-modal attention of the visual tokens of the visual text pair when the visual language model performs model reasoning on the visual text pair, the cross-modal attention is divided into sample attention when there is a hallucination or sample attention when there is no hallucination according to the reasoning result of the visual language model on the visual text pair, so as to construct a training sample for training the detector. Then, by inputting the sample attention into the first-stage hallucination detector, the first-stage hallucination detector obtains the first hallucination detection result of the sample attention, and according to the first hallucination detection result and the attention category of the sample attention, the first-stage hallucination detector is updated to achieve a high recall rate of attention to the hallucinated samples during the first-stage training process. Next, from the sample attention input to the first-stage hallucination detector, the sample attention with the attention category of hallucination is screened out, and the attention is continuously input to the second-stage hallucination detector for more detailed classification. By obtaining the second hallucination detection result of the sample attention by the second-stage hallucination detector, the second-stage hallucination detector is updated according to the second hallucination detection result and the sample category of the sample attention, so as to achieve a high detection rate of hallucinated samples during the training process of the second stage. Then, the first-stage hallucination detector obtained after the training is cascaded with the second-stage hallucination detector to obtain a hallucination detection model, which is used to detect whether hallucination occurs in the visual language model during model inference, and can ensure the accuracy of detecting whether hallucination occurs in the text language model. At the same time, compared with the existing technology, the hallucination detection model can directly detect whether hallucination occurs, with high detection efficiency and strong practicality.

[0134] In one embodiment, the step of obtaining cross-modal attention of visual tokens of a visual-text pair when a visual-language model performs model reasoning on a visual-text pair specifically includes:

[0135] A visual-text pair is obtained, where the visual-text pair includes an image and a text, and the text is used to inquire whether an object is included in the image; a visual token is obtained based on the image through a visual language model, a text token is obtained based on the text, and cross-modal attention of the visual token is obtained based on the visual token and the text token.

[0136] For example, refer to Figure 2 The visual language model includes an image encoding network, a modality alignment network, and a large language model, and the large language model includes a text encoding network and a text decoding network. In an exemplary embodiment, Fig.10 As shown, obtaining visual tokens based on images and text tokens based on texts through a visual language model, and obtaining cross-modal attention of visual tokens based on visual tokens and text tokens specifically includes the following steps 1002 to 1008:

[0137] Step 1002: Encode the image through the image encoding network in the visual language model to obtain image encoding features.

[0138] The image coding network of the visual language model is used to encode the image to obtain image coding features. The image coding network can be implemented using any visual encoder, such as the CLIP model.

[0139] Step 1004, modality alignment of the image encoding features is performed through the modality alignment network of the visual language model to obtain visual tokens of the visual-text pair.

[0140] Among them, the modality alignment network, also known as the vision-language alignment module or visual tokenization network or vision-language adapter, is used to align image representation and text representation to convert image encoding features into a format compatible with the large language model, that is, to convert an image into a series of visual tokens, which has the same dimension as the word embedding space in the large language model. The visual token is the result of expressing the image encoding features in a way that is easier for the text decoding network to understand.

[0141] Step 1006, embedding the text through the text encoding network of the visual language model to obtain the text token of the visual text pair.

[0142] Among them, the text encoding network is used to embed the text (embedding encoder) to obtain text tokens.

[0143] Step 1008, through the text decoding network of the visual language model, text decoding is performed based on the cross-modal features obtained by splicing the visual token and the text token, to obtain the inference result of the visual language model on the visual text pair, and obtain the cross-modal attention of the visual language model to the visual token of the visual text pair when outputting the inference result.

[0144] In the above embodiment, cross-modal attention to visual tokens is obtained during the model reasoning process so that it is possible to efficiently and accurately detect whether hallucinations occur when the visual language model is reasoned.

[0145] In one embodiment, in order to obtain the model's cross-modal attention to the visual tokens of the visual-text pair to construct the sample attention, such as Fig.11As shown, the method provided in the embodiment of the present application further includes the step of constructing a visual text pair, specifically including the following steps 1102 to 1106:

[0146] Step 1102 , obtaining a sample image set, and for each sample image in the sample image set, obtaining a labeled object of the sample image, where the labeled object refers to an object existing in the sample image.

[0147] The server can obtain sample images from any image dataset to obtain a sample image set. The image dataset can be, for example, the COCO image dataset. The COCO image dataset is a large-scale dataset that can be used for image detection, semantic segmentation, and image caption generation. It contains a large number of annotated images. The annotation information includes objects, categories, image description sentences, etc. in the image. The server can obtain sample images from these images to form a sample image set.

[0148] Exemplarily, the server can obtain images including no less than 3 objects from these images as sample images, and obtain annotation information of these sample images. The annotation information may include objects, categories, image description statements, etc. existing in the sample images. In this embodiment, the annotated objects of the sample images are objects existing in the sample images recorded in the annotation information.

[0149] Step 1104 , obtaining a text template, where the text template is used to inquire whether the image includes an object.

[0150] The text template refers to a template used to construct the text in the visual text pair. For example, the text template can be a question in the form of "is there a / an {object} in the image?", where {object} is an object name. It can be understood that this embodiment is an example, and a visual text pair is constructed by constructing a text template based on the annotated objects existing in the sample image. The server can set the text template according to actual needs. The set text template can be a text question about the image, or a text instruction about the image, etc.

[0151] Step 1106, generating a query text according to the annotated object and the text template of the sample image, and obtaining a visual-text pair according to the sample image and the query text.

[0152] In one of the examples, a query text is generated based on the annotated objects and text templates of a sample image, and a visual text pair is obtained based on the sample image and the query text, including: generating a query text based on the annotated objects and text templates of the sample image, wherein one query text is generated for one annotated object, and the query text is used to inquire whether the sample image includes the annotated object; and constructing a visual text pair as a positive example based on the sample image and the query text.

[0153] In one of the examples, a query text is generated based on the annotated objects and text template of a sample image, and a visual text pair is obtained based on the sample image and the query text, including: determining a negative example object of the sample image based on the annotated objects of the sample image, where the negative example object is an object that does not exist in the sample image; generating a query text based on the negative example object and text template of the sample image, wherein one query text is generated for one negative example object, and the query text is used to inquire whether the sample image includes the negative example object; and constructing a visual text pair as a negative example based on the sample image and the query text.

[0154] Specifically, generating a query text based on the annotated objects and text templates of the sample image can include two aspects. On the one hand, generating a query text based on the annotated objects and text templates of the sample image; on the other hand, determining a negative example object based on the annotated objects of the sample image, where the negative example object is an object that does not exist in the sample image, and generating a query text based on the negative example object and text templates of the sample image.

[0155] For example, the labeled objects of sample image A are "vase", "cat", and "table". Based on these labeled objects, three different query texts can be generated, thereby obtaining three visual text pairs as positive examples. Then, three negative example objects, such as "water-bottle", "dog", and "piano", can be determined based on these labeled objects. Based on these negative example objects, three different query texts can be generated, thereby obtaining three visual text pairs as negative examples. Furthermore, depending on whether {object} is located in the sample image, the standard result corresponding to the visual text pair is determined to be {Yes} or {No}, that is, the standard result of the visual text pair as a positive example is {Yes}, and the standard result of the visual text pair as a negative example is {No}.

[0156] In the above example, the six visual-text pairs and their corresponding standard results are:

[0157] Visual-text pair 1: sample image A and “is there a vase in the image?”, standard result {Yes}.

[0158] Visual-text pair 2: sample image A and “is there a cat in the image?”, standard result {Yes}.

[0159] Visual-text pair 3: sample image A and “is there a table in the image?”, standard result {Yes}.

[0160] Visual-text pair 4: sample image A and “is there a water-bottle in the image?”, standard result {No}.

[0161] Visual-text pair 5: sample image A and “is there a dog in the image?”, standard result {No}.

[0162] Visual-text pair 6: sample image A and “is there a piano in the image?”, standard result {No}.

[0163] It can be understood that this embodiment is only an example, and a text template is constructed based on the annotated objects in the sample image to construct a visual text pair. In other embodiments, the server can also construct other forms of text templates based on other aspects of the annotation information of the sample image. For example, assuming that the annotation information of the sample image includes the position information of the pedestrian in the sample image (left side\middle\right side), the constructed text template can also be in the form of "Which side of the image is the person on?", so that a visual text pair can be constructed based on the text template and the sample image, and according to the position information of the pedestrian in the sample image, the standard result corresponding to the constructed visual text pair is determined to be (left side\middle\right side).

[0164] In the above embodiment, by constructing a text template based on the annotated objects existing in the sample image, thereby constructing a visual-text pair and its corresponding standard result, the visual-text pair used to observe the cross-modal attention of the visual language model can be obtained quickly and accurately.

[0165] In order to obtain the cross-modal attention of the visual language model to the visual tokens in various situations, to characterize the attention pattern of the visual language model when hallucination occurs, and thus to improve the reliability and accuracy of the subsequent hallucination detection model, the following strategy can be adopted when determining the negative example objects to construct the visual-text pairs as negative examples.

[0166] In one embodiment, determining a negative example object of the sample image according to the annotated object of the sample image includes: determining multiple negative example objects of the sample image according to the annotated object of the sample image, obtaining a negative example object set of the sample image, and randomly sampling the negative example object set to obtain a negative example object as the negative example object of the sample image. In this embodiment, the multiple negative example objects can be any objects other than the annotated object of the sample image, which can ensure the diversity of the constructed visual text pairs.

[0167] In one embodiment, determining the negative example object of the sample image according to the annotated object of the sample image includes: obtaining the annotated objects of other sample images other than the sample image in the sample image set, filtering out the negative example objects of the sample image from the annotated objects of other sample images, obtaining the negative example object set of the sample image, and randomly sampling the negative example object set to obtain a negative example object as the negative example object of the sample image. In this embodiment, the negative example object is an annotated object of other sample images that does not belong to the annotated object of the current sample image.

[0168] In one embodiment, the negative object of the sample image is determined according to the annotated object of the sample image, including: determining the co-occurrence object of the annotated object of the sample image, the co-occurrence object is an object that does not exist in the sample image and has a co-occurrence relationship with the annotated object, obtaining a set of negative object of the sample image according to the co-occurrence object, and randomly sampling the set of negative object to obtain a negative object as the negative object of the sample image. In this embodiment, the negative object is an object that has a co-occurrence relationship with the annotated object of the sample image and does not exist in the sample image. The co-occurrence relationship between two objects can be understood as the two objects having a correlation. The correlation can mean that the two objects usually appear in the same picture at the same time, for example, a bowl and chopsticks, a plate and a table, a car and a street lamp, etc. The correlation can also mean that the two objects have similarities, for example, an eagle and a seagull flying in the sky, a duck and a platypus, a car and a car model, etc.

[0169] In a specific embodiment, for a sample image, the server can construct three visual-text pairs as positive examples based on three different objects existing in the image, obtain three negative objects according to the above three methods respectively, and construct three visual-text pairs as negative examples. Thus, six visual-text pairs can be constructed based on a sample image to meet the need of the visual language model to observe the cross-modal attention of the same sample image in different situations.

[0170] In an optional embodiment, whether there is a co-occurrence relationship between two objects can be determined through the following steps: obtaining the annotated object of each sample image in the sample image set; based on the sample image set, counting the co-occurrence degree between any two annotated objects, wherein two different annotated objects appearing simultaneously in the same sample image are regarded as a co-occurrence; when the co-occurrence degree of two different annotated objects meets the set conditions, determining that the two different annotated objects have a co-occurrence relationship.

[0171] In this embodiment, the co-occurrence degree can be the number of co-occurrences between any two annotated objects, or the ratio of the number of co-occurrences between any two annotated objects to the number of occurrences. For example, the sample image set includes n sample images, and the annotated object O1 and the annotated object O2 co-appear in m of the sample images, then the number of co-occurrences is m times, the annotated object O1 appears in m1 sample images, and the annotated object O2 appears in m2 sample images. The intersection of the m1 sample images and the m2 sample images has a total of m images, and the union has a total of u sample images. The co-occurrence degree can be the number of co-occurrences, or the ratio of the intersection to the union, that is, m / u. When the co-occurrence degree of two different annotated objects is greater than a set threshold, it is determined that the two different annotated objects have a co-occurrence relationship.

[0172] In an optional embodiment, the co-occurrence objects of the annotated objects of the sample images can be obtained in the following manner: obtain the annotated objects of each sample image in the sample image set, obtain the image features of the annotated objects, calculate the similarity between any two different annotated objects based on the image features, and determine that the two different annotated objects have a co-occurrence relationship when the similarity meets the set conditions.

[0173] In the above embodiments, negative example objects of sample images are obtained in various ways, so that various visual-text pairs can be constructed based on the sample images. Based on these visual-text pairs, the attention pattern of the visual language model when hallucinations occur can be characterized, thereby improving the reliability and accuracy of the subsequent hallucination detection model.

[0174] In one of the embodiments, the step of dividing the cross-modal attention into sample attention when there is hallucination or sample attention when there is no hallucination according to the inference result of the visual language model on the visual text pair specifically includes: obtaining the standard result corresponding to the visual text pair; when the standard result is inconsistent with the inference result of the visual language model on the visual text pair, dividing the cross-modal attention into sample attention when there is hallucination; when the standard result is consistent with the inference result of the visual language model on the visual text pair, dividing the cross-modal attention into sample attention when there is no hallucination.

[0175] Among them, the standard result is the accurate content that the visual language model should generate. If the standard result is inconsistent with the inference result, it means that the inference result is wrong. The server can mark the cross-modal attention corresponding to the wrong inference result as the sample attention when there is hallucination, recorded as AH. If the standard result is consistent with the inference result, it means that the inference result is accurate. The server can mark the cross-modal attention corresponding to the accurate inference result as the sample attention when there is no hallucination, recorded as A.

[0176] For example, in the binary question-answering scenario mentioned above, the standard result of the visual-text pair is the accurate answer that the visual language model should answer, that is, one of {Yes} and {No}. The reasoning result of the visual language model is also one of {Yes} and {No}. For each visual-text pair, the reasoning result of the visual language model is compared with the standard answer to determine whether the visual language model has hallucinations when performing model reasoning on the visual-text pair, thereby dividing the acquired cross-modal attention into two categories.

[0177] For example, in the visual question-answering scenario, the standard result of the visual-text pair is the accurate answer that the visual language model should answer. Assuming that the text in the visual-text pair is "Which side of the image is the personon", then the corresponding standard answer is one of "left side\middle\right side", and the reasoning result of the visual language model may also be one of "left side\middle\right side". For each visual-text pair, the reasoning result of the visual-language model is compared with the standard answer to determine whether the visual-language model has hallucinations when performing model reasoning on the visual-text pair, thereby dividing the acquired cross-modal attention into two categories.

[0178] In order to further explore the differences in the patterns of cross-modal attention when the visual language model gives different answers, the cross-modal attention can be further subdivided according to the type (value) of the model's reasoning results. In one embodiment, based on the reasoning results of the visual language model on the visual text pair, the cross-modal attention is divided into sample attention with hallucination or sample attention without hallucination, including: obtaining the reasoning results of the visual language model on the visual text pair, the reasoning results are one of the first type of reasoning results and the second type of reasoning results; based on the type of reasoning results and the annotation information of the visual text pair, the cross-modal attention is divided into the first type of sample attention with hallucination, the second type of sample attention with hallucination, the first type of sample attention without hallucination, or the second type of sample attention without hallucination.

[0179] Still taking the binary question-answering scenario as an example: the cross-modal attention is further subdivided according to the type (value) of the model's reasoning results: the cross-modal attention when the visual language model answers {Yes} and the answer is accurate (no hallucination) is denoted as AY, the cross-modal attention when the visual language model answers {No} and the answer is accurate (no hallucination) is denoted as AN, the cross-modal attention when the visual language model answers {Yes} and the answer is wrong (hallucination) is denoted as AYH, and the cross-modal attention when the visual language model answers {No} and the answer is wrong (hallucination) is denoted as ANH.

[0180] It is understandable that in different question-answering scenarios, the types of reasoning results may not be limited to the first and second types of reasoning results, but may also be three or even four types, and so on. According to the type of reasoning results in the actual scenario, cross-modal attention can be further subdivided into six or eight attention categories.

[0181] Taking the example that there are two types of inference results, including first-category inference results and second-category inference results, the server can divide the second-stage hallucination detector into two types: first-category second-stage hallucination detector and second-category second-stage hallucination detector. The obtained cross-modal attention can be input into the corresponding category of the second-stage hallucination detector for hallucination detection according to the type of inference result.

[0182] In one embodiment, step 506 may include: from the sample attention input to the first-stage hallucination detector, screening out the sample attention whose attention category is the first category with hallucination, continuing to input the first-category second-stage hallucination detector, obtaining the second hallucination detection result of the first-category second-stage hallucination detector on the sample attention, and updating the first-category second-stage hallucination detector according to the second hallucination detection result and the sample category of the sample attention; from the sample attention input to the first-stage hallucination detector, screening out the sample attention whose attention category is the second category with hallucination, continuing to input the second-category second-stage hallucination detector, obtaining the second hallucination detection result of the second-category second-stage hallucination detector on the sample attention, and updating the second-category second-stage hallucination detector according to the second hallucination detection result and the sample category of the sample attention.

[0183] Accordingly, step 508 may include: cascading the first-stage hallucination detector obtained after the training is completed with the first type of second-stage hallucination detector to obtain a hallucination detection model, the hallucination detection model is used to detect whether hallucinations occur when the visual language model performs model inference to obtain a first type of reasoning result; and cascading the first-stage hallucination detector obtained after the training is completed with the second type of second-stage hallucination detector to obtain a hallucination detection model, the hallucination detection model is used to detect whether hallucinations occur when the visual language model performs model inference to obtain a second type of reasoning result.

[0184] like Fig.12 FIG. 1 is a schematic diagram of a first-stage hallucination detector in one embodiment. Fig.12 , in the binary question-answering scenario, Fig.12 The input of the first-stage hallucination detector Stage-1 MLP (model parameter is denoted as c1) can be the cross-modal attention when the answer is {Yes} and the answer is accurate (no hallucination), denoted as AY, the cross-modal attention when the answer is {No} and the answer is accurate (no hallucination), denoted as AN, the cross-modal attention when the answer is {Yes} and the answer is wrong (hallucination), denoted as AYH, and the cross-modal attention when the answer is {No} and the answer is wrong (hallucination), denoted as ANH. For each sample attention, the corresponding hallucination detection result may be True or False. In order to improve the recall rate of the first-stage hallucination detector, the loss function can be calculated according to the difference between the hallucination detection result and the attention category of the sample attention, and the first-stage hallucination detector can be updated according to the loss function.

[0185] like Fig.13 FIG. 1 is a schematic diagram of a first type second stage hallucination detector in one embodiment. Fig.13 , Fig.13 The input of the first-class second-stage hallucination detector Stage-2 MLP (model parameter is denoted as c2-YH) is the cross-modal attention AYH when the answer is {Yes} and the answer is wrong (hallucination occurs). For the input AYH, the corresponding hallucination detection result may be True AYH or False AYH. In order to improve the accuracy of the first-class second-stage hallucination detector, the loss function can be calculated according to the difference between the hallucination detection result and the attention category of the sample attention, and the first-class second-stage hallucination detector is updated according to the loss function.

[0186] like Fig.14 FIG. 1 is a schematic diagram of a second type second stage hallucination detector in one embodiment. Fig.14 , Fig.14The input of the second-class second-stage hallucination detector Stage-2 MLP (model parameter is denoted as c2-NH) is the cross-modal attention ANH when the answer is {No} and the answer is wrong (hallucination occurs). For the input ANH, the corresponding hallucination detection result may be True ANH or False ANH. In order to improve the accuracy of the second-class second-stage hallucination detector, the loss function can be calculated according to the difference between the hallucination detection result and the attention category of the sample attention, and the second-class second-stage hallucination detector can be updated according to the loss function.

[0187] Based on this, in an exemplary embodiment, Fig.15 As shown, another method for detecting model hallucination is also provided, which is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following steps 1502 to 1510. Among them:

[0188] Step 1502, obtaining cross-modal attention of the visual tokens of the visual-text pair when the visual-language model performs model reasoning on the visual-text pair;

[0189] Step 1504, obtaining the inference result of the visual language model on the visual text pair, where the inference result is one of the first type of inference result and the second type of inference result;

[0190] Step 1506, according to the type of the inference result and the annotation information of the visual-text pair, the cross-modal attention is divided into the first type of sample attention when there is hallucination, the second type of sample attention when there is hallucination, the first type of sample attention when there is no hallucination, or the second type of sample attention when there is no hallucination;

[0191] Step 1508A, from the sample attention input to the first stage hallucination detector, filter out the sample attention whose attention category is the first category with hallucination, continue to input the first category second stage hallucination detector, obtain the second hallucination detection result of the first category second stage hallucination detector on the sample attention, and update the first category second stage hallucination detector according to the second hallucination detection result and the sample category of the sample attention;

[0192] Step 1508B, from the sample attention input to the first stage hallucination detector, filter out the sample attention whose attention category is the second category with hallucination, continue to input the second category second stage hallucination detector, obtain the second hallucination detection result of the second category second stage hallucination detector on the sample attention, and update the second category second stage hallucination detector according to the second hallucination detection result and the sample category of the sample attention;

[0193] Step 1510A, cascading the first-stage hallucination detector obtained after training with the first-type second-stage hallucination detector to obtain a hallucination detection model, where the hallucination detection model is used to detect whether hallucination occurs when the visual language model performs model inference to obtain the first-type inference result;

[0194] Step 1510B, cascade the first-stage hallucination detector obtained after training with the second-type second-stage hallucination detector to obtain a hallucination detection model. The hallucination detection model is used to detect whether hallucination occurs when the visual language model performs model inference to obtain the second type of inference result.

[0195] Based on this, in an exemplary embodiment, Fig.16 As shown, another method for detecting model hallucination is also provided, which is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following steps 1602 to 1608. Among them:

[0196] Step 1602, obtaining cross-modal attention and target reasoning results of visual tokens of the target visual text pair when the visual language model performs model reasoning on the target visual text pair, where the target reasoning result is one of the first type of reasoning results and the second type of reasoning results;

[0197] Step 1604, inputting the cross-modal attention into the hallucination detection model, the hallucination detection model is obtained by cascading the first-stage hallucination detector obtained after training and the second-stage hallucination detector, and outputting the first hallucination detection result through the first-stage hallucination detector;

[0198] Step 1606A, when the first hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning, and the target reasoning result is a first-category reasoning result, inputting the cross-modal attention into the first-category second-stage hallucination detector to obtain a first-category second hallucination detection result;

[0199] In one of the examples, the method further includes: when the first hallucination detection result indicates that the visual language model does not hallucinate when performing model reasoning on the target visual-text pair, determining that the visual language model does not hallucinate when performing model reasoning on the target visual-text pair.

[0200] Step 1606B, when the first hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning, and the target reasoning result is a second-category reasoning result, inputting the cross-modal attention into a second-category second-stage hallucination detector to obtain a second-category second hallucination detection result;

[0201] Step 1608A: When the first type of second hallucination detection result indicates that the visual language model hallucinates when performing model reasoning, determine whether the visual language model hallucinates when performing model reasoning on the target visual text pair.

[0202] In this case, the target reasoning result can be further flipped to the second type of reasoning result.

[0203] In one of the examples, the method further includes: when the first type of second hallucination detection result indicates that the visual language model does not hallucinate when performing model reasoning, determining that the visual language model does not hallucinate when performing model reasoning on the target visual text pair.

[0204] Step 1608B, when the second type of second hallucination detection result indicates that the visual language model hallucinates when performing model reasoning, determine that the visual language model hallucinates when performing model reasoning on the target visual text pair.

[0205] In this case, the target reasoning result can be further flipped to the first type of reasoning result.

[0206] In one of the examples, the method further includes: determining that the visual language model does not hallucinate when performing model reasoning on the target visual-text pair, when the second type of second hallucination detection result indicates that the visual language model does not hallucinate when performing model reasoning.

[0207] like Fig.17 FIG. 1 is a schematic diagram of using a hallucination detection model to detect and mitigate hallucinations in one embodiment. Fig.17 , the input is the cross-modal attention A(v, t) extracted by the visual language model when performing model reasoning on the target visual text pair, which is detected by the first-stage hallucination detector to obtain the first hallucination detection result, and the first hallucination detection result indicates that the attention category of the cross-modal attention A(v, t) is AYH, ANH, AY or AN. If it is AYH, the extracted cross-modal attention A(v, t) is further detected by the first-class second-stage hallucination detector to obtain the first-class second hallucination detection result. If it is ANH, the extracted cross-modal attention A(v, t) is further detected by the second-class second-stage hallucination detector to obtain the second-class second hallucination detection result. If it is AY or AN, it is determined that the visual language model does not hallucinate when performing model reasoning on the target visual text pair, and the final classification result is AY or AN, and the final reasoning result is {Yes} or {No}.

[0208] The first type of second hallucination detection result indicates that the attention category of the cross-modal attention A(v, t) is True AYH or False AYH. If it is True AYH, the final classification result is AYH. If it is False AYH, it is determined that the visual language model does not hallucinate when performing model reasoning on the target visual text pair. The final classification result is AY, and the final reasoning result is {Yes}. In the case where the final classification result is AYH, the target reasoning result can be flipped, that is, {Yes} is flipped to {No}.

[0209] The second type of second hallucination detection result indicates that the attention category of the cross-modal attention A(v, t) is True ANH or False ANH. If it is True ANH, the final classification result is ANH. If it is False ANH, it is determined that the visual language model does not hallucinate when performing model reasoning on the target visual text pair, and the final classification result is AN, and the final reasoning result is {No}. In the case where the final classification result is ANH, the obtained target reasoning result can be flipped, that is, {No} is flipped to {Yes}.

[0210] In a detailed embodiment, a method for detecting hallucinations is provided, comprising the following steps:

[0211] 1. Obtain a sample image set, and for each sample image in the sample image set, obtain a labeled object of the sample image, where the labeled object refers to an object existing in the sample image;

[0212] 2. Get a text template, which is used to inquire whether the image includes an object;

[0213] 3. Generate query text based on the annotated objects and text templates of the sample image, and obtain visual text pairs based on the sample image and the query text;

[0214] 4. Perform model reasoning on the visual-text pair through the visual language model to obtain a reasoning result and obtain cross-modal attention to the visual tokens of the visual-text pair. The reasoning result is one of the first and second reasoning results.

[0215] 5. Obtain the standard results corresponding to the visual text pairs;

[0216] 6. When the standard result is inconsistent with the inference result of the visual language model for the visual text pair, and the inference result is the first type of inference result, the cross-modal attention is divided into the first type of sample attention when there is hallucination; when the standard result is inconsistent with the inference result of the visual language model for the visual text pair, and the inference result is the second type of inference result, the cross-modal attention is divided into the second type of sample attention when there is hallucination; when the standard result is consistent with the inference result of the visual language model for the visual text pair, and the inference result is the first type of inference result, the cross-modal attention is divided into the first type of sample attention when there is no hallucination; when the standard result is consistent with the inference result of the visual language model for the visual text pair, and the inference result is the second type of inference result, the cross-modal attention is divided into the second type of sample attention when there is no hallucination;

[0217] 7. Input the sample attention into the first-stage hallucination detector to obtain the first hallucination detection result of the first-stage hallucination detector on the sample attention, and update the first-stage hallucination detector according to the first hallucination detection result and the attention category of the sample attention;

[0218] 8. From the sample attention input to the first-stage hallucination detector, filter out the sample attention whose attention category is the first category with hallucination, continue to input the first-category second-stage hallucination detector, obtain the second hallucination detection result of the first-category second-stage hallucination detector on the sample attention, and update the first-category second-stage hallucination detector according to the second hallucination detection result and the sample category of the sample attention;

[0219] 9. From the sample attention input to the first-stage hallucination detector, filter out the sample attention whose attention category is the second category with hallucination, continue to input the second category second-stage hallucination detector, obtain the second hallucination detection result of the second category second-stage hallucination detector on the sample attention, and update the second category second-stage hallucination detector according to the second hallucination detection result and the sample category of the sample attention;

[0220] 10. Cascading the first-stage hallucination detector obtained after the training is completed with the first-type second-stage hallucination detector to obtain a hallucination detection model, where the hallucination detection model is used to detect whether hallucination occurs when the visual language model performs model inference to obtain the first-type inference result; and

[0221] 11. The first-stage hallucination detector obtained after the training is cascaded with the second-stage hallucination detector of the second type to obtain a hallucination detection model. The hallucination detection model is used to detect whether hallucination occurs when the visual language model performs model inference to obtain the second type of inference result.

[0222] 12. Obtain the target text pair, perform model reasoning on the target visual text pair through the visual language model, and obtain the cross-modal attention and target reasoning results. The target reasoning result is one of the first and second reasoning results.

[0223] 13. Input the cross-modal attention into the hallucination detection model. The hallucination detection model is obtained by cascading the first-stage hallucination detector obtained after the training and the second-stage hallucination detector. The first-stage hallucination detector outputs the first hallucination detection result.

[0224] 14. When the first hallucination detection result indicates that the visual language model hallucinates when performing model reasoning, and the target reasoning result is a first-category reasoning result, the cross-modal attention is input into the first-category second-stage hallucination detector to obtain a first-category second hallucination detection result; when the first-category second hallucination detection result indicates that the visual language model hallucinates when performing model reasoning, it is determined that the visual language model hallucinates when performing model reasoning on the target visual text pair. When the first-category second hallucination detection result indicates that the visual language model does not hallucinate when performing model reasoning on the target visual text pair, it is determined that the visual language model does not hallucinate when performing model reasoning on the target visual text pair.

[0225] 15. When the first hallucination detection result indicates that the visual language model hallucinates when performing model reasoning, and the target reasoning result is a second-category reasoning result, the cross-modal attention is input into the second-category second-stage hallucination detector to obtain a second-category second hallucination detection result; when the second-category second hallucination detection result indicates that the visual language model hallucinates when performing model reasoning, it is determined that the visual language model hallucinates when performing model reasoning on the target visual text pair. When the second-category second hallucination detection result indicates that the visual language model does not hallucinate when performing model reasoning on the target visual text pair, it is determined that the visual language model does not hallucinate when performing model reasoning on the target visual text pair.

[0226] 16. When the first hallucination detection result indicates that the visual language model does not hallucinate when performing model reasoning, determine that the visual language model does not hallucinate when performing model reasoning on the target visual text pair.

[0227] 17. When it is determined that the visual language model hallucinates when performing model reasoning on the target visual text pair, the target reasoning result is flipped and the flipped reasoning result is used as the reasoning result corresponding to the target visual text. When it is determined that the visual language model does not hallucinate when performing model reasoning on the target visual text pair, the target reasoning result is used as the reasoning result corresponding to the target visual text.

[0228] like Fig.18As shown in Figure 1, it is a schematic diagram of the relevant statistics of the hallucination detection model. 22,670 images with at least 3 objects in the image are selected from COCO-val2014, of which 18,136 images are used as the training set and 4,534 images are used as the test set. Fig.18 Part (a) of the graph shows the statistical results of the first-stage hallucination detector on the test set. Fig.18 Part (b) of shows the statistics of the effect of using the second-stage hallucination detector on the test set, as Fig.18 Part (c) shows the statistical data of the hallucination detection effect of jointly using the first-stage hallucination detector and the second-stage hallucination detector on the test set. It can be seen that the cascaded hallucination detection model can achieve a practical level of hallucination detection effect (with an accuracy of 93.86%).

[0229] like Fig.19 The figure shows a schematic diagram of relevant statistical data of the hallucination detection model. The visual text pairs as negative examples in the first test set (Randomset) are obtained in the following manner: multiple negative example objects of the sample image are determined according to the annotated objects of the sample image to obtain a negative example object set of the sample image, the negative example object set is randomly sampled to obtain a negative example object as the negative example object of the sample image, and a visual text pair as a negative example is generated according to the negative example object.

[0230] The visual-text pairs used as negative examples in the second test set (Popular set) are obtained in the following manner: obtaining annotated objects of other sample images in the sample image set except the sample image, filtering out negative example objects of the sample image from the annotated objects of other sample images to obtain a negative example object set of the sample image, randomly sampling the negative example object set to obtain a negative example object as the negative example object of the sample image, and generating a visual-text pair used as a negative example based on the negative example object.

[0231] The visual-text pairs used as negative examples in the third test set (Adversarial set) are obtained in the following manner: co-occurrence objects of the annotated objects of the sample image are determined, where the co-occurrence objects are objects that do not exist in the sample image and have a co-occurrence relationship with the annotated objects; a set of negative example objects of the sample image is obtained based on the co-occurrence objects; the set of negative example objects is randomly sampled to obtain a negative example object as the negative example object of the sample image; and a visual-text pair used as a negative example is generated based on the negative example objects.

[0232] Fig.19In the figure, DHCPa refers to the use of four classifiers in the first stage and two classifiers in the second stage, and DHCPb refers to the use of two classifiers instead of four classifiers in the first stage, and the use of two classifiers in the first stage, that is, the cascade of two two classifiers is used to determine whether the model has hallucinations. Of course, based on the inventive concept of the present application, more two classifiers or cascades or serial-parallel connections of various classifiers can be used to achieve higher performance hallucination detection. In this way, the above-mentioned different test sets are constructed based on the POPE evaluation standard (a method for evaluating hallucinations of objects) to calculate the relevant effect statistical data of the obtained hallucination detection model, thereby objectively evaluating the performance of the hallucination detection model.

[0233] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0234] Based on the same inventive concept, the embodiment of the present application also provides a model hallucination detection device for implementing the above-mentioned model hallucination detection method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiments of one or more model hallucination detection devices provided below can refer to the limitations of the model hallucination detection method above, and will not be repeated here.

[0235] In an exemplary embodiment, Fig. 20 As shown, a model hallucination detection device 2000 is provided, comprising: an acquisition module 2002, a detector update module 2004 and a cascade module 2006, wherein:

[0236] An acquisition module 2002 is used to acquire cross-modal attention of visual tokens of a visual text pair when a visual language model performs model reasoning on the visual text pair, and divide the cross-modal attention into sample attention when there is hallucination or sample attention when there is no hallucination according to the reasoning result of the visual language model on the visual text pair;

[0237] The detector updating module 2004 is used to input the sample attention into the first stage hallucination detector, obtain the first hallucination detection result of the first stage hallucination detector on the sample attention, and update the first stage hallucination detector according to the first hallucination detection result and the attention category of the sample attention;

[0238] The detector updating module 2004 is further used to filter out sample attentions with an attention category of hallucination from the sample attentions input to the first-stage hallucination detector, continue to input the sample attentions to the second-stage hallucination detector, obtain the second-stage hallucination detection result of the sample attention by the second-stage hallucination detector, and update the second-stage hallucination detector according to the second hallucination detection result and the sample category of the sample attention;

[0239] The cascade module 2006 is used to cascade the first-stage hallucination detector obtained after training with the second-stage hallucination detector to obtain a hallucination detection model. The hallucination detection model is used to detect whether hallucination occurs when the visual language model performs model inference.

[0240] In one embodiment, the acquisition module 2002 is also used to acquire a visual-text pair, which includes an image and a text, and the text is used to inquire whether an object is included in the image; obtain a visual token based on the image through a visual language model, obtain a text token based on the text, and obtain cross-modal attention of the visual token based on the visual token and the text token.

[0241] In one embodiment, the visual language model includes an image encoding network, a modality alignment network, and a large language model, and the large language model includes a text encoding network and a text decoding network;

[0242] The acquisition module 2002 is also used to encode the image through the image encoding network in the visual language model to obtain image encoding features, modally align the image encoding features through the modal alignment network of the visual language model to obtain visual tokens of the visual text pair; embed the text through the text encoding network of the visual language model to represent the text and obtain text tokens of the visual text pair; decode the text based on the cross-modal features obtained by concatenating the visual tokens with the text tokens through the text decoding network of the visual language model to obtain the reasoning result of the visual language model on the visual text pair, and obtain the cross-modal attention of the visual language model to the visual tokens of the visual text pair when outputting the reasoning result.

[0243] In one embodiment, the acquisition module 2002 is also used to acquire a sample image set, and for each sample image in the sample image set, acquire the annotated object of the sample image, where the annotated object refers to an object present in the sample image; acquire a text template, where the text template is used to inquire whether the image includes the object; generate a query text based on the annotated object and the text template of the sample image, and obtain a visual text pair based on the sample image and the query text.

[0244] In one embodiment, the acquisition module 2002 is further used to generate a query text based on the annotated object and text template of the sample image, wherein one annotated object generates one query text, and the query text is used to inquire whether the sample image includes the annotated object; based on the sample image and the query text, a visual text pair as a positive example is constructed.

[0245] In one embodiment, the acquisition module 2002 is further used to determine the negative example object of the sample image based on the annotated object of the sample image, where the negative example object is an object that does not exist in the sample image; generate a query text based on the negative example object of the sample image and a text template, wherein one negative example object generates one query text, and the query text is used to inquire whether the sample image includes the negative example object; and construct a visual text pair as a negative example based on the sample image and the query text.

[0246] In one embodiment, the acquisition module 2002 is further used to determine multiple negative example objects of the sample image according to the annotated objects of the sample image, obtain a negative example object set of the sample image, randomly sample the negative example object set, and obtain a negative example object as the negative example object of the sample image; or, obtain the annotated objects of other sample images in the sample image set except the sample image, filter out the negative example objects of the sample image from the annotated objects of other sample images, obtain a negative example object set of the sample image, randomly sample the negative example object set, and obtain a negative example object as the negative example object of the sample image; or, determine the co-occurrence objects of the annotated objects of the sample image, the co-occurrence objects are objects that do not exist in the sample image and have a co-occurrence relationship with the annotated objects, obtain the negative example object set of the sample image according to the co-occurrence objects, randomly sample the negative example object set, and obtain a negative example object as the negative example object of the sample image.

[0247] In one of the embodiments, the acquisition module 2002 is further used to obtain the annotated objects of each sample image in the sample image set; based on the sample image set, the co-occurrence between any two annotated objects is counted, wherein two different annotated objects appearing simultaneously in the same sample image are regarded as a co-occurrence; when the co-occurrence of two different annotated objects meets the set conditions, it is determined that the two different annotated objects have a co-occurrence relationship.

[0248] In one embodiment, the acquisition module 2002 is also used to obtain the standard result corresponding to the visual text pair; when the standard result is inconsistent with the inference result of the visual language model for the visual text pair, the cross-modal attention is divided into sample attention when there is hallucination; when the standard result is consistent with the inference result of the visual language model for the visual text pair, the cross-modal attention is divided into sample attention when there is no hallucination.

[0249] In one embodiment, the acquisition module 2002 is further used to obtain the inference result of the visual language model on the visual text pair, where the inference result is one of the first type of inference result and the second type of inference result; according to the type of the inference result and the annotation information of the visual text pair, the cross-modal attention is divided into the first type of sample attention when there is hallucination, the second type of sample attention when there is hallucination, the first type of sample attention when there is no hallucination, or the second type of sample attention when there is no hallucination;

[0250] The detector updating module 2004 is further used to screen out sample attentions whose attention category is the first category with hallucination from the sample attentions input to the first-stage hallucination detector, continue to input the sample attentions to the first-category second-stage hallucination detector, obtain the second hallucination detection result of the sample attention by the first-category second-stage hallucination detector, and update the first-category second-stage hallucination detector according to the second hallucination detection result and the sample category of the sample attention;

[0251] The detector updating module 2004 is further used to filter out the sample attention whose attention category is the second category with hallucination from the sample attention input to the first-stage hallucination detector, continue to input the sample attention to the second-category second-stage hallucination detector, obtain the second-category second-stage hallucination detector's second-category hallucination detection result on the sample attention, and update the second-category second-stage hallucination detector according to the second hallucination detection result and the sample category of the sample attention;

[0252] The cascade module 2006 is further used to cascade the first-stage hallucination detector obtained after the training is completed with the first-type second-stage hallucination detector to obtain a hallucination detection model, and the hallucination detection model is used to detect whether hallucination occurs when the visual language model performs model inference to obtain the first-type inference result;

[0253] The cascade module 2006 is also used to cascade the first-stage hallucination detector obtained after the training is completed with the second-type second-stage hallucination detector to obtain a hallucination detection model. The hallucination detection model is used to detect whether hallucination occurs when the visual language model performs model inference to obtain the second type of inference result.

[0254] In one embodiment, the acquisition module 2002 is further used to obtain a target reasoning result obtained by performing model reasoning on a target visual text pair by a visual language model, where the target visual text pair includes a target image and a target text, and the target text is used to inquire whether a target image includes an object;

[0255] The model hallucination detection device 2000 also includes:

[0256] An inference module to obtain the cross-modal attention of the visual language model on the visual tokens of the target visual-text pair;

[0257] A hallucination detection module is used to input cross-modal attention into a hallucination detection model, where the hallucination detection model is obtained by cascading the first-stage hallucination detector obtained after training and the second-stage hallucination detector, and outputting a first hallucination detection result through the first-stage hallucination detector;

[0258] The hallucination detection module is further configured to input the cross-modal attention into the second-stage hallucination detector to obtain a second hallucination detection result when the first hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning;

[0259] The hallucination detection module is further used to determine that the visual language model hallucinates when performing model reasoning on the target visual text pair when the second hallucination detection result indicates that the visual language model hallucinates when performing model reasoning.

[0260] In one of the embodiments, a hallucination detection module is used to determine that the visual language model does not hallucinate when performing model reasoning on a target visual text pair when a first hallucination detection result indicates that the visual language model does not hallucinate when performing model reasoning, or when a second hallucination detection result indicates that the visual language model does not hallucinate when performing model reasoning.

[0261] In one of the embodiments, the model hallucination detection device 2000 also includes a hallucination mitigation module, which is used to flip the target reasoning result when it is determined that the visual language model has hallucinations when performing model reasoning on the target visual text pair, and use the flipped reasoning result as the reasoning result corresponding to the target visual text.

[0262] The above-mentioned model hallucination detection device 2000 pays attention to the difference in cross-modal attention between the visual language model when hallucination occurs and when hallucination does not occur. Therefore, firstly, by obtaining the cross-modal attention of the visual token of the visual text pair when the visual language model performs model reasoning on the visual text pair, the cross-modal attention is divided into sample attention when there is hallucination or sample attention when there is no hallucination according to the reasoning result of the visual language model on the visual text pair, so as to construct a training sample for training the detector. Then, by inputting the sample attention into the first-stage hallucination detector, the first-stage hallucination detector obtains the first hallucination detection result of the sample attention, and according to the first hallucination detection result and the attention category of the sample attention, the first-stage hallucination detector is updated to achieve a high recall rate of attention to the hallucinated samples during the first-stage training process. Next, from the sample attention input to the first-stage hallucination detector, the sample attention with the attention category of hallucination is screened out, and the attention is continuously input to the second-stage hallucination detector for more detailed classification. By obtaining the second hallucination detection result of the sample attention by the second-stage hallucination detector, the second-stage hallucination detector is updated according to the second hallucination detection result and the sample category of the sample attention, so as to achieve a high detection rate of hallucinated samples during the training process of the second stage. Then, the first-stage hallucination detector obtained after the training is cascaded with the second-stage hallucination detector to obtain a hallucination detection model, which is used to detect whether hallucination occurs in the visual language model during model inference, and can ensure the accuracy of detecting whether hallucination occurs in the text language model. At the same time, compared with the existing technology, the hallucination detection model can directly detect whether hallucination occurs, with high detection efficiency and strong practicality.

[0263] In an exemplary embodiment, Fig.21 As shown, a model hallucination detection device 2100 is provided, comprising: an acquisition module 2102, a detection module 2104 and a detection result determination module 2106, wherein:

[0264] An acquisition module 2102 is used to acquire cross-modal attention of a visual token of a target visual text pair when a visual language model performs model reasoning on the target visual text pair;

[0265] A detection module 2104 is used to input the cross-modal attention into a hallucination detection model, where the hallucination detection model is obtained by cascading the first-stage hallucination detector obtained after training and the second-stage hallucination detector, and outputting a first hallucination detection result through the first-stage hallucination detector;

[0266] The detection module 2104 is further configured to input the cross-modal attention into the second-stage hallucination detector to obtain a second hallucination detection result when the first hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning;

[0267] The detection result determination module 2106 is used to determine whether the visual language model hallucinates when performing model reasoning on the target visual text pair when the second hallucination detection result indicates that the visual language model hallucinates when performing model reasoning.

[0268] In one embodiment, the detection result determination module 2106 is used to determine that the visual language model does not hallucinate when performing model reasoning on a target visual text pair when a first hallucination detection result indicates that the visual language model does not hallucinate when performing model reasoning, or when a second hallucination detection result indicates that the visual language model does not hallucinate when performing model reasoning.

[0269] In one of the embodiments, the model hallucination detection device 2100 also includes a hallucination mitigation module, which is used to flip the target reasoning result when it is determined that the visual language model has hallucinations when performing model reasoning on the target visual text pair, and use the flipped reasoning result as the reasoning result corresponding to the target visual text.

[0270] The above-mentioned model hallucination detection device 2100 pays attention to the difference in cross-modal attention between the visual language model when hallucination occurs and when hallucination does not occur. Therefore, by obtaining the cross-modal attention of the visual language model to the visual tokens of the target visual text pair when performing model reasoning on the target visual text pair, the cross-modal attention is input into the hallucination detection model, and the hallucination detection model is obtained by cascading the first-stage hallucination detector and the second-stage hallucination detector obtained after the training is completed. The first-stage hallucination detector outputs a first hallucination detection result. When the first hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning, the cross-modal attention is further input into the second-stage hallucination detector to obtain a second hallucination detection result. When the second hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning, it can be determined that the visual language model has hallucinations when performing model reasoning on the target visual text pair. In this way, efficient and accurate detection of whether the visual language model has hallucinations when performing model reasoning based on cross-modal attention can be achieved.

[0271] Each module in the above device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0272] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 22As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store training sample data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for detecting model hallucination is implemented.

[0273] Those skilled in the art will understand that Fig. 22 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0274] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of a method for detecting model hallucination when executing the computer program.

[0275] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for detecting model hallucination are implemented.

[0276] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the method for detecting model hallucination when executed by a processor.

[0277] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0278] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0279] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0280] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for detecting model hallucination, characterized in that: The method comprises: Obtaining cross-modal attention of a visual token of a visual text pair when a visual language model performs model reasoning on the visual text pair, and dividing the cross-modal attention into sample attention when there is hallucination or sample attention when there is no hallucination according to the reasoning result of the visual language model on the visual text pair; Inputting the sample attention into a first-stage hallucination detector, obtaining a first hallucination detection result of the first-stage hallucination detector on the sample attention, and updating the first-stage hallucination detector according to the first hallucination detection result and the attention category of the sample attention; From the sample attention input to the first-stage hallucination detector, filter out the sample attention when the attention category is hallucination, continue to input into the second-stage hallucination detector, obtain the second-stage hallucination detector's second-stage hallucination detection result of the sample attention, and update the second-stage hallucination detector according to the second hallucination detection result and the attention category of the sample attention; The first-stage hallucination detector obtained after training is cascaded with the second-stage hallucination detector to obtain a hallucination detection model, and the hallucination detection model is used to detect whether hallucination occurs when the visual language model performs model inference.

2. The method according to claim 1, characterized in that The step of obtaining cross-modal attention of visual tokens of a visual text pair when a visual language model performs model reasoning on a visual text pair includes: Acquire a visual-text pair, the visual-text pair comprising an image and a text, the text being used to inquire whether the image includes an object; A visual token is obtained based on the image through the visual language model, a text token is obtained based on the text, and cross-modal attention of the visual token is obtained according to the visual token and the text token.

3. The method according to claim 2, characterized in that The visual language model includes an image encoding network, a modality alignment network and a large language model, and the large language model includes a text encoding network and a text decoding network; The step of obtaining a visual token based on the image through the visual language model, obtaining a text token based on the text, and obtaining cross-modal attention of the visual token according to the visual token and the text token includes: By using the image encoding network in the visual language model, the image is encoded to obtain image encoding features, and by using the modality alignment network in the visual language model, the image encoding features are modally aligned to obtain visual tokens of the visual-text pair; The text is embedded and represented by a text encoding network of the visual language model to obtain a text token of the visual-text pair; Through the text decoding network of the visual language model, text decoding is performed based on the cross-modal features obtained by splicing the visual token and the text token, so as to obtain the inference result of the visual language model on the visual-text pair, and obtain the cross-modal attention of the visual language model to the visual token of the visual-text pair when outputting the inference result.

4. The method according to claim 1, characterized in that: The method further comprises: Acquire a sample image set, and for each sample image in the sample image set, acquire a labeled object of the sample image, where the labeled object refers to an object existing in the sample image; Acquire a text template, wherein the text template is used to inquire whether the image includes an object; A query text is generated according to the annotated object of the sample image and the text template, and a visual text pair is obtained according to the sample image and the query text.

5. The method according to claim 4, characterized in that The step of generating a query text according to the annotated object of the sample image and the text template, and obtaining a visual text pair according to the sample image and the query text comprises: Generate an inquiry text according to the annotated object of the sample image and the text template, wherein one inquiry text is generated for one annotated object, and the inquiry text is used to inquire whether the sample image includes the annotated object; A visual-text pair serving as a positive example is constructed according to the sample image and the query text.

6. The method according to claim 5, characterized in that The method further comprises: Determine a negative example object of the sample image according to the annotated object of the sample image, wherein the negative example object is an object that does not exist in the sample image; Generate a query text according to the negative example object of the sample image and the text template, wherein one query text is generated for one negative example object, and the query text is used to inquire whether the sample image includes the negative example object; According to the sample image and the query text, a visual-text pair serving as a negative example is constructed.

7. The method according to claim 6, characterized in that The determining the negative example object of the sample image according to the annotated object of the sample image includes: Determine multiple negative example objects of the sample image according to the annotated object of the sample image to obtain a negative example object set of the sample image, and randomly sample the negative example object set to obtain a negative example object as the negative example object of the sample image; or Obtaining annotated objects of other sample images in the sample image set except the sample image, screening out negative example objects of the sample image from the annotated objects of the other sample images to obtain a negative example object set of the sample image, and randomly sampling the negative example object set to obtain a negative example object as the negative example object of the sample image; or Determine a co-occurrence object of the annotated object of the sample image, where the co-occurrence object is an object that does not exist in the sample image and has a co-occurrence relationship with the annotated object, obtain a set of negative example objects of the sample image based on the co-occurrence object, and randomly sample the set of negative example objects to obtain a negative example object as the negative example object of the sample image.

8. The method according to claim 7, characterized in that The method further comprises: Obtaining a label object for each sample image in the sample image set; Based on the sample image set, counting the co-occurrence between any two annotated objects, wherein two different annotated objects appearing simultaneously in the same sample image is regarded as one co-occurrence; When the co-occurrence degrees of two different annotated objects satisfy a set condition, it is determined that the two different annotated objects have a co-occurrence relationship.

9. The method according to claim 1, characterized in that: The step of dividing the cross-modal attention into sample attention when there is hallucination or sample attention when there is no hallucination according to the inference result of the visual language model on the visual text pair includes: Obtaining a standard result corresponding to the visual-text pair; When the standard result is inconsistent with the inference result of the visual language model on the visual text pair, dividing the cross-modal attention into sample attention when there is hallucination; When the standard result is consistent with the inference result of the visual language model on the visual-text pair, the cross-modal attention is divided into sample attention without hallucination.

10. The method according to claim 1, characterized in that The step of dividing the cross-modal attention into sample attention when there is hallucination or sample attention when there is no hallucination according to the inference result of the visual language model on the visual text pair includes: Obtaining an inference result of the visual language model on the visual text pair, wherein the inference result is one of a first type of inference result and a second type of inference result; According to the type of the inference result and the annotation information of the visual-text pair, the cross-modal attention is divided into the first type of sample attention when there is hallucination, the second type of sample attention when there is hallucination, the first type of sample attention when there is no hallucination, or the second type of sample attention when there is no hallucination; The method comprises: filtering out sample attentions with an attention category of hallucination from the sample attentions input to the first-stage hallucination detector, continuously inputting the sample attentions into the second-stage hallucination detector, obtaining a second hallucination detection result of the sample attention by the second-stage hallucination detector, and updating the second-stage hallucination detector according to the second hallucination detection result and the attention category of the sample attention, including: From the sample attention input to the first-stage hallucination detector, filter out the sample attention whose attention category is the first category with hallucination, continue to input into the first-category second-stage hallucination detector, obtain the second hallucination detection result of the first-category second-stage hallucination detector on the sample attention, and update the first-category second-stage hallucination detector according to the second hallucination detection result and the attention category of the sample attention; From the sample attention input to the first-stage hallucination detector, filter out the sample attention whose attention category is the second category with hallucination, continue to input the second category second-stage hallucination detector, obtain the second hallucination detection result of the second category second-stage hallucination detector on the sample attention, and update the second category second-stage hallucination detector according to the second hallucination detection result and the attention category of the sample attention; The first-stage hallucination detector obtained after the training is cascaded with the second-stage hallucination detector to obtain a hallucination detection model, including: Cascading the first-stage hallucination detector obtained after training with the first-type second-stage hallucination detector to obtain a hallucination detection model, wherein the hallucination detection model is used to detect whether hallucination occurs when the visual language model performs model inference to obtain the first-type inference result; and The first-stage hallucination detector obtained after training is cascaded with the second-type second-stage hallucination detector to obtain a hallucination detection model, which is used to detect whether hallucination occurs when the visual language model performs model inference to obtain the second type of inference result.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: Obtaining a target inference result obtained by performing model inference on a target visual text pair by the visual language model, wherein the target visual text pair includes a target image and a target text, and the target text is used to inquire whether the target image includes an object; Obtaining the cross-modal attention of the visual language model to the visual token of the target visual text pair, inputting the cross-modal attention into the hallucination detection model, wherein the hallucination detection model is obtained by cascading the first-stage hallucination detector obtained after training and the second-stage hallucination detector, and outputting a first hallucination detection result through the first-stage hallucination detector; In a case where the first hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning, inputting the cross-modal attention into the second-stage hallucination detector to obtain a second hallucination detection result; When the second hallucination detection result indicates that the visual language model hallucinates when performing model reasoning, it is determined that the visual language model hallucinates when performing model reasoning on the target visual-text pair.

12. The method according to claim 11, characterized in that The method further comprises: When the first hallucination detection result indicates that the visual language model does not hallucinate when performing model reasoning, or when the second hallucination detection result indicates that the visual language model does not hallucinate when performing model reasoning, it is determined that the visual language model does not hallucinate when performing model reasoning on the target visual text pair.

13. The method according to claim 11, characterized in that The method further comprises: When it is determined that the visual language model has hallucinations when performing model reasoning on the target visual text pair, the target reasoning result is flipped, and the flipped reasoning result is used as the reasoning result corresponding to the target visual text.

14. A method for detecting model hallucination, characterized in that: The method comprises: Obtaining cross-modal attention of a visual language model to a visual token of a target visual text pair when performing model reasoning on the target visual text pair; Inputting the cross-modal attention into a hallucination detection model, wherein the hallucination detection model is obtained by cascading a first-stage hallucination detector obtained after training and a second-stage hallucination detector, and outputting a first hallucination detection result through the first-stage hallucination detector; In a case where the first hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning, inputting the cross-modal attention into the second-stage hallucination detector to obtain a second hallucination detection result; When the second hallucination detection result indicates that the visual language model hallucinates when performing model reasoning, it is determined that the visual language model hallucinates when performing model reasoning on the target visual-text pair.

15. A device for detecting model hallucination, characterized in that: The device comprises: an acquisition module, configured to acquire cross-modal attention of visual tokens of a visual text pair when a visual language model performs model inference on the visual text pair, and divide the cross-modal attention into sample attention when there is hallucination or sample attention when there is no hallucination according to the inference result of the visual language model on the visual text pair; a detector updating module, configured to input the sample attention into a first-stage hallucination detector, obtain a first hallucination detection result of the first-stage hallucination detector on the sample attention, and update the first-stage hallucination detector according to the first hallucination detection result and the attention category of the sample attention; The detector updating module is further used to screen out sample attentions with an attention category of hallucination from the sample attentions input to the first-stage hallucination detector, continue to input the sample attentions to the second-stage hallucination detector, obtain a second hallucination detection result of the second-stage hallucination detector on the sample attention, and update the second-stage hallucination detector according to the second hallucination detection result and the attention category of the sample attention; The cascade module is used to cascade the first-stage hallucination detector obtained after training with the second-stage hallucination detector to obtain a hallucination detection model, and the hallucination detection model is used to detect whether hallucination occurs when the visual language model performs model inference.

16. The device according to claim 15, characterized in that The acquisition module is also used to acquire a visual-text pair, wherein the visual-text pair includes an image and a text, wherein the text is used to inquire whether the image includes an object; obtain a visual token based on the image through the visual language model, obtain a text token based on the text, and obtain cross-modal attention of the visual token based on the visual token and the text token.

17. The device according to claim 16, characterized in that The visual language model includes an image encoding network, a modality alignment network and a large language model, and the large language model includes a text encoding network and a text decoding network; The acquisition module is further used to perform image encoding on the image through the image encoding network in the visual language model to obtain image encoding features, and perform modality alignment on the image encoding features through the modality alignment network of the visual language model to obtain visual tokens of the visual text pair; The text is embedded and represented by a text encoding network of the visual language model to obtain a text token of the visual-text pair; Through the text decoding network of the visual language model, text decoding is performed based on the cross-modal features obtained by splicing the visual token and the text token, so as to obtain the inference result of the visual language model on the visual-text pair, and obtain the cross-modal attention of the visual language model to the visual token of the visual-text pair when outputting the inference result.

18. The device according to claim 15, characterized in that The acquisition module is further used to acquire a set of sample images, and for each sample image in the set of sample images, acquire the annotated object of the sample image, where the annotated object refers to an object existing in the sample image; acquire a text template, where the text template is used to inquire whether an object is included in the image; generate an inquiry text based on the annotated object of the sample image and the text template, and obtain a visual text pair based on the sample image and the inquiry text.

19. The device according to claim 18, characterized in that The acquisition module is further used to generate a query text based on the annotated object of the sample image and the text template, wherein one annotated object generates one query text, and the query text is used to inquire whether the sample image includes the annotated object; and construct a visual text pair as a positive example based on the sample image and the query text.

20. The device according to claim 19, characterized in that The acquisition module is further used to determine a negative example object of the sample image according to the annotated object of the sample image, wherein the negative example object is an object that does not exist in the sample image; Generate a query text based on the negative example object of the sample image and the text template, wherein one query text is generated for one negative example object, and the query text is used to inquire whether the sample image includes the negative example object; construct a visual text pair as a negative example based on the sample image and the query text.

21. The device according to claim 20, characterized in that The acquisition module is further used to determine multiple negative example objects of the sample image according to the annotated objects of the sample image, obtain a negative example object set of the sample image, randomly sample the negative example object set, and obtain a negative example object as the negative example object of the sample image; or, obtain the annotated objects of other sample images in the sample image set except the sample image, filter out the negative example objects of the sample image from the annotated objects of the other sample images, obtain a negative example object set of the sample image, randomly sample the negative example object set, and obtain a negative example object as the negative example object of the sample image; or, determine the co-occurrence objects of the annotated objects of the sample image, the co-occurrence objects are objects that do not exist in the sample image and have a co-occurrence relationship with the annotated objects, obtain the negative example object set of the sample image according to the co-occurrence objects, and randomly sample the negative example object set to obtain a negative example object as the negative example object of the sample image.

22. The device according to claim 21, characterized in that The acquisition module is further used to acquire the annotated object of each sample image in the sample image set; based on the sample image set, count the co-occurrence between any two annotated objects, wherein two different annotated objects appearing simultaneously in the same sample image are regarded as a co-occurrence; and when the co-occurrence of two different annotated objects meets a set condition, determine that the two different annotated objects have a co-occurrence relationship.

23. The device according to claim 15, characterized in that The acquisition module is further used to acquire a standard result corresponding to the visual text pair; when the standard result is inconsistent with the inference result of the visual language model on the visual text pair, the cross-modal attention is divided into sample attention when there is hallucination; When the standard result is consistent with the inference result of the visual language model on the visual-text pair, the cross-modal attention is divided into sample attention without hallucination.

24. The device according to claim 15, characterized in that The acquisition module is further used to obtain the inference result of the visual language model on the visual text pair, where the inference result is one of the first type of inference result and the second type of inference result; according to the type of the inference result and the annotation information of the visual text pair, the cross-modal attention is divided into the first type of sample attention when there is hallucination, the second type of sample attention when there is hallucination, the first type of sample attention when there is no hallucination, or the second type of sample attention when there is no hallucination; The detector updating module is further used to filter out sample attentions whose attention category is the first category with hallucination from the sample attentions input to the first-stage hallucination detector, continue to input the sample attentions into the first-category second-stage hallucination detector, obtain the second hallucination detection result of the sample attention by the first-category second-stage hallucination detector, and update the first-category second-stage hallucination detector according to the second hallucination detection result and the attention category of the sample attention; The detector updating module is further used to filter out sample attentions whose attention category is the second category with hallucination from the sample attentions input to the first-stage hallucination detector, continue to input the second category second-stage hallucination detector, obtain the second hallucination detection result of the second category second-stage hallucination detector on the sample attention, and update the second category second-stage hallucination detector according to the second hallucination detection result and the attention category of the sample attention; The cascade module is further used to cascade the first-stage hallucination detector obtained after the training is completed with the first-type second-stage hallucination detector to obtain a hallucination detection model, and the hallucination detection model is used to detect whether hallucination occurs when the visual language model performs model inference to obtain the first-type inference result; The cascade module is also used to cascade the first-stage hallucination detector obtained after training with the second-type second-stage hallucination detector to obtain a hallucination detection model, and the hallucination detection model is used to detect whether hallucination occurs when the visual language model performs model inference to obtain the second type of inference result.

25. The device according to claim 15, characterized in that The device also includes: An inference module, used to obtain a target inference result obtained by the visual language model performing model inference on a target visual text pair, wherein the target visual text pair includes a target image and a target text, and the target text is used to inquire whether the target image includes an object; a hallucination detection module, configured to obtain the cross-modal attention of the visual language model to the visual token of the target visual text pair, input the cross-modal attention to the hallucination detection model, wherein the hallucination detection model is obtained by cascading the first-stage hallucination detector obtained after training and the second-stage hallucination detector, and output a first hallucination detection result through the first-stage hallucination detector; The hallucination detection module is further configured to input the cross-modal attention into the second-stage hallucination detector to obtain a second hallucination detection result when the first hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning; The hallucination detection module is further used to determine that the visual language model hallucinates when performing model reasoning on the target visual text pair when the second hallucination detection result indicates that the visual language model hallucinates when performing model reasoning.

26. The device according to claim 25, characterized in that The hallucination detection module is used to determine that the visual language model does not hallucinate when performing model reasoning on the target visual text pair when the first hallucination detection result indicates that the visual language model does not hallucinate when performing model reasoning, or when the second hallucination detection result indicates that the visual language model does not hallucinate when performing model reasoning.

27. The device according to claim 25, characterized in that The device also includes: The hallucination mitigation module is used to flip the target reasoning result when it is determined that the visual language model has hallucinations when performing model reasoning on the target visual text pair, and use the flipped reasoning result as the reasoning result corresponding to the target visual text.

28. A device for detecting model hallucination, characterized in that: The device comprises: An acquisition module, used for acquiring cross-modal attention of a visual token of a target visual text pair when a visual language model performs model reasoning on the target visual text pair; A detection module, configured to input the cross-modal attention into an illusion detection model, wherein the illusion detection model is obtained by cascading a first-stage illusion detector obtained after training and a second-stage illusion detector, and outputting a first illusion detection result through the first-stage illusion detector; The detection module is further configured to input the cross-modal attention into the second-stage hallucination detector to obtain a second hallucination detection result when the first hallucination detection result indicates that the visual language model has hallucinations when performing model reasoning; A detection result determination module is used to determine that the visual language model hallucinates when performing model reasoning on the target visual text pair when the second hallucination detection result indicates that the visual language model hallucinates when performing model reasoning.

29. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 14 are implemented.

30. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

31. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

Images