A fine-grained large language model hallucination detection method, device, and storage medium

Abstract

The present invention relates to a fine-grained large language model hallucination detection method, device, and storage medium. The method comprises: training a hallucination detection model based on hallucination detection training data, and performing hallucination detection on responses from a target large language model based on the trained hallucination detection model. The hallucination detection training data acquisition process comprises the following steps: acquiring topic data and obtaining corresponding reference data through retrieval; generating multiple question data based on the reference data using a language model different from the target large language model and screening the question data; inputting the screened question data into the language model to obtain corresponding response data, and forming hallucination detection training data by performing fine-grained annotation on a sentence-by-sentence basis. Compared with existing technologies, the present invention has the advantages of strong detection capability and high accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a fine-grained large language model hallucination detection method, device, and storage medium. Background Art

[0002] Currently, large language models generally suffer from the "hallucination" problem, that is, when answering user questions, especially those that require a lot of knowledge, the model will produce information that sounds credible but is untrue or meaningless, which greatly hinders the application of large language models in the real world.

[0003] Hallucination detection in large language models involves using certain technical means to detect hallucinations in large language model responses. These hallucinations may be misleading information, inaccurate descriptions, misunderstandings, or ambiguities, which may negatively impact the understanding and dissemination of information.

[0004] Existing large language model hallucination detector designs, such as FACTOR and HaluEval, primarily use training corpora containing hallucinations and annotated with hallucination types to train large models, turning them into hallucination detectors. However, these solutions have the following problems:

[0005] (1) Existing methods for constructing hallucination annotation data for large language model hallucination detector training often only perform hallucination annotation on an entire complete response, lacking more fine-grained analysis. For example, it is impossible to determine the location, number of times the hallucination appears in the entire response, and thus it is impossible to locate the specific location for hallucination repair.

[0006] (2) Existing open-source models have poor hallucination detection capabilities. Even when hallucination annotated data is used to train large language model hallucination detectors, they only follow a simple fine-tuning paradigm. Therefore, the constructed hallucination detectors still perform poorly, especially when facing more fine-grained hallucination detection tasks.

[0007] In summary, there is currently a lack of a large language model hallucination detection method to solve or partially solve the aforementioned problems. Summary of the Invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a fine-grained large language model hallucination detection method, device, and storage medium, which improves the hallucination detection effect by fine-grained labeling of question-answers in units of sentences.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] One aspect of the present invention provides a fine-grained large language model hallucination detection method, wherein a hallucination detection model is trained based on hallucination detection training data, and hallucination detection is performed on responses of a target large language model based on the trained hallucination detection model. The process of acquiring the hallucination detection training data includes the following steps:

[0011] Obtain subject data and obtain corresponding reference data through retrieval;

[0012] Based on the reference document data, a language model different from the target large language model is used to generate a plurality of question data and the question data is screened;

[0013] Input the filtered question data into the language model to obtain the corresponding answer data;

[0014] The answer data is annotated at a fine-grained level on a sentence-by-sentence basis to form hallucination detection training data.

[0015] As a preferred technical solution, the retrieval process of the reference data includes:

[0016] For terms with multiple meanings, entry selection is achieved through exact matching, semantic similarity matching and model judgment.

[0017] As a preferred technical solution, the process of generating the problem data includes:

[0018] The language model configured with prompt words in advance is used, and the reference document data is used as input of the language model to obtain generated question data.

[0019] As a preferred technical solution, the screening process of the problem data includes:

[0020] Question data was screened based on authenticity, answerability, difficulty, and diversity.

[0021] As a preferred technical solution, the process of fine-grained annotation of response data includes:

[0022] Using a language model pre-configured with prompt words, the system takes the response data as input to obtain the corresponding hallucination type, reference segment information, and correction information, thus achieving automatic labeling.

[0023] Manual error correction and completion are performed on the results of automated annotation.

[0024] As a preferred technical solution, the types of hallucinations include no hallucination, unverifiable, no facts, and contradiction.

[0025] As a preferred technical solution, the training process of the hallucination detection model includes:

[0026] By constructing prompt words and using language dialogue, the hallucination detection model is made to judge hallucinations, thereby realizing the training of the hallucination detection model.

[0027] As a preferred technical solution, the training process of the hallucination detection model further includes:

[0028] The last layer of the hallucination detection model is replaced with a linear classification layer. After inputting questions and responses, the hallucination type judgment result is obtained, thus realizing the training of the hallucination detection model.

[0029] Another aspect of the present invention provides an electronic device comprising: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the aforementioned fine-grained large language model hallucination detection method.

[0030] Another aspect of the present invention provides a computer-readable storage medium comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the aforementioned fine-grained large language model hallucination detection method.

[0031] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0032] (1) Finer detection granularity: The present invention refines the annotation granularity and performs hallucination annotation on a sentence basis. The hallucination detection training corpus constructed in this way has denser annotation information, and the large language model hallucination detector trained on this corpus has better hallucination detection capabilities.

[0033] (2) High detection accuracy: After constructing sentence-level hallucination annotation data, the present invention adopts a more effective hallucination detector training scheme, using the generative paradigm of the language model and the discriminative paradigm of the classification model for training, achieving better hallucination detection results.

[0034] (3) More diverse training methods: The present invention provides training schemes using the generative paradigm of the language model and the discriminative paradigm of the classification model, which can be flexibly applied to various scenarios with different requirements for training methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of a process for acquiring hallucination detection training data during hallucination detection of a fine-grained large language model in an embodiment;

[0036] Figure 2 Schematic diagram of prompt words used in the problem screening and selection process in the embodiment;

[0037] Figure 3Schematic diagram of prompt words used for automatic annotation in an embodiment;

[0038] Figure 4 A diagram of the labeling process for an example. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0040] Example 1

[0041] See also Figure 1 To address the problems of the aforementioned prior art, this embodiment provides a fine-grained large language model hallucination detection method, which mainly includes the following steps:

[0042] S1, select topic & search references.

[0043] To ensure diverse and comprehensive information, our topic selection is categorized into celebrities, events, places, and things. We also cover a variety of fields, including but not limited to politics and military, art, science and technology, religion, and text. After selecting a topic (i.e., subject information), we retrieve corresponding references (i.e., reference data) from pre-trained databases, including Wikipedia, Baidu Encyclopedia, Encyclopedia Britannica, and the Encyclopedia of China. During the reference retrieval process, terminology varies across sources, and a single name can have multiple meanings. This method employs a strategy that transitions from hard matching to soft matching. First, we perform an exact match (i.e., hard matching) on ​​the entries. Then, we rank candidate terms based on semantic similarity within the sentences (i.e., soft matching) and further apply InternLM to select the correct terms. This decision is based on two aspects: whether the term is unique or likely to have synonyms; and whether the term is famous or important, ensuring that the selected terms are both unique and important.

[0044] S2, question generation and selection.

[0045] In order to prevent data leakage, that is, the model has seen the test data, GPT4 was used to regenerate the questions instead of directly using the existing dataset. In addition, we designed the questions so that they can be answered entirely based on the reference documents provided, avoiding being too subjective or open-ended. To ensure the diversity of questions, our questions cover different types (that is, a variety of question modes, such as "what", "when", "why", "where", etc.), spanning different levels of knowledge (that is, from common sense and basic knowledge to more specialized and complex subject knowledge or domain knowledge). After generating the questions, GPT4 is used again to select the questions based on the four indicators of authenticity, answerability, difficulty and diversity to obtain the final high-quality questions. The prompt words used to generate and select questions are as follows: Figure 2 As shown in (a) and (b) in .

[0046] S3, response generation.

[0047] The question obtained by S2 is input into the large language model to generate a response.

[0048] S4, Fine-grained hallucination annotation.

[0049] The model's responses are annotated as hallucinations on a sentence-by-sentence basis, and the hallucination type, reference fragment, and corrections are returned. There are four types of hallucinations: no hallucination, unverifiable, no facts, and contradiction. If a sentence contains factual information and is consistent with the reference, it is classified as "no hallucination," and the annotator should also identify the specific reference fragment in the original document. If a sentence contradicts the reference, it is classified as "contradictory hallucination," requiring a specific reference fragment and correction suggestions. If the sentence lacks supporting evidence and cannot be verified, it is classified as "unverifiable hallucination," and correction suggestions are required. If the sentence contains no factual information that can be evaluated, it falls into the "no facts" category and requires no further annotation.

[0050] The entire annotation process is divided into two steps. First, GPT4 is used for automatic annotation, and then human annotators correct errors and improve the GPT4 annotation results. Among them, GPT4 can also be replaced by other language models. The prompt words used for GPT4 automatic annotation are as follows: Figure 3 shown.

[0051] See also Figure 4 An example annotation process.

[0052] After the above process of constructing hallucination annotated data, we obtained 12k+ pieces of hallucination detection training data with annotated information. We used the InternLM series as the base large language model for training to obtain the hallucination detector (other models can also be used to construct hallucination detectors). Two training paradigms were used for training:

[0053] On the one hand, the language model paradigm is used for training. Through prompt word construction, the model is allowed to make hallucination judgments, extract reference fragments, and correct hallucination content through language dialogue. The overall prompt words are similar to the prompt words used in fine-grained hallucination annotation.

[0054] On the other hand, training using a classification model paradigm replaces the last layer of the original language model with a linear classification layer. After inputting questions and responses, the model directly determines the type of hallucination. Using both training paradigms, we achieved large language model hallucination detectors with performance far exceeding that of current open-source models.

[0055] By using the trained large language model hallucination detector, the question of the target large language model, the output of the target large language model and the reference data are input, and the hallucination detection results can be obtained, which include the hallucination type, reference fragment and content correction information.

[0056] To illustrate the effectiveness of this method, this example trained a hallucination detector based on the Internlm2 series model and tested its hallucination detection capabilities before and after the test. The test results are shown in Table 1, which effectively demonstrates the effectiveness of our two hallucination detector training schemes.

[0057] Table 1 Test results

[0058]

[0059]

[0060] Among them, 7B and 20B represent the model parameter scale, respectively. G- represents that the model uses the generative hallucination detector training method, and D- represents that the model uses the discriminative hallucination detector training method. Models without G or D prefix represent untrained original models.

[0061] We counted the detection accuracy of these models on the test set of the hallucination detection benchmark. The T and Q in the table refer to the use of different criteria to divide the test set. T refers to dividing the test set by different topics, and Q refers to dividing the test set by different questions.

[0062] In summary, after using the training method of this embodiment, the model's hallucination detection ability is greatly improved, proving the effectiveness of our solution.

[0063] This method further refines the annotation granularity, annotating hallucinations on a sentence-by-sentence basis. After locating the specific location where the hallucination occurs, it retrieves reference documents based on the corresponding content to correct the hallucination. The hallucination detection training corpus constructed in this way possesses more densely labeled information, and the large language model hallucination detector trained on this corpus has superior hallucination detection capabilities. Therefore, the training data used by the hallucination detector of this embodiment has finer annotation granularity and higher information density than the prior art. Furthermore, after hallucination detection, it can automatically perform reference retrieval and correction, enabling more detailed statistical analysis.

[0064] Furthermore, after constructing sentence-level hallucination annotated data, a more effective hallucination detector training scheme was designed. Training was performed using both a generative paradigm for language models and a discriminative paradigm for classification models, achieving good hallucination detection results. Unlike existing techniques that primarily use inductive cues to guide large language models into hallucinating, this method's data collection process focuses on capturing the large language model's natural hallucinatory reactions to information, better reflecting the model's inherent hallucination issues.

[0065] Example 2

[0066] Compared with Example 1, this embodiment uses a different database for topic selection, uses different question and answer generation methods, and uses other open source or closed source models to replace manual annotation in the annotation process. The other processes are the same.

[0067] Example 3

[0068] Compared with Example 1, this embodiment is based on the usage paradigm of the language model, constructs prompt words, and uses language dialogue to enable the model to perform hallucination judgment. Other processes are the same.

[0069] Example 4

[0070] Compared with Example 1, this embodiment modifies the model structure in the hallucination detector based on the usage paradigm of the classification model so that it directly outputs the judgment result. Other processes are the same.

[0071] Example 5

[0072] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the fine-grained large language model hallucination detection method as described in any one of Embodiments 1-4.

[0073] In a typical configuration, an electronic device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0074] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0075] Example 6

[0076] This embodiment provides a computer-readable storage medium, including one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the fine-grained large language model hallucination detection method as described in any one of Embodiments 1-4.

[0077] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0078] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A fine-grained large language model hallucination detection method, characterized in that The hallucination detection model is trained based on the hallucination detection training data, and hallucination detection is performed on the response of the target large language model based on the trained hallucination detection model, wherein the acquisition process of the hallucination detection training data includes the following steps: Obtain subject data and obtain corresponding reference data through retrieval; Based on the reference document data, a language model different from the target large language model is used to generate a plurality of question data and the question data is screened; Input the filtered question data into the language model to obtain the corresponding answer data; For the response data, fine-grained annotation is performed on a sentence-by-sentence basis to form hallucination detection training data. Among them, the InternLM series is used as the base large language model for training to obtain the hallucination detector, which is trained using two training paradigms: Use the language model paradigm for training, construct prompt words, and use language dialogue to enable the model to make hallucination judgments; Use the classification model paradigm for training, replace the last layer of the language model with a linear classification layer, and after inputting questions and responses, the model directly determines the hallucination type. The process of obtaining corresponding reference data through retrieval includes: We perform an exact match on the entries, then sort the candidate entries based on the semantic similarity of the sentences, and use InternLM to judge them and select the entries. When judging, we consider two aspects: one is whether the entry is unique or has some synonyms, and the other is whether the entry is famous or important. We ensure that the selected entries are unique and important. The process of generating problem data includes: Use GPT4 to regenerate questions. After generating questions, use GPT4 again to select questions based on authenticity, answerability, difficulty, and diversity indicators to obtain the final high-quality questions. The process of fine-grained annotation includes: If the sentence contains factual information and is consistent with the reference information, its type is non-hallucination, indicating the specific reference fragment in the original document; If the sentence contradicts the reference, its type is illusion of contradiction, and a specific fragment of the reference and correction suggestions are provided; If a sentence lacks supporting evidence and cannot be verified, it is classified as an unverifiable hallucination and a revision is suggested; If the sentence does not contain any factual information that can be evaluated, it belongs to the no-fact category.

2. A fine-grained large language model hallucination detection method according to claim 1, characterized in that: The retrieval process of the reference data includes: For terms with multiple meanings, entry selection is achieved through exact matching, semantic similarity matching and model judgment.

3. The fine-grained large language model hallucination detection method according to claim 1, characterized in that: The process of generating the problem data includes: The language model configured with prompt words in advance is used, and the reference document data is used as input of the language model to obtain generated question data.

4. A fine-grained large language model hallucination detection method according to claim 1, characterized in that: The screening process of the problem data includes: Question data was screened based on authenticity, answerability, difficulty, and diversity.

5. The fine-grained large language model hallucination detection method according to claim 1, characterized in that: The process of fine-grained annotation of response data includes: Using a language model pre-configured with prompt words, the system takes the response data as input to obtain the corresponding hallucination type, reference segment information, and correction information, thus achieving automatic labeling. Manual error correction and completion are performed on the results of automated annotation.

6. A fine-grained large language model hallucination detection method according to claim 5, characterized in that: The types of hallucinations include no hallucination, unverifiable, no fact, and contradiction.

7. The fine-grained large language model hallucination detection method according to claim 1, characterized in that: The training process of the hallucination detection model includes: By constructing prompt words and using language dialogue, the hallucination detection model is made to judge hallucinations, thereby realizing the training of the hallucination detection model.

8. The fine-grained large language model hallucination detection method according to claim 1, characterized in that: The training process of the hallucination detection model also includes: The last layer of the hallucination detection model is replaced with a linear classification layer. After inputting questions and responses, the hallucination type judgment result is obtained, thus realizing the training of the hallucination detection model.

9. An electronic device, characterized in that: include: One or more processors and a memory, wherein the memory stores one or more programs, wherein the one or more programs include instructions for executing the fine-grained large language model hallucination detection method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that The method comprises one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the fine-grained large language model hallucination detection method according to any one of claims 1 to 8.

Images