A hallucination mitigation method for large language models based on contrast decoding

Abstract

The present application relates to a kind of big language model hallucination relief method based on contrast decoding, by eliminating potential hallucination component from the prediction information of the final layer of big model, the authenticity of output content is enhanced, design multi-layer fusion mechanism, make full use of the probability distribution information predicted by each layer of model.Introduce hallucination information screening model, in a simple and efficient way, integrate and analyze the hallucination information that big language model low layer can contain.By training hallucination information screening model on hallucination question and answer data set, it can accurately capture and fit the hallucination data characteristics commonly existing in data set, so as to realize the effective identification and filtering of hallucination information.In decoding stage, the probability distribution output by the final layer of model is compared with the probability distribution output by hallucination information screening model, and the difference is used as a guide to dynamically adjust the probability distribution of the next token generation, so as to guide big language model to produce more real, hallucination-free content in the decoding process.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and more particularly to a method for alleviating hallucinations in large language models based on contrastive decoding. Background Technology

[0002] The rapid development of large language models has brought revolutionary changes to the field of artificial intelligence research. Large language models have demonstrated powerful capabilities in natural language understanding and handling complex tasks, and their widespread application has provided a significant impetus for the development of natural language processing and artificial intelligence technologies. However, despite their excellent performance on many tasks, large models sometimes generate text that deviates from the facts; this phenomenon is known as illusion. Illusion can lead to the spread of misleading information, impairing the accuracy of decision-making, especially in high-risk fields such as healthcare and finance, where illusory information can have irreversible consequences. Therefore, solving the illusion problem in large language models has significant practical application value.

[0003] Currently, mainstream methods in this field include manually or automatically removing noisy training data, reinforcement learning based on human feedback, designing decoding strategies, and introducing external knowledge. Methods for removing noisy training data operate on the dataset. Initially, researchers used manual data selection, but with the increasing size of pre-training corpora in recent years, rule-based or large language models (such as ChatGPT) are generally used to automatically organize the pre-training corpora to minimize unverifiable or unreliable data. Reinforcement learning based on human feedback is a commonly used method. To alleviate hallucinations, researchers have designed a special reward function that encourages the model to express "I don't know" for uncertain answers. This honesty-oriented reward function is more advantageous in alleviating hallucinations. Another approach is designing decoding strategies. This method does not require changing the pre-trained knowledge of the original large language model but tends to change the model's final prediction. The most common methods are comparative decoding with models containing hallucination information and decoding with inference time intervention. Methods for introducing external knowledge generally involve two steps: first, acquiring external knowledge, and second, using external knowledge to guide response generation. This method transfers real-time updated knowledge to the large language model, providing interpretability.

[0004] Existing technical solutions include the following: Corpus filtering, which manually removes illusory data from the dataset or filters inaccurate data based on rule retrieval. Supervised fine-tuning, which first annotates or collects a large amount of factual data, then fine-tunes the pre-trained base language model using a training objective based on illusion settings instead of maximum likelihood estimation (e.g., minimum risk training). This approach, employing carefully designed supervised fine-tuning strategies, can mitigate the illusion of the large model. Illusion scorer design, which involves designing an additional fact scorer. The scorer model is typically designed based on understanding-based models like BERT, allowing the language model to generate multiple outputs for the same question. These outputs are then factually scored, and the highest-scoring result is selected as the final output. Inducing illusions before comparative decoding, which first induces illusions from the original model, then fine-tunes the original model on the illusion dataset to construct a relatively weak large model. Finally, by comparing the original model with the fact-weak model during decoding, the internalized non-factual knowledge in the weak model is eliminated from the output space of the original model, guiding the model to generate more factual content. Prompt engineering allows large language models to change significantly based on user-provided prompts. It enables models to understand and perform various tasks through predefined prompts without requiring large-scale retraining or fine-tuning.

[0005] The aforementioned existing technical solutions have shortcomings. Manually removing fake data from datasets may be influenced by subjective judgment, leading to insufficient objectivity and accuracy in data filtering. Furthermore, manual filtering requires significant time and human resources, especially when dealing with large-scale datasets, resulting in low efficiency. Rule-based retrieval methods may not cover all types of false data, thus failing to completely remove problematic data from the corpus. While supervised fine-tuning can significantly improve the factual accuracy of model outputs, it also has some significant drawbacks. Firstly, it relies on a large amount of high-quality labeled data, which is difficult to obtain. Secondly, the fine-tuning process places high demands on hardware resources. For example, the memory of a common NVIDIA RTX 3090 graphics card typically only supports fine-tuning models with a maximum of approximately 1.5 billion parameters, while most models currently in use have between 6 billion and 13 billion parameters. Training the illusion scorer remains a supervised training process, still requiring high-quality labeled training data, and the accuracy of the illusion scorer significantly impacts the final output. If the scorer's accuracy is low, it may be unable to accurately identify and evaluate illusions, thus affecting its overall performance and the reliability of the output results. Meanwhile, because the entire process requires generating multiple responses before scoring, it is quite lengthy and computationally intensive. While the comparative decoding of two large models has indeed shown significant effectiveness in mitigating the illusion of large language models, this method also introduces a significant problem: a substantial increase in decoding time. This is because the system needs to process information from two large models simultaneously during the comparative decoding process. This time delay not only affects the overall system performance but also leads to a poor user experience in actual use. Constructing prompt words through prompt engineering to positively influence the output is relatively easy to implement, but designing effective prompt templates requires deep domain knowledge and is quite challenging. Furthermore, current large language models are probabilistic generative models, and the uncertainty and poor interpretability of their output make it very difficult to correct responses based on prompt words. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a hallucination mitigation method based on a large language model using contrastive decoding. This method uses the predicted information from the final layer of the large language model to subtract hallucination information, thereby reducing hallucinations. It fully utilizes the probability distributions predicted by all layers of the large language model and employs an additional hallucination information filtering model to fuse all hallucination information present in the lower layers. By training the hallucination information filtering model on some hallucination question-and-answer datasets, the model is made to fit the characteristics of most hallucination data in the dataset. The difference between the probability distribution of the final layer and the probability distribution obtained from the hallucination information filtering model is used as the distribution for the next label. The hallucination mitigation method specifically includes:

[0007] Step 1: Load the pre-trained hallucination information filtering model. If the hallucination information filtering model does not exist, execute the following subdivided training steps 11 to 17; otherwise, execute step 2.

[0008] Step 11: Design a hallucination information filtering model. Set the input of the hallucination information filtering model to N dimensions, where N is the number of layers in the large language model, and the output to 1 dimension.

[0009] Step 12: Prepare the hallucination question-and-answer dataset, which can be a public dataset or a self-constructed question-and-answer dataset, and needs to contain two labels: question and hallucination answer;

[0010] Step 13: Input the hallucination question-answering dataset into the large language model to obtain the labels of the hallucination responses in the hallucination question-answering dataset. Use a tokenizer to segment the hallucination responses of each data point into tokens, and then map the tokens to the indexes in the vocabulary of the large language model for use in calculating the loss.

[0011] Step 14: Obtain the probability distribution of the hallucination response predictions for the layers other than the final layer in the large language model. Take the question of each data in the dataset as the input of the large language model, and according to the character-by-character prediction characteristic of the large language model, obtain the probability distribution of each character predicted by the 0 to N-1 layers of the large oracle model (excluding the final layer) for the hallucination response.

[0012] Step 15: Obtain the label predicted by the hallucination information filtering model. Stack the probability distributions predicted by the large language model layers 0 to N-1 for the hallucination response as the input of the hallucination information filtering model. Obtain the probability distribution calculated by the hallucination information filtering model. Based on the greedy decoding, select the token with the highest probability as the predicted label of the hallucination information filtering model.

[0013] Step 16: Train the hallucination information filtering model. Calculate the loss between the predicted labels from Step 15 and the labels of the hallucination answers from Step 13. Specifically, use the cross-entropy loss function and set parameters including the learning rate and the number of learning rounds for training.

[0014] Step 17: Backpropagate the loss to the hallucination information filtering model, repeat step 16, and when the number of learning iterations in step 16 is reached, save the hallucination information filtering model and then execute step 2.

[0015] Step 2: Obtain the probability distribution of the final layer of the large language model for the prediction of the next token;

[0016] Step 3: Obtain the probability distribution of the next token prediction for layers 0 to N-1 of the large language model, excluding the final layer;

[0017] Step 4: Obtain the probability distribution results of the hallucination information filtering model, stack the probability distributions obtained in Step 3 as the input of the hallucination information filtering model, and output the probability distribution of the hallucination information filtering model.

[0018] Step 5: Compare the decoding and use the difference between the probability distribution predicted by the final layer of the large language model and the probability distribution obtained by the illusion information filtering model as the probability distribution of the next token.

[0019] Step 6: Decode the probability distribution after comparison decoding into text. Use a greedy decoding algorithm to decode the probability distribution into text.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0021] 1. This invention uses a comparative decoding technique to remove hallucinatory information from the predictions of the final layer of a large language model, thereby reducing hallucinations and increasing the factuality of the output.

[0022] 2. This invention alleviates hallucinations through an additional hallucination information filtering model, and a simple linear layer can meet the requirements, saving computational resources and time costs.

[0023] 3. This invention designs an illusion information filtering model that is independent of the large language model. It is used as an external component and does not require changing the knowledge originally learned by the large language model. This ensures the integrity of the model and the consistency of the data, while effectively improving the authenticity and reliability of the generated content. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the structure of the IEAH model proposed in this invention;

[0025] Figure 2 This is a flowchart of the processing method of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0027] The following is a detailed explanation with reference to the accompanying drawings.

[0028] In this invention, a token refers to the basic unit when a large language model processes input text. It can be a Chinese character, a word, a sub-word, a character, or other text fragments.

[0029] The Illusion Question-Answer Dataset of this invention is a collection of illusion data focused on question-answering scenarios. It covers misleading information or "illusion" phenomena that large language models may encounter when generating text. Each sample consists of a real question and a corresponding response that seems reasonable but is actually inconsistent with the question context or contains logical errors.

[0030] This invention proposes a novel decoding strategy. The core idea is to subtract predictions containing hallucination information from the final layer predictions of a large language model, thereby mitigating the hallucination effect of the large model. Therefore, this invention designs an additional hallucination information filtering model (preferably a single-linear-layer model) to fuse hallucination information present in the lower-level predictions of the model. This decoding strategy effectively reduces hallucination information in the model output, enhancing the factuality and reliability of the model output. More importantly, the method of this invention has a low dependence on the complexity of the hallucination information filtering model. For example, the preferred single-linear-layer model structure in this invention is very simple, the training process is relatively easy, and the training cost is low, making the model more efficient and economical.

[0031] The hallucination information filtering model of this invention can be a single-linear layer model, or it can be a convolutional neural network, a recurrent neural network, a BERT model, or a large language model. Larger models are not recommended because as the model size increases, the required computing resources, memory usage, and energy consumption will increase significantly.

[0032] This case study uses the large language model LLaMA-7B as an example. The output of layer 32 is the model's raw output. When asked, "Which goddess is associated with Easter?", the model incorrectly outputs "Ishtar," while the correct answer is "Istr." One possible reason is that the pre-training objective of the large language model, namely prediction of the next label based on maximum likelihood, does not always align with real-world facts but tends to rely on surface patterns in the training corpus. Simply put, "Ishtar" might appear frequently in the training corpus, leading the model to tend to output it. However, the model does not completely discard other knowledge but stores it in lower layers. The purpose of this invention is to extract useful information from the lower layers of the large language model by constructing an additional illusion information filtering model (preferably an illusion information filtering model composed of a single-linear layer model) to alleviate illusions.

[0033] The invention will now be described in detail with reference to the accompanying drawings.

[0034] Figure 1The present invention provides a schematic diagram of a model structure for mitigating large-scale hallucinations, which is referred to as the IEAH model (Integrate the hallucination information existing in all Evolving layers to Alleviate Hallucinations). Figure 1 This paper uses the LLaMA-7B model as an example of a large language model, but its practical applications are not limited to the LLaMA-7B model. The core idea of ​​this invention is to remove hallucinatory information from the probability distribution predicted by the final layer of the large language model to alleviate hallucinations. The IEAH model fully utilizes the predictive information from all layers of the large language model. Figure 1 As shown, firstly, for the input large language model problem, the probability distribution of the next token prediction for all layers of the large language model (including layer 0, i.e., the word embedding layer) is obtained. Next, this invention stacks the remaining layers (except the final layer) as input to a single-linear layer model. Internally, this single-linear layer model performs a weighted sum of these input probability distributions to obtain the result of the single-linear layer model. Finally, this invention compares and decodes the probability distribution predicted by the final layer of the large language model with the probability distribution obtained by the single-linear layer model, and uses the difference as the probability distribution of the next token.

[0035] from Figure 1 As can be seen, the IEAH model corrects the original erroneous output of the large language model.

[0036] Step 1: Load the pre-trained hallucination information filtering model. The preferred hallucination information filtering model in this invention is a fully connected network model composed of single linear layers. If the hallucination information filtering model does not exist, perform the following subdivided training step 11; otherwise, perform step 2.

[0037] Step 11: Using a single-linear-layer network model as an example, design a single-linear-layer model. Set the input of the hallucination information filtering model to N dimensions, where N is the number of layers in the large language model, and the output to 1 dimension.

[0038] Step 12: Prepare the hallucination question-and-answer dataset. This can be a self-constructed dataset that includes two labels: question and hallucination answer, or it can be a public dataset, such as the HaluEval dataset.

[0039] Step 13: Input the hallucination question-answering dataset into the large language model to obtain the labels of the hallucination responses in the dataset. Use a tokenizer to segment the hallucination responses of each data point into tokens, and then map these tokens to the indexes in the model's vocabulary for use in calculating the loss.

[0040] Step 14: Obtain the probability distribution of the hallucination response predictions for all layers in the large language model except the final layer. Take the question of each data point in the dataset as the input to the large language model, and based on the character-by-character prediction characteristic of the large language model, obtain the probability distribution of each character predicted by the 0 to N-1 layers of the large oracle model (excluding the final layer) for the hallucination response. The mathematical expression is as follows:

[0041] q(x t |x <t ,j)=softmax(logit(x t |x <t ,j)),j∈{0,1,2…N-1} (1)

[0042] Among them, logit(x t |x <t (j) is the next label x predicted by the j-th layer (j∈{0,1,2,…,N-1}) of the large language model. t The probability is calculated, and finally, it is normalized into a probability distribution through the softmax operation.

[0043] Step 15: Obtain the labels predicted by the hallucination information filtering model. Stack the probability distributions predicted by the large language model layers 0 to N-1 for the hallucination response as the input of the hallucination information filtering model. Obtain the probability distribution calculated by the hallucination information filtering model, and select the token with the highest probability as the predicted label of the single linear layer model according to the greedy decoding.

[0044] Step 16: Train the hallucination information filtering model. Calculate the loss between the predicted labels from Step 15 and the labels of the hallucination answers from Step 13. Here, the cross-entropy loss function is preferred. Set parameters such as learning rate and number of learning rounds for training.

[0045] Step 17: Backpropagate the loss to the hallucination information filtering model, repeat step 16, and save the hallucination information filtering model when the number of learning iterations in step 16 is reached, and then execute step 2.

[0046] Step 2: Obtain the probability distribution of the final layer of the large language model for the prediction of the next token.

[0047] This is similar to the formula in step 14, but for the sake of convenience in the following description, the probability of the last layer prediction of the large language model is listed separately here.

[0048] p(x t |x <t = softmax(logit(x) t |x <t ,N)) (2)

[0049] Step 3: Obtain the probability distribution of the prediction of the next token for the remaining layers of the large language model, excluding the final layer. The formula is the same as in step 14 of the training phase.

[0050] q(x t |x <t ,j)=softmax(logit(x t |x <t ,j)),j∈{0,1,2…N-1} (3)

[0051] Step 4: Obtain the probability distribution results of the hallucination information filtering model. Stack the probability distributions obtained in Step 3 and use them as input to the hallucination information filtering model to obtain the predicted results. The mathematical expression is as follows:

[0052]

[0053] in, This represents stacking the probability distributions predicted from layers 0 to N-1. LinearModel() performs the operation of the hallucination information filtering model, and LM represents the probability distribution after the hallucination information filtering model is operated on.

[0054] Step 5: Comparative Decoding. The difference between the probability distribution predicted by the final layer of the large language model and the probability distribution obtained by the hallucination information filtering model is used as the distribution of the next label.

[0055] To avoid grammatical issues after comparison and decoding, this invention introduces an adaptive rationality constraint V. head Only a few labels with high probabilities predicted by the final layer of the model are retained, where α ranges from [0,1]. The larger α is, the fewer labels are retained. The mathematical expression is as follows:

[0056]

[0057] Step 6: Decode the probability distribution after contrastive decoding into words. Through greedy decoding, the probability distribution is decoded into words. The probability distribution obtained by the hallucination information filtering model contains hallucination information; contrastive decoding is equivalent to removing hallucination information from the predictions of the large language model. This method can effectively reduce hallucination information and generate more accurate and logical output.

[0058] It should be noted that the specific embodiments described above are exemplary, and those skilled in the art can devise various solutions inspired by the disclosure of this invention. These solutions all fall within the scope of this invention and its protection. Those skilled in the art should understand that this specification and its accompanying drawings are illustrative and not intended to limit the scope of the claims. The scope of protection of this invention is defined by the claims and their equivalents.

Claims

1. A method for alleviating hallucinations in a large language model based on contrastive decoding, characterized in that, The hallucination relief method uses the predicted information from the final layer of a large language model to subtract hallucination information to alleviate hallucinations. It fully utilizes the probability distributions predicted by all layers of the large language model and employs an additional hallucination information filtering model to fuse all hallucination information present in the lower layers. This model is trained on a hallucination question-and-answer dataset to fit the characteristics of the hallucination data in the dataset. The difference between the probability distribution of the final layer and the probability distribution obtained from the hallucination information filtering model is used as the distribution for the next label. Specifically, the hallucination relief method includes: Step 1: Load the pre-trained hallucination information filtering model. If the hallucination information filtering model does not exist, execute the following subdivided training steps 11 to 17; otherwise, execute step 2. Step 11: Design a hallucination information filtering model. Set the input of the hallucination information filtering model to N dimensions, where N is the number of layers in the large language model, and the output to 1 dimension. Step 12: Prepare the hallucination question-and-answer dataset, which can be a public dataset or a self-constructed question-and-answer dataset, and needs to contain two labels: question and hallucination answer; Step 13: Input the hallucination question-answering dataset into the large language model to obtain the labels of the hallucination responses in the hallucination question-answering dataset. Use a tokenizer to segment the hallucination responses of each data point into tokens, and then map the tokens to the indexes in the vocabulary of the large language model for use in calculating the loss. Step 14: Obtain the probability distribution of the hallucination response predictions for the layers other than the final layer in the large language model. Take the question of each data in the dataset as the input of the large language model, and according to the character-by-character prediction characteristic of the large language model, obtain the probability distribution of each character predicted by the 0 to N-1 layers of the large oracle model (excluding the final layer) for the hallucination response. Step 15: Obtain the label predicted by the hallucination information filtering model. Stack the probability distributions predicted by the large language model layers 0 to N-1 for the hallucination response as the input of the hallucination information filtering model. Obtain the probability distribution calculated by the hallucination information filtering model. Based on the greedy decoding, select the token with the highest probability as the predicted label of the hallucination information filtering model. Step 16: Train the hallucination information filtering model. Calculate the loss between the predicted labels from Step 15 and the labels of the hallucination answers from Step 13. Specifically, use the cross-entropy loss function and set parameters including the learning rate and the number of learning rounds for training. Step 17: Backpropagate the loss to the hallucination information filtering model, repeat step 16, and when the number of learning iterations in step 16 is reached, save the hallucination information filtering model and then execute step 2. Step 2: Obtain the probability distribution of the final layer of the large language model for the prediction of the next token; Step 3: Obtain the probability distribution of the next token prediction for layers 0 to N-1 of the large language model, excluding the final layer; Step 4: Obtain the probability distribution results of the hallucination information filtering model, stack the probability distributions obtained in Step 3 as the input of the hallucination information filtering model, and output the probability distribution of the hallucination information filtering model. Step 5: Compare the decoding and use the difference between the probability distribution predicted by the final layer of the large language model and the probability distribution obtained by the illusion information filtering model as the probability distribution of the next token. Step 6: Decode the probability distribution after comparison decoding into text. Use a greedy decoding algorithm to decode the probability distribution into text.

Images