Large language model output authenticity detection method, system, device and medium
By inserting trainable vectors into the hidden state of a large language model to learn prototypes of real and illusory categories, the accuracy problem of realism detection of the output of the large language model is solved, and realism detection of the output is realized.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XI AN JIAOTONG UNIV
- Filing Date
- 2026-01-29
- Publication Date
- 2026-05-01
AI Technical Summary
Existing large language models cannot accurately detect the authenticity of output. Existing detection methods focus more on language fluency than factual accuracy, leading to the generation of illusions.
By acquiring a training dataset of questions and answers, a large language model is trained, trainable vectors are inserted to intervene in the hidden state, real and illusion category prototypes are learned, the authenticity of the output is detected using category prototypes, and training is performed using contrastive loss and maximum likelihood estimation loss functions, combined with unlabeled datasets to enhance training.
It achieves realism detection of the output of large language models, can accurately distinguish between real and illusion generation, and improves the accuracy of realism detection of model output.
Smart Images
Figure CN121958784A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing technology, and in particular to a method, system, device, and medium for detecting the authenticity of output from a large language model. Background Technology
[0002] With the widespread application of Large Language Models (LLMs) in natural language understanding and generation, the illusion problem of large models has also become apparent. Illusion refers to the frequent generation of seemingly reasonable but inaccurate statements by large language models, which can lead to serious consequences. This necessitates that large language models not only generate text identical to user prompts but also possess the ability to reduce illusion generation (i.e., ensure the authenticity of the large language model's output).
[0003] Existing large language models primarily rely on scoring functions that assess uncertainty in their output to detect errors. For example, logit-based methods use word-level probabilities as uncertainty scores for the large language model's output. However, these detection methods typically focus more on the fluency of the large model's output than on factual accuracy, failing to accurately detect the authenticity of the large language model's output. Summary of the Invention
[0004] The purpose of this invention is to provide a method, system, device, and medium for detecting the authenticity of large language model outputs, which can solve the problem of not being able to accurately detect the authenticity of large language model outputs.
[0005] To address the aforementioned technical problems, embodiments of the present invention provide a method for detecting the authenticity of large language model outputs, comprising the following steps: We obtain several questions and the answers generated by a large language model for each question, which serve as the training dataset; each answer is labeled to indicate whether the answer is true or false. The large language model is trained using a training dataset. In each iteration of training, selected layers of the large language model are intervened in the hidden state generated by the question through an inserted vector. The truthfulness of the output of the large language model is predicted by fusing the hidden states after intervention of all selected layers. The inserted vector is adjusted according to the predicted truthfulness and the corresponding answer label. The adjusted vector is then used to intervene in the hidden state generated by each layer in the next iteration. The trained large language model generates fusion hidden states based on questions with real answers in the training dataset, which are then used as real hidden states. The center point of all real hidden states is then used as the prototype of the real class. The system obtains the fusion hidden state generated by the trained large language model based on the target question to be detected, and performs authenticity detection on the output of the large language model based on the similarity between the fusion hidden state of the target question and the real category prototype.
[0006] Furthermore, in each iteration of training, the large language model performs a weighted fusion of the hidden states after intervention in all selected layers based on the importance of the hidden states generated by each selected layer to the authenticity of the predicted output of the large language model, in order to obtain a fused hidden state.
[0007] Furthermore, the large language model is trained using the following contrastive loss function and maximum likelihood estimation loss function: ; ; In the formula, The maximum likelihood estimation loss function is used, where N is the number of samples and c is the target class. = This is the normalized embedding vector of the last token output from the last layer after intervention in the large language model. It is a category Category prototype, This is a concentration parameter used to control the distribution around the average direction. The degree of aggregation; ; In the formula, To compare loss functions, To be consistent with the sample A set of positive samples of the same type The temperature parameter is used for comparison of the samples. and For the sample And the internal feature vector of sample j.
[0008] Furthermore, during training, the large language model updates the real category prototype and the hallucination category prototype using the following exponential moving average strategy: ; In the formula, Represents the category prototype vector. The attenuation rate, Indicates that the current training sample belongs to the category The average value of the eigenvectors, This is a normalization operation.
[0009] After training the large language model using the training dataset, the method further includes: Obtain an unlabeled dataset consisting of several questions and their corresponding unlabeled answers; Soft labels are assigned to each answer in the unlabeled dataset using the optimal transfer algorithm; We use an unlabeled dataset with soft-labeled answer assignments to enhance the training of a large language model trained on the training dataset.
[0010] Embodiments of the present invention also provide a large language model output authenticity detection system, comprising the following modules: The data acquisition module is used to acquire several questions and the answers generated by the initial large language model for each question, as a training dataset; each answer has a label to indicate whether the answer is true or false. The model training module is used to train a large language model using a training dataset. In each iteration of training, selected layers of the large language model are intervened in the hidden state generated by the question through an inserted vector. The hidden state is fused from the hidden states after intervention of all selected layers to predict the authenticity of the output of the large language model. The inserted vector is adjusted according to the predicted authenticity and the corresponding answer label. The adjusted vector is then used to intervene in the hidden state generated by each layer in the next iteration. The model application module is used to obtain the fusion hidden states generated by the trained large language model based on the questions in the training dataset where the answers are real, as the real hidden states, and to use the center point of all the real hidden states as the prototype of the real category. The realism detection module is used to obtain the fusion hidden state generated by the trained large language model based on the target question to be detected, and to perform realism detection on the output of the large language model based on the similarity between the fusion hidden state of the target question and the real category prototype.
[0011] Embodiments of the present invention also provide a computer device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described large language model output authenticity detection method.
[0012] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for detecting the authenticity of large language model outputs.
[0013] The method for detecting the authenticity of large language model outputs provided by this invention has at least the following beneficial effects: The large language model is trained using several questions and the answers generated for each question. During training, a trainable vector is inserted into the hidden states of selected layers within the model to intervene in their generation. This vector training adjusts the hidden states to better predict the authenticity of the model's output. This yields hidden states corresponding to questions with true answers (true hidden states) and questions with illusory answers (illusory hidden states). These two hidden states divide the model's output into two clusters: true and illusory. The centers of the true and illusory clusters represent the true and illusory category prototypes, respectively. The learned category prototypes of the trained model accurately reflect the authenticity of the model's output. Therefore, the authenticity of the model's output can be detected by comparing the similarity between the hidden state of the trained model for the question to be detected (which is more likely to predict the authenticity of the model's output) and the true category prototype. Attached Figure Description
[0014] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:
[0015] Figure 1 A flowchart illustrating the method for detecting the authenticity of output from a large language model provided by this invention; Figure 2 A schematic diagram illustrating the principle of a large language model output authenticity detection provided by this invention; Figure 3 This is a schematic diagram of the training process of a large language model provided by the present invention. Detailed Implementation
[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0017] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0018] One embodiment of the present invention relates to a method for detecting the authenticity of large language model outputs. The specific process of the method for detecting the authenticity of large language model outputs in this embodiment can be as follows: Figure 1 As shown, it includes: Step 101: Obtain several questions and answers generated by a large language model for each question as a training dataset; where each answer has a label indicating whether the answer is true or false.
[0019] Step 102: Train the large language model using the training dataset. In each iteration of training, select layers in the large language model are intervened in the hidden state generated by the question through an inserted vector. The hidden state is fused from the hidden states after intervention by all selected layers to predict the authenticity of the output of the large language model. The inserted vector is adjusted according to the predicted authenticity and the corresponding answer label. The adjusted vector is then used to intervene in the hidden state generated by each layer in the next iteration.
[0020] Step 103: Obtain the fusion hidden states generated by the trained large language model based on the questions with real answers in the training dataset, and use them as the real hidden states. Then, use the center point of the aggregation of all real hidden states as the prototype of the real category.
[0021] Step 104: Obtain the fusion hidden state generated by the trained large language model based on the target question to be detected, and perform authenticity detection on the output of the large language model based on the similarity between the fusion hidden state of the target question and the real category prototype.
[0022] This embodiment introduces a lightweight approach without changing the model parameters: trainable vectors are inserted into multiple layers of the large language model. By training and adjusting the vectors, the internal representation space of the large model is reshaped into two highly distinguishable clusters (the real cluster and the illusion cluster), enabling the large language model to learn two class prototypes (real and illusion). Authenticity detection is then performed using these two class prototypes, which simplifies the process of calculating the similarity between a new sample and the "real" prototype.
[0023] See below. Figure 2 The implementation details of the large language model output authenticity detection method in this embodiment are explained in detail. The following content is only for the convenience of understanding and is not necessary for implementing this solution.
[0024] First see Figure 2 The first stage involves defining a labeled dataset for pre-training, with input prompts (i.e., questions). Answers generated by large language models Use real labels The examples are labeled to indicate whether they are true or false, thus forming a set of examples. } (i.e., the training dataset), where N is the total number of examples.
[0025] Secondly, to distinguish between the representations of true and false answers within the model, this approach introduces a lightweight trainable vector into multiple target layers during the forward propagation process without changing any parameters of the pre-trained large language model. Specifically, a fixed layer is selected here. For each selected layer Define trainable vectors For example, trainable vectors can be inserted into layers 4, 6, 8, and 10 of a large language model, allowing each data stream after intervention to independently complete forward propagation and extract features from four final layers (feature vectors corresponding to the last extracted token). It is the hidden state dimension of the model.
[0026] As shown below, in each target layer Insert trainable vectors: ; in, For the first The insertion strength coefficient of the layer.
[0027] Since different layers exhibit different levels of realism—the bottom layer focuses on factual details, the middle layer on logical coherence, and the top layer on semantic rationality—an importance coefficient is set to measure the weight of each layer, thereby integrating information from these different levels.
[0028] Specifically, for the target layer to be intervened, the global feature representation of the layer is first obtained through global average pooling, then the importance score of the layer is calculated through a lightweight MLP, and finally the importance score of the target layer is normalized: ; ; in, It is a global average pooling operation. Finally, the hidden state of the last token in each layer is extracted and weighted and fused to obtain the overall feature vector (fused hidden state): ; in, To extract the hidden state at the last position of the sequence.
[0029] This embodiment utilizes the labels of the annotated data and jointly optimizes the parameters of all vectors and the importance MLP through backpropagation to obtain the best large model for separation. The importance parameter is used to determine which intervention layer's subsequent path produces the best features for the current input.
[0030] In the specific implementation, the model training process of this embodiment can be found in [reference needed]. Figure 3To effectively detect hallucinations, we first propose training by performing maximum likelihood estimation, explicitly modeling the probability distribution using von Mises_Fisher: ; in, = Represents the normalized feature vector. It is a category Category prototype, The concentration parameter controls the distribution to be centered around the average direction. The degree of aggregation.
[0031] Under the above model, the MLE loss function is defined as: ; In the formula, N is the number of samples, and c is the target category. = This is the normalized embedding vector of the last token output from the last layer after intervention in the large language model. It is a category Category prototype, This is a concentration parameter used to control the distribution around the average direction. The degree of aggregation.
[0032] In addition, to better train the two class prototypes, a contrastive loss is introduced to enhance inter-class separation. The core idea is to bring similar samples closer together and push dissimilar samples further apart, thereby enhancing inter-class compactness and separability. ; In the formula, To be consistent with the sample A set of positive samples of the same type The temperature parameter is used for comparison of the samples. and For the sample And the internal feature vector of sample j.
[0033] Combining the maximum likelihood loss function and the contrastive loss function yields the overall loss function: ; As can be seen, this embodiment uses supervised learning for model pre-training, utilizing an example set. Training begins with initializing all parameters. In each iteration, the input text undergoes forward propagation through the large model, inserting corresponding vectors at specified layers. Importance weights for each layer are calculated using defined importance evaluation coefficients, and the multi-layer features are weighted and fused to obtain the final feature representation. Based on this feature representation, a hybrid loss function is calculated, and all trainable parameters are updated using the backpropagation algorithm.
[0034] Specifically, for training the category prototypes, an exponential moving average strategy is used to update the category prototypes: ; In the formula, The attenuation rate, Indicates the category in the current training sample The average characteristic representation.
[0035] In one example, see Figure 2 In the second stage, since the existing manually labeled datasets have few samples, but the unlabeled datasets are massive, and while using a small dataset can help with hallucination detection, it may not be able to fully capture the difference between reality and hallucination, this embodiment uses the optimal transfer algorithm to assign soft labels to the unlabeled data for model enhancement training, thereby improving the model's reasoning ability, as shown below: ; ; in, Here is the cost matrix. It is a priori distribution.
[0036] Specifically, samples with high confidence levels are selected for training: ; Samples with higher confidence can be inserted into the labeled dataset mentioned above, and the model parameters can be trained on the expanded dataset.
[0037] See last. Figure 2 In the third stage, after the training described above, two separate clusters (real and hallucination) are shaped in the model's representation space. At the same time, the model learns the centroids of these two clusters, which are the prototype vectors of the real and hallucination categories. Therefore, during the model's inference process, input prompts (i.e., questions) are given first. And the text generated by the model (i.e., the answer generated by the large language model). Define a binary predictor. : ; Predictor The output reflects the consistency between the generated result and the true distribution.
[0038] Define a realism scoring function based on the hallucination detection (i.e., similarity detection) performed by the predictor using learned category prototypes. Essentially, it detects how far the test sample is from the trained true category prototype: ; =[s(x)(scoring function)>threshold], where 1 represents reality and 0 represents illusion.
[0039] For a new sample problem, its feature vector is extracted and then calculated using a scoring function to determine the authenticity of the output of the large model.
[0040] To verify the effectiveness, the method described in this invention was adopted, and the mainstream open-source model LLaMA-3.1-8b was used to evaluate the scheme. The evaluation was carried out on TruthfulQA, TriviaQA, and domain-specific question answering datasets (Sciq). 25% of the question-answer pairs in each dataset were reserved for testing.
[0041] The effectiveness of this scheme was verified using AUROC and AUPRC metrics. The results are shown in Table 1:
[0042] Table 1 The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this invention. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the protection scope of this invention.
[0043] Another embodiment of the present invention relates to a large language model output authenticity detection system. The implementation details of this embodiment's large language model output authenticity detection system are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this solution. This embodiment's large language model output authenticity detection system includes: The data acquisition module is used to acquire several questions and the answers generated by the initial large language model for each question, as a training dataset; each answer has a label to indicate whether the answer is true or false. The model training module is used to train a large language model using a training dataset. In each iteration of training, selected layers of the large language model are intervened in the hidden state generated by the question through an inserted vector. The hidden state is fused from the hidden states after intervention of all selected layers to predict the authenticity of the output of the large language model. The inserted vector is adjusted according to the predicted authenticity and the corresponding answer label. The adjusted vector is then used to intervene in the hidden state generated by each layer in the next iteration. The model application module is used to obtain the fusion hidden states generated by the trained large language model based on the questions in the training dataset where the answers are real, as the real hidden states, and to use the center point of all the real hidden states as the prototype of the real category. The realism detection module is used to obtain the fusion hidden state generated by the trained large language model based on the target question to be detected, and to perform realism detection on the output of the large language model based on the similarity between the fusion hidden state of the target question and the real category prototype.
[0044] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.
[0045] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by this invention; however, this does not mean that other units are absent from this embodiment.
[0046] Another embodiment of the present invention relates to a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the large language model output authenticity detection method of the above embodiments.
[0047] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.
[0048] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.
[0049] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.
[0050] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0051] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing the present invention, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of the present invention.
Claims
1. A method for detecting the authenticity of output from a large language model, characterized in that, The method includes: We obtain several questions and the answers generated by a large language model for each question, which serve as the training dataset; each answer is labeled to indicate whether the answer is true or false. The large language model is trained using a training dataset. In each iteration of training, selected layers of the large language model are intervened in the hidden state generated by the question through an inserted vector. The truthfulness of the output of the large language model is predicted by fusing the hidden states after intervention of all selected layers. The inserted vector is adjusted according to the predicted truthfulness and the corresponding answer label. The adjusted vector is then used to intervene in the hidden state generated by each layer in the next iteration. The trained large language model generates fusion hidden states based on questions with real answers in the training dataset, which are then used as real hidden states. The center point of all real hidden states is then used as the prototype of the real class. The system obtains the fusion hidden state generated by the trained large language model based on the target question to be detected, and performs authenticity detection on the output of the large language model based on the similarity between the fusion hidden state of the target question and the real category prototype.
2. The method for detecting the authenticity of large language model output according to claim 1, characterized in that, In each iteration of training, the large language model performs a weighted fusion of the hidden states after intervention in all selected layers, based on the importance of the hidden states generated by each selected layer to the authenticity of the predicted output of the large language model, in order to obtain a fused hidden state.
3. The method for detecting the authenticity of large language model output according to claim 1, characterized in that, The large language model is trained using the following contrastive loss function and maximum likelihood estimation loss function: ； ； In the formula, The maximum likelihood estimation loss function is used, where N is the number of samples and c is the target class. = This is the normalized embedding vector of the last token output from the last layer after intervention in the large language model. It is a category The category prototype, This is a concentration parameter used to control the distribution around the average direction. The degree of aggregation; ； In the formula, To compare loss functions, To be consistent with the sample A set of positive samples of the same type The comparison temperature parameter for the sample. and For the sample And the internal feature vector of sample j.
4. The method for detecting the authenticity of large language model output according to claim 3, characterized in that, During training, the large language model updates the real category prototypes and illusion category prototypes using the following exponential moving average strategy: ； In the formula, Represents the category prototype vector. The attenuation rate, Indicates that the current training sample belongs to the category The average value of the eigenvectors, This is a normalization operation.
5. The method for detecting the authenticity of large language model output according to claim 1, characterized in that, After training the large language model using the training dataset, the method further includes: Obtain an unlabeled dataset consisting of several questions and their corresponding unlabeled answers; Soft labels are assigned to each answer in the unlabeled dataset using the optimal transfer algorithm; We use an unlabeled dataset with soft-labeled answer assignments to enhance the training of a large language model trained on the training dataset.
6. A system for detecting the authenticity of output from a large language model, characterized in that, The method includes: The data acquisition module is used to acquire several questions and the answers generated by the initial large language model for each question, as a training dataset; each answer has a label to indicate whether the answer is true or false. The model training module is used to train a large language model using a training dataset. In each iteration of training, selected layers of the large language model are intervened in the hidden state generated by the question through an inserted vector. The hidden state is fused from the hidden states after intervention of all selected layers to predict the authenticity of the output of the large language model. The inserted vector is adjusted according to the predicted authenticity and the corresponding answer label. The adjusted vector is then used to intervene in the hidden state generated by each layer in the next iteration. The model application module is used to obtain the fusion hidden states generated by the trained large language model based on the questions in the training dataset where the answers are real, as the real hidden states, and to use the center point of all the real hidden states as the prototype of the real category. The realism detection module is used to obtain the fusion hidden state generated by the trained large language model based on the target question to be detected, and to perform realism detection on the output of the large language model based on the similarity between the fusion hidden state of the target question and the real category prototype.
7. A computer device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the large language model output authenticity detection method as described in any one of claims 1 to 5.
8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for detecting the authenticity of output of a large language model as described in any one of claims 1 to 5.