Detection device and detection method

Abstract

A detection device (10) comprises: a generation unit (12) that causes an LLM (121) to generate a plurality of answer sentences to a question sentence; and a detection unit (13) that causes an LLM (131) to determine whether or not there is any inconsistency between the plurality of answer sentences generated by the LLM (121), and detects whether or not the LLM (121) exhibits a hallucination on the basis of the determination result.

Description

Detection device and detection method

[0001] The present invention relates to a detection device and a detection method.

[0002] A large-scale language model (LLM) is a natural language processing model trained using large amounts of text data. LLMs have made remarkable progress in recent years due to the development of sequence-to-sequence deep learning techniques such as Transformer-based language models. By fine-tuning LLMs, they can be applied to various natural language processing tasks such as text classification, sentiment analysis, information extraction, text summarization and generation, and question answering.

[0003] Furthermore, advances in LLM have led to more fluent natural language generation, improving many downstream tasks such as proofreading and editing text, summarizing text and concepts, bouncing ideas off each other and brainstorming, identifying research and issues, proposing ideas, etc. One model that has attracted particular attention is OpenAI's GPT, which is available for free and has quickly gained a large number of users since its release.

[0004] International Publication No. 2022 / 059209

[0005] Theodore Zhao, Mu Wei, J. Samuel Preston, Hoifung Poon, “LLM Calibration and Automatic Hallucination Detection via Pareto Optimal Self-supervision”, arXiv:2306.16564v2 [cs.CL] 6 Jul 2023., [Retrieved August 29, 2020], Internet <URL: https: / / arxiv.org / abs / 2306.16564>

[0006] However, it is known that conventional deep learning-based language generation may generate incorrect sentences (hallucination), which means that sentences generated by LLM are not necessarily correct as information, which could be a hurdle to the widespread use of LLM.

[0007] However, conventional methods have the problem of requiring the use of external knowledge to detect the accuracy of the sentences generated by LLM, which makes the system large-scale.

[0008] The present invention has been made in view of the above, and aims to provide a detection device and a detection method that can easily detect the correctness of a sentence generated by an LLM.

[0009] In order to solve the above-mentioned problems and achieve the objectives, the detection device is characterized by having a generation unit that causes a first language model to generate multiple answer sentences to a question sentence, and a detection unit that causes a second language model to determine whether there are any contradictions between the multiple answer sentences generated by the first language model, and detects the presence or absence of hallucination in the first language model based on the determination result.

[0010] According to the present invention, the correctness of a sentence generated by an LLM can be easily detected.

[0011] FIG. 1 is a diagram illustrating the Transformer architecture. FIG. 2 is a diagram illustrating an example of a GPT response. FIG. 3 is a diagram illustrating an example of a post. FIG. 4 is a diagram illustrating Ensemble. FIG. 5 is a diagram illustrating an example of a GPT response. FIG. 6 is a diagram illustrating an example of the configuration of a detection device according to an embodiment. FIG. 7 is a flowchart showing the procedure of a detection method according to an embodiment. FIG. 8 is a diagram illustrating a first calculation process. FIG. 9 is a diagram illustrating an example of an input to a GPT and a determination result. FIG. 10 is a diagram illustrating an example of an input to a GPT and a determination result. FIG. 11 is a diagram illustrating an example of a computer on which a detection device is implemented by executing a program.

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A detailed description of embodiments of a detection device and a detection method according to the present invention will be given below with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described below.

[0013] [Embodiment] [GPT (Generative Pre-trained Transformer)] GPT is a neural network model with a Transformer architecture, and is a type of LLM.

[0014] Figure 1 explains the Transformer architecture. Transformer is a network architecture that connects an encoder and decoder using a model called attention, and it has significantly improved the scores of models that used the previously mainstream CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network), and has achieved a significant reduction in training time.

[0015] The encoder part of the Transformer architecture is called BERT (Bidirectional Encoder Representations from Transformers), and the decoder part is called GPT. BERT is a natural language processing model that converts natural language into vector representations. GPT is a generative model that decodes vector representations into natural language.

[0016] As of July 2023, the latest version, GPT-4, employs the concept of reinforcement learning based on human feedback, enabling it to generate natural-sounding sentences that sound as if they are being conversed with a human. GPT's use cases include proofreading and correcting sentences, summarizing sentences and concepts, bouncing ideas off each other and brainstorming, identifying research and issues, and proposing ideas, and it is used by many users.

[0017] However, GPT often outputs incorrect information with confidence (hallucination), which can confuse users. We believe this problem can be an obstacle to developing applications that use GPT. In the evaluation experiments of the embodiment, we use GPT as a representative example of LLM.

[0018] [Problems with LLM] One problem with LLM is that it can generate information that is not factual (Hallucination).

[0019] There are several possible causes of hallucination, but the two main causes are thought to be that the task LLM solves is next word prediction, and that learning is performed based on training data.

[0020] LLM solves next-word prediction problems, but does not evaluate the semantic accuracy of the generated sentences. Furthermore, training data often contains information that is not factual (such as opinions and ideas), and an LLM trained on such data may generate sentences that are not factual. Furthermore, LLMs are not trained to respond "I don't know" to user questions, and they may generate plausible answers even when they lack sufficient information to provide a correct answer, which can lead to hallucination.

[0021] Figure 2 shows an example of a response from GPT. Figure 2 shows the response from chatGPT, a chatbot using GPT, when asked, "Tell me about Perfume."

[0022] The answer in Figure 2 contains an error in the member section, but it is a confident answer and is likely to confuse users. There is also a post on OpenAI's official Q&A site about GPT confidently answering incorrect information, which is shown in Figure 3. Figure 3 shows an example of a post.

[0023] The post in Figure 3 requests that the output also include how accurate GPT thinks the answer is. However, there has been no response to this post, and the issue is currently being overlooked.

[0024] Furthermore, a method for detecting hallucination has been proposed as described in Non-Patent Document 1. However, the method described in Non-Patent Document 1 has a problem in that it requires external knowledge for detection, resulting in an increase in the scale of the system.

[0025] In this embodiment, by applying Ensemble to the output of the GPT, it is possible to detect whether the GPT is experiencing hallucination.

[0026] [Ensemble] First, we will explain the Ensemble method. The Ensemble method is a method for generating a single machine learning model using multiple machine learning models, with the aim of improving prediction accuracy.

[0027] A common ensemble method is to take a majority vote of the outputs of multiple machine learning models. Figure 4 is a diagram explaining the ensemble.

[0028] As shown in Figure 4, for the same input image, Model 1 outputs a predicted value of "2," Model 2 outputs a predicted value of "4," and Model 3 outputs a predicted value of "2." In the Ensemble method, the predicted value of "2" for the input image is determined by majority vote among the outputs of Models 1 to 3.

[0029] In this way, by determining the final predicted value by majority vote, ensemble learning can achieve high overall accuracy even if the accuracy of each machine learning model is somewhat poor. Note that while Figure 4 shows an ensemble for a classification problem, it is common for ensembles for regression problems to take the average of the outputs.

[0030] [Detection method according to the embodiment] Even when the information necessary to produce a correct answer is insufficient, the LLM generates a plausible sentence, which causes hallucination. In this embodiment, Ensemble is applied to detect hallucination by having the LLM itself determine whether the output content contains hallucination.

[0031] The detection method according to this embodiment is based on the hypothesis that "when the information necessary to generate a correct answer is insufficient, the consistency of the LLM output is lost." As an example, Figure 5 shows three answer sentences when chatGPT is asked, "Tell me about Perfume."

[0032] All three of these response sentences contain errors in the member portion. What is noteworthy here is that the three response sentences are inconsistent. In this embodiment, hallucination is detected based on the hypothesis that consistency is lost when hallucination occurs.

[0033] The detection device according to the embodiment will be specifically described below. Fig. 6 is a diagram showing an example of the configuration of the detection device according to the embodiment. Fig. 7 is a flowchart showing the procedure of the detection method according to the embodiment.

[0034] 6 is a device that is realized by loading a predetermined program into a computer or the like including a read-only memory (ROM), a random access memory (RAM), a central processing unit (CPU), etc., and having the CPU execute the predetermined program. The detection device 10 also has a communication interface for transmitting and receiving various types of information to and from other devices connected via a network or the like.

[0035] The detection device 10 includes a question sentence input unit 11 , a generation unit 12 having an LLM 121 (first language model) of a detection target, and a detection unit 13 .

[0036] The question input unit 11 acquires a question to be input to the LLM 121 and inputs it to the generation unit 12 (step S1 in FIG. 7). The question may be created by the question input unit 11 or may be input from an external source.

[0037] The generation unit 12 inputs the question sentence input from the question sentence input unit 11 to the LLM 121 multiple times, and causes the LLM 121 to generate multiple (n) answer sentences to the question sentence (step S2 in FIG. 7 ). The generation unit 12 outputs the multiple answer sentences generated by the LLM 121 to the detection unit 13.

[0038] The detection unit 13 has an LLM 131 (second language model) and detects whether or not there is hallucination in the LLM 121 by determining whether or not there is a contradiction between the answer sentences generated by the LLM 121 (step S3 in Figure 7).

[0039] The detection unit 13 causes the LLM 131 to determine whether or not there is a contradiction between the multiple answer sentences generated by the LLM 121. Based on the result of this determination, the detection unit 13 detects whether or not there is hallucination in the LLM 121. The detection unit 13 uses one of the following two methods to determine whether or not there is a contradiction between the answer sentences. Note that the LLM that determines whether or not there is a contradiction between the answer sentences may be the LLM 121.

[0040] The first method will be described. In the first method, the detection unit 13 inputs two answer sentences from a plurality of answer sentences as a pair to the LLM 131, and executes a process of providing a prompt asking whether there is a contradiction between the two selected answer sentences for all combinations of the plurality of answer sentences. The LLM 131 determines whether there is a contradiction between the two answer sentences, and generates and outputs a determination statement indicating the determination result. The detection unit 13 counts the number of combinations of answer sentences that have contradictions from the determination result by the LLM 131. The detection unit 13 detects the presence or absence of hallucination in the LLM 131 based on the number of combinations of answer sentences that have contradictions.

[0041] In the first method, the prompt given when asking about contradictions is, in Japanese, "Please tell me if there is a contradiction between the following two sentences. \n\nAnswer Sentence 1\n\nAnswer Sentence 2." The prompt given when asking about contradictions is, in English, "Answer 'yes' if there is a contradiction between the following two statements and if not answer 'no'. \n\nAnswer Sentence 1\n\nAnswer Sentence 2." Note that the prompts are just examples.

[0042] Next, the second method will be described. In the second method, the detection unit 13 vectorizes each of the multiple response sentences using the LLM 131. The detection unit 13 inputs n response sentences to an encoder (see FIG. 1, for example) of the LLM 131 and vectorizes them. The detection unit 13 calculates the cosine similarity between the vectors, which correspond to the multiple response sentences respectively, by grouping two vectors into a pair.

[0043] When there is a combination of answer sentences in which the cosine similarity between the answer sentences is smaller than a predetermined value, the detection unit 13 interprets the two answer sentences as being inconsistent. In other words, the detection unit 13 determines that two answer sentences corresponding to a combination in which the calculated cosine similarity is smaller than a predetermined value are inconsistent. The detection unit 13 counts the number of combinations of answer sentences that are inconsistent. The detection unit 13 detects the presence or absence of hallucination in the LLM 121 based on the number of combinations of answer sentences that are inconsistent.

[0044] The detection unit 13 detects whether the LLM 121 is causing hallucination based on the number of combinations of contradictory answer sentences. The detection unit 13 performs the calculation by dividing the number of contradictions by nC2. For example, if the calculation result of dividing the number of contradictions by nC2 is greater than a predetermined threshold, the detection unit 13 detects that the LLM 121 is causing hallucination. On the other hand, if the calculation result of dividing the number of contradictions by nC2 is equal to or less than the predetermined threshold, the detection unit 13 determines that hallucination is not occurring.

[0045] The detection unit 13 detects whether or not there is a contradiction, but the method of outputting the detection result is not limited to whether or not there is a contradiction, and may estimate the possibility of hallucination occurring from the number of contradictions and the magnitude of the cosine similarity between the answer sentences, and output the answer sentence based on a probability.

[0046] [Experiment] In the experiment, GPT is used as the LLM 131 for determination in the detection unit 13. In the experiment, it is confirmed whether the LLM 131 can detect contradictory response sentences, and it is verified whether there is a possibility of detecting hallucination.

[0047] In the experiment, two of the three answer sentences were input, and the LLM 131 was made to determine whether there was a contradiction between the answer sentences. An answer sentence in which the contradiction had been corrected was also prepared, and the LLM 131 was made to make a judgment. Figures 8 and 9 are diagrams showing an example of input to the GPT and the judgment results.

[0048] [Experimental Results] The case where there is a contradiction between two reply sentences will be described with reference to Fig. 8. As shown in Fig. 8, when there is a contradiction between two reply sentences (see box W11), the LLM 131 detects that there is a contradiction between the two reply sentences and outputs a notice indicating that there is a contradiction (box W12).

[0049] A case where there is no contradiction between two reply sentences will be described with reference to Fig. 9. As shown in Fig. 9, for the two reply sentences after the contradiction has been corrected (box W21), the LLM 131 detects that there is no contradiction between the two reply sentences and outputs that there is no contradiction (box W22).

[0050] This experiment showed that LLM131 (GPT) can determine inconsistencies between response sentences and detect hallucination.

[0051] [Effects of the embodiment] Experiments have confirmed that the detection device 10 can detect hallucination. For this reason, in the embodiment, the LLM inside the detection device 10 itself can be made to detect whether or not hallucination is included in the output content. Therefore, the detection device 10 does not need to use external knowledge to detect hallucination. In this way, according to the embodiment, the correctness of sentences generated by the LLM can be easily determined.

[0052] Furthermore, we believe that being able to easily determine the correctness of sentences generated by LLMs will lead to increased trust in LLMs and expand the applicability of applications using LLMs.

[0053] [Example] An example of application to a program (thesis Q&A system) that delivers the contents of a thesis and answers questions about the thesis will be described. Fig. 10 is a diagram for explaining an example in the embodiment.

[0054] [Answer Sentence Acquisition Phase] First, the arXiV parser 11A acquires the paper HTML from the arXiv server based on the URL entered by the user. Then, the arXiV parser 11A formats the paper information and inputs it multiple times along with the question sentence to the LLM 121A to obtain multiple answer sentences.

[0055] [Contradiction Determination Phase] The generation unit 12 inputs the question sentence to the LLM 121A multiple times to generate multiple answer sentences. The detection unit 13 inputs the answer sentences generated by the LLM 121A again to the LLM 121A to detect whether there are any contradictions between the answer sentences.

[0056] Here, the detection unit 13 selects two of the n answer sentences as inputs of answer sentences and inputs them to the LLM 121 A. The detection unit 13 performs this process for all combinations and counts the number of combinations with contradictions.

[0057] In this case, the prompt given to LLM121A in Japanese is, "If there is a contradiction between the following two statements, answer yes; if not, answer no. \n\nAnswer Sentence 1 \n\nAnswer Sentence 2," and the presence or absence of a contradiction is determined by whether the answer contains "yes" or "no." The prompt given to LLM121A in English is, "Answer 'yes' if there is a contradiction between the following two statements and if not answer 'no'. \n\nAnswer Sentence 1 \n\nAnswer Sentence 2," and the presence or absence of a contradiction is determined by whether the answer contains "yes" or "no."

[0058] The detection unit 13 estimates whether the LLM 121A is experiencing hallucination based on the number of contradictions. It performs a calculation by dividing the number of contradictions by nC2. If there are no contradictions, the detection unit 13 outputs a response statement from the LLM 121A. If there are contradictions, the detection unit 13 searches for additional information and performs further detection.

[0059] In the embodiment, the detection unit 13 determines whether or not there is a contradiction, but the method of outputting the determination result is not limited to whether or not there is a contradiction, and may estimate the possibility of hallucination occurring from the number of contradictions or the magnitude of the cosine similarity between the answer sentences, and output the answer sentence with a probability.

[0060] [System Configuration of the Embodiment] Each component of the detection device 10 is a functional concept and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of the functions of the detection device 10 is not limited to that shown in the figure, and all or part of the functions can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc.

[0061] Furthermore, all or any part of the processes performed by the detection device 10 may be realized by a CPU and a program analyzed and executed by the CPU. Furthermore, each process performed by the detection device 10 may be realized as hardware using wired logic.

[0062] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters described above and illustrated can be changed as appropriate unless otherwise specified.

[0063] 11 is a diagram showing an example of a computer in which the detection device 10 is realized by executing a program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0064] The memory 1010 includes a ROM 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0065] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the detection device 10 is implemented as a program module 1093 in which code that can be executed by the computer 1000 is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing processes similar to those of the functional configuration of the detection device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0066] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, CPU 1020 reads out program module 1093 or program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as necessary and executes them.

[0067] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0068] Although the present invention has been described above as an embodiment, the present invention is not limited to the description and drawings that form part of the disclosure of the present invention. In other words, other embodiments, examples, and operational techniques that can be made by those skilled in the art based on the present invention are all included in the scope of the present invention.

[0069] 10 Detection device 11 Question sentence input unit 12 Generation unit 13 Detection unit 121, 131 LLM

Claims

1. A detection device comprising: a generation unit that causes a first language model to generate multiple answer sentences to a question sentence; and a detection unit that causes a second language model to determine whether there are any contradictions between the multiple answer sentences generated by the first language model, and detects the presence or absence of hallucination in the first language model based on the determination result.

2. The detection device described in claim 1, characterized in that the detection unit inputs two answer sentences out of the multiple answer sentences as a pair into the second language model, and executes a process of providing a prompt asking whether there is a contradiction between the two selected answer sentences for all combinations of the multiple answer sentences, and detects the presence or absence of hallucination in the first language model based on the number of combinations of answer sentences that have contradictions.

3. The detection device described in claim 1, characterized in that the detection unit vectorizes each of the multiple answer sentences using the second language model, calculates the cosine similarity between the vectors for each pair of vectors corresponding to the multiple answer sentences, determines that two answer sentences for which the calculated cosine similarity is smaller than a predetermined value are inconsistent, and detects the presence or absence of hallucination in the first language model based on the number of combinations of answer sentences that are inconsistent.

4. A detection method executed by a detection device, comprising: a step of causing a first language model to generate a plurality of answer sentences to a question sentence; and a step of causing a second language model to determine whether or not there is a contradiction between the plurality of answer sentences generated by the first language model, and detecting the presence or absence of hallucination in the first language model based on the determination result.

Images