Illusion detection and processing of domain-specific dialog systems based on large language models

Abstract

The invention relates to hallucination detection and processing of domain-specific dialog systems based on large language models. Systems and methods for detecting and reducing illusion and improving reliability of LLM-based QA systems are described herein. In particular, the present disclosure provides a set of illusion detection and processing methods that address different types of illusions of a QA system. The illusion detection method described herein includes comparing natural language answers to contextual information by means of sentence similarity estimation and keyword matching. The illusion processing method described herein includes removing an illusion sentence from the natural language answer or regenerating the natural language answer using better contextual information depending on a level of illusion detected in the natural language answer. A hybrid framework is also provided that systematically combines a hallucination detection method and a hallucination processing method into one system to achieve optimal hallucination reduction performance for a QA system.

Description

Technical Field

[0001] The devices and methods disclosed in this document relate to machine learning, and more particularly to hallucination detection and processing for dialogue systems based on large language models. Background Art

[0002] Unless otherwise indicated herein, the materials described in this section are not admitted to be prior art by inclusion in this section.

[0003] Large language models (LLMs) have become a recent focus of artificial intelligence and are used in many applications. LLMs such as ChatGPT have been found to be particularly effective for the development of dialogue systems such as question-answering (QA) systems. LLMs are trained on a huge amount of training data and typically have multiple natural language processing-related training objectives. The use of LLMs significantly improves the performance of downstream dialogue tasks. In particular, LLMs enable the system to better understand the user's input, learn richer knowledge and better natural language processing capabilities (e.g., in coreference resolution, summarization, etc.). Additionally, LLMs enable the system to generate more natural and higher-quality responses. Compared to previous state-of-the-art technologies, recently developed LLM-based dialogue systems (e.g., ChatGPT, GPT4) can more accurately understand the user's intent and generate much smarter dialogues with the user.

[0004] However, LLM still has multiple weaknesses. One major problem with LLM is the so-called "hallucination" problem. The hallucination problem can generally be understood as a phenomenon in which, given a query from a user, the LLM generates an answer that includes untrue or "hallucination" information. In other words, the LLM may fabricate some content in its answer that is not based on facts but is instead included to make the answer look more likely. The hallucination problem is particularly critical for domain-specific conversational applications (e.g., conversational assistants for vehicle maintenance operations, cooking assistants involving stove / oven operations), where reliability is typically a key factor affecting user satisfaction with the system. In addition, the hallucination content in the answers provided by LLM-based systems may cause significant liability issues (e.g., may cause damage or injury) for those applications that require high reliability.

[0005] Various methods have been proposed to alleviate the hallucination problem of natural language generation based on LLM. For domain-specific QA systems, an efficient way to improve system reliability would be to use information retrieval to improve question answering. That is, given a question from a user, we can first perform information retrieval to find relevant domain-specific information for a given question, and then ask the LLM to generate constrained answers based on (one or more) retrieval results, rather than relying solely on the LLM to generate answers to the question. In this way, since the answers generated by the LLM will be largely based on the retrieved domain-specific content, reliability can be significantly improved. However, even if information retrieval is added to the domain-specific QA system based on LLM, hallucination problems will still occur from time to time, usually due to imperfect information retrieval or because the previously learned knowledge (learned during the training of the LLM) is too strong. What is needed is a method to further reduce hallucinations and improve the reliability of domain-specific QA based on LLM to a satisfactory level. Summary of the invention

[0006] A method for answering a natural language question is disclosed. The method includes receiving a natural language question using a processor. The method further includes retrieving natural language context information from an information source using a first information retrieval technique based on the natural language question using the processor. The method further includes generating a natural language answer using a language model based on the natural language question and the natural language context information using the processor. The method further includes detecting hallucination content in the natural language answer based on the natural language context information using the processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The foregoing aspects and other features of the systems and methods are explained in the following description considered in conjunction with the accompanying drawings.

[0008] Figure 1 Example queries and answers generated by a domain-specific LLM-based QA system are shown.

[0009] Figure 2 An exemplary embodiment of a domain-specific LLM-based QA system is shown.

[0010] Figure 3 A flow chart of a method for processing hallucinated content in a question-answering system is shown.

[0011] Figure 4 Summarize the process used for sentence similarity based hallucination detection techniques.

[0012] Figure 5 Pseudocode summarizing a sentence similarity based hallucination detection technique is shown.

[0013] Figure 6Pseudo-code summarizing a keyword-based hallucination detection technique is shown.

[0014] Figure 7 Pseudocode summarizing the sentence similarity based hallucination processing technique is shown.

[0015] Figure 8 Pseudo-code summarizing the keyword-based hallucination processing technique is shown.

[0016] Fig. 9 A flow chart summarizing the mixed hallucination processing framework is shown.

[0017] Fig.10 Pseudocode summarizing the sentence similarity based hallucination processing technique is shown. DETAILED DESCRIPTION

[0018] For the purpose of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the accompanying drawings and described in the following written specification. It should be understood that no limitation is intended to the scope of the present disclosure, therefore. It should be further understood that the present disclosure includes any changes and modifications to the illustrated embodiments, and includes additional applications of the principles of the present disclosure as would normally occur to a person skilled in the art to which the present disclosure belongs.

[0019] Overview

[0020] Systems and methods for detecting and reducing hallucinations and improving the reliability of LLM-based QA systems are described herein. In particular, the present disclosure provides a set of hallucination detection and processing methods that address different types of hallucinations for QA systems. The hallucination detection methods described herein include comparing natural language answers with contextual information with the aid of sentence similarity estimation and keyword matching. The hallucination processing methods described herein include removing hallucinated sentences from natural language answers or regenerating natural language answers with better contextual information, depending on the level of hallucination detected in the natural language answers. A hybrid framework is provided that systematically combines hallucination detection methods and hallucination processing methods into one system to achieve optimal hallucination reduction performance for QA systems.

[0021] Figure 1 Example prompts and responses generated by a domain-specific LLM-based QA system are shown. A user provides a domain-specific natural language question related to operating an in-vehicle infotainment system (e.g., "How do I activate the voice recognition system?"). The QA system retrieves contextual information including natural language text (e.g., "Introducing voice recognition, Getting started with Uconnect voice recognition with ...") from one or more domain-specific resources (e.g., a user manual or web page for a car or in-vehicle infotainment system).

[0022] Next, the QA system constructs a prompt 10 that includes both a natural language question and natural language context information (e.g., "Given the context: Introducing speech recognition, Getting started with speech recognition with..., answer the following question: How do I activate the speech recognition system in my vehicle?"). Prompt 10 is fed as input into the LLM of the QA system (e.g., OpenAI GPT). The LLM of the QA system generates a natural language answer based on prompt 10 (e.g., "To activate the speech recognition system, press and hold the VR button on your steering wheel for a few seconds until you hear a beep. Alternatively, you can also say "Hey, system" or "Hey, car" to activate the system. Once the system is activated, you can use voice commands to control various functions of your vehicle, such as making calls, adjusting the temperature, and navigating to a destination.").

[0023] Although feeding contextual information to the LLM of the QA system will help reduce hallucinations in the generated natural language answers, such hallucinations generated by the LLM are unlikely to be completely avoided and may still occur from time to time. For example, in the illustrated example answer, both the second and third sentences are hallucinations, i.e., not mentioned or supported by the provided contextual information. However, a conventional QA system will simply output the natural language answers generated by the QA system to the user. However, a QA system according to the present disclosure will attempt to eliminate or reduce the amount of hallucination content in the generated natural language answers, such as by deleting the second and third sentences, or by retrieving better contextual information and using it to regenerate a natural language answer with less hallucination content.

[0024] Exemplary Question Answering System

[0025] Figure 2 An exemplary embodiment of a domain-specific LLM-based QA system 100 is shown. In the illustrated embodiment, the QA system 100 includes a server 110 and one or more client devices 130. The client device 130 (which may also be referred to herein as a "personal electronic device") may be a desktop computer, a laptop computer, a smart phone, a tablet device, a smart watch, or other smart device. The client device 130 is configured to enable a user to ask a natural language question and transmit the question to the server 110 for processing. The server 110 is configured to receive the natural language question from the client device 130 and process the question to determine an appropriate answer, which is transmitted back to the client device 130. The client device 130 is configured to receive the answer from the server 110 and present the answer to the user.

[0026] The server 110 and the client device 130 are configured to communicate with each other via one or more networks 140. The network 140 may include one or more subnets that work together to enable communication between the server 110 and the client device 130. The network 140 may include, in part, one or more pre-existing wired or wireless networks, such as a local area network, the Internet, a telephone network, or any combination thereof. It will be appreciated that although the QA system 100 is described as having distinct servers 110 and client devices 130, in some embodiments, a single computing device (not shown) may include combined components and features of both the server 110 and the client device 130.

[0027] Continue to refer Figure 2 , illustrates exemplary components of a server 110. In the illustrated embodiment, the server 110 includes at least one processor 112, a memory 114, a user interface 116, and a network communication module 118. It will be appreciated that the illustrated embodiment of the server 110 is merely one exemplary embodiment of the server 110 and is merely representative of any of various ways or configurations of a server, a remote computer, or any other data processing system operating in the manner set forth herein.

[0028] Processor 112 can be any of a variety of processors as will be recognized by those of ordinary skill in the art. It will be appreciated by those of ordinary skill in the art that "processor" as used herein includes any hardware system, hardware mechanism, or hardware component that processes data, signals, and / or other information. Processor 112 may include a system with a central processing unit, a plurality of processing units, a dedicated circuit for implementing functionality, and / or other systems. Exemplary processors include a microprocessor (μP), a microcontroller (μC), a digital signal processor (DSP), a graphics processing unit (GPU), or any combination thereof. Processor 112 is operably connected to memory 114, a user interface 116, and a network communication module 118.

[0029] The server 110 can be operated locally or remotely by a user. To facilitate local operation, the server 110 may include an interactive user interface 116. Via the user interface 116, a user can modify and / or update program instructions stored on the memory 114, as well as collect data from the memory 114 and store data to the memory 114. In one embodiment, the user interface 116 may appropriately include an LCD display screen, a mouse or other pointing device, a keyboard or other keypad, a speaker, and a microphone, as will be appreciated by those of ordinary skill in the art. Alternatively, in some embodiments, a user can remotely operate the server 110 from another computing device, which communicates with it via a network communication module 118 and has a similar user interface.

[0030] The network communication module 118 of the server 110 provides an interface that allows communication with any of a variety of devices or networks, and includes at least a transceiver or other hardware configured to communicate with the client device 30. In particular, the network communication module 118 may include a local area network port that allows communication with any of a variety of local computers housed in the same or nearby facilities. In some embodiments, the network communication module 118 further includes a wide area network port that allows communication with a remote computer via the Internet. Alternatively, the server 110 communicates with the Internet via a separate modem and / or router of a local area network. In one embodiment, the network communication module 118 is equipped with a Wi-Fi transceiver or other wireless communication device. Therefore, it will be appreciated that communication with the server 110 can occur via wired communication or via wireless communication. Any of various known communication protocols can be used to achieve communication.

[0031] The memory 114 of the server 110 is configured to store information including both data and instructions. The memory 114 can be any type of device capable of storing information accessible by the processor 112, such as a memory card, ROM, RAM, writable memory, read-only memory, hard drive, disk, flash memory, or any of a variety of other computer-readable media used as data storage devices as will be recognized by those of ordinary skill in the art. The memory 114 is configured to store program instructions that, when executed by the processor 112, enable the server 110 to provide features, functionalities, characteristics, and / or the like as described herein. In particular, the memory 114 includes a question-answering system backend 120 that includes an LLM capable of processing natural language questions received from a client device 130 and determining appropriate answers to the natural language questions.

[0032] The memory 114 also stores information sources 122, which include a corpus of text information that may relate to one or more specific knowledge domains of the application system 100. In some embodiments, the information sources 122 may include a database with a large number of different documents, web pages, etc. converted into a text format. Alternatively, the information sources 122 may include only a single comprehensive document or a small number of documents for a specific knowledge domain of the application system 100. Therefore, it should be appreciated that the nature of the information sources 122 may vary significantly depending on the scope of the (one or more) knowledge domains of the application system 100.

[0033] Continue to refer Figure 2, illustrating exemplary components of a client device 130. In the illustrated embodiment, the client device 130 includes a processor 132, a memory 134, a transceiver 136, a display screen 138, and a microphone 140. It should be appreciated that the illustrated embodiment of the client device 130 is only one exemplary embodiment of the client device 130 and is merely representative of any of various ways or configurations of a client device, personal electronic device, or other device operating in the manner set forth herein.

[0034] Processor 132 can be any of a variety of processors as will be recognized by those of ordinary skill in the art. It will be appreciated by those of ordinary skill in the art that "processor" as used herein includes any hardware system, hardware mechanism, or hardware component that processes data, signals, and / or other information. Processor 132 may include a system with a central processing unit, a plurality of processing units, a dedicated circuit for implementing functionality, and / or other systems. Exemplary processors include a microprocessor (μP), a microcontroller (μC), a digital signal processor (DSP), a graphics processing unit (GPU), or any combination thereof. Processor 132 is operably connected to memory 134, transceiver 136, display screen 138, and microphone 140.

[0035] The processor 132 is configured to operate and / or communicate with the display screen 138 and microphone 140 and any other interfaces (such as tactile buttons, switches, triggers, speakers and / or connection ports). The display screen 138 can be an LCD screen or any of a variety of other screens suitable for personal electronic devices. The processor 132 operates the display screen 138 to visually display graphics, text and other data to the user via the display screen 138. In some embodiments, the microphone 140 is configured to record the user's voice asking natural language questions to be answered using the QA system 100. In other embodiments, the user uses a virtual keyboard on the display screen 138 or a physical keyboard connected to the processor 132 to type in natural language questions.

[0036] The transceiver 136 includes at least a transceiver such as a Wi-Fi transceiver configured to communicate with the server 110 via the network 140, but may also include any of a variety of other devices configured to communicate with other electronic devices, including the ability to send and receive communication signals. In one embodiment, the transceiver 136 further includes additional transceivers common to smart phones, smart watches, laptops, tablet computers, desktop computers, such as Bluetooth transceivers, Ethernet adapters, and transceivers configured to communicate via wireless telephone networks.

[0037] The memory 134 of the server 110 is configured to store information including both data and instructions. The memory 134 can be any type of device capable of storing information accessible by the processor 132, such as a memory card, ROM, RAM, writable memory, read-only memory, hard drive, disk, flash memory, or any of various other computer-readable media used as data storage devices as will be recognized by those of ordinary skill in the art. The memory 134 is configured to store program instructions that, when executed by the processor 132, enable the client device 130 to provide features, functionalities, characteristics, and / or the like as described herein. In particular, the memory 134 includes a QA system application 142 that enables a user to ask natural language questions and receive corresponding answers to the natural language questions provided from the server 110. It will be appreciated that the QA system application 142 can be an application executed locally or a network application executed via a network browser.

[0038] Methods for handling hallucinated content in question answering systems

[0039] Various operations and processes for operating the server 110 and the client device 130 are described below to provide a question-answering system that better detects and handles hallucinated content. In these descriptions, statements that methods, processors, and / or systems are performing certain tasks or functions refer to a controller or processor (e.g., processor 112 of the server 110 or processor 132 of the client device 130) executing programming instructions stored in a non-transitory computer-readable storage medium (e.g., memory 120 of the server 110 or memory 134 of the client device 130) that is operably connected to the controller or processor to manipulate data or operate one or more components in the server 110 and the client device to perform the task or function. Additionally, the steps of the method may be performed in any feasible time order, regardless of the order shown in the various figures or the order in which the steps are described therein.

[0040] Figure 3 A flow chart of a method 200 for handling hallucinated content in an LLM-based QA system is shown. The method 200 advantageously utilizes a set of hallucination detection and handling methods that address different types of hallucinations for a QA system. In particular, the method 200 utilizes sentence similarity estimates and keyword matching between retrieved contextual information and natural language answers to detect hallucinated content in natural language answers. When hallucinated content is detected, the method 200 reduces the amount of hallucinated content in the natural language answer by removing the hallucinated content or regenerating the natural language answer with better contextual information, depending on the level of hallucination in the natural language answer. Thus, the method 200 advantageously improves the reliability and credibility of the LLM-based QA system.

[0041] The method 200 begins by receiving a natural language question (block 210). In particular, the processor 112 receives a natural language question, for example, from a user. In some embodiments, the processor 132 of the client device 130 executes program instructions of the question-answering system application 142 to receive a natural language question from a user. In one embodiment, the processor 132 operates the display screen 138 to display a graphical user interface on the display screen 138, the graphical user interface including a question prompt, via which the user can type the natural language question in text form using a keyboard or the like. The processor 132 receives text corresponding to the natural language question via a physical or virtual keyboard of the client device 130. Alternatively, the user can speak the natural language question into the microphone 140, and the processor 132 receives audio corresponding to the natural language question. In one embodiment, the processor 132 transcribes the received audio of the natural language question into text form. Finally, the processor 132 operates the transceiver 136 to transmit the natural language question to the server 110 in text form or audio form for processing. The processor 112 of the server 110 operates the network communication module 118 to receive the natural language question from the client device 130.

[0042] The method 200 continues by retrieving natural language context information based on the natural language question using information retrieval techniques (block 220). In particular, the processor 112 retrieves natural language context information from the information source 122 based on the natural language question and using corresponding information retrieval techniques. The information retrieval techniques utilized by the processor 112 are designed to retrieve natural language information related to the answer to the natural language question, which is referred to herein as "context information."

[0043] As discussed above, the information source 122 may include a database with a large number of text documents, etc., or may include only a single comprehensive document or a small number of documents for a particular knowledge domain of the application system 100. The retrieved contextual information may include, for example, one or more paragraphs of natural language text extracted from one or more sections of documents from the information source 122. Depending on the information retrieval technique, multiple paragraphs may be retrieved from the same section of the same document, from different sections of the same document, or from different sections of different documents.

[0044] It should be appreciated that a variety of information retrieval techniques, such as Lucene search, can be utilized to retrieve relevant contextual information from information source 122. Different information retrieval techniques can provide retrieval results of varying quality with different computational costs. Generally speaking, information retrieval techniques that are more computationally expensive provide better retrieval results, while information retrieval techniques that are less computationally expensive provide lower quality retrieval results.

[0045] In at least some embodiments, processor 112 selects an information retrieval technology from a plurality of available information retrieval technologies and uses the selected information retrieval technology to perform an initial retrieval of natural language context information. As will be discussed below, in some cases, processor 112 will later perform information retrieval again using a different information retrieval technology. In at least one embodiment, processor 112 selects an information retrieval technology with the smallest computational cost for performing an initial information retrieval.

[0046] The method 200 continues to generate a natural language answer based on the natural language question and the natural language context information using a language model (block 230). In particular, the processor 112 generates a natural language answer based on the natural language question and the natural language context information using a language model. The language model is, for example, a so-called "large language model" (LLM) such as OpenAI's GPT language model, which is configured to generate natural language output text based on natural language input text and has been trained on a very large amount of training text. In at least some embodiments, the language model incorporates an artificial neural network.

[0047] In some embodiments, processor 112 constructs a natural language prompt based on the natural language question and the natural language context information, which is designed to request an answer from a language model constrained by the natural language context information. Processor 112 generates a natural language answer by providing the natural language prompt as input to the language model. Return to reference Figure 1 , prompt 10 is generated by including additional text in prompt 10 that identifies natural language context information (e.g., "Given the context:") followed by the text of the natural language context information. Similarly, prompt 10 is generated by including additional text in prompt 10 that identifies a natural language question (e.g., "Answer the following question:") followed by the text of the natural language question. However, it should be appreciated that prompts that incorporate and identify both natural language questions and natural language context information may be constructed in a variety of ways, which will result in differences in the answers generated and may depend on the language model utilized in QA system 100.

[0048] Method 200 continues by detecting hallucinated content in the natural language answer based on the natural language context information (block 240). In particular, processor 112 detects hallucinated content in the natural language answer based on the previously retrieved natural language context information. Generally speaking, processor 112 detects hallucinated content in the natural language answer by comparing sentences and / or keywords in the natural language answer to sentences and words in the previously retrieved natural language context information. However, various specifications of hallucination detection techniques are described in detail below.

[0049] In the sentence similarity-based hallucination detection technique, processor 112 determines the similarity between sentences in the natural language answer and sentences in the natural language context information. Next, processor 112 marks the sentence as hallucination content in response to the similarity between one or more sentences in the natural language answer and each sentence in the natural language context information being less than a threshold.

[0050] Figure 4 The process for hallucination detection technology based on sentence similarity is summarized. Each sentence of the natural language answer is compared with the sentences in the natural language context information to identify one or more sentences that provide support for the sentences in the natural language answer. In the diagram, similar sentences in the natural language answer and the natural language context information are represented by arrows. As can be seen, among the five sentences in the answer, it is found that the first sentence will be supported by (similar to) the second sentence in the context information, the second sentence is supported by the fifth and sixth sentences in the context information, and the fourth and fifth sentences are both supported by the eighth sentence in the context information. However, the third sentence is found to be not similar to any sentence in the context information, and is therefore marked as hallucination content.

[0051] The basic idea of ​​the hallucination detection technology based on sentence similarity is to determine whether each sentence in the natural language answer is based on or supported by the retrieval results, and the retrieval results are natural language context information. If the sentence is not supported by the retrieval results, it is likely to be an illusion, or in other words, the language model has fabricated it. In at least some embodiments, the processor 112 utilizes two different types of similarity assessments to determine whether the sentence in the natural language answer is similar to one or more sentences in the natural language context information (that is, based on one or more sentences in the natural language context information). In particular, the processor 112 utilizes similarity calculations based on sentence embedding and similarity calculations based on pattern overlap rates. The processor 112 uses these two types of similarity assessments in a hybrid manner to make a joint decision about whether a specific sentence in the natural language answer is based on natural language context information. If not, the processor 112 marks the sentence as hallucination content.

[0052] In the similarity calculation based on sentence embedding, the processor 112 determines the corresponding sentence embedding (e.g., vector) for each sentence in the natural language answer and each sentence in the natural language context information. The processor 112 generates each sentence embedding using a language model and / or its encoder. However, it should be appreciated that the language model used to generate the sentence embedding vector is not necessarily the same language model as that used to generate the natural language answer. In one example, a first language model (e.g., OpenAI's GPT) may be used to generate a natural language answer, and a second language model (e.g., Sentence-BERT) may be used to generate the sentence embedding required for sentence similarity determination. In at least one embodiment, the second language model used to generate sentence embedding has higher efficiency and lower cost than the first language model used to generate the natural language answer.

[0053] For each sentence in the natural language answer, the processor 112 determines the corresponding embedding similarity of each sentence in the natural language context information based on the comparison of the corresponding sentence embeddings. In at least one embodiment, the processor 112 determines the embedding similarity between each sentence combination by calculating the cosine similarity between the corresponding sentence embeddings. In response to a pair of sentences having an embedding similarity exceeding a first predetermined threshold (Threshold_A), the processor 112 identifies and / or marks the pair of sentences as similar to each other. The predetermined threshold can be tuned on the development data set.

[0054] However, it will be appreciated that, in some cases, a sentence in a natural language answer may combine multiple sentences from natural language context information. Likewise, in some cases, a sentence from natural language context information may be divided into multiple sentences in a natural language answer. Similarity calculations based on sentence embeddings do not solve this situation where there is a one-to-one or many-to-one mapping between sentences in a natural language answer and sentences in natural language context information. In particular, even if the sentences support each other, the sentence embeddings in this situation may not be completely similar to each other, because longer sentences may contain significant additional information compared to shorter sentences. In such a case, similarity calculations based on pattern overlap help capture such partial similarities between sentences.

[0055] In the similarity calculation based on the pattern overlap rate, the processor 112 uses dynamic programming with the goal of maximizing overlap to determine the corresponding mapping between the words in the corresponding sentence in the natural language answer and the corresponding words in the natural language context information. The dynamic programming compares the sentences in the two focuses with the goal of maximizing overlap and outputs the optimal path containing the mapping results of each step (for example, 0011111001111001000000, where 1 indicates a word match and otherwise 0). Next, the processor 112 determines the word pattern overlap rate between the corresponding sentence in the natural language answer and the corresponding sentence in the natural language context information based on the determined mapping and / or optimal path.

[0056] More specifically, the processor 112 first determines the overlap length based on the determined mapping and / or optimal path as the sum of the overlapping words between the corresponding sentence in the natural language answer and the corresponding sentence in the natural language context information. In one embodiment, the processor 112 determines the sum of the overlapping words as the total number of overlapping words (i.e., the sum of 1s in the optimal path). In an alternative embodiment, the processor 112 determines the sum of the overlapping words as only the words in the overlapping phrases (i.e., the sequence of overlapping words) that include at least a predetermined minimum number of words (e.g., overlapping phrases with at least 4 words). In at least one embodiment, the predetermined minimum number of words is a predetermined percentage (e.g., 30%) of the number of words in the shorter sentence of the corresponding sentence in the natural language answer and the corresponding sentence in the natural language context information. In this way, relatively short overlapping phrases (e.g., high-frequency words such as "the") are ignored to reduce the impact of noise.

[0057] Finally, processor 112 determines the word pattern overlap rate by dividing the corresponding overlap length by the number of words in the shorter sentence of the corresponding sentence in the natural language answer and the corresponding sentence in the natural language context information. In this way, word pattern overlap is defined independently of the different lengths of the sentences, and the above-discussed problem of one-to-many or many-to-one mapping between the sentences in the natural language answer and the sentences in the natural language context information is solved. In response to a pair of sentences having a word pattern overlap rate exceeding a second predetermined threshold (Threshold_B), processor 112 identifies and / or marks the pair of sentences as similar. The predetermined threshold can be tuned on the development data set.

[0058] As noted above, in at least some embodiments, the processor 112 utilizes both sentence embedding-based similarity calculations and pattern overlap-based similarity calculations in a hybrid manner to make a joint decision about whether a particular sentence in a natural language answer is a hallucination. These two types of similarity calculations have different situations in which they are most useful. Similarity calculations based on sentence embeddings can handle the mapping of a sentence in a natural language answer to another sentence in natural language context information, while similarity calculations based on pattern overlap rates are good at handling one-to-many or many-to-one sentence mappings. Because they have complementary advantages, they are advantageously combined into a hybrid hallucination detection method.

[0059] Figure 5 Pseudo code summarizing a hallucination detection technique based on sentence similarity is shown. In particular, for each sentence in the natural language answer (sen_A), for each sentence in the natural language context information (labeled as sen_C), the processor 112 determines the embedding similarity using a similarity calculation based on sentence embeddings, and determines the word pattern overlap rate using a similarity calculation based on pattern overlap rate. If the similarity calculation based on sentence embeddings results in an embedding similarity that exceeds a first predetermined threshold (Threshold_A), the processor 112 identifies and / or marks the sentences (sen_A and sen_C) as similar. Otherwise, if the similarity calculation based on pattern overlap rate results in a word pattern overlap rate that exceeds a second predetermined threshold (Threshold_B), the processor 112 identifies and / or marks the sentences (sen_A and sen_C) as similar. Otherwise, the processor 112 identifies and / or marks the sentences (sen_A and sen_C) as dissimilar. If the sentence (sen_A) in the natural language answer is not similar to any sentence in the natural language context information, the processor 112 identifies and / or marks the sentence (sen_A) in the natural language answer as hallucination content. In other words, the processor 112 identifies and / or marks the corresponding sentence in the natural language answer as hallucination content in response to (i) the embedding similarity of each sentence in the natural language context information being less than a first predetermined threshold (Threshold_A) and (ii) the word pattern overlap rate of each sentence in the natural language context information being less than a second predetermined threshold (Threshold_B).

[0060] In contrast to sentence similarity-based hallucination detection techniques that detect hallucinated content on a sentence-by-sentence level, keyword-based hallucination detection techniques are also provided that detect whether a natural language answer as a whole is heavily hallucinated.

[0061] In the keyword-based hallucination detection technique, processor 112 identifies a keyword in a natural language answer and determines whether the keyword is also present in the natural language context information. Next, if at least a predetermined threshold amount of the keyword does not appear in the natural language context information, processor 112 marks the natural language answer as including hallucination content. The predetermined threshold amount may be a fixed number of keywords or a percentage of all keywords found in the natural language answer.

[0062] The main idea of ​​keyword-based hallucination detection technology is that, given a natural language answer generated by a language model, if a significant portion of the keywords in the answer are not present in the natural language context information, it is very likely that the language model is essentially hallucinating the answer (i.e., it is not supported by the natural language context information). The effectiveness of this method depends on the accuracy and efficiency of the keyword extraction technology. If the keyword extraction technology adopted can reliably detect keywords with high efficiency, the keyword-based hallucination detection technology can be more robust and more efficient than the hallucination detection technology in the sentence similarity-based hallucination detection technology for the detection of highly hallucinated answers.

[0063] Figure 6 Pseudo code summarizing the keyword-based hallucination detection technique is shown. In particular, processor 112 extracts keywords from the natural language answer using state-of-the-art keyword extraction techniques. For each extracted keyword in the natural language answer, if the keyword does not appear in the natural language context information, processor 112 identifies and / or marks the keyword as a potential hallucination keyword. If the ratio of the number of potential hallucination keywords to the number of total keywords is greater than a predetermined threshold, processor 112 identifies and / or marks the natural language answer as a whole as being a severe hallucination.

[0064] In at least some embodiments, the keyword-based hallucination detection technique is combined with the sentence similarity-based hallucination detection technique previously discussed. In particular, in some embodiments, the processor 112 first performs the keyword-based hallucination detection technique to determine whether the natural language answer as a whole is a severe hallucination. If the natural language answer as a whole is a severe hallucination, the hallucination detection can stop here. However, if the natural language answer as a whole is not a severe hallucination according to the keyword-based hallucination detection technique, the processor 112 then performs the sentence similarity-based hallucination detection technique to determine whether individual sentences in the natural language answer are hallucinations.

[0065] Back to Figure 3, method 200 continues to generate a modified natural language answer that reduces the hallucinated content in the natural language answer (block 250). In particular, when the natural language answer is determined to include hallucinated content, processor 112 generates a modified natural language answer that reduces the hallucinated content in the natural language answer. In some embodiments, processor 112 generates the modified natural language answer by removing the hallucinated content from the natural language answer. Alternatively or additionally, in some embodiments, processor 112 uses a different information retrieval technique to retrieve modified natural language context information, and generates a modified natural language answer using a language model based on the natural language question and the modified natural language context information.

[0066] In the hallucination processing technique based on sentence similarity, combined with the hallucination detection technique based on sentence similarity, modified natural language context information can be generated in a manner that depends on the hallucination level of the natural language answer. If the answer is only a mild hallucination, the processor 112 deletes the detected hallucination sentence from the answer. However, if the answer is a severe hallucination, this usually indicates that the LLM cannot find enough information from the retrieval results so that it makes up something to make the answer look possible. In this case, the processor 112 switches to a better (most likely more computationally expensive) information retrieval technique and reruns the information retrieval, prompt construction, and LLM-based answer generation. Repeat this process until the modified natural language answer includes less than a threshold amount of hallucination content, or until no additional information retrieval technique results in a reduction in the amount of hallucination content in the modified natural language answer.

[0067] Figure 7 Pseudo code summarizing a sentence similarity-based hallucination handling technique is shown. In particular, given a natural language question, processor 112 generates a natural language answer using natural language context information retrieved using a baseline information retrieval technique. Next, processor 112 determines a hallucination level that classifies the amount of hallucination content in the natural language answer. In one embodiment, the hallucination level is a count of how many or what percentage of sentences in the natural language answer are hallucinations.

[0068] When hallucination content is detected in a natural language answer, if the hallucination level is less than a predetermined threshold (e.g., less than 30% of the sentence) - indicating that the natural language answer is only a mild hallucination, processor 112 generates a modified natural language answer by deleting the hallucinated sentence from the natural language answer, and provides the shortened natural language answer as the final modified natural language answer.

[0069] Otherwise, if the hallucination level is greater than a predetermined threshold (e.g., greater than 30% of the sentence) - indicating that the natural language answer is severely hallucinated, and if there are still better / different information retrieval techniques available, the processor 112 retrieves modified natural language context information using the better / different information retrieval techniques, and regenerates the natural language answer using the language model based on the natural language question and the modified natural language context information. This process is repeated for the regenerated natural language answer until the hallucination level is less than a predetermined threshold (e.g., less than 30% of the sentence) - indicating that the natural language answer is only mildly hallucinated, or until there are no longer any better / different information retrieval techniques available. In either case, the processor 112 then finally generates a modified natural language answer by deleting the hallucinated content from the natural language answer, and the shortened natural language answer is provided as the final modified natural language answer.

[0070] In the keyword-based hallucination handling technique, combined with the keyword-based hallucination detection technique, a modified natural language answer can be generated by regenerating the modified natural language context information using a better / different information retrieval technique. In particular, if the answer is a heavy hallucination, the processor 112 switches to a better (most likely more computationally expensive) information retrieval technique and reruns the information retrieval, prompt construction, and LLM-based answer generation. This process is repeated until no hallucination is detected in the generated answer, or no better retriever is available.

[0071] Figure 8 Pseudo code summarizing a sentence similarity-based hallucination processing technique is shown. In particular, given a natural language question, processor 112 generates a natural language answer using natural language context information retrieved using a baseline information retrieval technique. When hallucinated content is detected in a natural language answer, if there are still better / different information retrieval techniques available, processor 112 uses the better / different information retrieval techniques to retrieve modified natural language context information, and regenerates the natural language answer using a language model based on the natural language question and the modified natural language context information. The process is repeated for the regenerated natural language answer until the natural language answer is no longer heavily hallucinated (i.e., no longer has a predetermined threshold amount of hallucinated keywords), or until there are no longer any better / different information retrieval techniques available.

[0072] In at least some embodiments, keyword-based hallucination detection and processing techniques are combined with sentence similarity-based hallucination detection and processing techniques. In particular, sentence similarity-based techniques and keyword-based techniques have complementary advantages. That is, keyword-based techniques can be useful for detecting severe hallucination answers in an efficient manner, while sentence similarity-based techniques can detect / process mild hallucination answers. Therefore, it is advantageous to combine sentence similarity-based techniques and keyword-based techniques into a hybrid hallucination processing framework that can efficiently detect severe hallucination cases and can also process mild hallucination cases at the same time.

[0073] Fig. 9 A flow chart summarizing a hybrid hallucination processing framework 300 is shown, which combines both sentence similarity-based and keyword-based hallucination detection and processing. Similar to the above discussion, given a user query 310 (i.e., a natural language question), the processor 112 selects a baseline information retrieval technique, such as the most efficient retrieval method (box 320A), performs information retrieval using the baseline information retrieval technique (box 330), and generates a natural language answer 350 using a language model (box 340). After generating the natural language answer 350, the processor 112 performs keyword-based hallucination detection to determine whether the natural language answer as a whole is heavily hallucinated (box 360). If the natural language answer 350 is determined to be heavily hallucinated based on the keyword-based hallucination detection, the processor 112 selects a different information retrieval technique, such as a better but less efficient retrieval method (box 320B). Otherwise, if the natural language answer 350 is not determined to be a severe hallucination based on the keyword-based hallucination detection, the processor 112 performs a sentence similarity-based hallucination detection to determine whether each sentence in the natural language answer is a hallucination (box 370). Finally, the processor 112 removes the sentences determined to be hallucinations from the natural language answer (box 380). The resulting shortened natural language answer is provided as a final modified natural language answer (box 390).

[0074] use Fig. 9 With the hybrid framework, the hallucination problem associated with LLM-based answer generation will be greatly alleviated. At the same time, high system efficiency can be maintained due to two factors. First, computationally expensive and better information retrieval techniques will be applied only when needed (i.e., when severe hallucinations are detected). Second, the combined use of keyword-based and similarity-based hallucination detection can further improve the overall efficiency of hallucination detection. In particular, the generally more efficient keyword-based hallucination detection is first applied to identify the need for using better information retrieval techniques. Then, sentence similarity-based hallucination detection is subsequently applied to eliminate any remaining hallucination content from the natural language answers.

[0075] Back to Figure 3 , method 200 continues to output the modified natural language answer (block 260). In particular, at least one user interface device is operated to output the final modified natural language answer to the user (e.g., the user who provided the natural language question). In some embodiments, a speaker is operated to output the final modified natural language answer in audio form. use Output the final modified natural language answer. In some embodiments, the operation display screen displays the final modified natural language answer in a graphical form.

[0076] In at least some embodiments, the processor 112 of the server 110 operates the network communication module 118 to transmit the final modified natural language answer to the client device 130. The processor 132 of the client device 130 operates the transceiver 136 to receive the final modified natural language answer from the server 110. The processor 132 operates at least one output device to output the final modified natural language answer to the user perceptibly. In one embodiment, the processor 132 operates the display screen 138 to display a graphical user interface on the display screen 138, the graphical user interface including the final modified natural language answer in graphical form. In one embodiment, the processor 132 operates the speaker of the client device 130 to read aloud the natural language text string using a text-to-speech engine.

[0077] In some embodiments, in addition to generating the final modified natural language answer, the QA system 100 is further configured to provide information identifying which sentences in the natural language context information provide support for each sentence in the final modified natural language answer. Indications of supporting sentences in the natural language context information are output to the user along with the natural language answer in the same manner as discussed above with respect to outputting the natural language answer (i.e., transmitted to the client device 130 and output via a speaker or display screen). This advantageously provides traceability for domain-specific QA systems and provides useful insights to users.

[0078] When performing hallucination detection based on sentence similarity in the manner discussed above, processor 112 identifies and / or marks sentences in natural language answers and natural language context information that are similar to each other. Additionally, in some embodiments, processor 112 determines a similarity score for each pair of sentences (e.g., based on cosine similarity and / or word pattern overlap). Based on these previous similarity determinations and / or similarity scores, processor 112 identifies sentences in natural language context information that provide support for sentences in the modified natural language sentences. In particular, in one embodiment, sentences in natural language context information that are determined to be similar to a specific sentence in the natural language answer that is finally modified are considered to support that sentence in the natural language answer that is finally modified. Alternatively, in another embodiment, sentences in natural language context information that have a similarity score greater than a predetermined threshold with a specific sentence in the natural language answer that is finally modified are considered to support that sentence in the natural language answer that is finally modified.

[0079] Fig.10 Pseudo code summarizing the hallucination processing technique based on sentence similarity is shown. In particular, for each sentence (sen_A) in the natural language answer, for each sentence (labeled as sen_C) in the natural language context information, the processor 112 uses a similarity calculation based on sentence embedding to determine the embedding similarity, and uses a similarity calculation based on the pattern overlap rate to determine the word pattern overlap rate. If the similarity calculation based on sentence embedding results in an embedding similarity exceeding a first predetermined threshold (Threshold_A), the processor 112 identifies and / or marks the sentence (sen_C) in the natural language context information as supporting the sentence (sen_A) in the natural language answer. Otherwise, if the similarity calculation based on the pattern overlap rate results in a word pattern overlap rate exceeding a second predetermined threshold (Threshold_B), the processor 112 identifies and / or marks the sentence (sen_C) in the natural language context information as supporting the sentence (sen_A) in the natural language answer. Finally, as discussed above, the supporting sentences or indications thereof in the natural language context information are output to the user.

[0080] Embodiments within the scope of the present disclosure may also include non-transitory computer-readable storage media or machine-readable media for carrying or storing computer-executable instructions (also referred to as program instructions) or data structures. Such non-transitory computer-readable storage media or machine-readable media can be any available medium that can be accessed by a general-purpose or special-purpose computer. As an example and not limitation, such non-transitory computer-readable storage media or machine-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code components in the form of computer-executable instructions or data structures. The above combination should also be included in the scope of non-transitory computer-readable storage media or machine-readable media.

[0081] Computer executable instructions include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a specific function or group of functions. Computer executable instructions also include program modules that are executed by computers in a stand-alone or network environment. Typically, program modules include routines, programs, objects, components, and data structures that perform specific tasks or implement specific abstract data types. Computer executable instructions, associated data structures, and program modules represent examples of program code components for performing the method steps disclosed herein. A specific sequence of such executable instructions or associated data structures represents an example of corresponding actions for implementing the functions described in such steps.

[0082] Although the present disclosure has been described and illustrated in detail in the drawings and the foregoing description, these should be considered illustrative and not restrictive in nature. It should be understood that only preferred embodiments have been presented, and all changes, modifications, and additional applications within the spirit of the present disclosure are desired to be protected.

Claims

1. A method for answering a natural language question, the method comprising: receiving a natural language question using a processor; Retrieving, with a processor, natural language context information from an information source using a first information retrieval technique based on the natural language question; generating a natural language answer using a language model based on the natural language question and the natural language context information, using a processor; and Using a processor, hallucinated content in a natural language answer is detected based on natural language context information.

2. The method of claim 1, wherein detecting hallucination content further comprises: Compare the natural language answer to the natural language context information.

3. The method of claim 2, wherein detecting hallucination content further comprises: determining similarities between sentences in the natural language answer and sentences in the natural language context information; and In response to at least one sentence in the natural language answer having a similarity with each sentence in the natural language context information that is less than a threshold, the at least one sentence is marked as hallucinated content.

4. The method according to claim 3, determining the similarity further comprises: Determine sentence embeddings for each sentence in the natural language answer; Determine sentence embeddings for each sentence in natural language context information; and Based on the sentence embeddings of the corresponding sentences in the natural language answer and the sentence embeddings of the corresponding sentences in the natural language context information, a corresponding embedding similarity between the corresponding sentence in the natural language answer and each sentence in the natural language context information is determined for each corresponding sentence in the natural language answer.

5. The method according to claim 4, determining the corresponding embedding similarities further comprises: Determine corresponding cosine similarities between sentence embeddings of corresponding sentences in the natural language answer and sentence embeddings of corresponding sentences in the natural language context information.

6. The method according to claim 4, determining the similarity further comprises: For each corresponding sentence in the natural language answer, a corresponding word pattern overlap rate between the corresponding sentence in the natural language answer and each corresponding sentence in the natural language context information is determined.

7. The method according to claim 6, wherein determining the corresponding word pattern overlap ratio further comprises: determining a corresponding mapping between a corresponding sentence in the natural language answer and a corresponding sentence in the natural language context information; and Based on the corresponding mapping, a corresponding word pattern overlap rate between the corresponding sentence in the natural language answer and the corresponding sentence in the natural language context information is determined.

8. The method according to claim 7, wherein determining the corresponding word pattern overlap ratio further comprises: determining the corresponding overlap length as the sum of overlapping words between the corresponding sentence in the natural language answer and the corresponding sentence in the natural language context information; and The corresponding word pattern overlap rate between the corresponding sentence in the natural language answer and the corresponding sentence in the natural language context information is determined by dividing the corresponding overlap length by the number of words in the shorter sentence of the corresponding sentence in the natural language answer and the corresponding sentence in the natural language context information.

9. The method of claim 8, wherein determining the corresponding overlap length further comprises: The sum of overlapping words between the corresponding sentence in the natural language answer and the corresponding sentence in the natural language context information is determined to include only words in overlapping word sequences having at least a predetermined minimum number of words.

10. The method of claim 6, marking at least one sentence in the natural language answer as hallucination content further comprises: At least one sentence in the natural language answer is marked as hallucinated content in response to: (i) the corresponding embedding similarity between at least one sentence in the natural language answer and each sentence in the natural language context information is less than a first predetermined threshold; and (ii) The corresponding word pattern overlap rate between at least one sentence in the natural language answer and each sentence in the natural language context information is less than a second predetermined threshold.

11. The method of claim 2, detecting hallucination content further comprising: Identify keywords in natural language answers; Determine whether the keyword is also present in the natural language context information; and In response to an amount of keywords in the natural language answer that are not present in the natural language context information exceeding a third predetermined threshold, the natural language answer is marked as including hallucination content.

12. The method according to claim 1, further comprising: A processor is utilized to generate a modified natural language answer that reduces hallucinated content in the natural language answer.

13. The method of claim 12, generating a modified natural language answer further comprising: A modified natural language answer is generated by removing the hallucinated content from the natural language answer.

14. The method according to claim 13, further comprising: determining, with a processor, a hallucination level that quantifies an amount of hallucination content in the natural language answer, Wherein removing the hallucinated content from the natural language answer is performed in response to the hallucination level being less than a predetermined threshold.

15. The method of claim 12, generating a modified natural language answer further comprising: retrieving modified natural language context information based on the natural language question using a second information retrieval technique, the second information retrieval technique being different from the first information retrieval technique; Based on the natural language question and the modified natural language context information, a language model is used to generate a modified natural language answer.

16. The method according to claim 15, further comprising: determining, with a processor, a hallucination level that classifies an amount of hallucination content in the natural language answer, Wherein, in response to the hallucination level being greater than a predetermined threshold, retrieving modified natural language context information and generating a modified natural language answer based on the modified natural language context information are performed.

17. The method according to claim 15, wherein: In each case, modified natural language context information is repeatedly retrieved using a different information retrieval technique and a modified natural language answer is generated based on the modified natural language context information until the modified natural language answer includes less than a threshold amount of hallucinated content.

18. The method according to claim 15, wherein: In each case, modified natural language context information is repeatedly retrieved using a different information retrieval technique and a modified natural language answer is generated based on the modified natural language context information until no additional information retrieval technique results in a reduction in the amount of hallucination content in the modified natural language answer.

19. The method of claim 1, further comprising: The modified natural language answer is output to the user using an output device.

20. The method according to claim 19, further comprising: identifying, using a processor, sentences in the natural language context information that provide support for sentences in the modified natural language sentence; and An indication of a sentence in the natural language context information that provides support for a sentence in the modified natural language sentence is outputted using an output device.

Images