Computing system for identifying hallusions in generative artificial intelligence outputs
By employing entity extraction and semantic comparison techniques within the computational system, the illusions of generative language models are detected and corrected, thus resolving the issue of inaccurate responses generated by generative language models in the aviation field and improving the accuracy and security of text output.
Patent Information
- Application Number
- CN202510867644.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-07-10
- Filing Date
- 2025-06-26
- Publication Date
- 2026-01-13
AI Technical Summary
Generative language models (LLMs) can produce inaccurate, illogical, or incoherent illusions when generating responses, posing a high risk, particularly in the generation of safety and regulatory reports in the aviation sector, and existing technologies struggle to effectively detect and correct these illusions.
The system receives the text output generated by LLM through a computing system, forms semantic pairs using an entity extraction model and a semantic pairing model, compares them with the semantic pairs of the original source text data, detects hallucinations using a semantic similarity model and an n-gram sequence model, and outputs a classification indicator by combining the hallucination classifier.
It improves the accuracy and efficiency of detecting hallucinations in generative AI output, reduces the need for human intervention, and ensures the accuracy and safety of generated text in the aviation field.
Smart Images
Figure HDA0005469023560000011 
Figure HDA0005469023560000021 
Figure HDA0005469023560000031
Abstract
Description
Technical Field
[0001] This disclosure relates generally to artificial intelligence, and more specifically to error reduction in large language models (LLMs). Background Technology
[0002] Generative language models such as LLMs are known to occasionally produce responses that are inaccurate, illogical, or incoherent—a phenomenon known as “illusion.” In many fields and domains where AI is increasingly being incorporated, information must be accurate, and responses from generative models must not contain inaccurate or misleading statements. While efforts to combat illusions (such as parameter tuning and reinforcement learning from human feedback) can have positive effects, the required levels of human intervention are often unsustainable or unscalable for many organizations and may even introduce additional errors.
[0003] At the same time, the aviation industry requires numerous documents and reports to maintain safety and comply with regulations. Historically, these reports have been entirely human-generated, a time-consuming task for trained individuals with other responsibilities. However, attempting to alleviate this drafting burden with the help of generative language models carries a high risk of generating illusions, which is unacceptable. Summary of the Invention
[0004] To address the aforementioned problems, according to one aspect of this disclosure, a computational system for identifying hallucinations in generative artificial intelligence (AI) output is provided. In this aspect, the computational system includes a processing circuitry configured to receive text output generated by a generative large language model (LLM) in response to input cues including origin source text data, and to extract entities from the text output and from the origin source text data using an entity extraction model. The processing circuitry uses a semantic pairing model to form first semantic pairs from the entities in the text output and second semantic pairs from the entities in the origin source text data, and uses a semantic similarity model to perform a semantic comparison between the first and second semantic pairs. The processing circuitry classifies, at least based on the comparison, whether either of the first semantic pairs is a hallucination and outputs a classification indication.
[0005] Another aspect of this disclosure relates to a method for identifying hallucinations in generative artificial intelligence (AI) output. The method includes receiving text output generated by a generative large language model (LLM) in response to input cues including original source text data, and extracting entities from the text output and from the original source text data. The method includes forming a first semantic pair from the entities in the text output and forming a second semantic pair from the entities in the original source text data, and semantically comparing the first semantic pair with the second semantic pair. The method includes classifying whether either of the first semantic pair is a hallucination, at least based on the comparison, and outputting an indication of the classification.
[0006] Another aspect of this disclosure relates to a method for training a computational system for identifying hallucinations in generative artificial intelligence (AI) output. The computational system includes a processing circuitry configured to receive text output generated by a generative large language model (LLM) in response to input cues including raw source text data; extract entities from the text output and from the raw source text data using an entity extraction model; form a first semantic pair from the entities in the text output and a second semantic pair from the entities in the raw source text data using a semantic pairing model; perform a semantic comparison between the first and second semantic pairs using a semantic similarity model; classify whether either of the first semantic pairs is a hallucination based at least on the comparison; and output a classification indication. The training method includes receiving or generating a domain knowledge base comprising text entities in a predetermined domain; training the entity extraction model on the domain knowledge base to extract entities related to the predetermined domain from the input text; and training the semantic similarity model on the domain knowledge base to perform a semantic comparison of the input text in a manner sensitive to the predetermined domain. Attached Figure Description
[0007] Figure 1 A schematic diagram of an example computing system based on this disclosure is shown.
[0008] Figure 2 The display shows Figure 1 An example graphical user interface (GUI) of the output of the computing system.
[0009] Figure 3A This is a flowchart of a method for identifying hallucinations in generative artificial intelligence (AI) output, and Figure 3B This is a flowchart of a method for training a computational system to identify hallucinations.
[0010] Figure 4 It is one of the feasible ones. Figure 1 A schematic diagram of an example computing environment for a computing system. Detailed Implementation
[0011] To solve the above problems, Figure 1 A schematic diagram of an example computing system 100 for identifying illusions in generative artificial intelligence (AI) output, according to this disclosure, is shown. The computing system 100 includes one or more computing devices 10, which include a processing circuitry system 12 configured to perform various functions. For example, the processing circuitry system 12 may be configured to receive text output 14 generated by a generative large language model (LLM) 16 in response to an input prompt 18 including original source text data 20. The input prompt 18 may be, for example, “Summarize the attached document,” where the attached document includes the original source text data. That is, the text output 14 of the LLM 16 may be a summary of the original source text data 20. In some cases, the summary may be in a specified format (e.g., a format that meets the requirements of a specific regulatory report). In the aviation field, this may include FAA safety reports, such as Continuing Operational Safety Plan (COSP) reports. As mentioned above, the ability to accurately incorporate the use of generative AI in this regard can significantly reduce the administrative burden on staff responsible for drafting such reports.
[0012] As mentioned above, the text output 14 generated by LLM 16 may or may not include unwanted illusions. Attempting to detect such illusions using alternative generative models presents numerous challenges, including improper model training, unavailable domain knowledge, contradictory or lacking factual training data, biased models, and inappropriate detection granularity. Therefore, this disclosure instead utilizes semantic modeling and a deep domain knowledge base 22, as described below.
[0013] In some cases, the original source text data may initially be presented in a format unsuitable for semantic processing (e.g., as a chain of emails with headings, salutations, signature blocks, footers, special characters, spaces, images, etc.). In such cases, the quality of the output can be improved by further configuring the processing circuitry system 12 to extract the original source text data 20 from the original source and normalize it before proceeding to the next step (e.g., using a text normalizer 24). The text normalizer 24 can also process the text output 14 similarly. The text normalizer 24 can be configured to strip non-text data, format and programmatically specify irrelevant text that is unrelated to the input prompt 18 and the text output 14, and otherwise remove noise to bring the original source text data 20 and / or the text output 14 into a good state for various other models discussed below to produce suitable output.
[0014] Next, using entity extraction model 26, the processing circuitry can be configured to extract entities 28A and 28B from text output 14 and from the original source text data 20. Entities (i.e., the first entity 28A and the second entity 28B) can be well-known nouns, verbs, adjectives, etc., found in the processed text. Focusing on entities 28A and 28B ensures that the computing system 100 knows which substantial topics, objects, actions, etc., are discussed in the original source text data 20 and text output 14, which helps in subsequent steps when identifying hallucinations. In the aviation field, examples of entities 28A and 28B might include “depleted,” “unable,” and “vibration.” Suitable examples of entity extraction model 26 include named entity recognition (NER) models or term frequency-inverse document frequency (TF-IDF) models. Both of these example models are capable of extracting substantial entities from text, thereby improving the accuracy of hallucination determination. A concrete example of entity extraction model 26 is the en_core_web_sm published by spaCy. Each extracted entity 28A, 28B may include, for example, an index (e.g., how many characters entity 28A, 28B begins to enter in a line each time it appears); the text constituting entity 28A, 28B; and an entity type. The processing circuitry system 12 may also be configured to receive or generate, during training time, a domain knowledge base 22 comprising text entities from a predetermined domain (such as aviation or a specific organization or company). The processing circuitry system can then be configured to train an entity extraction model 26 on the domain knowledge base 22 to extract entities 28A, 28B from input text that are relevant to the predetermined domain. By training in this manner, the entity extraction model 26 can become sensitive to words and phrases used in the domain or field, improving the accuracy and relevance of illusion detection.
[0015] Using semantic pairing model 30, the processing circuit system 12 can be configured to form a first semantic pair 32A from entity 28A of text output 14 and a second semantic pair 32B from entity 28B of the original source text data 20. Semantic pairs 32A and 32B can be between two entities 28A and 28B, or between one entity 28A and 28B and other text from the corresponding source text (i.e., the original source text data 20 and text output 14). Semantic pairs 32A and 32B can pair the paired entities 28A and 28B with contextual or relevance interpretations. Examples of semantic pairs 32A and 32B for the example entities 28A and 28B described above could include “depleted-hydraulic fluid quantity”, “unable-reengage lnavmode”, and “vibration-engine no.2”, where “lnav” is an abbreviation for “logfile navigator”. An example of semantic pairing model 30 is a question-and-answer (Q&A) LLM. A Q&A LLM can be, for example, a retrieval-enhanced generative system that includes a pre-trained retrieval unit and a generator configured to invoke the Q&A LLM. A concrete example of semantic pairing model 30 is tinyroberta-squad2 released by Hugging Face. The Q&A LLM can be configured to receive entities 28A and 28B and process the prompt “What is [ENTITY]?” or similar, and the output can be a phrase or word paired with entities 28A and 28B.
[0016] In some cases, the processing circuitry 12 can also be configured to use an n-gram model 34 to perform an n-gram comparison 36 between the first semantic pair 32A and the data in the domain knowledge base 22 of human-generated text. The n-gram comparison 36 compares the first semantic pair 32A with a known domain data corpus in the domain knowledge base 22 to check whether the phrase pairing is feasible within that domain. In the aviation domain, the domain knowledge base 22 could be, for example, a large repository of years of data from a company, including lists of parts, functions, processes, and historical write-ups and reports drafted by humans rather than an LLM. The n-gram sequence could be a binary sequence, a ternary sequence, etc. The n-gram model 34 can be configured to indicate the probability that each first semantic pair 32A generated by the generative LLM 16 is real or probable in the domain. Optionally, the probability could be a simple binary number relating whether the first semantic pair 32A has been found in the domain knowledge base 22, or whether it has been found at least a preset threshold number of times. When detecting hallucinations, the results of n-gram sequence comparison 36 can be considered because pairs that are not found or are rarely found in the domain knowledge base 22 are more likely to be hallucinations from generative LLM 16. For example, the pairing of "wing flap" and "inch" in close proximity is somewhat odd, and no judgment / determination that could affect the text output 14 including hallucinations can be found in the domain knowledge base 22.
[0017] The processing circuit system 12 can use a semantic similarity model 38 to perform a semantic comparison (i.e., perform comparison 40) between a first semantic pair 32A and a second semantic pair 32B. Comparison 40 is performed on a semantic basis (i.e., by meaning / connotation), rather than on a lexical basis (i.e., by character or word). Therefore, the semantic similarity model 38 can be able to consider words or phrases with similar meanings, even if they are spelled differently, and output a judgment that semantic pairs 32A and 32B are still semantically similar. For example, "remove" and "replace," or "depleted" and "run low," are two sets of semantically similar terms, and text output 14 using one of these words or phrases while the original source text data 20 uses the other will not be incorrectly labeled as a word-selection-based illusion. However, if semantic pairs 32A and 32B are too far apart in meaning, the first semantic pair 32A is labeled as suspicious for the illusion judgment.
[0018] The threshold for marking something as suspicious can be adjusted by the user or preset by the developer. The first semantic pair 32A does not have a corresponding second semantic pair 32B; in other words, the phrase in the text output 14 is not even found in a roughly similar form in the original source text data 20, which may lead to the discovery of "suspicious objects." An example of the semantic similarity model 38 is a sentence converter. A concrete example of the semantic similarity model 38 is bert-base-uncased released by Hugging Face. The semantic similarity model 38 can be open source and ready to be used as is, or the processing circuit system 12 can also be configured to train the semantic similarity model 38 on the domain knowledge base 22 during training time, thereby performing semantic comparisons of the input text in a pre-determined domain-sensitive manner. That is, the model can be trained or fine-tuned to not only know the vocabulary used in the domain or field, but also be biased towards finding suspicious phrases not used in the domain knowledge base 22.
[0019] The computing device 10 may also include a hallucination classifier 42. The processing circuitry 12 may be configured to classify whether any of the first semantic pairs 32A is a hallucination, at least based on a comparison 40 of the semantic similarity model 38, i.e., to generate a classification 44. If the above-described n-gram sequence comparison is performed, classification 44 can be performed at least based on the (semantic) comparison 40 and the n-gram sequence comparison 36 to improve classification accuracy. The hallucination classifier 42 may be a model predicting the hallucination classification of each of the first semantic pairs 32A, thereby classifying whether any of the first semantic pairs 32A is a hallucination, or it may be a simple programming function that computes the output based on comparison 40 and / or the n-gram sequence comparison 36. Finally, the processing circuitry may be configured to output an indication of classification 44, as referenced below. Figure 2 This will be discussed in more detail. For example, the output may be displayed in the graphical user interface (GUI) 46 of the display 48 of the computing device 10.
[0020] Figure 2 An example of a GUI 46 on display 48 is shown. It should be understood that display 48 displaying GUI 46 (see...) Figure 1 This can be associated with a separate computing device 10, other than the computing device performing illusion classification, such as in a client-server or cloud computing configuration. Furthermore, Figure 2 The GUI 46 shown is merely an example for illustration and can be modified in many ways. For instance, the example shown may be better suited as a developer view, and the client view or customer view may be simplified and include more graphics.
[0021] As in Figure 2In the diagram, the left side of GUI 46 can be dedicated to the original source, with the original source text data 20 displayed at the top, while the right side can be dedicated to the output of the generative LLM 16, with the text output 14 displayed at the top. Below the corresponding text data, the first and second entities 28A, 28B extracted by entity extraction model 26 can be displayed. The associated probability 50 of each of the first entities 28A can indicate the results of n-gram sequence comparison 36 and / or semantic comparison 40. In the example shown, the first entities 28A FUEL, TANK, PANEL, and WING all have a probability 50 of approximately 1, indicating that they are all confirmed and none are suspicious. In this paper, an indication 52 of 0 suspicious and 4 confirmed is displayed corresponding to classification 44. Classification 44 can include a positive hallucination classification (e.g., 0 suspicious) and / or a negative hallucination classification (e.g., 4 confirmed).
[0022] In some cases, GUI 46 may display an indication such that it includes an overall probability or judgment 54 that at least one of the displayed first semantic pairs 32A is a hallucination. In this document, the overall probability or judgment 54 is 89%, meaning the hallucination classifier 42 finds the generated text output 14 to have an accuracy of 89%. More specifically, this percentage can be calculated by determining how many of the first semantic pairs 32A and the first entity 28A are found to be suspicious or confirmed. For example, one in ten being classified as suspicious might have an accuracy of 90%. However, the accuracy can be weighted based on, for example, the certainty of a suspicious hallucination. Alternatively or additionally, icon 56 may alert the user that the probability or accuracy is below a threshold.
[0023] The first semantic pair 32A and the second semantic pair 32B have been sorted into separate categories in order to... Figure 2 The indicator is displayed in the middle, but can be changed to display all without categorization. This indicator can be visually indicated in the output text using one or more of the following font formats (e.g., bold, underline, italic, name, size, highlight, background, etc.), colors, labels, shapes, symbols, and icons to indicate at least one of the first semantic pairs 32A that classifies it as a hallucination. These various options provide numerous clear methods to quickly and accurately communicate to the user whether a hallucination has been detected. Figure 2Numerous such indications are shown. For example, icon 56 can also be placed next to the first semantic pair 32A LOSS—THE AMOUNT OF FUEL LOSS COULD NOT BEDETERMINED—which has a low score below a preset threshold, causing it to be flagged as suspicious. Throughout the GUI, the format of the suspicious indication 58 can be similar (e.g., red and bold) and contrasted with the confirmed indication 60 (e.g., green and italicized). The probability 50 for each first entity 28A and first semantic pair 32A can itself serve as an indication. In this way, the indication can individually show the classification of each of the first semantic pairs 32A as a probability that each corresponding first semantic pair 32A is a hallucination, as an alternative to or supplement to the overall probability or judgment 54. Furthermore, as shown in the text output 14 in the upper left corner of GUI 46, various first semantic pairs 32A can be indicated online as suspicious or confirmed using corresponding font formats, colors, labels, shapes, symbols, and / or icons. Rather than simply notifying users that a hallucination has been detected or may occur, this online indication can pinpoint the potential problem so that users can quickly and easily confirm or correct it. Therefore, generative LLM 16 can be used as a useful tool for users to produce text output 14, while also allowing users the opportunity to conveniently verify or correct the final output.
[0024] Figure 3A This is a flowchart of method 300A for identifying illusions in generative AI output, and Figure 3B This is a flowchart of method 300B for training a computational system for identifying hallucinations. (Refer to the above...) Figure 1 The computing system 100 shown provides the following descriptions of methods 300A and 300B. It should be understood that methods 300A and 300B can also be performed using other suitable components in other contexts.
[0025] refer to Figure 3A In 302, method 300A includes receiving text output generated by a generative large language model (LLM) in response to input prompts including original source text data. LLMs are increasingly used to assist humans in drafting various types of writing. In some cases, the text output of an LLM can be a summary of the original source text data. In such cases, human drafters may be cautious about illusions contained in the summary, but both the generative techniques used to detect illusions and human censorship have significant drawbacks in terms of efficiency and time.
[0026] Optionally, at 304, method 300A may include extracting and standardizing the original source text data before extracting entities from the original source text data in 306. As described above, standardizing the original source text data (and optionally the output text to ensure downstream comparisons using equal inputs) can remove noise and improve the quality of the output. At 306, method 300A may include extracting entities from the text output and from the original source text data. Entity extraction may be performed, for example, using a Named Entity Recognition (NER) model or a Term Frequency-Inverse Document Frequency (TF-IDF) model. These examples enable the extraction of substantial entities from text, improving the accuracy of downstream illusion determination.
[0027] In 308, method 300A may include forming first semantic pairs from entities in the text output and second semantic pairs from entities in the original source text data. These pairs may be formed between an entity and a context word or phrase that concisely explains the relevance of the entity as used in the text. Examples of semantic pairs in the aviation domain may include “takeoff – IAS inconsistency,” “crack – innerchord,” and “notice – very strong odor,” where IAS stands for “indicated airspeed.” Forming semantic pairs ensures that any analysis used to detect hallucinations considers how entities are used, not just their presence or absence. Therefore, if the original context is a positive statement that is reversed into a negative statement in the text output generated by the generative LLM, then simply checking for the same unique words before and after the LLM may not detect the hallucination.
[0028] In 310, method 300A may include performing a semantic comparison between the first semantic pair and the second semantic pair. As mentioned above, semantic comparison is more advantageous than lexical comparison because it takes into account the meaning of the words used (including synonyms, etc., up to the specified degree of similarity). In 312, method 300A may optionally include performing an n-gram sequence comparison between the first semantic pair and data from a domain knowledge base of human-generated text. The n-gram sequence comparison can further improve the accuracy of illusion detection by ensuring that the content generated from the LLM is meaningful within a domain or field context such as aviation or a specific company.
[0029] In 314, method 300A may include classifying whether either of the first semantic pairs is an illusion, at least based on comparison. If an n-gram sequence comparison is performed in 312, the classification is performed at least based on the comparison and the n-gram sequence comparison. In 316, method 300A may include an indication of the output classification. Figure 2The example shown illustrates an output, such as the overall probability or judgment that at least one of the first semantic pairs is a hallucination. This example allows a user to easily see at a glance how well the generative LLM performs on this particular task and whether further review is needed. However, computational systems can provide more specific indications, such as when the indication visually indicates in the output text that at least one of the first semantic pairs is classified as a hallucination using one or more of the following: font formatting, color, labels, shapes, symbols, and icons. This type of online indication is more sophisticated and helps to point out to the user which semantic pair and where in the generated text a problem might exist or not exist, allowing the user to quickly and easily review and / or correct the generated output text. Furthermore, the indication can show the classification of each of the first semantic pairs individually as the probability that each corresponding first semantic pair is a hallucination, allowing the user to be informed how certain the judgment is and thus make a more impartial correction.
[0030] Go to Figure 3B The following explains method 300B, used to train a computational system capable of executing method 300A. At 318, method 300B may include receiving or generating a domain knowledge base comprising textual entities from a predetermined domain. The domain knowledge base may be, for example, a large repository of data from a company or a specific domain over many years, including lists of parts, functions, processes, and historical reviews and reports. The domain knowledge base has various uses in training / fine-tuning and runtime applications. For example, at 320, method 300B may include training an entity extraction model on the domain knowledge base to extract entities from input text that are relevant to the predetermined domain. Thus, the entity extraction model can be trained to understand the terminology within the domain or field, improving extraction accuracy. Furthermore, at 322, method 300B may include training a semantic similarity model on the domain knowledge base to perform semantic comparisons of input text in a manner sensitive to the predetermined domain. In this way, the semantic similarity model can understand the meaning of words used in the domain and field, which may differ from lay use, and thus improve accuracy in pairing entities with context.
[0031] In some embodiments, the methods and processes described herein may be associated with a computing system of one or more computing devices. In particular, such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.
[0032] Figure 4 A non-limiting embodiment of a computing system 400 is schematically shown, which may implement one or more of the methods and processes described above. The computing system 400 is shown in a simplified form. The computing system 400 may embody the above-described and Figure 1 The computing system 100 is shown. Components of the computing system 400 may be included in one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, video game devices, mobile computing devices, mobile communication devices (e.g., smartphones) and / or other computing devices, as well as wearable computing devices (such as smartwatches and head-mounted augmented reality devices).
[0033] The computing system 400 includes a logic processor 402, volatile memory 404, and non-volatile storage device 406. The computing system 400 may optionally include a display subsystem 408, an input subsystem 410, a communication subsystem 412, and / or... Figure 4 Other components not shown.
[0034] The logic processor 402 includes one or more physical devices configured to execute instructions. For example, the logic processor may be configured to execute instructions that are part of one or more application programs, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions can be implemented to perform tasks, implement data types, transition the state of one or more components, achieve technical effects, or otherwise achieve desired results.
[0035] A logic processor may include one or more physical processors configured to execute software instructions. Additionally or alternatively, a logic processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of logic processor 402 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. The various components of the logic processor may optionally be distributed across two or more separate devices that may be remotely located and / or configured for coordinated processing. Aspects of the logic processor may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration. In this context, it will be understood that these virtualized aspects operate on different physical logic processors on various different machines.
[0036] Non-volatile storage device 406 includes one or more physical devices configured to retain instructions that can be executed by a logic processor to implement the methods and processes described herein. When these methods and processes are implemented, the state of non-volatile storage device 406 can be transformed—for example, to retain different data.
[0037] Non-volatile storage device 406 may include removable and / or built-in physical devices. Non-volatile storage device 406 may include optical memory, semiconductor memory, and / or magnetic memory, or other high-capacity storage technologies. Non-volatile storage device 406 may include non-volatile, dynamic, static, read / write, read-only, sequential access, location-addressable, file-addressable, and / or content-addressable devices. It should be understood that non-volatile storage device 406 is configured to retain instructions even when power to the non-volatile storage device 406 is cut off.
[0038] Volatile memory 404 may include a physical device comprising random access memory. Volatile memory 404 is typically utilized by logic processor 402 to temporarily store information during the processing of software instructions. It should be understood that when power to volatile memory 404 is cut off, volatile memory 404 typically does not continue storing instructions.
[0039] The logic processor 402, volatile memory 404, and non-volatile storage device 406 can be integrated together into one or more hardware logic components. For example, such hardware logic components may include field-programmable gate arrays (FPGAs), program-specific integrated circuits and application-specific integrated circuits (PASICs / ASICs), program-specific standard products and application-specific standard products (PSSPs / ASSPs), system-on-a-chip (SOCs), and complex programmable logic devices (CPLDs).
[0040] The terms "module," "program," and "engine" are used to describe aspects of computing system 400 that are typically implemented in software by a processor to perform specific functions using portions of volatile memory. These functions involve translational processing specifically configured for the processor to perform those functions. Therefore, a module, program, or engine can be instantiated via logic processor 402 using portions of volatile memory 404 to execute instructions stored in non-volatile storage device 406. It should be understood that different modules, programs, and / or engines can be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine can be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" can include individual or grouped executable files, data files, libraries, drivers, scripts, database records, etc.
[0041] When included, the display subsystem 408 can be used to present a visual representation of the data stored in the non-volatile storage device 406. The visual representation may take the form of a graphical user interface (GUI). As the methods and processes described herein change the data stored in the non-volatile storage device, thereby changing the state of the non-volatile storage device, the state of the display subsystem 408 can also be changed to visually represent the change in the underlying data. The display subsystem 408 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with the logic processor 402, volatile memory 404, and / or non-volatile storage device 406 in a shared housing, or such display devices may be peripheral display devices.
[0042] When included, the input subsystem 410 may include one or more user input devices (e.g., keyboard, mouse, touchscreen, camera, or microphone) or interface with such one or more user input devices (e.g., keyboard, mouse, touchscreen, camera, or microphone).
[0043] When included, the communication subsystem 412 can be configured to communicatively couple the various computing devices described herein to each other and to communicatively couple the various computing devices to other devices. The communication subsystem 412 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem may be configured to communicate via wired or wireless local area networks or wide area networks, broadband cellular networks, etc. In some embodiments, the communication subsystem may allow the computing system 400 to send messages to and / or receive messages from other devices via a network such as the Internet.
[0044] The following paragraphs provide additional description of the subject matter of this application. One aspect provides a computational system for identifying hallucinations in generative artificial intelligence (AI) output. The computational system includes a processing circuitry configured to receive text output generated by a generative large language model (LLM) in response to input cues including raw source text data; extract entities from the text output and from the raw source text data using an entity extraction model; form first semantic pairs from the entities in the text output and second semantic pairs from the entities in the raw source text data using a semantic pairing model; perform a semantic comparison between the first and second semantic pairs using a semantic similarity model; classify whether either of the first semantic pairs is a hallucination based at least on the comparison; and output a classification indication. Additionally or alternatively, in this respect, the indication includes a displayed overall probability or judgment that at least one of the first semantic pairs is a hallucination. Additionally or alternatively, in this respect, the indication visually indicates, using one or more of font formatting, color, labels, shapes, symbols, and icons, that at least one of the first semantic pairs within the output text is classified as a hallucination. In this regard, additionally or alternatively, the classification of each of the first semantic pairs is indicated separately as the probability that each corresponding first semantic pair is an illusion. In this regard, additionally or alternatively, the entity extraction model is a Named Entity Recognition (NER) model or a Term Frequency-Inverse Document Frequency (TF-IDF) model. In this regard, additionally or alternatively, the semantic pairing model is a Question-Answering (Q&A) LLM. In this regard, additionally or alternatively, the text output of the LLM is a summary of the original source text data. In this regard, additionally or alternatively, the processing circuitry is also configured to extract the original source text data from the original source and normalize the original source text data before extracting entities from the original source text data using the entity extraction model. In this regard, additionally or alternatively, the processing circuitry is also configured to perform an n-gram sequence comparison between the first semantic pairs and data from a domain knowledge base of human-generated text using an n-gram sequence model, and to perform classification at least based on said comparison and the n-gram sequence comparison. In this regard, additionally or alternatively, the processing circuitry is also configured to receive or generate a domain knowledge base including text entities in a predetermined domain during training time, and to train an entity extraction model on the domain knowledge base to extract entities related to the predetermined domain from the input text. In this regard, additionally or alternatively, the semantic similarity model is a sentence converter, and the processing circuitry is also configured to receive or generate a domain knowledge base including text entities related to the predetermined domain during training time, and to train a semantic similarity model on the domain knowledge base, thereby performing semantic comparison of the input text in a manner sensitive to the predetermined domain.
[0045] Another aspect provides a method for identifying hallucinations in generative artificial intelligence (AI) output. The method includes receiving text output generated by a generative large language model (LLM) in response to input cues including original source text data; extracting entities from the text output and from the original source text data; forming first semantic pairs from the entities in the text output and second semantic pairs from the entities in the original source text data; semantically comparing the first and second semantic pairs; classifying whether any one of the first semantic pairs is a hallucination based at least on the comparison; and outputting a classification indication. In this respect, additionally or alternatively, the indication includes a displayed overall probability or judgment that at least one of the first semantic pairs is a hallucination. In this respect, additionally or alternatively, the indication visually indicates, using one or more of font formatting, color, labels, shapes, symbols, and icons, that at least one of the first semantic pairs within the output text is classified as a hallucination. In this respect, additionally or alternatively, the indication shows the classification of each of the first semantic pairs individually as a probability that each corresponding first semantic pair is a hallucination. In this regard, additionally or alternatively, a Named Entity Recognition (NER) model or a Term Frequency-Inverse Document Frequency (TF-IDF) model is used to perform entity extraction. In this regard, additionally or alternatively, the text output of the LLM is a summary of the original source text data. In this regard, additionally or alternatively, the method further includes extracting and standardizing the original source text data before extracting entities from it. In this regard, additionally or alternatively, the method further includes performing an n-gram sequence comparison between the first semantic pair and data from a domain knowledge base of human-generated text, and performing classification at least based on said comparison and the n-gram sequence comparison.
[0046] Another aspect provides a method for identifying hallucinations in generative artificial intelligence (AI) output, comprising receiving text output generated by a generative large language model (LLM) in response to input cues including original source text data, and extracting entities from the text output and from the original source text data using an entity extraction model, which is a named entity recognition (NER) model or a term frequency inverse document frequency (TF-IDF) model. The method further comprises forming a first semantic pair from the entities in the text output and forming a second semantic pair from the entities in the original source text data, semantically comparing the first semantic pair with the second semantic pair using a semantic comparison model, classifying whether any one of the entities in the first semantic pair is a hallucination based at least on the comparison, and outputting a classification indicator that visually indicates at least one entity in the first semantic pair in the output text as a hallucination using one or more of font formatting, color, labels, shapes, symbols, and icons. In this aspect of the method, the entity extraction model has been trained on a domain knowledge base that includes text entities in a predetermined domain to extract entities related to the predetermined domain from the input text, and the semantic similarity model has been trained on the domain knowledge base to perform semantic comparison of the input text in a manner sensitive to the predetermined domain.
[0047] It should be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered in a limiting sense, as many variations are possible. The particular routines or methods described herein may represent one or more of any number of processing strategies. Therefore, the various actions shown and / or described may be performed in the order shown and / or described, in other orders, in parallel, or omitted. Similarly, the order of the above processes may be changed.
[0048] The subject matter of this disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems, and configurations disclosed herein, as well as other features, functions, actions, and / or properties, and any and all equivalents thereof.
[0049] Parts list
[0050] Computing device 10
[0051] Processing circuit system 12
[0052] Text output 14
[0053] Generative LLM 16
[0054] Input prompt 18
[0055] Original source text data 20
[0056] Domain Knowledge Base 22
[0057] Text Normalizer 24
[0058] Entity extraction model 26
[0059] Entities 28A and 28B
[0060] Semantic Pairing Model 30
[0061] First semantic pair 32A
[0062] The second semantic pair 32B
[0063] n-ary sequence model 34
[0064] n-ary sequence comparison 36
[0065] Semantic similarity model 38
[0066] Compare 40
[0067] Hallucination Classifier 42
[0068] Category 44
[0069] Monitor 48
[0070] Probability 50
[0071] Instructions 52, 58, 60
[0072] Verdict 54
[0073] Icon 56
[0074] Computing System 100
[0075] Methods 300A and 300B
[0076] Computing System 400
[0077] Logic Processor 402
[0078] volatile memory 404
[0079] Non-volatile storage device 406
[0080] Display Subsystem 408
[0081] Input Subsystem 410
[0082] Communication Subsystem 412
Claims
1. A computing system (100) for identifying hallucinations in generative artificial intelligence (AI) output, the computing system (100) comprising: Processing circuit system (12), which is configured as follows: Receive text output (14) generated by a generative large language model (LLM) (16) in response to an input prompt (18) including the original source text data (20); Using the entity extraction model (26), entities (28A, 28B) are extracted from the text output (14) and from the original source text data (20); Using a semantic pairing model (30), a first semantic pair (32A) is formed from the entity (28A) of the text output (14) and a second semantic pair (32B) is formed from the entity (28B) of the original source text data (20); The first semantic pair (32A) and the second semantic pair (32B) are semantically compared using a semantic similarity model (38); At least based on comparison (40), classify whether either of the first semantic pairs (32A) is a hallucination; and Output the classification (44) instructions (52, 56, 58, 60).
2. The computing system (100) according to claim 1, wherein, The indications (52, 56, 58, 60) include the overall probability or determination (54) that at least one of the first semantic pairs (32A) is a hallucination.
3. The computing system (100) according to claim 1, wherein, The indications (52, 56, 58, 60) visually indicate, using one or more of font format, color, label, shape, symbol and icon, at least one first semantic pair in the first semantic pair (32A) within the text output (14) that is classified as an illusion.
4. The computing system (100) according to claim 1, wherein, The instructions (52, 56, 58, 60) individually show the classification (44) of each of the first semantic pairs (32A) as the probability that each corresponding first semantic pair (32A) is a hallucination.
5. The computing system (100) according to claim 1, wherein, The entity extraction model (26) is a named entity recognition (NER) model or a term frequency-inverse document frequency (TF-IDF) model.
6. The computing system (100) according to claim 1, wherein, The semantic pairing model (30) is a question-and-answer (Q&A) LLM.
7. The computing system (100) according to claim 1, wherein, The text output (14) of the LLM is a summary of the original source text data (20).
8. The computing system (100) according to claim 1, wherein the processing circuit system (12) is further configured to extract the original source text data (20) from the original source and normalize the original source text data (20) before extracting the entity (28B) from the original source text data (20) using the entity extraction model (26).
9. The computing system (100) according to claim 1, wherein The processing circuitry (12) is also configured to perform an n-gram sequence comparison (36) between the first semantic pair (32A) and data from the domain knowledge base (22) of human-generated text using an n-gram sequence model (34), and The classification (44) is performed based at least on the comparison (40) and the n-gram sequence comparison (36).
10. The computing system (100) according to claim 1, wherein the processing circuitry (12) is further configured to: Receive or generate a domain knowledge base including text entities in a predetermined domain (22); and The entity extraction model (26) is trained on the domain knowledge base (22) to extract entities related to the predetermined domain from the input text.