A large language model illusion detection method and system based on type perception

Abstract

The application discloses a type-aware-based large language model hallucination detection method and system. The method comprises the following steps: receiving a question text input by a user, obtaining a main answer text and a plurality of sampling answer texts output by a large language model, and identifying whether the question text is a single-answer type or a multi-answer type; performing effectiveness and semantic relationship calculation on the sampling answer texts to form an answer effectiveness score, a text semantic cluster and a contradiction relationship; constructing a diagnosis feature according to the question type, calculating a hallucination risk score of the main answer text, and performing risk identification, review or output restriction according to the hallucination risk score. The application can reduce false positives in a multi-answer scenario and improve output monitoring accuracy.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, natural language processing, and large language model credibility assessment. Specifically, it relates to a type-aware large language model illusion detection method and system, which is particularly suitable for answer text risk identification, risk labeling, and output control scenarios in open-domain question answering, ambiguous question answering, set-type question answering, and generative artificial intelligence applications. Background Technology

[0002] With the widespread application of large language models in question-answering systems, intelligent customer service, knowledge retrieval, content generation, and decision support, these models can generate fluent natural language answers based on user-input questions. However, in the absence of external evidence, ambiguous questions, or insufficient internal model knowledge, large language models may generate seemingly reasonable but actually incorrect, poorly supported, or mismatched answers—a phenomenon often referred to as model illusion. For application systems that directly output answer text to users, failure to promptly identify and control such risky outputs can easily lead to the spread of misinformation, user misunderstanding, and a decline in system credibility.

[0003] Among existing hallucination detection schemes for large language models, a common approach is based on output consistency. This type of scheme typically samples the same question text multiple times to generate candidate answer texts, and then estimates the model's uncertainty based on the similarity, semantic clustering results, or contradictory relationships among the multiple answer texts. When multiple sampled answer texts are concentrated in the same semantic cluster, the model is generally considered relatively deterministic; when multiple sampled answer texts are scattered across multiple semantic clusters, the model is considered to have a higher risk of hallucination. This type of method can detect unstable answers in single-answer factual questions to some extent, but its implicit premise is that the question text corresponds to only one definite semantic answer target.

[0004] In open-domain question answering, ambiguous question answering, and set-based question answering, the same question text may naturally contain multiple semantically different but all valid answers, or multiple valid interpretations corresponding to different answers. For such multi-answer questions, the distribution of different sampled answer texts across multiple semantic clusters does not necessarily mean that the model is experiencing hallucinations; it may simply mean that the model outputs different correct set members or reasonable answers under different valid interpretations. If the number of semantic clusters, semantic entropy, or answer dispersion is still directly used as the basis for hallucination risk, it is easy to misjudge the diversity of legitimate answers as uncertainty or hallucination risk, thus causing false positives.

[0005] Furthermore, some existing solutions rely on internal state signals such as hidden layers, logit, attention distribution, or model structure, making them less applicable to closed-source large language models or models that can only be accessed via interfaces. Other solutions directly use the large language model as a judge to determine answer correctness, but the judgment results are easily affected by prompt word design, example selection, and the judge model's own knowledge bias, lacking explicit differentiation of the question text's answer structure. Therefore, existing technologies still need a hallucination detection scheme that can process output text and distinguish between single-answer and multi-answer questions to improve the reliability of large language model output monitoring in multi-answer scenarios. Summary of the Invention

[0006] Purpose of the invention: The purpose of this invention is to provide a type-aware large language model illusion detection method and system to solve the technical problem that existing output consistency illusion detection methods do not distinguish the structure of the question text answer and are prone to misjudging the diversity of legitimate answers in multi-answer questions as illusion risks.

[0007] Technical Solution: A type-aware illusion detection method for large language models, applied to the output monitoring module of large language models, includes the following steps:

[0008] S1, receive the question text input by the user and obtain the question type signal of the question text. The question type signal includes single answer type and multiple answer type. Single answer type means that the valid answers of the question text correspond to the same semantic answer target. Multiple answer type means that the question text has multiple semantically different valid answers or multiple valid interpretations.

[0009] S2, obtain the main answer text generated by the large language model to be detected around the question text and multiple sampled answer texts to form a set of sampled answer texts, where the main answer text is the target output text to be detected;

[0010] S3, perform text validity calculation and text semantic relation calculation on the sampled answer text set to obtain the answer validity score of each sampled answer text relative to the question text, the text semantic relation between sampled answer texts, and the text semantic cluster formed based on the text semantic relation, wherein the text semantic relation includes at least text semantic equivalence relation and text contradiction relation;

[0011] S4. Based on the question type signal, perform type condition interpretation on the answer validity score, text semantic relationship, and text semantic cluster, and construct a type feature vector; calculate type stability based on the type label set, which is used to characterize the consistency of the question type signal in multiple generation processes; obtain type condition diagnostic features based on the type feature vector and type stability; wherein, the type feature vector of a single answer type includes at least the answer dispersion risk obtained based on the text semantic cluster distribution, and the type feature vector of a multi-answer type includes at least the multi-answer invalid risk and the cross-contradiction risk;

[0012] S5, calculate the hallucination risk score of the main answer text based on the type condition diagnostic features, and perform output control processing on the main answer text based on the hallucination risk score. The output control processing includes at least one of adding a risk label to the main answer text, sending the main answer text to the review queue, or restricting the output of the main answer text.

[0013] 2. The method according to claim 1, wherein in step S1, obtaining the question type signal of the question text includes:

[0014] Input the question text into an external question type classifier, which then outputs a single-answer or multi-answer question type signal; or...

[0015] Input the question text into the type judgment model used to identify the answer structure, and the type judgment model outputs a question type signal of single answer type or multiple answer type;

[0016] If the valid answers to a question are only aliases, abbreviations, spelling variations, or synonyms of the same answer, the question text will be marked as a single-answer type. If the valid answers to a question form multiple semantically different answer clusters, or if the question text has multiple valid interpretations and different interpretations correspond to different answers, the question text will be marked as a multi-answer type.

[0017] Further, in step S2, the main answer text generated by the large language model to be detected around the question text, as well as multiple sampled answer texts, are obtained, including:

[0018] A low-temperature decoding method is used to enable the large language model to be tested to generate the main answer text based on the question text;

[0019] High-temperature sampling and kernel sampling methods are used to generate K sampled answer texts based on the same question text, forming a sampled answer text set;

[0020] When the question type signal is a multi-answer type, each sampling prompts the large language model to output a valid answer text, instead of requiring all valid answer texts to be listed at once.

[0021] Furthermore, in step S3, the text validity calculation of the sampled answer text set includes:

[0022] Input the question text and any sampled answer text into the answer validity validator, which is implemented using a large language model;

[0023] For an open-source large language model, a validation prompt is constructed, requiring the model to judge only between Yes and No, where Yes indicates a valid candidate answer and No indicates an invalid candidate answer. The length-normalized logarithmic probabilities of the model outputs Yes and No are calculated separately, and the valid probability of the candidate answer is obtained through Softmax. Define the answer validity score: For closed-source large language models, the question text q and candidate answer texts are input via API. The model is required to return a confidence score. When the label is "Yes", the confidence level is directly used as the answer validity score. When the label is "No", insufficient confidence is used as the answer validity score. ;

[0024] Calculate the risk of invalid answers based on the answer validity scores of K sampled answer texts in the sampled answer text set: The validity of the text is characterized by an answer validity score and an answer invalidity risk.

[0025] Further, in step S3, calculating the text semantic relationships of the sampled answer text set and forming text semantic clusters includes:

[0026] Perform bidirectional natural language reasoning on any two sampled answer texts to obtain the semantic implications and contradiction probabilities of the texts in two directions;

[0027] Based on the semantic implications of the text in two directions, the predicted categories in the two directions are determined. If the predicted categories in both directions are both implied, then the two sampled answer texts are semantically equivalent.

[0028] Clustering semantically equivalent sampled answer texts yields one or more text semantic clusters.

[0029] Furthermore, when the question text is interpreted as a single-answer type, the construction of the type feature vector in step S4 includes:

[0030] Based on the answer validity scores of the sampled answer texts in each text semantic cluster, calculate the validity-weighted quality of each text semantic cluster: ,in, This indicates that the sampled answer falls into the c-th text semantic cluster. The validity weighting ratio, and satisfying: m is the number of semantic clusters in the text;

[0031] The macroscopic dispersion is calculated based on the effectiveness-weighted quality of multiple text semantic clusters: And use macroscopic dispersion as the answer to diversify risk;

[0032] The risk of micro-inconsistency is calculated based on the textual semantic consistency among sampled answer texts within the same textual semantic cluster. Textual semantic consistency is represented as: The risk of micro-inconsistency is: ,in For two sampled answer texts and The average semantic entailment probability between them;

[0033] The risk of invalid answers, the risk of answer dispersion, and the risk of micro-inconsistency are used as type feature vectors under a single answer type: .

[0034] Furthermore, when the question text is interpreted as a multiple-answer type, the construction of the type feature vector in step S4 includes:

[0035] Calculate the risk of multiple invalid answers based on the answer validity scores of each sampled answer text in the sampled answer text set: ;

[0036] Based on the probability of textual contradictions between different sampled answer texts, and combined with the answer validity scores of the corresponding sampled answer texts, the risk of cross-contradiction is calculated: ,in Represents the sampled answer text and The probability of textual contradictions between them;

[0037] The risks of invalid multiple answers and cross-contradiction are used as type feature vectors under the multiple answer type: .

[0038] Furthermore, in step S5, calculating the hallucination risk score of the main answer text based on the type condition diagnostic features includes using at least one of the following methods:

[0039] Method 1: Set up single-answer risk headers and multi-answer risk headers. The single-answer risk header is used to output the basic risk of a single answer based on the type condition diagnostic features under a single answer type. The multi-answer risk header is used to output the basic risk of multiple answers based on the type condition diagnostic features under multiple answer types. The basic risks of single answers and / or multiple answers are calibrated based on type stability. The illusion risk score of the main answer text is obtained based on the calibrated risks.

[0040] Method 2: Convert the type condition diagnostic features into natural language feature summaries; input the question text, main answer text, natural language feature summaries, and few sample examples into the large language model evaluator; the large language model evaluator outputs the correct probability of the main answer text, and obtains the illusion risk score of the main answer text based on the correct probability.

[0041] A type-aware large language model-based hallucination detection system, used in the above-mentioned method, includes:

[0042] The question text acquisition module is used to receive the question text input by the user and obtain the question type signal of the question text. The question type signal includes single answer type and multiple answer type. The single answer type means that the valid answers of the question text correspond to the same semantic answer target, and the multiple answer type means that there are multiple semantically different valid answers or multiple valid interpretations in the question text.

[0043] The answer text generation module is used to obtain the main answer text generated by the large language model to be detected around the question text and multiple sampled answer texts to form a sampled answer text set. The main answer text is the target output text to be detected, and the multiple sampled answer texts are used to represent the output distribution of the large language model to be detected under the same question text.

[0044] The sampled answer text diagnosis module is used to calculate the text validity and text semantic relationship of the sampled answer text set, and obtain the answer validity score of each sampled answer text, the text semantic relationship between sampled answer texts, and the text semantic cluster formed based on the text semantic relationship.

[0045] The type condition feature construction module is used to interpret the type conditions of answer validity scores, text semantic relations, and text semantic clusters based on the question type signal, and construct type feature vectors; calculate type stability based on the type label set, which is used to characterize the consistency of question type signals in multiple generation processes; and obtain type condition diagnostic features based on the type feature vectors and type stability; wherein, the type feature vector of a single answer type includes at least the answer dispersion risk obtained based on the text semantic cluster distribution, and the type feature vector of a multi-answer type includes at least the multi-answer invalidity risk and the cross-contradiction risk;

[0046] The risk output control module is used to calculate the illusion risk score of the main answer text based on the type condition diagnostic features, and to perform output control processing on the main answer text based on the illusion risk score. The output control processing includes at least one of adding a risk label to the main answer text, sending the main answer text to the review queue, or restricting the output of the main answer text.

[0047] The present invention also provides an electronic device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the type-aware large language model-based illusion detection method as described above.

[0048] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the type-aware large language model-based illusion detection method described above.

[0049] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the type-aware large language model-based illusion detection method described above.

[0050] The present invention has the following beneficial effects:

[0051] (1) This invention utilizes the fact that by dividing the question text into single-answer and multi-answer types, and interpreting the diagnostic information of the sampled answer text under different types, the hallucination detection process can adapt to the answer structure of the question text itself. For single-answer type questions, if multiple sampled answer texts are distributed in different text semantic clusters, it usually indicates that the model has uncertainty about the unique semantic answer target. Therefore, incorporating the answer dispersion risk obtained based on the distribution of text semantic clusters into the type condition diagnostic features helps to identify unstable outputs in single-answer factual questions. For multi-answer type questions, multiple valid answer texts may correspond to different semantic clusters. This invention does not increase the hallucination risk by the number or dispersion of text semantic clusters alone, but constructs diagnostic features by combining the answer validity score and the contradictory relationship between answers, which can reduce the situation of misjudging the differences between legitimate multiple answers as hallucinations.

[0052] (2) This invention calculates the text validity of the sampled answer text set to determine whether each sampled answer text can serve as a valid answer to the question text, and thereby establishes the risk of invalid answers. This makes hallucination detection no longer solely dependent on the consistency between answer texts, but further focuses on whether the sampled answer texts themselves are valid. Especially in multi-answer type questions, even if the sampled answer texts are diverse, as long as multiple answer texts are valid answers and there are no obvious contradictions, the system will not increase the hallucination risk simply because the text semantic clusters are scattered, thereby improving the accuracy of risk judgment in open-domain question answering, ambiguous question answering, and aggregate question answering scenarios.

[0053] (3) This invention calculates the semantic relationships between sampled answer texts and forms semantic clusters, thereby simultaneously obtaining semantic equivalence and contradiction relationships. Semantic equivalence relationships are used to identify whether different expressions point to the same semantic answer target, while contradiction relationships are used to identify whether there is a factual conflict between different sampled answer texts. This enables the system to measure the degree of answer dispersion using semantic cluster distribution under a single answer type, and to identify inconsistent or conflicting outputs using the risk of contradiction between answers under multiple answer types, thereby improving the interpretability of hallucination detection results.

[0054] (4) After calculating the illusion risk score of the main answer text, this invention does not merely output an abstract score, but further performs output control processing on the main answer text based on the risk score, such as risk identification, sending to a review queue, or restricting output. By combining the text diagnostic results with the output control process of the large language model application system, this invention can prompt, intercept, or manually review high-risk outputs before the user receives the answer text, thereby improving the security and credibility of generative artificial intelligence applications. It also helps to meet the requirements of controllable risk, reviewable results, and interferable output in actual system deployment.

[0055] (5) This invention mainly relies on externally available information such as question text, main answer text, sampled answer text, text validity score, text semantic relations, and text semantic clusters, without requiring access to the hidden layers, logit, attention weights, or model structure of the large language model to be detected. Therefore, this scheme can be used for both open-source large language models and closed-source large language models that can only be called through an interface, and has good deployment versatility. At the same time, since the detection logic revolves around text input and output, and differentiates the diagnostic features through question type signals, this invention can improve the ability to identify the hallucination risk of multi-answer questions while ensuring its applicability. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments are briefly introduced below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort, wherein:

[0057] Figure 1 This is the overall flowchart of the method of the present invention;

[0058] Figure 2 This is a schematic diagram of the construction of type condition diagnostic features;

[0059] Figure 3 This is a schematic diagram of two optional hallucination detection routes. Detailed Implementation

[0060] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to specific embodiments and accompanying drawings.

[0061] This invention provides a type-aware large language model illusion detection method, applied to a large language model output monitoring module. In this embodiment, the large language model application system can be an open-domain question-answering system, an intelligent customer service system, a retrieval enhancement generation system, a content generation system, or other application systems that call the large language model to generate answer text. The large language model to be detected is the model in the application system that generates answer text based on the user-input question text; it can be a locally deployed open-source large model or a closed-source large language model called through an application programming interface (API). The output monitoring module is located between the large language model to be detected and the user's output. For ease of explanation, the system mentioned below refers to the illusion detection system or output monitoring module that executes the method of this invention. (See also...) Figure 1 The method specifically includes the following steps:

[0062] S1 receives the question text input by the user and obtains the question type signal of the question text. The question type signal includes single answer type and multiple answer type. The single answer type indicates that the valid answers of the question text correspond to the same semantic answer target, and the multiple answer type indicates that there are multiple semantically different valid answers or multiple valid interpretations in the question text.

[0063] It receives the question text q submitted by the user through a question-and-answer interface, application programming interface, or other text input terminal, and feeds it into the large language model to be tested. The question can be a factual open-domain question, an ambiguous question, a set-based question, or other questions that can be answered with a short answer.

[0064] Obtain the question type signal t from the question text q. The question type signal t represents the valid answer structure of question text q, including single-answer type SA and multiple-answer type MA. Single-answer type SA indicates that valid answers to question text q may exist in different textual forms, but all correspond to the same semantic answer target, such as the full name, abbreviation, alias, acronym, spelling variant, or synonym of the same entity name. Multiple-answer type MA indicates that question text q has multiple semantically different but all valid answers, or that question text q has multiple valid interpretations, with each valid interpretation corresponding to a different answer.

[0065] In one implementation, the system can input the question text q into an external question type classifier, which then outputs an SA or MA type label as the question type signal t. The external question type classifier can be a text binary classification model that takes the question text as input, such as a classifier trained based on a natural language understanding model. In another implementation, the system can also determine the answer structure of the question text q using a prompting large language model or an auxiliary large language model, and output an SA or MA type label. For models that can control the output format, the type label can also be output in a preset text format, such as outputting a field containing "...". <tag> <answer>The text of "", in which <tag>This field can be either SA or MA. Parsing this field yields the question type signal t, which determines whether the question text q is more suitable for further processing as a single-answer or multi-answer question.

[0066] S2, obtain the main answer text generated by the large language model to be detected around the question text and multiple sampled answer texts to form a sampled answer text set. The main answer text is the target output text to be detected, and the multiple sampled answer texts are used to represent the output distribution of the large language model to be detected under the same question text.

[0067] Obtain the main answer text generated by the large language model to be tested around the question text q. And multiple sampled answer texts, forming a sampled answer text set. Main answer text This represents the main output of the large language model to be tested for the question text q, and is also the target output text for subsequent hallucination risk detection. Multiple sampled answer texts are used to represent the different answer expressions or content that the large language model to be tested might give under the same question text q.

[0068] Specifically, a low-temperature decoding method can be used to call the large language model to be tested to generate the main answer text. To reduce the randomness of the output and make the main answer text... It more closely approximates the model's deterministic answer to the current question text q. For example, in labeled generation mode, the system generates a response from "..." <tag> <answer>The main type observation is obtained from the format parsing. and the main answer text In external classifier mode, the output is determined by the classifier. Output by the model to be detected .

[0069] Subsequently, the large language model to be detected is called multiple times under the same question text q, using a temperature parameter higher than that used when generating the main answer text, and K sampled answer texts are generated using kernel sampling. For example, the temperature parameter can be set to 0.8 and the top-p parameter to 0.9, and K sampled answer texts can be generated. K can be set according to the detection accuracy and computational cost, for example, 9, but is not limited to this.

[0070] If the model output contains type labels, then the set of sampled type labels is also parsed.

[0071]

[0072] When the question type signal t is a single-answer type (SA), multiple sampled answer texts are mainly used to reflect the output stability of the large language model under test around the same semantic answer target. When the question type signal t is a multi-answer type (MA), the sampling prompts can require the large language model under test to output one valid answer text at a time, instead of requiring all valid answer texts to be listed at once, so as to obtain the answer texts corresponding to different valid answers or different valid interpretations that may exist under the question text.

[0073] S3, calculate the text validity and text semantic relations of the sampled answer text set to obtain the answer validity score of each sampled answer text relative to the question text, the text semantic relations between sampled answer texts, and the text semantic clusters formed based on the text semantic relations, wherein the text semantic relations include at least text semantic equivalence relations and text contradiction relations.

[0074] Text validity calculation is used to determine the validity of each sampled answer text. Whether it can serve as a valid answer to the question text q. Specifically, the question text q and the sampled answer text can be compared. Input the answer validity validator, and output the sampled answer text. The probability of a valid answer text is used as the sampled answer text. Validity score of answers ,in, For example, construct a validation prompt, inputting the question text q and candidate answer texts. The validator is required to output the "valid / invalid" probability. The validator in this invention is implemented using a large language model to determine candidate answers. Can this be considered a valid answer to question q? For an open-source large language model, construct validation hints, requiring the model to judge only between Yes and No. Here, Yes indicates a valid candidate answer, and No indicates an invalid candidate answer. Calculate the length-normalized logarithmic probabilities of the model outputs Yes and No respectively, and obtain the valid probability of the candidate answer through Softmax. If the probability of the valid answer output by the validator is... Then the definition is: For closed-source large language models, the question text q and candidate answer texts are input via API. The model is required to return a confidence score. When the label is Yes, the confidence score is directly used as the answer validity score. When the label is "No", insufficient confidence is used as the validity score. Answer validity score The higher the value, the better the sampled answer text. The more likely it is to be a valid answer to the question text q; the more likely it is to be a valid answer score. The lower the value, the lower the value of the sampled answer text. The more likely it is to be an invalid answer, a wrong answer, or an answer that does not match the question text q.

[0075] Furthermore, the answer invalidity risk can be calculated based on the answer validity scores of multiple sampled answer texts. For example, the overall validity can be determined based on the average or weighted value of the validity scores of each sampled answer text, and the overall validity can be converted into answer invalidity risk. For K sampled answer texts, the invalidity risk is defined as: When the overall validity of the sampled answers is lower, The larger the value, the higher the risk of hallucination. Therefore, the system not only focuses on the consistency between multiple sampled answer texts, but also determines whether these sampled answer texts themselves have the potential to be valid answers.

[0076] This invention also performs text semantic relation calculations, on any two sampled answer texts in the sampled answer text set. and Perform semantic relation determination. This can be done using either a natural language inference model or a semantic relation determination model, respectively. Contains probability, Contains The system can determine the semantic equivalence between two sampled answer texts based on the implied probabilities in both directions, and the degree of textual contradiction between them based on the contradictory probabilities in both directions. If both sampled answer texts have high probabilities in bidirectional implied relations, they can be considered to point to the same semantic answer target; if the two sampled answer texts have high contradictory probabilities, they can be considered to have factual or semantic conflicts.

[0077] Specifically, let: .in, express Contains The probability, express Contains The probability of their semantic implications is defined as follows:

[0078]

[0079] set up: .in, and Let represent the probability of conflict in each of the two directions. The maximum probability of conflict in one direction is defined as:

[0080]

[0081] This invention constructs semantic clusters based on bidirectional natural language reasoning results, dividing all semantically equivalent answers into several semantic clusters. Specifically, each sampled answer text is first treated as a node, and each answer text is initialized to belong to an independent set.

[0082] For any two candidate answer texts and First, determine if the two answers are completely identical. If they are identical after removing leading and trailing spaces and converting to lowercase, then they are considered semantically equivalent and merged into the same semantic cluster.

[0083]

[0084] If the two texts are not completely identical, a natural language inference model is used for bidirectional judgment. Construct two-way promise-hypothesis inputs:

[0085]

[0086]

[0087]

[0088]

[0089] The natural language inference model outputs predicted categories in two directions:

[0090]

[0091]

[0092] If the predicted categories in both directions are entailment, then the answer is considered... and Semantic equivalence:

[0093]

[0094]

[0095] When the above conditions are met, using a disjoint-set data structure will... and Merge them into the same set. After traversing all answer pairs, each connected set in the disjoint-set data structure is a semantic cluster, resulting in:

[0096]

[0097] Where m is the number of semantic clusters.

[0098] Each text semantic cluster represents a set of semantically equivalent or substantially equivalent sampled answer texts. Different text semantic clusters typically represent answers with different objectives, content, or interpretations. The resulting sampled answer diagnostic information includes the answer validity score of each sampled answer text, the text semantic equivalence relationships between sampled answer texts, text contradiction relationships, and text semantic clusters formed by these relationships.

[0099] S4. Based on the question type signal, perform type condition interpretation on the answer validity score, text semantic relationship, and text semantic cluster, and construct type feature vector; calculate type stability based on the type label set, which is used to characterize the consistency of question type signal in multiple generation processes; obtain type condition diagnostic features based on type feature vector and type stability; wherein, the type feature vector of a single answer type includes at least the answer dispersion risk obtained based on the text semantic cluster distribution, and the type feature vector of a multi-answer type includes at least the multi-answer invalid risk and cross-contradiction risk.

[0100] In step S4, based on the question type signal t obtained in step S1, type condition interpretation is performed on the answer validity score, text semantic relationship, and text semantic cluster obtained in step S3 to construct a type feature vector. That is, this invention does not apply the same risk interpretation rule to all question texts, but rather assigns different risk meanings to the same sampled answer diagnostic information based on whether the question text belongs to a single-answer type (SA) or a multi-answer type (MA). The construction process of the type condition diagnostic features is as follows: Figure 2 As shown in the image.

[0101] When the question type signal t indicates that the question text q is of single-answer type SA, the valid answers to question text q should correspond to the same semantic answer target. In this case, if multiple sampled answer texts are distributed across multiple different text semantic clusters, it usually indicates that the large language model being tested has uncertainty in its response to the question text or that its output is unstable. Therefore, the validity-weighted quality of each text semantic cluster can be calculated based on the answer validity scores of the sampled answer texts in each text semantic cluster. Furthermore, the macroscopic dispersion can be calculated based on the validity-weighted quality of multiple text semantic clusters, and this macroscopic dispersion is used as the answer dispersion risk. The higher the macroscopic dispersion, the more dispersed the sampled answer texts are, and the higher the illusion risk under single-answer type questions. The microscopic inconsistency risk can also be calculated based on the semantic consistency between sampled answer texts within the same text semantic cluster, reflecting whether the expression within the same answer cluster is stable. That is, under the SA type, the conditional feature vector is constructed based on the answer invalidity risk, answer dispersion risk, and microscopic inconsistency risk.

[0102] Specifically, when a question is interpreted as an SA type, scattered answers typically indicate model uncertainty. Therefore, in this case, the validity-weighted semantic cluster quality is calculated. For any semantic cluster... Its effectiveness-weighted quality is defined as:

[0103]

[0104] in, Let represent the weighted proportion of the validity of the sampled answer falling into the c-th semantic cluster, and satisfy: If m > 1, calculate the macroscopic dispersion:

[0105]

[0106] The larger the value, the more dispersed the sampled answers, and the higher the uncertainty in the single-answer case. Further, we calculate the semantic consistency within the cluster. For semantic clusters... If a cluster contains at least two answers, then it is defined as follows:

[0107]

[0108] This defines the risk of micro-inconsistency:

[0109]

[0110] The larger the value, the more unstable or inconsistent the answer representations are within the same semantic cluster. Ultimately, the SA type feature vector is defined as:

[0111]

[0112] When the question type signal t indicates that the question text q is of the multiple-answer type (MA), the question text q itself may have multiple semantically different but all valid answers, or multiple valid interpretations corresponding to different answers. In this case, the distribution of multiple sampled answer texts in different text semantic clusters does not necessarily indicate that the model is experiencing hallucinations; it may also indicate that the model has sampled different valid answers. Therefore, this invention does not increase the hallucination risk solely by the number or dispersion of text semantic clusters, but focuses on utilizing answer validity scores and text contradiction relationships to construct type conditional diagnostic features. The risk of multiple-answer invalidity can be calculated based on the answer validity scores of each sampled answer text in the sampled answer text set; simultaneously, the risk of cross-contradiction can be calculated based on the degree of text contradiction between different sampled answer texts, combined with the answer validity scores of the corresponding sampled answer texts. For two sampled answer texts with high validity scores and mutual contradiction, their contradiction relationship contributes significantly to the risk of cross-contradiction; for sampled answer texts with low validity scores, their contradiction relationship contributes less to the overall risk. That is, under the MA type, using the risk of multiple-answer invalidity and the risk of cross-contradiction as type feature vectors can distinguish between the diversity of legitimate answers and invalid or conflicting answers caused by hallucinations.

[0113] Specifically, when a question is interpreted as an MA type, multiple semantic clusters may represent different correct answers; therefore, semantic diversity is not directly considered a risk. The primary focus is on whether the answers are valid and whether different answers contradict each other. First, we define the risk of multiple invalid answers: Secondly, calculate the risk of cross-contradiction. For all sampled answer pairs, define:

[0114]

[0115] in, Indicate the answer and The probability of contradiction, This is used to emphasize the contradictory relationship between two highly effective answers. If the two answers themselves have low effectiveness, the contribution of their contradictory relationship to the overall risk is correspondingly reduced. Ultimately, the MA type feature vector is defined as:

[0116]

[0117] Furthermore, this invention also calculates type stability, which characterizes the consistency of the question type signal across multiple generation processes. Specifically, if a labeled generation mode is used, the set of type labels is parsed from the sampled answers: Let the principal type observation be denoted as Then type stability is defined as:

[0118]

[0119] Where 1[·] is an indicator function. When If the value is 1, then the value is 0; otherwise, the value is 0. The larger the value, the more stable the model's judgment of the problem type across multiple generation attempts. If an external classifier is used, and the classifier only outputs a single label, then we can let: .

[0120] like Figure 2 As shown, the final type conditional diagnostic features include , and .

[0121] S5, calculate the hallucination risk score of the main answer text based on the type condition diagnostic features, and perform output control processing on the main answer text based on the hallucination risk score. The output control processing includes at least one of adding a risk label to the main answer text, sending the main answer text to the review queue, or restricting the output of the main answer text.

[0122] Reference Figure 3 This invention provides two routes for calculating the illusion risk score of the main answer text. Either route can be chosen, or both can be selected.

[0123] Route A: Set up single-answer risk headers and multi-answer risk headers. The single-answer risk header is used to output the basic risk of a single answer based on the type condition diagnostic features under a single answer type. The multi-answer risk header is used to output the basic risk of multiple answers based on the type condition diagnostic features under multiple answer types. The basic risks of single answers and / or multiple answers are calibrated based on type stability. The illusion risk score of the main answer text is obtained based on the calibrated risks.

[0124] Specifically, SA risk headers and MA risk headers are set separately. The SA risk header uses... Input, output basic risk: ,in, and The parameters are learnable, and σ(·) is the Sigmoid function. The MA risk head is... Input, output basic risk: ,in, and These are learnable parameters. The mechanism of the true answer is denoted as a latent variable: First, estimate the prior probability that the problem belongs to SA. In labeled sampling mode, the following can be estimated from the sampling labels: In the external classifier mode, the following options are available: This allows for further calibration of the prior probabilities: ,in, and These are learnable calibration parameters. Therefore, we obtain: , Simultaneously, type stability is calibrated: .in, and These are learnable parameters. Use them. The value is adjusted for the underlying risk: , When type stability is low, The smaller the risk, the higher the risk will ultimately be.

[0125] Further study of the type observation matrix: Where z,t∈{SA,MA}. Calculate the posterior of the latent variables using Bayes' theorem: Ultimately, route A outputs the posterior expected risk: Where z = SA, = When z = MA, = .

[0126] During training, if the training sample labels are ∈{0,1}, where =1 indicates that the main answer is incorrect or a hallucination. =0 indicates the main answer is correct, minimizing the marginal negative log-likelihood:

[0127]

[0128] Route B: Convert the type condition diagnostic features into natural language feature summaries; input the question text, main answer text, natural language feature summaries, and few sample examples into the large language model evaluator; the large language model evaluator outputs the correct probability of the main answer text, and obtains the illusion risk score of the main answer text based on the correct probability.

[0129] In this approach, the numerical features obtained in steps S3 to S5 are converted into natural language evidence. Specifically, the 20th, 40th, 60th, and 80th percentile thresholds for each feature are calculated on the training set, and the features of the target sample are mapped to descriptions such as "very low," "low," "medium," "high," and "very high," while retaining the numerical values. The system selects static examples from the training pool and retrieves dynamic examples based on the semantic similarity of the question and the Mahalanobis distance in the feature space, constructing a few-shot prompt. The final prompt includes a task description, static examples, dynamic examples, and a feature summary of the target sample. The large model evaluator outputs the probability of the main answer being correct. Where φ represents the summary of linguistic features. The hallucination risk of Route B is defined as:

[0130]

[0131] Finally, the risk scores obtained from either route A or route B are normalized to between 0 and 1. If route A is used, then: If route B is used, then: The closer the risk score U is to 1, the stronger the main answer. The more likely the answer is to be wrong; the closer the risk score U is to 0, the stronger the likelihood of the correct answer. The more likely it is to be the correct answer.

[0132] The hallucination risk score can be used as a basis for performing output control processing on the main answer text. The output control processing includes at least one of adding a risk label to the main answer text, sending the main answer text to the review queue, or restricting the output of the main answer text.

[0133] To verify the performance of the proposed method, verification experiments were conducted in the embodiments. The experimental results are shown in Table 1.

[0134] Table 1. Comparison of AUROC between the present invention and traditional semantic clustering methods.

[0135] Model Dataset Traditional semantic clustering AUROC This invention AUROC Increase Qwen2.5-7B-Instruct NQOpen 0.7275 0.8262 0.0987 Qwen2.5-7B-Instruct AmbigQA 0.7156 0.853 0.1374 Qwen2.5-7B-Instruct WebQA 0.6858 0.7617 0.0759 LLaMA-3-8B-Instruct NQOpen 0.7027 0.8006 0.0979 LLaMA-3-8B-Instruct AmbigQA 0.7375 0.8335 0.096 LLaMA-3-8B-Instruct WebQA 0.6803 0.7318 0.0515

[0136] As shown in Table 1, on the three open-domain question-answering datasets NQOpen, AmbigQA, and WebQA, the present invention achieves an AUROC improvement over traditional semantic clustering methods. Specifically, on the Qwen2.5-7B-Instruct model, the AUROC of traditional semantic clustering methods on NQOpen, AmbigQA, and WebQA are 0.7275, 0.7156, and 0.6858, respectively; after adopting the type-aware uncertainty estimation of the present invention, the corresponding AUROCs are improved to 0.8262, 0.8530, and 0.7617, respectively, with improvements of 0.0987, 0.1374, and 0.0759. On the LLaMA-3-8B-Instruct model, the AUROC of traditional semantic clustering methods on NQOpen, AmbigQA, and WebQA are 0.7027, 0.7375, and 0.6803, respectively. After adopting this invention, the corresponding AUROCs are improved to 0.8006, 0.8335, and 0.7318, respectively, representing improvements of 0.0979, 0.0960, and 0.0515. These results demonstrate that this invention can effectively improve the risk ranking and differentiation ability between illusory answers and correct answers.

[0137] Existing detection methods based on the model's internal state typically require access to hidden layers, logit, attention weights, or model structure, making them difficult to apply to closed-source large language models. This invention relies solely on the model's output text, question type signal, answer validity judgment, and external semantic relationship judgment, without requiring access to the internal structure of the model being detected. Therefore, this invention can be used with open-source models such as Qwen, LLaMA, Mistral, and GLM, as well as closed-source models that can only be accessed via API, offering greater deployment versatility.

[0138] In ablation experiments, using meaningful SA / MA type signals is generally superior to random type signals or processing all signals as SA. For example, on the AmbigQA dataset of Qwen2.5-7B-Instruct, the AUROC of the LLM evaluation path is 0.7815 when using true type signals, 0.7624 for random type signals, and 0.6687 when processing all signals as SA; on the NQOpen dataset of GLM-4-9B-Chat, the AUROC is 0.7540 when using true type signals, 0.6666 for random type signals, and 0.6138 when processing all signals as SA. These results demonstrate that the performance improvement of this invention does not simply come from increasing the number of samples or adjusting the threshold, but directly from the type condition interpretation mechanism for single-answer and multi-answer questions.

[0139] This invention also provides a type-aware large language model-based hallucination detection system for performing the above-described method, comprising:

[0140] The question text acquisition module is used to receive the question text input by the user and obtain the question type signal of the question text. The question type signal includes single answer type and multiple answer type. The single answer type means that the valid answers of the question text correspond to the same semantic answer target, and the multiple answer type means that there are multiple semantically different valid answers or multiple valid interpretations in the question text.

[0141] The answer text generation module is used to obtain the main answer text generated by the large language model to be detected around the question text and multiple sampled answer texts to form a sampled answer text set. The main answer text is the target output text to be detected, and the multiple sampled answer texts are used to represent the output distribution of the large language model to be detected under the same question text.

[0142] The sampled answer text diagnosis module is used to calculate the text validity and text semantic relationship of the sampled answer text set, and obtain the answer validity score of each sampled answer text, the text semantic relationship between sampled answer texts, and the text semantic cluster formed based on the text semantic relationship.

[0143] The type condition feature construction module is used to interpret the type conditions of answer validity scores, text semantic relations, and text semantic clusters based on the question type signal, and construct type feature vectors; calculate type stability based on the type label set, which is used to characterize the consistency of question type signals in multiple generation processes; and obtain type condition diagnostic features based on the type feature vectors and type stability; wherein, the type feature vector of a single answer type includes at least the answer dispersion risk obtained based on the text semantic cluster distribution, and the type feature vector of a multi-answer type includes at least the multi-answer invalidity risk and the cross-contradiction risk;

[0144] The risk output control module is used to calculate the illusion risk score of the main answer text based on the type condition diagnostic features, and to perform output control processing on the main answer text based on the illusion risk score. The output control processing includes at least one of adding a risk label to the main answer text, sending the main answer text to the review queue, or restricting the output of the main answer text.

[0145] The present invention also provides an electronic device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein the programs, when executed by the processors, implement the above-described type-aware large language model-based illusion detection method.

[0146] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described type-aware large language model-based illusion detection method.

[0147] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described type-aware large language model-based illusion detection method.

[0148] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems (apparatus), electronic devices, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] This invention is described with reference to a flowchart of a method according to embodiments of the invention. It should be understood that each step in the flowchart and combinations thereof can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1 A device for performing a specified function in one or more processes. The processor involved in each embodiment may be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a quantum computing-based data processing logic device, an artificial intelligence processor, etc., and is not limited thereto.

[0150] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 The function specified in one or more processes.

[0151] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 Steps of a specified function in one or more processes.

[0152] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, apparatus, or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, apparatus, or device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, apparatus, or device that includes said element.< / answer> < / tag> < / tag> < / answer> < / tag>

Claims

1. A type-aware illusion detection method for large language models, applied to the output monitoring module of a large language model, characterized in that, Includes the following steps: S1, receive the question text input by the user and obtain the question type signal of the question text. The question type signal includes single answer type and multiple answer type. Single answer type means that the valid answers of the question text correspond to the same semantic answer target. Multiple answer type means that the question text has multiple semantically different valid answers or multiple valid interpretations. S2, obtain the main answer text generated by the large language model to be detected around the question text and multiple sampled answer texts to form a set of sampled answer texts, where the main answer text is the target output text to be detected; S3, perform text validity calculation and text semantic relation calculation on the sampled answer text set to obtain the answer validity score of each sampled answer text relative to the question text, the text semantic relation between sampled answer texts, and the text semantic cluster formed based on the text semantic relation, wherein the text semantic relation includes at least text semantic equivalence relation and text contradiction relation; S4. Based on the question type signal, perform type condition interpretation on the answer validity score, text semantic relationship, and text semantic cluster, and construct a type feature vector; calculate type stability based on the type label set, which is used to characterize the consistency of the question type signal in multiple generation processes; obtain type condition diagnostic features based on the type feature vector and type stability; wherein, the type feature vector of a single answer type at least includes the answer dispersion risk obtained based on the text semantic cluster distribution, and the type feature vector of a multi-answer type at least includes the multi-answer invalid risk and the cross-contradiction risk; S5, calculate the hallucination risk score of the main answer text based on the type condition diagnostic features, and perform output control processing on the main answer text based on the hallucination risk score. The output control processing includes at least one of adding a risk label to the main answer text, sending the main answer text to the review queue, or restricting the output of the main answer text.

2. The method according to claim 1, characterized in that, In step S1, obtaining the problem type signal from the problem text includes: Input the question text into an external question type classifier, which then outputs a single-answer or multi-answer question type signal; or... Input the question text into the type judgment model used to identify the answer structure, and the type judgment model outputs a question type signal of single answer type or multiple answer type; If the valid answers to a question are only aliases, abbreviations, spelling variations, or synonyms of the same answer, the question text will be marked as a single-answer type. If the valid answers to a question form multiple semantically different answer clusters, or if the question text has multiple valid interpretations and different interpretations correspond to different answers, the question text will be marked as a multi-answer type.

3. The method according to claim 1, characterized in that, In step S2, the main answer text generated by the large language model to be detected around the question text, as well as multiple sampled answer texts, are obtained, including: A low-temperature decoding method is used to enable the large language model to be tested to generate the main answer text based on the question text; High-temperature sampling and kernel sampling methods are used to generate K sampled answer texts based on the same question text, forming a sampled answer text set; When the question type signal is a multi-answer type, each sampling prompts the large language model to output a valid answer text, instead of requiring all valid answer texts to be listed at once.

4. The method according to claim 1, characterized in that, Step S3, which involves calculating the text validity of the sampled answer text set, includes: Input the question text and any sampled answer text into the answer validity validator, which is implemented using a large language model; For an open-source large language model, a validation prompt is constructed, requiring the model to judge only between Yes and No, where Yes indicates a valid candidate answer and No indicates an invalid candidate answer. The length-normalized logarithmic probabilities of the model outputs Yes and No are calculated separately, and the valid probability of the candidate answer is obtained through Softmax. Define the answer validity score: For closed-source large language models, the question text q and candidate answer texts are input via API. The model is required to return a confidence score. When the label is "Yes", the confidence level is directly used as the answer validity score. When the label is "No", insufficient confidence is used as the answer validity score. ; Calculate the risk of invalid answers based on the answer validity scores of K sampled answer texts in the sampled answer text set: The validity of the text is characterized by an answer validity score and an answer invalidity risk.

5. The method according to claim 4, characterized in that, In step S3, calculating the semantic relationships of the sampled answer text set and forming a text semantic cluster includes: Perform bidirectional natural language reasoning on any two sampled answer texts to obtain the semantic implications and contradiction probabilities of the texts in two directions; Based on the semantic implications of the text in two directions, the predicted categories in the two directions are determined. If the predicted categories in both directions are both implied, then the two sampled answer texts are semantically equivalent. Clustering semantically equivalent sampled answer texts yields one or more text semantic clusters.

6. The method according to claim 5, characterized in that, When the question text is interpreted as a single-answer type, the construction of the type feature vector in step S4 includes: Based on the answer validity scores of the sampled answer texts in each text semantic cluster, calculate the validity-weighted quality of each text semantic cluster: ,in, This indicates that the sampled answer falls into the c-th text semantic cluster. The validity weighting ratio, and satisfying: m is the number of semantic clusters in the text; The macroscopic dispersion is calculated based on the effectiveness-weighted quality of multiple text semantic clusters: And use macroscopic dispersion as the answer to diversify risk; The risk of micro-inconsistency is calculated based on the textual semantic consistency among sampled answer texts within the same textual semantic cluster. Textual semantic consistency is represented as: The risk of micro-inconsistency is: ,in For two sampled answer texts and The average semantic entailment probability between them; The risk of invalid answers, the risk of answer dispersion, and the risk of micro-inconsistency are used as type feature vectors under a single answer type: .

7. The method according to claim 6, characterized in that, When the question text is interpreted as a multiple-answer type, the construction of the type feature vector in step S4 includes: Calculate the risk of multiple invalid answers based on the answer validity scores of each sampled answer text in the sampled answer text set: ; Based on the probability of textual contradictions between different sampled answer texts, and combined with the answer validity scores of the corresponding sampled answer texts, the risk of cross-contradiction is calculated: ,in Represents the sampled answer text and The probability of textual contradictions between them; The risks of invalid multiple answers and cross-contradiction are used as type feature vectors under the multiple answer type: .

8. The method according to any one of claims 1 to 7, characterized in that, In step S5, calculating the hallucination risk score of the main answer text based on the type condition diagnostic features includes using at least one of the following methods: Method 1: Set up single-answer risk headers and multi-answer risk headers. The single-answer risk header is used to output the basic risk of a single answer based on the type condition diagnostic features under a single answer type. The multi-answer risk header is used to output the basic risk of multiple answers based on the type condition diagnostic features under multiple answer types. The basic risks of single answers and / or multiple answers are calibrated based on type stability. The illusion risk score of the main answer text is obtained based on the calibrated risks. Method 2: Convert the type condition diagnostic features into natural language feature summaries; input the question text, main answer text, natural language feature summaries, and few sample examples into the large language model evaluator; the large language model evaluator outputs the correct probability of the main answer text, and obtains the illusion risk score of the main answer text based on the correct probability.

9. A type-aware large language model illusion detection system, used to perform the method according to any one of claims 1 to 8, characterized in that, include: The question text acquisition module is used to receive the question text input by the user and obtain the question type signal of the question text. The question type signal includes single answer type and multiple answer type. The single answer type means that the valid answers of the question text correspond to the same semantic answer target, and the multiple answer type means that there are multiple semantically different valid answers or multiple valid interpretations in the question text. The answer text generation module is used to obtain the main answer text generated by the large language model to be detected around the question text and multiple sampled answer texts to form a sampled answer text set. The main answer text is the target output text to be detected, and the multiple sampled answer texts are used to represent the output distribution of the large language model to be detected under the same question text. The sampled answer text diagnosis module is used to calculate the text validity and text semantic relationship of the sampled answer text set, and obtain the answer validity score of each sampled answer text, the text semantic relationship between sampled answer texts, and the text semantic cluster formed based on the text semantic relationship. The type condition feature construction module is used to interpret the type conditions of answer validity scores, text semantic relations, and text semantic clusters based on the question type signal, and construct type feature vectors; calculate type stability based on the type label set, which is used to characterize the consistency of question type signals in multiple generation processes; and obtain type condition diagnostic features based on the type feature vectors and type stability; wherein, the type feature vector of a single answer type includes at least the answer dispersion risk obtained based on the text semantic cluster distribution, and the type feature vector of a multi-answer type includes at least the multi-answer invalidity risk and the cross-contradiction risk; The risk output control module is used to calculate the illusion risk score of the main answer text based on the type condition diagnostic features, and to perform output control processing on the main answer text based on the illusion risk score. The output control processing includes at least one of adding a risk label to the main answer text, sending the main answer text to the review queue, or restricting the output of the main answer text.

10. An electronic device, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the method of any one of claims 1 to 8.

Images