Model hallucination detection method, system, and storage medium

Abstract

The embodiment of the present specification provides a model hallucination detection method and system and a storage medium. In the method, attention information and hidden state information in the process of generating a response sequence of an inference model are obtained, the attention distribution of the answer content to the inference content in the response sequence is determined based on the attention information, the semantic similarity distribution between the answer content and the inference content in the response sequence is determined based on the hidden state information, and then the hallucination score of the inference model is determined based on the consistency degree of the attention distribution and the semantic similarity distribution.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence, and in particular to a method, system and storage medium for detecting model hallucinations. Background Technology

[0002] With the development of artificial intelligence technology, large language models have been widely applied to highly complex professional fields such as risk assessment, compliance logic deduction, and data predictive analysis. In these fields, users not only focus on the model's final answer but also need traceable and explainable reasoning to support relevant decisions. However, when handling highly complex tasks, large language models may experience a "reasoning illusion" phenomenon (i.e., the generated content appears logically consistent and clear in natural language expression, but contains factual errors in some details, such as arithmetic errors, logical contradictions, or inconsistencies with the input premises), affecting the reliability of user decisions.

[0003] The information in the background section is merely information known only to the inventor and does not imply that such information had entered the public domain before the date of this application, nor does it imply that it can be considered prior art in this disclosure. Summary of the Invention

[0004] This specification provides a model hallucination detection method, system, and storage medium that performs consistency verification on the attention distribution and semantic similarity between the model's reasoning content and the response content, which can effectively improve the accuracy of hallucination detection, thereby intercepting high-risk erroneous content and improving user experience.

[0005] In a first aspect, this specification provides a model hallucination detection method, comprising: obtaining attention information and hidden state information during the generation of a response sequence by an inference model, wherein the response sequence includes inference content and answer content for a target question; determining the attention distribution of the answer content to the inference content based on the attention information, and determining the semantic similarity distribution between the answer content and the inference content based on the hidden state information; and determining the hallucination score of the inference model based on the degree of consistency between the attention distribution and the semantic similarity distribution.

[0006] In some embodiments, the reasoning content includes multiple reasoning steps, the answer content includes at least one answering step, each answering step or reasoning step includes multiple lexical units, the attention information includes attention weights between lexical units, the hidden state information includes the hidden state of each lexical unit, and the step of determining the attention distribution of the answer content to the reasoning content based on the attention information, and determining the semantic similarity distribution between the answer content and the reasoning content based on the hidden state information, includes: aggregating the attention weights between lexical units at the step level to obtain a target attention weight matrix, determining the attention distribution of each answering step to each reasoning step based on the target attention weight matrix; and for each step, aggregating the hidden states of the lexical units belonging to the step to obtain the target hidden state of the step, determining the semantic similarity between each answering step and each reasoning step based on the target hidden state corresponding to each step, to obtain a step-level semantic similarity distribution.

[0007] In some embodiments, determining the attention distribution of each answer step to each inference step based on the target attention weight matrix includes: constructing an inference path for each answer step, the inference path representing a multi-hop dependency link tracing back from the answer step to the predecessor inference step; and correcting the target attention weight matrix based on the inference path of each answer step to obtain the attention distribution.

[0008] In some embodiments, the precursor reasoning step includes a direct precursor step and at least one indirect precursor step, and the correction of the target attention weight matrix based on the reasoning path of each answer step includes: for each answer step, determining the attention enhancement weight of the answer step for each indirect precursor step based on the attention weight between adjacent steps in its corresponding reasoning path; and correcting the attention weight corresponding to the answer step in the target attention weight matrix based on the attention enhancement weight corresponding to each answer step.

[0009] In some embodiments, determining the attention enhancement weight of the answer step for each indirect predecessor step based on the attention weights between adjacent steps in the corresponding inference path includes: for each indirect predecessor step, obtaining the attention enhancement weight of the answer step for the indirect predecessor step based on the attention weights of each pair of adjacent steps between the indirect predecessor step and the answer step; and correcting the attention weights corresponding to the answer step in the target attention weight matrix based on the attention enhancement weights corresponding to each answer step includes: for each indirect predecessor step, correcting the attention weights of the answer step for the indirect predecessor step in the target attention weight matrix based on the attention enhancement weights of the answer step for the indirect predecessor step.

[0010] In some embodiments, the reasoning path for each answering step includes: initializing a source step set; performing multiple iterative updates on the source step set to obtain the reasoning path for each answering step; in the first iteration, the source step set includes the at least one answering step; in each iteration: for each step in the source step set, determining a first preset number of candidate predecessor steps according to the attention weight from largest to smallest to obtain a candidate predecessor step set; determining the target predecessor steps in the candidate predecessor step set that meet preset conditions as the source step set for the next iteration; and determining the reasoning path for each answering step based on the source step set in each iteration.

[0011] In some embodiments, the target precursor step is obtained by: for each candidate precursor step in the candidate precursor step set, determining the cumulative attentional influence of the candidate precursor step based on the attentional weights of at least some steps in the source step set on the candidate precursor step; and selecting candidate precursor steps whose cumulative attentional influence is greater than or equal to a first preset threshold as the target precursor step.

[0012] In some embodiments, the inference model includes multiple attention heads, and the attention information includes a word-level attention matrix corresponding to each attention head. The step-by-step aggregation of the attention weights between words to obtain the target attention weight matrix includes: for each attention head, performing step-by-step aggregation of the attention weights between words in its corresponding word-level attention matrix to obtain an initial attention weight matrix corresponding to each attention head; and aggregating the initial attention weight matrices corresponding to at least one target attention head to obtain the target attention weight matrix.

[0013] In some embodiments, the method further includes: for each attention head, determining the information entropy of the attention head based on the initial attention weight matrix corresponding to the attention head; and selecting a second preset number of attention heads in ascending order of information entropy to obtain the at least one target attention head.

[0014] In some embodiments, the attention weights of step i to step j in the initial attention weight matrix are obtained as follows: for each word in step i, the attention weights of the word to each word in step j are aggregated by pooling operations to obtain the attention weights of the word to step j; and the attention weights of each word in step i to step j are aggregated by pooling operations to obtain the attention weights of step i to step j.

[0015] In some embodiments, determining the semantic similarity between each answer step and each inference step based on the target hidden state corresponding to each step includes: for each answer step, determining the semantic similarity based on the cosine similarity between the target hidden state of the answer step and the target hidden state of each inference step.

[0016] In some embodiments, determining the illusion score of the reasoning model based on the consistency between the attention distribution and the semantic similarity distribution includes: for each answer step, determining the reasoning steps with a semantic similarity greater than or equal to a second preset threshold to form a semantically related step set; aggregating the attention weights of the answer steps to each reasoning step in the semantically related step set to determine the illusion sub-score of the answer step; and determining the illusion score of the reasoning model based on the illusion sub-scores corresponding to each answer step.

[0017] In some embodiments, determining the illusion score of the reasoning model based on the illusion sub-scores corresponding to each answer step includes: selecting the maximum value from the illusion sub-scores corresponding to the at least one answer step as the illusion score of the reasoning model.

[0018] In some embodiments, the method further includes: determining that the reasoning model has reasoning illusions if the illusion score is greater than or equal to a third preset threshold; or determining that the reasoning model does not have reasoning illusions if the illusion score is less than the third preset threshold.

[0019] Secondly, this specification also provides a detection system, comprising: at least one storage medium storing at least one instruction set; and at least one processor communicatively connected to the at least one storage medium, wherein the at least one processor reads the at least one instruction set during operation and executes the method as described in the first aspect according to the instructions of the at least one instruction set.

[0020] Thirdly, this specification provides a computer-readable non-transitory storage medium, wherein the computer-readable non-transitory storage medium stores at least one instruction set, which, when executed by at least one processor, implements the method as described in the first aspect.

[0021] Other features of the model illusion detection methods, systems, and storage media provided in this specification will be partially listed in the following description. The inventive aspects of the model illusion detection methods, systems, and storage media provided in this specification can be fully understood through practice or by using the methods, systems, and combinations described in the detailed examples below. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic diagram illustrating an application scenario for model hallucination detection provided according to embodiments of this specification is shown.

[0024] Figure 2 A hardware structure diagram of a computing system provided according to an embodiment of this specification is shown;

[0025] Figure 3 A flowchart of a model hallucination detection method provided according to an embodiment of this specification is shown;

[0026] Figure 4 A schematic diagram of a response sequence generated by an inference model provided according to an embodiment of this specification is shown;

[0027] Figure 5 A detailed flowchart of hallucination detection provided according to embodiments of this specification is shown; and

[0028] Figure 6 A schematic diagram of a reasoning path provided according to an embodiment of this specification is shown. Detailed Implementation

[0029] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0030] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. Unless otherwise stated, the term “a plurality” means two or more, and “at least one” means one or more. Terms such as “first,” “second,” etc., may be used in this specification to describe various information, but such information should not be limited to these terms. These terms are used to distinguish information of the same type from one another and do not necessarily imply a specific order or sequence. For example, “first” may also be referred to as “second” without departing from the scope of embodiments described herein, and similarly, “second” may also be referred to as “first.”

[0031] The term "at least one of A, B, or C" includes seven cases: A only, B only, C only, both A and B, both A and C, both B and C, and both A, B, and C. Similarly, the statement "at least one of multiple items" refers to all possible combinations based on these items. The term "and / or" refers to any or all possible combinations of one or more related listed items. For example, "A and / or B" includes three cases: A only, B only, and both A and B. "A, B, and / or C" is equivalent to "at least one of A, B, or C". The character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0032] The term "comprising" is an open-ended description and should be understood as "including but not limited to," potentially including other content beyond what has been described. When used in this specification, the terms "comprising," "including," and / or "containing" mean the presence of the associated integers, steps, operations, elements, and / or components, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or the possibility of adding other features, integers, steps, operations, elements, components, and / or groups to the system / method.

[0033] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0034] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0035] There are currently several methods for detecting hallucinations in large language models. Some of them are shown in the following examples.

[0036] In some embodiments, the service system that invokes the large language model can determine the authenticity of the model-generated content through external knowledge base retrieval and programmatic tool calls (such as calculators and database queries). This method has a high computational cost, requiring multiple additional calls to external knowledge bases or tools for each detection, resulting in a significant increase in response time and failing to meet the needs of low-latency scenarios such as high-frequency trading analysis or real-time compliance checks. Furthermore, this method faces challenges in domain adaptation. Since intelligent assistant data is mostly private, unstructured, and dynamically updated, and public knowledge base coverage is insufficient, the toolchain construction for detection is costly and difficult to comprehensively cover.

[0037] In some embodiments, a service system that invokes a large language model can instruct the model to generate multiple independent inference trajectories and detect model illusions by evaluating the semantic consistency or logical conflicts among these trajectories. This approach is inefficient and inaccurate; to obtain stable detection results, the service system typically requires the model to generate a large number of inference trajectories, resulting in a linear increase in computational cost, which is unacceptable in batch concurrent processing scenarios. Furthermore, this approach is ineffective against systematic biases; if the model exhibits the same errors across a certain type of problem due to training data bias, multiple inference trajectories will only repeat the same illusion, leading to incorrect answers being misjudged as "credible."

[0038] In some embodiments, a service system that invokes a large language model can detect illusions by analyzing the probability distribution of the model's output layer or the inherent uncertainty of its hidden states. For example, the service system uses the predicted probability of the generated lexical units, word-by-word entropy, or sequence perplexity as confidence metrics. However, large language models, under reinforcement learning or procedural supervised training, tend to generate highly confident and fluent reasoning chains. Even if errors occur in intermediate steps, subsequent steps still make seemingly reasonable inferences based on these errors, leading to high probabilities for local lexical units and rendering the detection scheme ineffective.

[0039] In some embodiments, a service system that invokes a large language model can detect illusions by tracking the divergence of hidden states, changes in inference depth, or the geometry of potential trajectories. However, the core assumption of this approach is limited to the dynamics within a single stage and cannot quantify whether attention between the inference and response stages is hijacked by semantic representations, leading to a disconnect between the detection signal and the true root cause of the error.

[0040] In view of this, this specification provides a method, system, and storage medium for detecting model hallucinations to overcome the above-mentioned problems.

[0041] Figure 1 A schematic diagram illustrating an application scenario for model illusion detection provided according to embodiments of this specification is shown. For example... Figure 1 As shown, scenario 100 includes a service system 10, an inference model, and a detection system 20.

[0042] Target users can initiate specific operation requests to service system 10 (such as intelligent assistants) through their terminal devices. These specific operations include, but are not limited to: submitting risk assessment questions or data prediction analysis in financial consulting scenarios, and submitting code generation instructions in code development scenarios. In response to the specific operation, service system 10 invokes the inference model to execute the corresponding inference task.

[0043] The reasoning model in this specification refers to a pre-trained generative model with complex logical deduction capabilities. It can include general-purpose large language models as well as specialized large language models that are optimized or fine-tuned for multi-step reasoning tasks (such as mathematical problem solving, code generation, or scientific question answering).

[0044] In some embodiments, the inference model can be deployed locally on the server of service system 10 to ensure data privacy and security; alternatively, the inference model can be remotely deployed on a cloud server and invoked by service system 10 through an API interface. Service system 10 can embed an illusion detection mechanism at the end of the inference process of the inference model to effectively intercept high-risk erroneous answers generated by the model before presenting the results to the user. The detection system 20, which executes the illusion detection process, is also not restricted in its deployment location; it can be integrated into the local server of service system 10 or deployed independently on a cloud server.

[0045] During the inference task, the inference model can generate a response sequence containing intermediate inference content (generated during the inference phase) and the final answer content (generated during the answer phase). The detection system 20 obtains the response sequence and relevant internal data (such as attention weight matrix, hidden states, etc.) generated by the large language model during the response sequence generation process, and then analyzes whether the response sequence contains hallucinations based on the relevant internal data, and outputs the hallucination detection result.

[0046] After receiving the hallucination detection result from the detection system 20, the service system 10 determines whether to implement an intervention strategy based on the result. If the hallucination detection result indicates that the inference model exhibits inference hallucinations, the service system 10 may intercept the response sequence and not display it to the user, or trigger a retry mechanism to regenerate the response sequence, or return a risk warning message to the user. If the hallucination detection result indicates that the inference model does not exhibit inference hallucinations, the service system 10 may display the response sequence to the user.

[0047] In the embodiments of this specification, the Large Language Model (LLM) may also be simply referred to as the Large Model. A Large Language Model is a natural language processing model based on deep learning technology. It has a high number of parameters, typically ranging from billions to hundreds of billions or even higher, and possesses powerful language understanding and generation capabilities. Large Language Models can employ the Transformer architecture or its variants (such as GPT, BERT, etc.). This architecture utilizes an attention mechanism to achieve global modeling of sequential data, efficiently handling long-distance dependencies and thus performing excellently in natural language tasks. Large Language Models can be used to perform various tasks such as text generation, language translation, text summarization, and question answering reasoning.

[0048] In some embodiments, large language models can also have the ability to understand and generate data from other modalities (such as visual, audio, etc.). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound.

[0049] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v18, published on March 8, 2026), and will not be repeated here.

[0050] The detection system 20 is a device with certain data processing capabilities. The detection system 20 can execute the model illusion detection method described in this specification. For example, the detection system 20 stores data and instructions for implementing the model illusion detection method, and can execute or be used to execute said data and instructions. In some embodiments, the detection system 20 may include hardware devices with data information processing capabilities and the necessary programs required to drive the hardware devices to operate.

[0051] Figure 2A hardware structure diagram of a computing system 200 provided according to an embodiment of this specification is shown. The computing system 200 may be... Figure 1 The detection system 20 in the middle.

[0052] like Figure 2 As shown, the computing system 200 may include at least one storage medium 230 and at least one processor 220. In some embodiments, the computing system 200 may also include a communication port 250 and an internal communication bus 210. Furthermore, the computing system 200 may also include I / O components 260.

[0053] The internal communication bus 210 can connect to different system components. For example, the internal communication bus 210 can connect to storage medium 230, processor 220, communication port 250, and I / O component 260.

[0054] I / O component 260 supports input / output between computing system 200 and other components.

[0055] Communication port 250 is used for data communication between computing system 200 and the outside world. For example, communication port 250 can be used for data communication between computing system 200 and a network. Communication port 250 can be a wired communication port or a wireless communication port.

[0056] In some embodiments, the network can be any type of wired or wireless network, or a combination thereof. For example, the network may include a cable network, a wired network, a fiber optic network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network™, a ZigBee™ short-range wireless network, a near field communication (NFC) network, or a similar network.

[0057] In some embodiments, the network may include one or more network access points. For example, the network may include wired or wireless network access points, such as base stations or internet switching points. Through these access points, one or more components of various devices corresponding to the computing system 200 can connect to the network to exchange data or information.

[0058] Storage medium 230 may include a data storage device. The data storage device may be a non-transitory storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 232, a read-only storage medium (ROM) 234, or a random access storage medium (RAM) 236. Storage medium 230 also includes at least one instruction set stored in the data storage device. The instruction set may include computer program code, which may include programs, routines, objects, components, data structures, procedures, modules, etc., that execute the model illusion detection method provided in this specification.

[0059] Processor 220 can be communicatively connected to storage medium 230. Processor 220 is used to execute at least one of the above-described instruction sets. When computing system 200 is running, processor 220 reads the at least one instruction set and executes the model illusion detection method provided in this specification according to the instructions of the at least one instruction set.

[0060] Processor 220 may be in the form of one or more processors. In some embodiments, processor 220 may include one or more hardware processors, such as microcontrollers, microprocessors, reduced instruction set computers (RISC), application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), central processing units (CPUs), graphics processing units (GPUs), physical processing units (PPUs), microcontroller units, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), advanced RISC machines (ARMs), programmable logic devices (PLDs), any circuit or processor capable of performing one or more functions, or any combination thereof.

[0061] For the purpose of illustrating the point only, in the appendix Figure 2 Only one processor 220 is shown in the computing system 200. However, it should be noted that the computing system 200 may also include multiple processors. Therefore, the operation and / or method steps disclosed in this specification may be executed by one processor as described in this specification, or they may be executed jointly by multiple processors. For example, if processor 220 of computing system 200 in this specification executes steps A and B, it should be understood that steps A and B may also be executed jointly or separately by two different processors 220 (e.g., the first processor executes step A, the second processor executes step B, or the first and second processors jointly execute steps A and B).

[0062] Figure 3 A flowchart of a model hallucination detection method P300 according to an embodiment of this specification is shown. The detection system can execute the model hallucination detection method P300.

[0063] like Figure 3 As shown, the P300 model hallucination detection method includes the following steps.

[0064] S310: Obtain attention information and hidden state information during the process of the inference model generating a response sequence, which includes the inference content and the answer content for the target question.

[0065] The target question generally refers to the prompt content input into the inference model to trigger the generation of a response sequence. The target question can be either the original query entered by the user on the interactive interface, or a complete input text formed by the service system in response to the user's input, encapsulating the user's input along with preset instructions, background information, or format constraints within a prompt template. This specification does not limit the specific form or construction method of the target question.

[0066] A response sequence is an ordered data stream formed by an inference model predicting and outputting tokens based on an input target question using relevant mechanisms (such as autoregressive generation mechanisms). Logically, a response sequence includes reasoning content that demonstrates the derivation process and response content that provides the final conclusion.

[0067] Reasoning content refers to the part of the response sequence that precedes the answer content. It represents the intermediate thinking process performed by the reasoning model to solve the target problem, including but not limited to: chain-of-thought, logical deduction steps, numerical or symbolic calculation process, hypothesis testing analysis, and integration of background information based on internal knowledge or external retrieval.

[0068] The response content refers to the final conclusion, calculation result, or core summary given directly in response to the target question in the response sequence. This part is usually located after the reasoning content and constitutes a direct response to the user's question. It is the key semantic unit in the response sequence that carries the final decision or answer.

[0069] Taking a question-and-answer interaction scenario as an example, the target question, the reasoning content, and the answer content are as follows: Figure 4 As shown.

[0070] This manual does not impose any restrictions on the timing of the detection system's detection. For example, the detection system can acquire a portion of the generated response sequence in real time for parallel detection while the inference model is generating the response sequence, thereby improving detection efficiency and reducing the feedback delay in displaying results to the user. Alternatively, the detection system can acquire the complete response sequence all at once for detection after the inference model has completed all generation tasks.

[0071] In some embodiments, the detection system can divide the reasoning content into multiple logical units to form a sequence of reasoning steps (e.g., And divide the answer content into multiple logical units to form a sequence of answer steps (such as...) Each reasoning step and each response step contains multiple tokens, possessing complete semantics. Furthermore, the detection system can also divide the reasoning content into steps while treating the response content as a single whole; this specification does not restrict whether to perform fine-grained division of the response content. The number of tokens in each step can differ, and this specification does not impose any restrictions on this.

[0072] Taking reasoning content as an example, the detection system can divide it into multiple reasoning steps based on semantic integrity, such as segmentation according to paragraphs, sentences, or semantic logic breakpoints. This specification does not limit the specific granularity or method of segmentation. For the segmentation of answer content, the same or similar strategies as those for reasoning content can be adopted, which will not be elaborated here.

[0073] Attention information refers to the data generated by the internal attention mechanism of the inference model during the generation of response sequences, calculating the interaction relationships between queries, keys, and values. This information can be represented as a multi-dimensional attention weight matrix. The attention weight of term i to term j in the matrix characterizes the degree of attention or dependence of the inference model on term j when generating the hidden state of term i. This attention information reflects the actual citation of preceding inference content by the inference model when generating the answer content, and is key structured data for capturing logical dependencies between terms and analyzing the model's internal information routing mechanism. This attention information can cover the attention weights between terms within the response sequence, or further, the attention weights between terms in the response sequence and terms in the input content (i.e., the target question). This specification does not limit the specific scope of term interactions covered by the attention information.

[0074] Hidden state information refers to the high-dimensional vectors output by each layer of the neural network in the inference model during the generation of the response sequence. These vectors represent the semantic features of each word. This information integrates the global dependencies of words in a specific context and serves as the fundamental representation for the inference model to perform deep semantic understanding and logical reasoning. Hidden state information can be taken from the output of the last hidden layer of the inference model or from the output of other intermediate layers. This specification does not limit the specific source level of the hidden state information. The hidden state information can include the hidden states of each word in the response sequence, or it can further include the hidden states of each word in the input content (i.e., the target question). This specification does not limit the specific range of words covered by the hidden state information.

[0075] During the process of generating response sequences by the inference model, the aforementioned attention information and hidden state information can be stored in a relevant database, so that the detection system can obtain the aforementioned attention information and hidden state information from the relevant database.

[0076] S320: Based on this attention information, determine the attention distribution of the answer content to the reasoning content.

[0077] An attention distribution (also called an attention map) represents the attention weight of each semantic unit in the response content to each semantic unit in the reasoning content. The semantic units mentioned above can be words or steps, and this specification does not impose any restrictions on this. The attention distribution can be word-level or step-level, and can be represented as a matrix, and this specification does not impose any restrictions on this. A word-level attention distribution can include the attention weight of each word in the response content to each word in the reasoning content. A step-level attention distribution can include the attention weight of each response step in the response content to each reasoning step in the reasoning content.

[0078] Attention distribution can quantify the cross-stage attention or dependence of semantic units in the reasoning content on various semantic units during the generation process of the answer content. This attention distribution can be calculated directly based on the original attention information, or it can be the result of enhancing, correcting or aggregating the original attention information; this specification does not impose any restrictions on this.

[0079] When the attention and hidden state information obtained by the detection system from the inference model are word-level data, hallucination detection can be performed directly based on the fine-grained word-level data. Alternatively, the word-level data can be aggregated into step-level data first, and then hallucination detection can be performed based on the aggregated step-level data. This specification does not impose any restrictions on the data granularity used for hallucination detection. The following text mainly uses the example of hallucination detection after aggregating into step-level data. The method of directly performing hallucination detection using word-level data is similar and will not be elaborated further.

[0080] In some embodiments, the reasoning content includes multiple reasoning steps, and the answer content includes at least one answering step. Each answering step or reasoning step includes multiple lexical units. Attention information includes attention weights between lexical units. Hidden state information includes the hidden state of each lexical unit. The detection system aggregates the attention weights between lexical units at the step level to obtain a target attention weight matrix, and determines the attention distribution of each answering step on each reasoning step (i.e., the step-level attention distribution) based on the target attention weight matrix.

[0081] After the detection system aggregates the attention weights between each word at the step level, it can obtain an initial attention weight matrix. The initial attention weight matrix includes multiple elements, each element representing the attention weight between steps. As an example, the attention weight of step i to step j in the matrix represents the degree of attention or dependence of step i on step j.

[0082] The target attention weight matrix can be the initial attention weight matrix, or it can be the result obtained after further processing (such as filtering, weighting, or correction) of the initial attention weight matrix. This specification does not impose any restrictions on this.

[0083] For example, when the inference model contains only a single attention head, the target attention weight matrix can be equivalent to the initial attention weight matrix. However, when the inference model contains multiple attention heads, the detection system can first filter the multiple attention heads, and then further process the initial attention weight matrix based on the filtered attention heads to determine the target attention weight matrix.

[0084] In the above embodiments, the detection system reduces the data processing dimensionality and computational complexity of hallucination detection by aggregating fine-grained attention weights at the lexical level into coarse-grained attention weights at the step level. This allows the detection process to focus on units with complete semantics rather than isolated words, avoiding noise in hallucination detection at the lexical level. By detecting the consistency between the attention distribution and semantic similarity distribution at the step level, the detection system can improve the efficiency and accuracy of hallucination detection and reduce misjudgments caused by local lexical noise.

[0085] In some embodiments, the inference model includes multiple attention heads. For example... Figure 5 As shown, the attention information includes a word-level attention matrix corresponding to each attention head. The word-level attention matrices for each attention head can have the same dimension, and elements at the same position represent the attention weights between the same word pairs. For each attention head, the detection system performs step-by-step aggregation of the attention weights between words in its corresponding word-level attention matrix to obtain the initial attention weight matrix for each attention head.

[0086] The detection system can determine the mapping relationship between each term in the response sequence and its corresponding logical step. For example, the detection system defines a set S. r1 Let set S be the set of indices of all words belonging to the first reasoning step. a1 This is the set of indices for all terms belonging to the first response step, and so on.

[0087] The detection system can obtain a single-head attention matrix, that is, for each attention head h in the inference model, extract its corresponding word-level attention matrix W from the attention information. (h) .in, This represents the attention weight of the inference model on word j when generating word i under attention head h.

[0088] The following section uses attention head h as an example to introduce how to determine its corresponding initial attention weight matrix. The processing method for other attention heads is similar and will not be repeated here.

[0089] In some embodiments, the attention weights of step i to step j in the initial attention weight matrix corresponding to attention head h are obtained in the following manner: for each word in step i, the detection system aggregates the attention weights of that word to each word in step j through pooling operation to obtain the attention weight of that word to step j, and then aggregates the attention weights of each word in step i to step j through pooling operation to obtain the attention weights of step i to step j.

[0090] For example, the attention weight of step i to step j can be calculated based on the following formula:

[0091]

[0092] in, This represents the attention weight of step i to step j in the h-th attention head of layer l; Represents the set of lexical units in the i-th step; Represents the set of lexical units in step j; express Any word element in; express Any word element in; This represents the attention weight of word i' to word j' in the h-th attention head of layer l; This indicates a pooling operation.

[0093] The detection system performs two nested pooling operations: First, an inner pooling operation is performed. For each specific word in step i, all words in step j are traversed, and the original attention weights of that specific word to all words in step j are aggregated to obtain the overall attention weight of that specific word to step j. Then, an outer pooling operation is performed. It traverses all words in step i and aggregates the overall attention weights of each word in step i to step j obtained from the inner pooling operation, finally yielding the step-level attention weights of step i to step j.

[0094] In the above embodiments, the detection system achieves a smooth transition from micro-level word-based attention weights to macro-level step-based attention weights through two pooling operations. This hierarchical pooling operation preserves the contribution information of all words within a step, avoids information loss that may result from selecting representative words, and ensures that the step-level attention weights can comprehensively and evenly reflect the overall interaction strength between the two steps, laying the foundation for accurate hallucination detection.

[0095] See also Figure 5 After obtaining the initial attention weight matrix corresponding to each attention head in the manner described above, the detection system aggregates the initial attention weight matrices corresponding to at least one target attention head to obtain the target attention weight matrix. This at least one target attention head can be any of the attention heads in the inference model, or it can be a selected subset of attention heads; this specification does not impose any restrictions on this.

[0096] The detection system can reduce computational overhead and improve detection efficiency by selecting the initial attention weight matrix of a portion of the attention heads for aggregation, while selecting all attention heads for aggregation can ensure the comprehensiveness of the aggregation results.

[0097] For example, for each attention head, the detection system can determine the information entropy of that attention head based on the initial attention weight matrix corresponding to that attention head. Taking attention head h as an example, its information entropy can be a statistical indicator that quantifies the uncertainty or dispersion of the initial attention weight matrix corresponding to attention head h. Its value reflects whether the inference model's focus is concentrated when generating specific content. Lower information entropy indicates that attention is highly concentrated on a few key preceding steps, meaning that the inference model has a clear logical direction and strong dependencies. Higher information entropy, on the other hand, indicates that attention is evenly distributed across multiple preceding steps, suggesting that the inference model lacks a clear focus or has semantic ambiguity. Therefore, information entropy can serve as an important auxiliary feature for evaluating the certainty of the inference process and detecting potential illusions.

[0098] The detection system can determine the information entropy by the step-level distribution of attention weights in the initial attention weight matrix corresponding to the attention head h. A sharp attention weight distribution indicates low information entropy. A flat attention weight distribution indicates high information entropy, suggesting that the inference model lacks focus. For example, the information entropy of each attention head can be calculated using the following formula:

[0099]

[0100] Where H represents information entropy, This represents the initial attention weight matrix of the attention head.

[0101] The detection system can select k1 (i.e., the second preset number) attention heads in ascending order of information entropy to obtain at least one target attention head. Alternatively, the detection system can select at least one target attention head with information entropy less than a preset threshold; this specification does not impose any restrictions on this.

[0102] In this example, the detection system calculates information entropy and selects low-entropy target attention heads, filtering out attention heads with concentrated attention distribution and clear directionality. This effectively eliminates ambiguous attention signals, improves the signal-to-noise ratio of step-level attention weights, and makes hallucination detection more accurate.

[0103] After obtaining at least one target attention head, the detection system can aggregate the initial attention weight matrices corresponding to at least one target attention head to obtain the target attention weight matrix.

[0104] For example, when there are multiple target attention heads, the detection system determines the target attention weight matrix by performing aggregation operations on the initial attention weight matrices corresponding to each target attention head. Specifically, each element in the target attention weight matrix is ​​obtained by processing (such as weighted summation, pooling, or taking the maximum value) the elements in the same position of the initial attention weight matrices corresponding to each target attention head, or it can be obtained by concatenating the initial attention weight matrices corresponding to each target attention head. This specification does not limit the specific aggregation algorithm.

[0105] In the above scheme, the detection system first independently aggregates each attention head and then filters and merges the target heads, utilizing the specificity of different attention heads in capturing different semantic or syntactic relationships. This method avoids the signal dilution or noise superposition that may result from simply averaging the information from all attention heads. By selectively aggregating, it retains those attention patterns that are more discriminative in representing the relationships between steps, thus obtaining a target attention weight matrix that better reflects the real logical dependencies and improving the accuracy of hallucination detection.

[0106] In some embodiments, after aggregating the initial attention weight matrices corresponding to at least one target attention head, the detection system can further post-process the aggregation result to obtain a target attention weight matrix. This post-processing includes, but is not limited to, positional bias adjustment and normalization. Positional bias adjustment refers to the detection system introducing learnable or fixed bias values ​​based on the relative distances of terms or steps in the response sequence to compensate for attention decay caused by differences in sequence length or to enhance dependencies at specific positions. Normalization refers to the detection system rescaling the adjusted attention weights (e.g., using Softmax or L1 / L2 normalization) to satisfy probability distribution constraints or eliminate dimensional influences, thereby ensuring the comparability and numerical stability of attention weights between different steps. The final target attention weight matrix is, for example... ,in, This represents the sum of the number of reasoning steps and the number of response steps.

[0107] In some embodiments, see continue to see Figure 5 After obtaining the target attention weight matrix, the detection system can construct the inference path for each answer step. Then, based on the inference path of each answer step, the target attention weight matrix is ​​corrected to obtain the attention distribution of each answer step to each inference step. The inference path represents a multi-hop dependency link tracing backward from the answer step to the preceding inference step.

[0108] The detection system can construct the reasoning path for each answer step through a backtracking mechanism, or it can construct the reasoning path in other ways; this specification does not impose any restrictions on this. For example, the detection system can construct a step dependency graph based on the attention weights between steps, and use the message passing mechanism in a graph neural network to perform multiple rounds of information propagation on this step dependency graph, thereby determining the reasoning path for each answer step.

[0109] The backtracking mechanism refers to tracing backwards from each answer step to extract the key reasoning steps associated with that answer step, thereby constructing a reasoning path (or logical chain) from multiple key reasoning steps pointing to that answer step. Preceding reasoning steps can refer to steps that precede the answer step in time and / or logic, and are the premise or basis for the answer step.

[0110] In this embodiment, the detection system overcomes the limitations of traditional attention mechanisms that only focus on directly adjacent or locally related steps by introducing a backtracking mechanism to construct multi-hop dependency links and correct the target attention weight matrix, thus improving the accuracy of hallucination detection for long inference chains. Furthermore, the detection system can capture non-local long-distance inference dependencies, accurately locating the problem of long-distance error propagation (or non-local error propagation) caused by errors in early inference steps leading to deviations in subsequent results. This ensures that key evidence located early in the inference chain can be effectively traced and weighted, preventing the omission of key premises due to interference from intermediate steps. This method, which enhances the ability to capture implicit dependencies in complex multi-step inference without increasing additional model training costs, effectively improves the detection accuracy of hidden logical gaps and the root causes of hallucinations.

[0111] For example, the reasoning content includes ordered reasoning steps r1~r22, and the answer content includes answering steps a1~a5. Figure 6 Some steps unrelated to demonstrating the reasoning path have been omitted. Each answer step can have multiple reasoning paths, and this manual does not limit this. Taking answer step a4 as an example, its reasoning path includes: reasoning step r2 → reasoning step r10 → answer step a4; reasoning step r4 → reasoning step r16 → answer step a4. It should be noted that the reasoning path can also include the target question, and this manual does not limit this.

[0112] In some embodiments, the detection system initializes a source step set and performs multiple iterative updates on the source step set to obtain the inference path for each answer step. In the first iteration, the source step set includes all answer steps in the answer content. In each iteration: for each step in the source step set, the detection system determines a first preset number of candidate predecessor steps in descending order of attention weight, obtaining a candidate predecessor step set; then, the target predecessor steps in the candidate predecessor step set that meet preset conditions are determined as the source step set for the next iteration. After the detection system completes iteration, the inference path for each answer step can be determined based on the source step set obtained in each iteration.

[0113] In this embodiment, the detection system initializes the source step set and performs multiple iterative updates, sorting candidate predecessor steps according to attention weights to achieve reverse layer-by-layer tracing from result to cause. This iterative construction method can dynamically explore salient reasoning paths, avoiding blind traversal of all possible connections and effectively pruning low-weight irrelevant branches. Thus, while ensuring that key reasoning steps are not missed, it reduces the computational overhead and time complexity of constructing reasoning paths.

[0114] For example, the set of source steps obtained during the initialization of the detection system is as follows: , This includes every step in the response content. During multiple iterations, the detection system needs to determine which inference steps each response step in the source step set depends on, and which preceding steps these inference steps depend on, thus forming a complete dependency chain. In each iteration, the detection system backtracks from each step in the source step set to its directly dependent preceding steps. The detection system examines the attention weights of each step on all other steps using the target attention weight matrix.

[0115] See also Figure 6 Taking response step a4 in the source step set during the first iteration as an example, the detection system determines the attention weight of response step a4 to each reasoning step in reasoning steps r1~r22, and selects the first k2 (i.e., the first preset number) reasoning steps with the largest attention weights in descending order to obtain candidate precursor steps. In a similar way, the detection system can obtain the candidate precursor steps corresponding to each response step, thus obtaining a set of candidate precursor steps.

[0116] Suppose that the source step set in the first iteration is {response step a1, ..., response step a5}, where the top k2 reasoning steps with high attention to response step a4 include reasoning steps r4, r10, and r16. The top k2 reasoning steps with high attention to response step a5 include reasoning steps r10, r16, and r22. Then the detection system can determine the candidate predecessor step set as the deduplicated {reasoning step a4, reasoning step r10, reasoning step r16, reasoning step r22}.

[0117] In some embodiments, during each iteration, for each candidate predecessor step in the candidate predecessor step set, the detection system determines the cumulative attention influence of the candidate predecessor step based on the attention weights of at least some steps in the source step set on the candidate predecessor step, and selects the candidate predecessor steps whose cumulative attention influence is greater than or equal to a first preset threshold as the target predecessor step.

[0118] For example, in the first iteration, if the detection system determines that inference steps r10 and r16 are both followed by response steps a4 and a5, then the cumulative attentional influence θ(X) of inference step r10 can be a weighted sum of the attention weight of response step a4 on it (which can be obtained from the target attention weight matrix, for example, 0.18) and the attention weight of response step a5 on it (which can be obtained from the target attention weight matrix, for example, 0.23). For example, θ(X) = 0.18 + 0.23 = 0.41. The cumulative attentional influence of inference step r16 is calculated in the same way. For inference step r4, which is only followed by response step a4, its cumulative attentional influence is the attention weight of response step a4 on inference step r4 in the target attention weight matrix, for example, 0.2. The cumulative attentional influence of inference step r22 is calculated in the same way. If the first preset threshold is 0.3, then the detection system will use candidate predecessor steps with a cumulative attentional influence greater than or equal to 0.3 as target predecessor steps (e.g., inference steps r10 and r16). After the first iteration, the source step set is updated to ={Inference step r10, Inference step r16}. The second iteration checks the source step set. The same processing steps are performed as described above, and will not be repeated here. The reasoning path for each answer step can be obtained by continuing until the detection system reaches its maximum iteration count. (Where i is the index of the answer step, and p represents the iteration round,) (Indicates the maximum number of iterations).

[0119] The detection system can obtain the reasoning path for each answer step based on the backtracking relationship between the source step sets in different iterations. Assume the source step set for the first iteration... For {answering step a4, answering step a5}, the set of source steps for the second iteration For {reasoning step r10, reasoning step r16}, the set of source steps for the third iteration For {reasoning step r2, reasoning step r4}, the iteration ends here. See also... Figure 6 , Reasoning steps r10 and r16 are obtained by backtracking from answer step a4. In the middle reasoning step, r2 is obtained by backtracking backward from reasoning step r10, and inference step r4 is obtained by backtracking backward from reasoning step r16. Therefore, the detection system can determine the reasoning path for answering step a4 as: reasoning step r2 → reasoning step r10 → answering step a4; reasoning step r4 → reasoning step r16 → answering step a4. The detection system determines the reasoning path for answering step a5 in a similar way, which will not be described further.

[0120] Each answer step can have one or more reasoning paths. The maximum number of iterations can be a preset value, which can be less than or equal to the number of reasoning steps; this specification does not impose any restrictions on this.

[0121] In the above embodiments, the detection system filters candidate predecessor steps by calculating their cumulative attention impact and setting a threshold, enabling it to identify key inference steps that are commonly depended upon by multiple steps in the source step set. This method utilizes the idea of ​​"collective verification," where predecessor steps that are frequently referenced by multiple source steps often carry core logical information. By retaining these steps with high cumulative impact as source nodes for the next round, it effectively filters out weak connections caused by chance or noise, improving the logical tightness and reliability of the constructed inference path.

[0122] In some embodiments, the preceding reasoning steps in the reasoning path corresponding to each answer step include direct preceding steps (steps in the reasoning path that are directly adjacent to the answer step) and at least one indirect preceding step (steps in the reasoning path that are indirectly adjacent to the answer step).

[0123] For example, see [link to previous article] Figure 6 Reasoning step r16 is the direct predecessor of answering step a4, and reasoning step r4 is the indirect predecessor of answering step a4.

[0124] After obtaining the inference path for each answer step, the detection system can determine the attention enhancement weight of each answer step for each indirect predecessor step based on the attention weight between adjacent steps in its corresponding inference path, and correct the attention weight corresponding to the answer step in the target attention weight matrix based on the attention enhancement weight corresponding to each answer step.

[0125] Attention enhancement weights can be understood as quantifying the long-distance dependencies of the answer step on the indirect predecessor step in the reasoning path. This signal aims to capture and amplify long-distance logical connections. By feeding this indirect dependency information back to the target attention weight matrix, the attention weights are compensated and corrected, thereby strengthening the model's global perception of the reasoning path and ensuring that the corrected attention weights truly reflect the causality of the logical deduction.

[0126] In some embodiments, for each indirect precursor step in the reasoning path of each answer step, the detection system obtains the attention enhancement weight of the answer step on the indirect precursor step based on the attention weight of each adjacent step pair between the indirect precursor step and the answer step.

[0127] The detection system can perform weighted calculations on the attention weights of each adjacent step pair between the indirect predecessor step and the response step in the reasoning path to obtain the attention enhancement weight of the response step on that indirect predecessor step. This specification does not impose any restrictions on this.

[0128] For example, the attention enhancement weight of the response step to the indirect precursor step can be calculated using the following formula:

[0129]

[0130] in, This represents the attention enhancement weight for step k in the p-th iteration (or p-th hop). This represents the attention weight of step i on step j. This represents the attention weight of step j on step k; It is a set judgment condition, indicating that in the p-th jump reasoning sub-path of answering step i, step k is the direct predecessor of step j, and step j is the direct predecessor of answering step i.

[0131] Taking step i as an example, assuming it has a complete reasoning path: i→j→k→l→m→n, then the reasoning sub-path determined by the detection system in the p-th hop (i.e., the p-th iteration) covers the initial step i to the p-th predecessor step. Specifically, when p=2, the p-th hop reasoning sub-path is i→j→k (i.e., backtracking two steps to reach step k); when p=3, the p-th hop reasoning sub-path is i→j→k→l (i.e., backtracking three steps to reach step l), and so on.

[0132] In this specification, "hop count" can be understood as the number of iterations dependent on the backtracking mechanism, that is, the number of adjacent step pairs (or edges) traversed between the answer step and the target indirect predecessor step (the terminal step in the reasoning sub-path, such as step k in the second hop reasoning sub-path and step l in the third hop reasoning sub-path).

[0133] See also Figure 6 Taking the reasoning path for answering step a4 as an example: reasoning step r4 → reasoning step r16 → answering step a4, there are two adjacent step pairs between answering step a4 and reasoning step r4 (i.e., <reasoning step r4 and reasoning step r16>, <reasoning step r16 and answering step a4>). The attention enhancement weight of answering step a4 on reasoning step r4... The attention weights of step a4 to inference step r16 can be answered by the target attention weight matrix (e.g., ) and the attention weights of inference step r16 to inference step r4 (e.g. ) Determine. For example, .

[0134] It should be noted that the attention enhancement weights in this specification can also be called multi-hop attention weights, and this specification does not limit them to that term. For example, if there are two adjacent step pairs between the answering step a4 and the reasoning step r4, the attention enhancement weights calculated by the detection system... This can be referred to as 2-hop attention weights. If there are three adjacent step pairs between answer step a4 and reasoning step r4 in other reasoning paths, the attention enhancement weights calculated by the detection system... This can be called a 3-hop attention weight, and so on.

[0135] Suppose that response step a4 also includes the following reasoning path: reasoning step r4 → reasoning step r10 → reasoning step r16 → response step a4. In this case, the attention enhancement weight between response step a4 and reasoning step r4 can be calculated using the following formula:

[0136] .

[0137] in, Let r16 be the attention weight of inference step r10 in the target attention weight matrix. The attention weights of inference step r10 to inference step r4 in the target attention weight matrix.

[0138] If there are more intermediate steps between the answering step a4 and the reasoning step r4, the detection system can calculate the attention enhancement weights in the same way, which will not be elaborated further.

[0139] Using the above method, the detection system can obtain the attention enhancement weights of each answer step on each indirect predecessor step in its reasoning path. Taking the indirect predecessor step k in the reasoning path corresponding to answer step i as an example, the detection system corrects the attention weights of answer step i on indirect predecessor step j in the target attention weight matrix based on the attention enhancement weights of answer step i on indirect predecessor step k. The detection system can correct the attention weights corresponding to each answer step in the target attention weight matrix in a similar way, which will not be elaborated here.

[0140] For example, for each attention weight in the target attention weight matrix corresponding to step i... (e.g., each element in the row or column related to answer step i in the target attention weight matrix), the detection system can traverse all reasoning paths corresponding to answer step i to obtain at least one attention enhancement weight (e.g., 2-hop attention weight, 3-hop attention weight, etc.) of answer step i to step x, and then apply this at least one attention enhancement and attention weight to the target attention weight. The corrected attention weights are obtained by weighted summation. This manual does not limit the specific method of calibration. One method is illustrated below.

[0141] Taking answering step i as an example, the detection system can calculate the corrected attention weights using the following formula. :

[0142]

[0143] in, This represents the maximum number of iterations or the maximum number of adjacent step pairs between step i and step x. This represents the weighting coefficient (which can be preset; this specification does not impose any restrictions on this). Furthermore, the starting hop count for the above summation operation can also start from p=1 (i.e., including direct attention in the weighted summation range); this specification does not impose any restrictions on the specific setting of the starting position for the summation.

[0144] Assuming the reasoning path for answering step a4 includes: reasoning step r4 → reasoning step r16 → answering step a4; and reasoning step r4 → reasoning step r10 → reasoning step r16 → answering step a4, then the attention weight of the corrected answering step a4 to reasoning step r4 in the detection system is... + + .

[0145] In the above scheme, the detection system can effectively capture long-distance logical transmission paths and implicit inference chains in the response sequence by fusing direct attention between steps and multi-hop indirect attention, thereby improving the accuracy of attention weights between steps after correction and laying a solid and reliable data foundation for subsequent consistency detection.

[0146] After the detection system corrects the relevant weights in the target attention weight matrix, it obtains a step-level attention distribution. Each element in the step-level attention distribution represents the corrected attention weights between steps.

[0147] S330: Determine the semantic similarity distribution between the answer content and the reasoning content based on the hidden state information.

[0148] A semantic similarity distribution (also called a semantic similarity map) represents the semantic similarity between each semantic unit in the response content and each semantic unit in the reasoning content. Semantic similarity refers to the degree of similarity or association between two semantic units in their inherent meaning. Semantic similarity distributions can be at the word level or step level, and can be represented as a matrix; this specification does not impose any restrictions on this. A word-level semantic similarity distribution can include the semantic similarity between each word in the response content and each word in the reasoning content. A step-level semantic similarity distribution can include the semantic similarity between each response step in the response content and each reasoning step in the reasoning content.

[0149] See also Figure 5 The hidden state information includes the hidden states of each word element in the response sequence. The detection system can determine the semantic similarity between the response content and the inferred content based on the word element granularity. For example, for each word element in the response content, the detection system calculates the semantic similarity between its hidden state and the hidden states of each word element in the inferred content, thereby constructing a word element-level semantic similarity distribution. Furthermore, the detection system can also determine the semantic similarity distribution based on larger-granularity semantic units (such as step-level), which is not limited in this specification.

[0150] In some embodiments, see continue to see Figure 5 For each step, the detection system aggregates the hidden states of the words belonging to that step to obtain the target hidden state of that step. Then, based on the target hidden states corresponding to each step, it determines the semantic similarity between each answer step and each reasoning step to obtain the step-level semantic similarity distribution.

[0151] Taking step i as an example, the detection system can identify all words belonging to step i based on the mapping relationship between words and steps in the response sequence, and then determine the target hidden state of step i based on the hidden states of these words. In this specification, the hidden state is represented in vector form, and the aggregation operation can be understood as concatenating the hidden states of all words belonging to step i, or performing data operations such as mean pooling and max pooling to generate a feature vector (i.e., the target hidden state) representing the overall semantics of the step.

[0152] For example, the detection system can calculate the target hidden state at each step using the following formula:

[0153]

[0154] in, Let represent the target hidden state of step j at layer l, EmbPool represent the pooling operation used to aggregate word vectors, and c represent the word belonging to step j. This represents the original hidden state of word c in layer l. This represents the set of indices of all terms that constitute step j in the response sequence.

[0155] In some embodiments, for each answering step, the detection system determines the semantic similarity between the answering step and each inference step based on the cosine similarity between the target hidden state of the answering step and the target hidden state of each inference step, thereby obtaining a step-level semantic similarity distribution.

[0156] Each element in the step-level semantic similarity distribution represents the degree of semantic similarity between the response step and the inference step. Specifically, the detection system can determine the semantic similarity value by calculating the cosine similarity between the hidden state vectors corresponding to the response step and the inference step. The detection system can also determine semantic similarity in other ways, which are not limited in this specification. For example, the detection system can determine semantic similarity based on the dot product or bilinear function between the target hidden state of the response step and the target hidden states of each inference step.

[0157] In this embodiment, the detection system constructs step-level hidden states by aggregating word-level hidden states, effectively reducing the computational complexity of semantic similarity distribution and improving processing efficiency. Simultaneously, the detection system utilizes cosine similarity to quantify the content proximity between steps in a high-dimensional semantic space. This measurement method based on deep semantic representation overcomes the limitations of surface-level word matching, accurately identifying step associations with different wording but consistent logical meaning, thus providing a solid and reliable data foundation for subsequent consistency judgments.

[0158] S340: Determine the illusion score of the reasoning model based on the degree of consistency between the attention distribution and the semantic similarity distribution.

[0159] Consistency can also be understood as alignment. The illusion score of an inference model, also known as the routing focus score (RFS), is used to characterize the likelihood of illusion when the inference model generates a response sequence. It aims to quantify whether the cross-stage information routing of the inference model between the inference and response stages is robust, thereby identifying whether the inference model is at risk of "routing collapse".

[0160] "Route collapse" refers to a failure mode that inference models fall into when generating responses: the inference model focuses excessively on steps that are semantically close to the current context but are not key evidence, forming a self-confirmation loop, inhibiting the inference model's ability to correct early errors, and causing illusions to be amplified and spread.

[0161] Ideally, the information transmission path from the reasoning step to the response step should be driven by "key evidence," rather than merely guided by superficial semantic similarity. Here, "key evidence" refers to the original input information in the intermediate steps or target question that has logical necessity, causal determinacy, or factual support for arriving at the final correct answer. It differs from content that is merely literal or semantically similar but makes no substantial logical contribution. Therefore, when generating responses, the model should precisely focus on the key evidence that truly drives the logic forward and provides factual basis, avoiding being misled by seemingly relevant but actually ineffective "semantic proximity noise." If the reasoning model ignores key evidence and chooses semantically proximity but non-key steps, it may cause routing collapse, leading to illusions.

[0162] It is understandable that a higher illusion score indicates that the reasoning model's attention is more focused on steps involving semantically adjacent but non-critical evidence.

[0163] See also Figure 5 The detection system performs consistency checks on the attention distribution and semantic similarity distribution to obtain the illusion score of the reasoning model.

[0164] In some embodiments, for each answer step, the detection system determines inference steps with a semantic similarity greater than or equal to a second preset threshold, forming a semantically related step set; then, it aggregates the attention weights of the answer step to each inference step in the semantically related step set to determine the illusion sub-score of the answer step. After obtaining the illusion sub-score of each answer step, the detection system can determine the illusion score of the inference model based on the illusion sub-scores corresponding to each answer step.

[0165] For example, the detection system can determine the set of semantically relevant steps corresponding to each answer step by filtering with a second preset threshold. This involves traversing the semantic similarity distribution at the step level and filtering out steps with semantic similarity values ​​greater than the second preset threshold. Specifically, taking answer step i as an example, the detection system can obtain the set of semantically relevant steps using the following formula:

[0166]

[0167] in, This represents the set of semantically relevant steps corresponding to step i in the answer. This represents the semantic similarity between steps i and j in the step-level semantic similarity distribution, where j indicates that the condition is met. An index of all semantically relevant steps (i.e., steps semantically relevant to answer step i), This is the second preset threshold.

[0168] After obtaining the set of semantically relevant steps corresponding to each answer step, the detection system can sum up the attention weights (such as corrected attention weights) of the answer step to each step in the set of semantically relevant steps in the attention distribution to obtain the illusion sub-score of the answer step.

[0169] The illusion sub-score quantifies the total intensity of the inference model's attention being focused on inference steps deemed highly relevant (e.g., high semantic similarity) during the generation of answer step i, thus reflecting the alignment between the attention distribution and the semantic similarity distribution. Furthermore, the detection system can also obtain illusion sub-scores through other aggregation methods, which are not limited in this specification.

[0170] Specifically, taking the response to step i as an example, the detection system can obtain its illusion sub-score using the following formula:

[0171]

[0172] in, This represents the illusion sub-fraction in answering step i. This represents the attention weight of step i in the attention distribution for semantically relevant step j.

[0173] In the above embodiments, the detection system introduces a second preset threshold to filter out a set of reasoning steps that are highly semantically relevant to the answer steps, and aggregates the relevant discrete attention weights into a scalar index (i.e., illusion sub-score), which intuitively and quantitatively reflects the overall attention intensity of the model to "semantically relevant steps" when generating answer content. This avoids the overall routing deviation being masked by the randomness of a single high-weight step, thereby more robustly evaluating the alignment level between attention allocation and semantic consistency, effectively identifying the risk of "routing collapse" caused by over-focusing on semantically adjacent but non-logically critical steps, and improving the comprehensiveness and reliability of illusion detection.

[0174] For example, the detection system selects the maximum value from the illusion sub-scores corresponding to the at least one answering step as the illusion score of the reasoning model.

[0175] If the answer contains only one step, the detection system can use the illusion sub-score of that step as the illusion score of the inference model. If the answer contains multiple steps, the detection system can use the largest illusion sub-score as the illusion score of the inference model.

[0176] In this example, the detection system selects the maximum value as the overall hallucination score of the response sequence, which can identify the weakest or most suspicious part of the answer content. This scoring mechanism based on the worst-case principle ensures that the hallucination detection results are not masked by the hallucination sub-scores of other normal answer steps, thus maintaining high sensitivity to serious logical breaks or hallucinations that occur locally, and improving the accuracy of hallucination detection.

[0177] It should be noted that the detection system can also obtain the illusion score of the reasoning model through other calculation methods (such as averaging or weighted averaging of illusion sub-scores), and this specification does not limit this.

[0178] Furthermore, if both the attention distribution and semantic similarity distribution are at the word level, the detection system can determine the illusion sub-score for each word in the response content, thereby determining the illusion score of the reasoning model. The method by which the detection system determines the illusion score can be similar to the case where both the attention distribution and semantic similarity distribution are at the step level; this specification does not impose any restrictions on this.

[0179] In some embodiments, see continue to see Figure 5 If the illusion score is greater than or equal to the third preset threshold, the detection system determines that the reasoning model has reasoning illusion; or if the illusion score is less than the third preset threshold, the detection system determines that the reasoning model does not have reasoning illusion.

[0180] This embodiment sets a third preset threshold and compares it with the hallucination score, providing a simple and clear judgment standard, which improves the usability and execution efficiency of hallucination detection results in practical application scenarios.

[0181] Once the detection system determines that the reasoning model exhibits reasoning illusion, it can indicate the illusion detection results, representing the existence of reasoning illusion, to the relevant service system. The relevant service system can intercept the response sequence and not display it to the user, or it can trigger the re-execution of the reasoning task corresponding to the target question or directly generate a manual task after interception. Alternatively, the service system can mark or output the identification information of the answer steps with high illusion sub-scores in the answer content, facilitating manual review or automatic correction when the model re-executes the reasoning task. This specification does not restrict the specific operations performed after the model exhibits reasoning illusion.

[0182] During the inference model optimization phase, the relevant system can convert illusion scores into reward signals in reinforcement learning. For example, a penalty is given when the responses generated by the inference model exhibit higher attention and semantic consistency (i.e., high illusion risk and low route robustness), and a positive reward is given otherwise. This mechanism guides the inference model to actively adjust its internal parameters during optimization training, making it more inclined to choose logically sound and well-supported reasoning paths, thereby suppressing error patterns such as "route collapse" and improving the reliability of the inference model.

[0183] Once the detection system determines that the reasoning model does not have reasoning illusions, it can indicate the illusion detection results to the relevant service system, which can then display the response sequence of the target question to the user.

[0184] In summary, the model hallucination detection method, system, and storage medium provided in this specification utilize the attention information and hidden states within the inference model for hallucination detection. This eliminates the need for external retrieval verification and multiple inference sampling, thereby reducing hallucination detection costs and meeting low-latency requirements in specific scenarios. This specification uses the consistency between attention distribution and semantic similarity distribution as a detection signal to quantify the likelihood of the model generating hallucinations. This allows for more accurate identification of answers that appear semantically fluent but lack crucial logical support, providing a new paradigm of fully automated hallucination detection. This effectively improves detection accuracy and enhances the credibility and user satisfaction of service systems that call inference models. Furthermore, this detection system supports real-time embedding into service systems that call inference models, reducing the complexity of system integration.

[0185] It should be noted that the letter identifiers (such as i, j, etc.) appearing in this specification are only used to refer to specific terms, steps, etc., and are intended to clearly distinguish different technical elements. These variables may have different physical meanings in different embodiments, different formulas, or different contexts.

[0186] This specification, in another aspect, provides a computer-readable non-transitory storage medium storing at least one set of executable instructions for hallucination detection. When the executable instructions are executed by a processor, they instruct the processor to perform the steps of the hallucination detection method P300 described in this specification. In some possible embodiments, various aspects of this specification may also be implemented as a program product comprising program code. When the program product is run on a computing system 200, the program code causes the computing system 200 to perform the steps of the method P300 described in this specification. The program product for implementing the above method may employ a portable compact disc read-only memory (CD-ROM) containing program code and may run on the computing system 200. However, the program product of this specification is not limited thereto. In this specification, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system. The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing the operations described herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on computing system 200, partially on computing system 200, as a standalone software package, partially on computing system 200 and partially on a remote electronic device, or entirely on a remote electronic device.

[0187] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0188] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure is presented by way of example only and is not restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0189] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0190] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and aiding in the understanding of a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art will readily identify some of them as separate embodiments when reading this specification. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. It is also valid when each secondary embodiment contains fewer than all the features of a single foregoing disclosed embodiment.

[0191] Every patent, patent application, publication of a patent application, and other material cited herein, such as articles, books, specifications, publications, documents, and literature (excluding any related historical examination documents), is referenced for all purposes relevant to this document, including in the specification and claims herein. However, in the event of any inconsistency or conflict between the descriptions, definitions, and / or terms used in the foregoing and those used herein, the descriptions, definitions, and / or terms used herein shall prevail.

[0192] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. A method for detecting model hallucinations, comprising: Attention information and hidden state information are obtained during the process of the inference model generating a response sequence, wherein the response sequence includes inference content and answer content for the target question; Based on the attention information, the attention distribution of the answer content to the inference content is determined, and based on the hidden state information, the semantic similarity distribution between the answer content and the inference content is determined; and The illusion score of the reasoning model is determined based on the degree of consistency between the attention distribution and the semantic similarity distribution.

2. The method according to claim 1, wherein, The reasoning content includes multiple reasoning steps, the answer content includes at least one answering step, each answering step or reasoning step includes multiple word units, the attention information includes attention weights between word units, and the hidden state information includes the hidden state of each word unit. The step of determining the attention distribution of the answer content to the inference content based on the attention information, and determining the semantic similarity distribution between the answer content and the inference content based on the hidden state information, includes: The attention weights among each word are aggregated at the step level to obtain the target attention weight matrix. Based on the target attention weight matrix, the attention distribution of each answer step to each reasoning step is determined; and For each step, the hidden states of the words belonging to that step are aggregated to obtain the target hidden state of that step. Based on the target hidden states corresponding to each step, the semantic similarity between each answer step and each reasoning step is determined, resulting in a step-level semantic similarity distribution.

3. The method according to claim 2, wherein, The step of determining the attention distribution of each answer step to each reasoning step based on the target attention weight matrix includes: Construct a reasoning path for each answer step, wherein the reasoning path represents a multi-hop dependency link tracing backward from the answer step to the preceding reasoning step; and The target attention weight matrix is ​​corrected based on the reasoning path of each answer step to obtain the attention distribution.

4. The method according to claim 3, wherein, The precursor reasoning steps include direct precursor steps and at least one indirect precursor step. The correction of the target attention weight matrix based on the reasoning path for each response step includes: For each answer step, the attention enhancement weight of the answer step for each indirect predecessor step is determined based on the attention weights between adjacent steps in its corresponding inference path; and The attention weights corresponding to the answer steps in the target attention weight matrix are corrected based on the attention enhancement weights corresponding to each answer step.

5. The method according to claim 4, wherein, The determination of the attention enhancement weight of the answer step for each indirect predecessor step based on the attention weight between adjacent steps in its corresponding reasoning path includes: For each indirect precursor step, the attention enhancement weight of the response step on the indirect precursor step is obtained based on the attention weight of each adjacent step pair between the indirect precursor step and the response step. The step of correcting the attention weights corresponding to the answer steps in the target attention weight matrix based on the attention enhancement weights corresponding to each answer step includes: For each indirect precursor step, the attention weights of the response step on the indirect precursor step in the target attention weight matrix are adjusted based on the attention enhancement weights of the response step on the indirect precursor step.

6. The method according to claim 3, wherein, The reasoning path for constructing each answer step includes: An initialization process yields a source step set. Multiple iterative updates are performed on this source step set to obtain the reasoning path for each answer step. In the first iteration, the source step set includes at least one answer step. In each iteration: For each step in the source step set, a first preset number of candidate predecessor steps are determined according to the attention weight from large to small, to obtain the candidate predecessor step set. The target predecessor steps that meet the preset conditions in the candidate predecessor step set are determined as the source step set for the next iteration; and The reasoning path for each answer step is determined based on the set of source steps in each iteration.

7. The method according to claim 6, wherein, The target precursor step is obtained based on the following method: For each candidate predecessor step in the candidate predecessor step set, the cumulative attentional influence of the candidate predecessor step is determined based on the attentional weights of at least some steps in the source step set on the candidate predecessor step. as well as Candidate precursor steps whose cumulative attention influence is greater than or equal to a first preset threshold are selected as the target precursor steps.

8. The method according to claim 2, wherein, The inference model includes multiple attention heads, and the attention information includes a word-level attention matrix corresponding to each attention head. The step-by-step aggregation of attention weights among each word to obtain the target attention weight matrix includes: For each attention head, the attention weights between words in its corresponding word-level attention matrix are aggregated step-by-step to obtain the initial attention weight matrix for each attention head; and The target attention weight matrix is ​​obtained by aggregating the initial attention weight matrices corresponding to at least one target attention head.

9. The method according to claim 8, wherein, The method further includes: For each attention head, the information entropy of the attention head is determined based on the initial attention weight matrix corresponding to the attention head; and A second preset number of attention heads are selected according to the information entropy from smallest to largest to obtain the at least one target attention head.

10. The method according to claim 8, wherein, The attention weights for step i to step j in the initial attention weight matrix are obtained in the following manner: For each word in step i, the attention weights of the word to each word in step j are aggregated by pooling to obtain the attention weights of the word to step j. as well as The attention weights of each word in step i to step j are aggregated by pooling to obtain the attention weights of step i to step j.

11. The method according to claim 2, wherein, The determination of semantic similarity between each answer step and each inference step based on the target hidden state corresponding to each step includes: For each answering step, the semantic similarity is determined based on the cosine similarity between the target hidden state of the answering step and the target hidden state of each inference step.

12. The method according to claim 2, wherein, Determining the illusion score of the reasoning model based on the degree of consistency between the attention distribution and the semantic similarity distribution includes: For each answer step, determine the reasoning steps whose semantic similarity is greater than or equal to a second preset threshold, and form a set of semantically related steps; Aggregate the attention weights of the response step to each reasoning step in the semantically relevant step set to determine the illusion sub-score of the response step; and The illusion score of the reasoning model is determined based on the illusion sub-score corresponding to each answer step.

13. The method according to claim 12, wherein, Determining the illusion score of the reasoning model based on the illusion sub-scores corresponding to each answer step includes: The maximum value is selected from the illusion sub-scores corresponding to the at least one answer step and used as the illusion score of the reasoning model.

14. The method according to claim 1, wherein, The method further includes: If the hallucination score is greater than or equal to a third preset threshold, it is determined that the reasoning model exhibits reasoning hallucination; or If the hallucination score is less than the third preset threshold, it is determined that the reasoning model does not have reasoning hallucinations.

15. A detection system, comprising: At least one storage medium storing at least one instruction set; as well as At least one processor is communicatively connected to the at least one storage medium, wherein the at least one processor reads the at least one instruction set during operation and performs the method as described in any one of claims 1-14 according to the instructions of the at least one instruction set.

16. A computer-readable non-transitory storage medium, wherein, The computer-readable non-transitory storage medium stores at least one set of instructions, which, when executed by at least one processor, implement the method as described in any one of claims 1-14.

Images