Illusion detection method and device for retrieval enhancement generation system, equipment and medium
By a method of cross-layer aggregation of multi-head attention matrices and construction of target semantic graphs, the applicability problem of hallucination detection in retrieval enhancement generation systems is solved, and efficient and accurate automatic detection of hallucination risks is achieved.
Patent Information
- Application Number
- CN202511773772.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-02-17
AI Technical Summary
Existing retrieval enhancement generation systems are prone to hallucination phenomena during the generation process. Existing hallucination detection schemes mostly focus on the semantic level of text and lack understanding of the internal structural features of the model, resulting in insufficient applicability of detection.
By aggregating multi-head attention matrices across layers, a target semantic graph is constructed. The graph fusion results and preset topological structure indicators are used to detect hallucination risks, and automatic detection is performed by combining logistic regression or classification models.
It improves the interpretability, accuracy, and applicability of hallucination detection, and can automatically and efficiently identify deviations between generated content and factual knowledge.
Smart Images
Figure CN121543020A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and medium for detecting illusions in a retrieval enhancement generation system. Background Technology
[0002] With the rapid development of large language models (LLMs), retrieval-augmented generation (RAG) technology has become an important direction for improving the knowledge accuracy of large models. However, in practical applications, retrieval-augmented generation systems may produce illusions during the generation process, that is, the generated content may not match the factual knowledge or deviate from the original retrieved document.
[0003] To address this issue, existing hallucination detection schemes mostly focus on the semantic level of text, lacking an understanding of the internal structural features of the model. This makes it difficult to reflect the directionality and importance of information flow during the generation process. Furthermore, the detection relies heavily on manual rules or heuristic indicators, which limits the applicability of the detection. Summary of the Invention
[0004] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for hallucination detection in a retrieval enhancement generation system, which can automatically and efficiently realize hallucination detection in the retrieval enhancement generation system, and improve the interpretability, accuracy, comprehensiveness, and applicability of hallucination detection. The specific solution is as follows:
[0005] In a first aspect, this application provides a method for detecting hallucinations in a retrieval-enhanced generation system, comprising:
[0006] In the generation stage of the literature review generation task based on the retrieval enhancement generation system and the preset knowledge base, attention is aggregated across layers based on the multi-head attention matrix of each layer in the retrieval enhancement generation system to determine the attention aggregation result.
[0007] Based on the last layer of the retrieval enhancement generation system, word embedding vectors are extracted, and semantic similarity analysis and mapping are performed using the corresponding vector extraction results to determine the target semantic graph.
[0008] The attention aggregation result and the target semantic graph are weighted and fused, and the current structural index value is determined by using the corresponding graph fusion result and the preset graph topology index.
[0009] Based on the preset hallucination detection model, the structural index value, and the text fact tags corresponding to the documents in the preset knowledge base, hallucination risk detection is performed to determine the hallucination detection result corresponding to the literature review generation task; the preset hallucination detection model is a model constructed based on any one of the logistic regression model, classification model, and tree model.
[0010] Optionally, the step of performing cross-layer aggregation of attention based on the multi-head attention matrix of each layer in the retrieval enhancement generation system to determine the attention aggregation result includes:
[0011] The attention matrix of multiple attention heads in each layer of the retrieval enhancement generation system is extracted to determine the attention matrix extraction result;
[0012] Based on the attention matrix extraction results, the importance metric value corresponding to each attention head is determined;
[0013] Based on the magnitude of the importance metric, the attention heads are sorted from largest to smallest to determine the attention head sorting result;
[0014] Based on the attention head sorting results and the first preset number, attention heads are filtered to determine the set of filtered attention heads;
[0015] For any attention head belonging to any layer in the attention head set, based on the attention matrix extraction result, matrix averaging aggregation is performed between attention heads to determine the global attention matrix of the current layer;
[0016] Based on the preset layer set and the corresponding global attention matrix, cross-layer matrix averaging is performed to determine the aggregated matrix;
[0017] Based on the aggregated matrix, row normalization is triggered to determine the row-normalized matrix.
[0018] The row-normalized matrix is then symmetricized to determine the attention aggregation result.
[0019] Optionally, based on the last layer of the retrieval enhancement generation system, word embedding vectors are extracted, and semantic similarity analysis and mapping are performed using the corresponding vector extraction results, including:
[0020] From the last layer of the retrieval enhancement generation system, extract the lexical embedding vector to determine the vector extraction result;
[0021] Based on the vector extraction results and the preset semantic similarity analysis rules, the similarity values between the word embedding vectors are determined;
[0022] The similarity values are linearly mapped to a preset interval to determine the mapping result;
[0023] Using the word embedding vector as a node, for any word embedding vector, based on the corresponding similarity value, neighboring nodes are filtered to determine the target neighboring node set; the target neighboring node set includes a second preset number of neighboring nodes that are closest to the current word embedding vector;
[0024] Each word embedding vector is connected to the corresponding target neighbor node set by directed edges to determine the target semantic graph.
[0025] Optionally, the weighted fusion based on the attention aggregation result and the target semantic map includes:
[0026] The attention aggregation result and the target semantic map are subjected to deviation normalization processing to determine the normalization result;
[0027] Based on preset calibration rules, the preset fusion parameters are calibrated in the domain to determine the calibrated parameters;
[0028] Based on the calibrated parameters and the standardized processing results, a weighted fusion is performed to determine the weighted fusion result;
[0029] Based on a preset global threshold, the weighted fusion result is filtered to complete the sparsity processing operation and obtain the sparsity processing result.
[0030] The sparsification result is symmetricized to determine the graph fusion result; the graph fusion result is an undirected graph.
[0031] Optionally, determining the current structural index value using the corresponding graph fusion results and preset graph topology indexes includes:
[0032] Obtain the undirected weighted adjacency matrix corresponding to the graph fusion result; wherein, the undirected weighted adjacency matrix is a matrix used to represent the adjacency relationship between nodes in the graph fusion result;
[0033] Based on the undirected weighted adjacency matrix, the weighting degree corresponding to each node in the graph fusion result is determined;
[0034] The average weighted degree is determined based on the weighted degree corresponding to each node;
[0035] The degree distribution information is determined using the weighted degree and the average weighted degree; the degree distribution information includes degree variance and skewness.
[0036] Perform singular value decomposition on the undirected weighted adjacency matrix, and select a third preset number of target singular values from the corresponding decomposition results;
[0037] Based on the third preset number of target singular values, the spectral energy ratio is determined;
[0038] Based on the undirected weighted adjacency matrix, the distance matrix between nodes is transformed to determine the matrix transformation result;
[0039] Based on the matrix transformation results, Vitoris-Lipps filtering is performed, and the manifold topological divergence is determined using the corresponding filtering results.
[0040] Based on the weighted degree, the degree distribution information, the spectral energy ratio, and the manifold topological divergence, the current structural index value is determined.
[0041] Optionally, the hallucination risk detection based on the preset hallucination detection model, the structural index value, and the text fact tags corresponding to the documents in the preset knowledge base, to determine the hallucination detection result corresponding to the literature review generation task, includes:
[0042] The structural index values are concatenated to determine the concatenated feature vector;
[0043] Based on the text fact tags corresponding to the documents in the preset knowledge base, obtain the target text fact tags corresponding to the generation results of the literature review generation task;
[0044] Based on a preset hallucination detection model and the feature vector with the target text fact labels, hallucination risk detection is performed to determine the hallucination detection result; wherein, the hallucination detection result includes hallucination risk scores corresponding to each paragraph in the generated result.
[0045] Optionally, after determining the hallucination detection result corresponding to the literature review generation task, the method further includes:
[0046] Based on the hallucination detection results and the generation results corresponding to the literature review generation task, the retrieval enhancement generation system visualizes the information coupling and semantic jump regions between different documents in the preset knowledge base.
[0047] Secondly, this application provides a hallucination detection device for a retrieval enhancement generation system, comprising:
[0048] The multi-head attention aggregation module is used to perform cross-layer aggregation of attention based on the multi-head attention matrix of each layer in the retrieval enhancement generation system during the generation stage of the literature review generation task based on the retrieval enhancement generation system and the preset knowledge base, so as to determine the attention aggregation result.
[0049] The semantic graphing module is used to extract word embedding vectors based on the last layer of the retrieval enhancement generation system, and to use the corresponding vector extraction results to perform semantic similarity analysis and graphing to determine the target semantic graph.
[0050] The indicator value determination module is used to perform weighted fusion based on the attention aggregation result and the target semantic graph, and to determine the current structural indicator value using the corresponding graph fusion result and the preset graph topology structure indicator;
[0051] The hallucination detection result determination module is used to perform hallucination risk detection based on a preset hallucination detection model, the structural index value, and the text fact tags corresponding to the documents in the preset knowledge base, so as to determine the hallucination detection result corresponding to the literature review generation task; and to perform hallucination risk detection based on the preset hallucination detection model, the structural index value, and the text fact tags corresponding to the documents in the preset knowledge base.
[0052] Thirdly, this application provides an electronic device, comprising:
[0053] Memory, used to store computer programs;
[0054] A processor is configured to execute the computer program to implement the steps of the aforementioned hallucination detection method for the retrieval enhancement generation system.
[0055] Fourthly, this application provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the steps of the aforementioned hallucination detection method of the retrieval enhancement generation system.
[0056] As can be seen, in this application, during the generation stage of the literature review generation task based on the retrieval enhancement generation system and the preset knowledge base, attention is aggregated across layers based on the multi-head attention matrix of each layer in the retrieval enhancement generation system to determine the attention aggregation result; based on the last layer of the retrieval enhancement generation system, word embedding vectors are extracted, and semantic similarity analysis and graph construction are performed using the corresponding vector extraction results to determine the target semantic graph; weighted fusion is performed based on the attention aggregation result and the target semantic graph, and the current structural index value is determined using the corresponding graph fusion result and the preset graph topology index; based on the preset hallucination detection model, the structural index value, and the text fact tags corresponding to the documents in the preset knowledge base, hallucination risk detection is performed to determine the hallucination detection result corresponding to the literature review generation task; the preset hallucination detection model is a model constructed based on any one of the logistic regression model, classification model, and tree model. In other words, in this application, during the generation stage of the literature review generation task based on the retrieval enhancement generation system and a preset knowledge base, the multi-head attention matrix of each layer in the retrieval enhancement generation system is first aggregated across layers to determine the attention aggregation result. Then, based on the word embedding vector of the last layer in the retrieval enhancement generation system, semantic similarity analysis and graph construction are performed to determine the target semantic graph. Afterwards, the attention aggregation result and the target semantic graph are weighted and fused, and the corresponding graph fusion result is used to determine the current structural index value. Finally, based on the preset illusion detection model, the structural index value, and the text fact tags corresponding to the documents in the preset knowledge base, the illusion detection result is determined. This allows for automatic and efficient illusion detection of the retrieval enhancement generation system, and improves the interpretability, accuracy, comprehensiveness, and applicability of illusion detection. Attached Figure Description
[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0058] Figure 1 A flowchart of a retrieval enhancement generation system for hallucination detection provided in this application;
[0059] Figure 2 A flowchart of a specific retrieval enhancement generation system for hallucination detection provided in this application;
[0060] Figure 3 A schematic diagram of the illusion detection device of a retrieval enhancement generation system provided in this application;
[0061] Figure 4 This application provides a structural diagram of an electronic device. Detailed Implementation
[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0063] With the rapid development of large-scale language models, retrieval-enhanced generation technology has become an important direction for improving the accuracy of knowledge in large models. However, in practical applications, retrieval-enhanced generation systems may experience illusions during the generation process, meaning that the generated content may not match factual knowledge or may deviate from the original retrieved document. To address this issue, existing illusion detection schemes mostly focus on the semantic level of the text, lacking an understanding of the internal structural features of the model. This makes it difficult to reflect the directionality and importance of information flow during the generation process; furthermore, detection often relies on manual rules or heuristic indicators, thus limiting the applicability of the detection.
[0064] Therefore, this application provides a hallucination detection scheme for a retrieval enhancement generation system, which can automatically and efficiently realize hallucination detection in the retrieval enhancement generation system, and improve the interpretability, accuracy, comprehensiveness and applicability of hallucination detection.
[0065] See Figure 1 As shown in the figure, an embodiment of the present invention discloses a method for hallucination detection in a retrieval enhancement generation system, comprising:
[0066] Step S11: In the generation stage of the literature review generation task based on the retrieval enhancement generation system and the preset knowledge base, cross-layer aggregation of attention is performed based on the multi-head attention matrix of each layer in the retrieval enhancement generation system to determine the attention aggregation result.
[0067] In this embodiment, combined with Figure 2As shown, the multi-head attention matrix of each layer is extracted from the generation stage of the large model in the RAG system. The most important attention heads of the top-K layers are retained and then aggregated to reduce noise. The attention of the selected layers is averaged and aggregated into global attention. After cross-layer aggregation, the attention weight matrix is re-normalized and symmetricized. Specifically: The attention matrices of multiple attention heads at each layer of the retrieval enhancement generation system are extracted to determine the attention matrix extraction result; based on the attention matrix extraction result, the importance metric value corresponding to each attention head is determined; based on the magnitude of the importance metric value, the attention heads are sorted from largest to smallest to determine the attention head sorting result; based on the attention head sorting result and a first preset number, attention heads are filtered to determine the filtered attention head set; for attention heads belonging to any layer in the attention head set, matrix averaging aggregation is performed between attention heads based on the attention matrix extraction result to determine the global attention matrix of the current layer; based on a preset layer set and the corresponding global attention matrix, cross-layer matrix averaging aggregation is performed to determine the aggregated matrix; based on the aggregated matrix, row normalization processing is triggered to determine the row-normalized matrix; the row-normalized matrix is symmetricized to determine the attention aggregation result. The literature review generation task includes, but is not limited to, scenarios such as generating scientific literature reviews, compiling monthly analysis reports for enterprises, or assessing knowledge through educational Q&A. The application areas involved are tasks related to the financial field, but can also be tasks related to other fields such as the medical field. The preset knowledge base also varies depending on the field to which the task belongs, and the specific field to be selected can be determined based on actual needs.
[0068] It is important to understand that regarding the retrieval of multi-head attention, let's consider the... Layer The attention matrix of the head is Where T is the total number of tokens in the sequence. Let be the set of real numbers. Then, according to the expression shown below, calculate the _th _. Layer Head entropy (a measure of the degree of concentration / dispersion in a distribution) As a measure of head importance:
[0069] .
[0070] In the formula, To prevent small constants with a logarithm of zero (which can be set based on actual needs, such as 10⁻¹²), This indicates the row number of the attention matrix. This indicates the number of columns in the attention matrix. Indicates the first Layer First Line number The numerical values in the column. A lower entropy value indicates more "focused" attention, which can be considered as higher importance.
[0071] After that, according to Sort the heads and keep the order of each layer. (Values can be configured according to actual needs) Attention head set Based on the formula shown below, the aggregated attention for each layer is obtained by performing inter-head averaging on the selected head set. :
[0072] .
[0073] In the formula, This indicates the number of elements in the collection.
[0074] For the selected layer set (Configure according to actual needs, e.g., the last 3-6 layers of the model) Perform inter-layer averaging to obtain the global attention matrix A:
[0075] .
[0076] Then, perform row normalization according to the following formula (treating each row as a distribution):
[0077] ;
[0078] In the formula, Representation matrix No. Line number The values in the column, Represents the matrix A's first... Line number The values in the column.
[0079] Symmetry transformation (converting directed attention into undirected edges):
[0080] ;
[0081] In the formula, This represents the normalized attention matrix. This represents the attention matrix after symmetry processing. This indicates that the normalized attention matrix is transposed.
[0082] Step S12: Based on the last layer of the retrieval enhancement generation system, extract the word embedding vectors, and use the corresponding vector extraction results to perform semantic similarity analysis and graph construction to determine the target semantic graph.
[0083] In this embodiment, combined with Figure 2As shown, token embeddings, i.e., word vectors, are extracted from the last layer of the model. Cosine similarity is calculated between the vectors, and the cosine similarity is linearly mapped to [0,1]. For each token, K neighbor nodes with the highest similarity are selected, and the similarity of non-neighbor nodes is set to 0 to construct a semantic graph. Specifically: word embedding vectors are extracted from the last layer of the retrieval enhancement generation system to determine the vector extraction result; based on the vector extraction result and the preset semantic similarity analysis rules, the similarity value between the word embedding vectors is determined; the similarity value is linearly mapped to a preset interval to determine the mapping result; the word embedding vectors are used as nodes, and for any word embedding vector, based on the corresponding similarity value, neighbor nodes are filtered to determine the target neighbor node set; the target neighbor node set includes a second preset number of neighbor nodes that are closest to the current word embedding vector; each word embedding vector is connected to the corresponding target neighbor node set with directed edges to determine the target semantic graph.
[0084] Specifically, regarding the construction of the semantic graph, firstly, token embedding extraction: let the embedding of the i-th token be a vector. Calculate cosine similarity:
[0085] ;
[0086] In the formula, Represents the vector of the i-th word element. This represents the vector of the j-th word. This represents the cosine similarity value between the i-th and j-th word vectors. Represents the norm symbol.
[0087] Then, the similarity value is used as the distance and linearly mapped to [0,1] to obtain the mapped value. :
[0088] .
[0089] Then, for each row i, select the K sets of maximum similarity indices. Construct the KNN matrix (K-Nearest Neighbor). :
[0090] .
[0091] Then, the KNN matrix is symmetricized to obtain the target semantic map. :
[0092] .
[0093] Step S13: Perform weighted fusion based on the attention aggregation result and the target semantic graph, and determine the current structural index value using the corresponding graph fusion result and the preset graph topology index.
[0094] In this embodiment, after determining the target semantic graph and the attention aggregation result, normalization is first performed. Before merging, min-max normalization is used to ensure that the attention matrix and the similarity matrix are at a comparable scale. Then, weighted linear fusion of edge weights is performed. The fusion weights are set through cross-validation or prior experience. The fused weights are then sparsified and symmetricized. Global threshold filtering is performed to reduce computational overhead and noise. An undirected graph is obtained through symmetry. Specifically: the attention aggregation result and the target semantic graph are subjected to deviation normalization to determine the normalization result; based on preset calibration rules, preset fusion parameters are calibrated in the domain to determine the calibrated parameters; based on the calibrated parameters and the normalization result, weighted fusion is performed to determine the weighted fusion result; based on a preset global threshold, the weighted fusion result is filtered to complete the sparsification operation to obtain the sparsified result; the sparsified result is symmetricized to determine the graph fusion result; the graph fusion result is an undirected graph.
[0095] It's important to understand that the normalization formula is as follows, for the attention diagonal matrix, i.e., the attention aggregation result. And semantic matrix, i.e. target semantic graph Perform min-max normalization separately:
[0096] ;
[0097] In the formula, This represents the normalized attention diagonal matrix. This represents the normalized semantic matrix.
[0098] Then, weighted linear fusion is performed to obtain the weighted fusion result. :
[0099] ;
[0100] In the formula, The optimal value can be selected from a preset grid (e.g., {0.4, 0.5, 0.6, 0.7}) through cross-validation.
[0101] Next, global threshold filtering :reserve satisfy Set the rest to 0 to obtain the sparsification result. :
[0102] .
[0103] Re-symmetrize to ensure the graph is undirected, in order to obtain the graph fusion result. :
[0104] .
[0105] Furthermore, after the fusion is completed, basic connectivity and statistics, such as weighted degree and degree distribution statistics, are calculated; spectral features are calculated based on the adjacency matrix, the top k largest singular values are calculated and represented by spectral intervals; and manifold topological divergence (MTopDiv) is calculated to capture higher-order topological structures as a supplement to the spectral method. Specifically: The following steps are performed: 1. Obtain the undirected weighted adjacency matrix corresponding to the graph fusion result; wherein the undirected weighted adjacency matrix is a matrix used to represent the adjacency relationships between nodes in the graph fusion result; 2. Based on the undirected weighted adjacency matrix, determine the weighted degree corresponding to each node in the graph fusion result; 3. Based on the weighted degree corresponding to each node, determine the average weighted degree; 4. Using the weighted degree and the average weighted degree, determine the degree distribution information; the degree distribution information includes degree variance and skewness; 5. Perform singular value decomposition on the undirected weighted adjacency matrix, and select a third preset number of target singular values from the corresponding decomposition results; 6. Based on the third preset number of target singular values, determine the spectral energy ratio; 7. Based on the undirected weighted adjacency matrix, perform a transformation of the distance matrix between nodes to determine the matrix transformation result; 8. Based on the matrix transformation result, perform Vitoris-Lipps filtering, and use the corresponding filtering result to determine the manifold topological divergence; 9. Based on the weighted degree, the degree distribution information, the spectral energy ratio, and the manifold topological divergence, determine the current structural index value.
[0106] It is important to understand that, let the final undirected weighted adjacency matrix be... Calculate the weighted strength of node i. Average :
[0107] ;
[0108] ;
[0109] In the formula, Represents the adjacency matrix of the th Line number The values in the column. In addition, variance, skewness, etc., can be calculated according to conventional statistical definitions.
[0110] Next, perform singular value decomposition on W:
[0111] ;
[0112] In the formula, , , This is the eigenvector after singular value decomposition of matrix W.
[0113] Then take the first k (which can be configured according to actual needs) singular values and define the spectral energy ratio. :
[0114] ;
[0115] In the formula, Let be the i-th eigenvector of W.
[0116] Finally, the topological divergence of the manifold (MTopDiv) is calculated. First, convert the weights into a distance matrix. (or (e.g., monotonic mappings). Then, construct... Filtering and calculating 0-dimensional and 1-dimensional persistent barcode sets. (birth, death). Finally, define the sum of persistence or top-level persistence statistics as indicators:
[0117] .
[0118] Step S14: Based on the preset hallucination detection model, the structural index value, and the text fact tags corresponding to the documents in the preset knowledge base, perform hallucination risk detection to determine the hallucination detection result corresponding to the literature review generation task; perform hallucination risk detection based on the preset hallucination detection model, the structural index value, and the text fact tags corresponding to the documents in the preset knowledge base.
[0119] In this embodiment, after determining the current structural index value, the structural index value is correlated with the factual consistency labels of the RAG generation results. A hallucination detection model is constructed using logistic regression or a lightweight classification model to output a hallucination risk score. The labels are generated by manually or automatically using weakly supervised methods to annotate hallucination sentences / paragraphs present in the generated text. Specifically: the structural index values are concatenated to determine a concatenated feature vector; based on the textual factual labels corresponding to the documents in the preset knowledge base, target textual factual labels corresponding to the generation results of the literature review generation task are obtained; based on the preset hallucination detection model, the feature vector, and the target textual factual labels, hallucination risk detection is performed to determine the hallucination detection result; wherein, the hallucination detection result includes hallucination risk scores corresponding to each paragraph in the generated result.
[0120] It's important to understand that regarding the construction of labels, we first use weakly supervised annotation and define a consistency score C:
[0121] ;
[0122] In the formula, d represents a single text from the set of relevant texts retrieved from the knowledge base, and D represents the set of relevant texts retrieved from the knowledge base. This represents the output of the large language model to be tested for whether hallucinations are occurring. Methods for calculating text similarity include sentence vector cosine similarity and TF-IDF (Term Frequency-Inverse Document Frequency) scores. If C < It is then labeled as "possible hallucination". This is an empirical threshold.
[0123] At the same time, the structural index values are concatenated into an eigenvector f:
[0124] .
[0125] Then, logistic regression can be used to calculate:
[0126] .
[0127] In the formula, For the concatenated feature vector, and For trainable parameters, It is the sigmoid activation function.
[0128] Or use a tree model The output probability or score s∈[0,1] is given. Let the threshold be... If s≥ It is then determined to be a hallucination (this can be determined using the ROC curve (Receiver Operating Characteristic)). ).
[0129] Furthermore, after determining the hallucination detection result corresponding to the literature review generation task, in this embodiment: based on the hallucination detection result and the generation result corresponding to the literature review generation task, the retrieval enhancement generation system visualizes the information coupling and semantic jump regions between different documents in the preset knowledge base.
[0130] In summary, this embodiment constructs a more comprehensive semantic dependency network through a weighted fusion of attention and semantic similarity. Topological indices quantify the information flow and aggregation within the model, revealing the structural causes of hallucinations and improving interpretability. Furthermore, modeling based on topological features can automatically identify anomalous distributions in the generation process and automatically detect hallucination risks. Moreover, this scheme can be applied to any RAG-based question-answering, dialogue, or text generation system, demonstrating its versatility.
[0131] Therefore, in this application, during the generation stage of the literature review generation task based on the retrieval enhancement generation system and the preset knowledge base, the multi-head attention matrix of each layer in the retrieval enhancement generation system is first aggregated across layers to determine the attention aggregation result. Then, based on the word embedding vector of the last layer in the retrieval enhancement generation system, semantic similarity analysis and graph construction are performed to determine the target semantic graph. Afterwards, the attention aggregation result and the target semantic graph are weighted and fused, and the corresponding graph fusion result is used to determine the current structural index value. Finally, based on the preset illusion detection model, the structural index value, and the text fact tags corresponding to the documents in the preset knowledge base, the illusion detection result is determined. This allows for automatic and efficient illusion detection in the retrieval enhancement generation system, improving the interpretability, accuracy, comprehensiveness, and applicability of illusion detection.
[0132] The following is combined with Figure 2 The schematic diagram disclosed herein provides a detailed description of the technical solutions of the embodiments of this application.
[0133] The following example, using the generation of a scientific literature review, illustrates the specific process of hallucination detection:
[0134] 1) Data collection and knowledge construction: In this scenario, text is extracted from multiple scientific research databases or paper abstracts to construct a multi-source literature knowledge base, covering different perspectives and experimental results of the same research topic, providing semantic support for generating review-type answers;
[0135] 2) Semantic representation and attention modeling: The chapters, experimental conclusions and citation relationships of scientific literature are used as input units. The RAG large model is used to retrieve and generate preliminary answers. At the same time, the multi-head attention matrix and token-level embedding representation are extracted to preserve the attention distribution characteristics of the model in the information integration process.
[0136] 3) Adaptation of Fusion Graph Weights and Topological Indicators: Based on the semantic features of scientific literature, the fusion parameters of attention weights and similarity matrices are domain-calibrated to ensure that the weighted graph better reflects 'consistency of evidence' and 'convergence of conclusions'. On this basis, topological indicators (such as spectral energy ratio, MTopDiv, etc.) are calculated to quantify the model's degree of fusion balance of evidence from different sources during review generation.
[0137] 4) Hallucination Risk Assessment and Visualization: Using the topological features obtained in step 3), a regression or classification model is established with the fact labels obtained by manual annotation or automatic consistency detection. The hallucination risk score of each generated paragraph is output, and the information coupling and semantic jump regions of the model between different documents are displayed through visualization, providing an interpretable quality assurance mechanism for the automatic scientific review system.
[0138] In summary, although the implementation process of this solution is illustrated using a scenario involving the generation of a scientific literature review, the core technical framework—namely, the graph construction integrating attention and semantic similarity, the extraction of topological structure indicators, and the hallucination risk modeling based on graph features—remains consistent with the previous embodiment. The main changes focus on the data organization, similarity measurement methods, and domain adaptation of graph weight parameters, enabling the solution to achieve stable transfer and interpretable evaluation across various knowledge-intensive text generation tasks.
[0139] See Figure 3 As shown in the embodiments of this application, a hallucination detection device for a retrieval enhancement generation system is also disclosed, comprising:
[0140] The multi-head attention aggregation module 11 is used to perform cross-layer aggregation of attention based on the multi-head attention matrix of each layer in the retrieval enhancement generation system during the generation stage of the literature review generation task based on the retrieval enhancement generation system and the preset knowledge base, so as to determine the attention aggregation result.
[0141] The semantic graphing module 12 is used to extract word embedding vectors based on the last layer of the retrieval enhancement generation system, and to use the corresponding vector extraction results to perform semantic similarity analysis and graphing to determine the target semantic graph.
[0142] The indicator value determination module 13 is used to perform weighted fusion based on the attention aggregation result and the target semantic graph, and to determine the current structural indicator value using the corresponding graph fusion result and the preset graph topology structure indicator.
[0143] The hallucination detection result determination module 14 is used to perform hallucination risk detection based on a preset hallucination detection model, the structural index value, and the text fact tags corresponding to the documents in the preset knowledge base, so as to determine the hallucination detection result corresponding to the literature review generation task; and to perform hallucination risk detection based on the preset hallucination detection model, the structural index value, and the text fact tags corresponding to the documents in the preset knowledge base.
[0144] In some specific embodiments, the multi-head attention aggregation module 11 can be used to: extract the attention matrix of multiple attention heads in each layer of the retrieval enhancement generation system to determine the attention matrix extraction result; determine the importance metric value corresponding to each attention head based on the attention matrix extraction result; sort the attention heads from largest to smallest based on the importance metric value to determine the attention head sorting result; perform attention head filtering based on the attention head sorting result and a first preset number to determine the filtered attention head set; for attention heads belonging to any layer in the attention head set, perform matrix average aggregation between attention heads based on the attention matrix extraction result to determine the global attention matrix of the current layer; perform cross-layer matrix average aggregation based on a preset layer set and the corresponding global attention matrix to determine the aggregated matrix; trigger row normalization processing based on the aggregated matrix to determine the row normalized matrix; and perform symmetry processing on the row normalized matrix to determine the attention aggregation result.
[0145] In some specific embodiments, the semantic graphing module 12 can be used to: extract lexical embedding vectors from the last layer of the retrieval enhancement generation system to determine the vector extraction result; determine the similarity value between the lexical embedding vectors based on the vector extraction result and a preset semantic similarity analysis rule; linearly map the similarity value to a preset interval to determine the mapping result; use the lexical embedding vectors as nodes, and for any lexical embedding vector, filter neighbor nodes based on the corresponding similarity value to determine a target neighbor node set; the target neighbor node set includes a second preset number of neighbor nodes that are closest to the current lexical embedding vector; and connect each lexical embedding vector with the corresponding target neighbor node set using directed edges to determine the target semantic graph.
[0146] In some specific embodiments, the indicator value determination module 13 can be used to: perform deviation standardization processing on the attention aggregation result and the target semantic graph to determine the standardization processing result; perform domain calibration on the preset fusion parameters based on preset calibration rules to determine the calibrated parameters; perform weighted fusion based on the calibrated parameters and the standardization processing result to determine the weighted fusion result; filter the weighted fusion result based on a preset global threshold to complete the sparsity processing operation and obtain the sparsity processing result; perform symmetry processing on the sparsity processing result to determine the graph fusion result; the graph fusion result is an undirected graph.
[0147] In some specific embodiments, the index value determination module 13 can be specifically used to: obtain the undirected weighted adjacency matrix corresponding to the graph fusion result; wherein, the undirected weighted adjacency matrix is a matrix used to represent the adjacency relationship between nodes in the graph fusion result; determine the weighted degree corresponding to each node in the graph fusion result based on the undirected weighted adjacency matrix; determine the average weighted degree based on the weighted degree corresponding to each node; determine the degree distribution information using the weighted degree and the average weighted degree; the degree distribution information includes degree variance and skewness; and determine the undirected weighted adjacency matrix. Singular value decomposition is performed on the weighted adjacency matrix, and a third preset number of target singular values are selected from the decomposition results. Based on the third preset number of target singular values, the spectral energy ratio is determined. Based on the undirected weighted adjacency matrix, the distance matrix between nodes is transformed to determine the matrix transformation result. Based on the matrix transformation result, Vitoris-Lipps filtering is performed, and the manifold topological divergence is determined using the corresponding filtering result. Based on the weighted degree, the degree distribution information, the spectral energy ratio, and the manifold topological divergence, the current structural index value is determined.
[0148] In some specific embodiments, the hallucination detection result determination module 14 can be used to: concatenate the structural index values to determine the concatenated feature vector; obtain the target text fact tag corresponding to the generation result of the literature review generation task based on the text fact tag corresponding to the literature in the preset knowledge base; perform hallucination risk detection based on the preset hallucination detection model, the feature vector and the target text fact tag to determine the hallucination detection result; wherein, the hallucination detection result includes the hallucination risk score corresponding to each paragraph in the generation result.
[0149] In some specific embodiments, the hallucination detection device of the retrieval enhancement generation system can also be used to: based on the hallucination detection results and the generation results corresponding to the literature review generation task, to visualize the information coupling and semantic jump regions between different documents in the preset knowledge base of the retrieval enhancement generation system.
[0150] Furthermore, embodiments of this application also disclose an electronic device, Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.
[0151] Figure 4This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the hallucination detection method of the retrieval enhancement generation system disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0152] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0153] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0154] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the illusion detection method of the retrieval enhancement generation system executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.
[0155] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned hallucination detection method of the retrieval enhancement generation system. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.
[0156] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0157] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0158] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0159] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0160] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for hallucination detection in a retrieval-enhanced generation system, characterized in that, include: In the generation stage of the literature review generation task based on the retrieval enhancement generation system and the preset knowledge base, attention is aggregated across layers based on the multi-head attention matrix of each layer in the retrieval enhancement generation system to determine the attention aggregation result. Based on the last layer of the retrieval enhancement generation system, word embedding vectors are extracted, and semantic similarity analysis and mapping are performed using the corresponding vector extraction results to determine the target semantic graph. The attention aggregation result and the target semantic graph are weighted and fused, and the current structural index value is determined by using the corresponding graph fusion result and the preset graph topology index. Based on the preset hallucination detection model, the structural index value, and the text fact tags corresponding to the documents in the preset knowledge base, hallucination risk detection is performed to determine the hallucination detection result corresponding to the literature review generation task; the preset hallucination detection model is a model constructed based on any one of the logistic regression model, classification model, and tree model.
2. The hallucination detection method of the retrieval enhancement generation system according to claim 1, characterized in that, The process of performing cross-layer aggregation of attention based on the multi-head attention matrix of each layer in the retrieval enhancement generation system to determine the attention aggregation result includes: The attention matrix of multiple attention heads in each layer of the retrieval enhancement generation system is extracted to determine the attention matrix extraction result; Based on the attention matrix extraction results, the importance metric value corresponding to each attention head is determined; Based on the magnitude of the importance metric, the attention heads are sorted from largest to smallest to determine the attention head sorting result; Based on the attention head sorting results and the first preset number, attention heads are filtered to determine the set of filtered attention heads; For any attention head belonging to any layer in the attention head set, based on the attention matrix extraction result, matrix averaging aggregation is performed between attention heads to determine the global attention matrix of the current layer; Based on the preset layer set and the corresponding global attention matrix, cross-layer matrix averaging is performed to determine the aggregated matrix; Based on the aggregated matrix, row normalization is triggered to determine the row-normalized matrix. The row-normalized matrix is then symmetricized to determine the attention aggregation result.
3. The hallucination detection method of the retrieval enhancement generation system according to claim 1, characterized in that, The last layer of the retrieval enhancement generation system extracts word embedding vectors, and uses the corresponding vector extraction results to perform semantic similarity analysis and mapping, including: From the last layer of the retrieval enhancement generation system, extract the lexical embedding vector to determine the vector extraction result; Based on the vector extraction results and the preset semantic similarity analysis rules, the similarity values between the word embedding vectors are determined; The similarity values are linearly mapped to a preset interval to determine the mapping result; Using the word embedding vector as a node, for any word embedding vector, based on the corresponding similarity value, neighboring nodes are filtered to determine the target neighboring node set; the target neighboring node set includes a second preset number of neighboring nodes that are closest to the current word embedding vector; Each word embedding vector is connected to the corresponding target neighbor node set by directed edges to determine the target semantic graph.
4. The hallucination detection method of the retrieval enhancement generation system according to claim 1, characterized in that, The weighted fusion based on the attention aggregation result and the target semantic map includes: The attention aggregation result and the target semantic map are subjected to deviation normalization processing to determine the normalization result; Based on preset calibration rules, the preset fusion parameters are calibrated in the domain to determine the calibrated parameters; Based on the calibrated parameters and the standardized processing results, a weighted fusion is performed to determine the weighted fusion result; Based on a preset global threshold, the weighted fusion result is filtered to complete the sparsity processing operation and obtain the sparsity processing result. The sparsification result is symmetricized to determine the graph fusion result; the graph fusion result is an undirected graph.
5. The hallucination detection method of the retrieval enhancement generation system according to claim 1, characterized in that, The step of determining the current structural index value using the corresponding graph fusion results and preset graph topology indexes includes: Obtain the undirected weighted adjacency matrix corresponding to the graph fusion result; wherein, the undirected weighted adjacency matrix is a matrix used to represent the adjacency relationship between nodes in the graph fusion result; Based on the undirected weighted adjacency matrix, the weighting degree corresponding to each node in the graph fusion result is determined; The average weighted degree is determined based on the weighted degree corresponding to each node; The degree distribution information is determined using the weighted degree and the average weighted degree; the degree distribution information includes degree variance and skewness. Perform singular value decomposition on the undirected weighted adjacency matrix, and select a third preset number of target singular values from the corresponding decomposition results; Based on the third preset number of target singular values, the spectral energy ratio is determined; Based on the undirected weighted adjacency matrix, the distance matrix between nodes is transformed to determine the matrix transformation result; Based on the matrix transformation results, Vitoris-Lipps filtering is performed, and the manifold topological divergence is determined using the corresponding filtering results. Based on the weighted degree, the degree distribution information, the spectral energy ratio, and the manifold topological divergence, the current structural index value is determined.
6. The hallucination detection method of the retrieval enhancement generation system according to any one of claims 1 to 5, characterized in that, The process of performing hallucination risk detection based on a preset hallucination detection model, the structural index value, and the text fact tags corresponding to the documents in the preset knowledge base, to determine the hallucination detection result corresponding to the literature review generation task, includes: The structural index values are concatenated to determine the concatenated feature vector; Based on the text fact tags corresponding to the documents in the preset knowledge base, obtain the target text fact tags corresponding to the generation results of the literature review generation task; Based on a preset hallucination detection model and the feature vector with the target text fact labels, hallucination risk detection is performed to determine the hallucination detection result; wherein, the hallucination detection result includes hallucination risk scores corresponding to each paragraph in the generated result.
7. The hallucination detection method of the retrieval enhancement generation system according to claim 6, characterized in that, After determining the hallucination detection result corresponding to the literature review generation task, the method further includes: Based on the hallucination detection results and the generation results corresponding to the literature review generation task, the retrieval enhancement generation system visualizes the information coupling and semantic jump regions between different documents in the preset knowledge base.
8. A hallucination detection device for a retrieval enhancement generation system, characterized in that, include: The multi-head attention aggregation module is used to perform cross-layer aggregation of attention based on the multi-head attention matrix of each layer in the retrieval enhancement generation system during the generation stage of the literature review generation task based on the retrieval enhancement generation system and the preset knowledge base, so as to determine the attention aggregation result. The semantic graphing module is used to extract word embedding vectors based on the last layer of the retrieval enhancement generation system, and to use the corresponding vector extraction results to perform semantic similarity analysis and graphing to determine the target semantic graph. The indicator value determination module is used to perform weighted fusion based on the attention aggregation result and the target semantic graph, and to determine the current structural indicator value using the corresponding graph fusion result and the preset graph topology structure indicator; The hallucination detection result determination module is used to perform hallucination risk detection based on a preset hallucination detection model, the structural index value, and the text fact tags corresponding to the literature in the preset knowledge base, so as to determine the hallucination detection result corresponding to the literature review generation task. Hallucination risk detection is performed based on a preset hallucination detection model, the structural index value, and the text fact tags corresponding to the documents in the preset knowledge base.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the illusion detection method of the retrieval enhancement generation system as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the illusion detection method of the retrieval enhancement generation system as described in any one of claims 1 to 7.