A method of retrieval augmentation generation

Abstract

The present application belongs to the technical field of retrieval enhancement generation, and particularly relates to a retrieval enhancement generation method. The retrieval enhancement generation method comprises: performing first-level abnormality checking based on sensitive words, length and maximum character repetition rate in current demand; performing second-level abnormality checking according to semantic similarity between current demand index structure and the current demand after the first-level abnormality checking passes; performing composite risk scoring after the second-level abnormality checking passes; when the composite risk scoring is normal demand, recalling retrieval information to form a retrieval information set, and generating a preliminary reply text of the current demand based on the retrieval information set; after the preliminary reply text is divided into a plurality of reply segments, normal segments and illusion segments in the reply segments are determined; after the illusion segments are modified according to the context of the illusion segments and the retrieval information set, a modified text is obtained and output to a user. The present application can greatly improve the accuracy of the final reply text.

Description

Technical Field

[0001] This invention belongs to the field of search enhancement generation technology, and particularly relates to a search enhancement generation method. Background Technology

[0002] Retrieval-Augmented Generation (RAG) models have made significant progress in tasks such as open-domain question answering, dialogue systems, and content generation. RAG models combine an external knowledge retrieval module with a Large Language Model (LLM) to generate answers based on recalled retrieval information.

[0003] Some requests are considered anomalous (e.g., requests generated by accidental clicks, requests containing offensive language, etc.). Handling these anomalous requests in the RAG model not only wastes significant computational resources but also threatens a civilized online environment and cybersecurity. However, judging a request as anomalous solely based on keywords is too inaccurate. Sometimes, users' short and incomplete requests are mistakenly identified as anomalous, resulting in many legitimate requests going unanswered, which is highly detrimental to the widespread adoption of the RAG system.

[0004] Furthermore, existing RAG models only retrieve relevant search information for the current need, and then the LLM large language model generates response text based on this search information. However, some needs are strongly correlated with historical needs. Due to user expression habits, some key information belonging to the current need may not appear in the current need but may appear in some historical needs. Therefore, these historical needs can help improve the accuracy and relevance of the retrieved search information in the RAG system, and further improve the accuracy of the final generated response, and should be effectively utilized.

[0005] Existing RAG systems generally lack verification of hallucinatory content in the generated responses, resulting in inaccurate information in the responses provided to users, leading to low accuracy of the responses.

[0006] Therefore, how to accurately identify abnormal requests and effectively utilize historical requests to improve the accuracy of the final response text has become an urgent problem to be solved in the field of search enhancement generation technology. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a search enhancement generation method that can accurately identify abnormal requests, effectively utilize historical requests, reduce illusory fragments in the response text, and improve the accuracy of the final response text.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A retrieval enhancement generation method includes the following steps:

[0010] Step 1: Perform a first-level anomaly check based on the sensitive words, length, and maximum character repetition rate in the current requirement; after the first-level anomaly check passes, construct the current requirement index structure based on historical requirements, and perform a second-level anomaly check on the current requirement based on the semantic similarity between the current requirement index structure and the current requirement; after the second-level anomaly check passes, perform a composite risk score; if the composite risk score indicates a normal requirement, proceed to Step 2; otherwise, discard the current requirement.

[0011] Step 2: Based on the current needs and the current needs index structure, retrieve search information to form a search information set, and then generate a preliminary response text for the current needs based on the search information set;

[0012] Step 3: After dividing the initial response text into several response segments, identify the normal segments and hallucination segments within the response segments;

[0013] Step 4: Based on the context of the hallucination fragment and the retrieved information set, the hallucination fragment is corrected, and the corrected text is output to the user.

[0014] Preferably, in step 1, the specific content of the first-level anomaly check is as follows: the number of times the sensitive word appears in the current requirement q is less than j, and the character length of the current requirement q is within the first length threshold. The above, and the maximum percentage of repeated characters in the current requirement q, is based on the first repetition rate threshold. If the first level of exception check for the current requirement q is passed, then the first level of exception check for the current requirement q is considered to have failed and the current requirement q is considered to be an abnormal requirement.

[0015] Preferably, in step 1, the specific content of constructing the current demand index structure based on historical demands is as follows:

[0016] Step 11: After encoding the current demand q into a current demand vector v based on semantics, calculate the semantic similarity of the current demand vector v relative to all historical demand vectors. H represents the set of historical demand vectors, which includes all historical demand vectors except for the current demand vector v.

[0017] Step 12, if , If the dynamic drift threshold is set, it is determined that the current demand vector v has not undergone semantic drift relative to all historical demand vectors. Then, the current demand index structure is constructed based on the set of historical demand vectors H. ;like If the current demand vector v has undergone semantic drift relative to all historical demand vectors, then the memory scores of each historical demand vector relative to the current demand vector v are calculated and sorted in descending order. The historical demand vectors corresponding to the top k memory scores are then selected to construct the current demand index structure. k is a positive integer, and k < t:

[0018] ;

[0019] ;

[0020] in, Representing the historical demand vector The memory score relative to the current demand vector v; Indicates the weighting adjustment factor; This represents the time decay rate parameter; This represents the difference between the current demand vector v and the historical demand vector. The time difference between them; i and t are integers, and 1≤i≤t, the historical demand vector set H contains a total of t historical demand vectors, This represents the i-th historical demand vector generated in chronological order, where the current demand vector v is a historical demand vector. The next demand vector.

[0021] Preferably, in step 1, the specific content of the second-level anomaly check is as follows:

[0022] Step 11': Calculate the current demand vector v and the current average index vector using the cosine function. semantic similarity between ; where the current average index vector For the current requirement index structure The average value of all historical demand vectors accumulated in the data;

[0023] Step 12': Calculate the current demand vector v and the current demand index structure. Deviation between : If the deviation If the deviation is found to be within acceptable limits, then the second-level anomaly check for the current requirement q is deemed to have passed; if the deviation is within acceptable limits... If the second-level exception check for the current requirement q fails, then the current requirement q is an exception requirement. This represents the first drift threshold.

[0024] Preferably, in step 1, the specific content of the composite risk score is as follows:

[0025] ;

[0026] in, This represents the composite risk score of the current demand q. Indicates the first risk factor; Indicates the second risk factor; This represents the third risk factor; among which ∈[0,1], ∈[0,1] and ∈[0,1] and + + =1; This represents the sensitive word parameter in the current requirement q. If there are no sensitive words in the current requirement q, then... =0, otherwise =1; This indicates the maximum percentage of repeated characters in the current requirement q; when If so, then the current demand q is determined to be a normal demand; Indicates the first risk threshold; when Then, ask the user follow-up questions and return to step 1; This indicates the second risk threshold; if the user replies to the follow-up question, the user's reply is treated as a new request, and this process continues until the latest request is determined to be a normal or abnormal request; when If the current requirement q is determined to be an abnormal requirement, discard the abnormal requirement and return to step 1.

[0027] Preferably, step 11 also includes the following specific details:

[0028] ;

[0029] ;

[0030] in, Represents the historical average demand vector; Represents the norm.

[0031] Preferably, step 12 also includes the following specific details:

[0032] ;

[0033] in, This represents the current demand vector v and the historical window. The mean similarity of each historical demand vector; historical window It consists of x historical demand vectors preceding the current demand vector v; This represents the current demand vector v and the historical window. The standard deviation of the mean similarity of the historical demand vectors in the data; α represents the sensitivity adjustment parameter; Indicates the lower bound of semantic similarity. Indicates the upper limit of semantic similarity; if Then, an approximate nearest neighbor search algorithm is used to construct the current demand index structure based on all historical demand vectors in the historical demand vector set H. .

[0034] Preferably, step 3 further includes the following sub-steps:

[0035] Step 31: Divide the preliminary response text (PRT) into several response segments:

[0036] PRT={R1,...,R y ,...,R n}; where 1≤y≤n, R y This represents the y-th response segment in the preliminary response text (PRT).

[0037] Step 32: Perform uncertainty assessment on each response segment:

[0038] ;

[0039] in, Reply fragment Uncertainty score; ,in, Reply fragment The f-th character in the sequence, 1≤f≤m; Reply fragment The character segment consisting of the first to (f-1)th characters; Indicates in an existing character field Generate characters under the condition The conditional probability;

[0040] Step 33, if Then determine the current response segment. For normal segments; τ represents the uncertainty threshold; if , indicating the current response segment If hallucinations are suspected, several verification questions are used to evaluate the current response fragment based on the second RAG model. Perform a hallucination assessment to determine the current response fragment. Is it a hallucination fragment?

[0041] Hallucination assessment also includes the following: comparing the verification question with the current response fragment. As input to the second RAG model, the output of the second RAG model is used as the answer to the corresponding validation question. The answers to all corresponding validation questions are then compared with the current response fragment. Overall consistency score :

[0042] ;

[0043] Among them, A( A() represents the set of answers to the verification question. )={ },in, Let A represent the set of answers. The z-th answer in ) is also the answer to the z-th verification question, and 1≤z≤p; E(·) represents encoding into a vector;

[0044] If the overall consistency score Then determine the current response segment. This is a hallucination fragment; if the overall consistency score is... Then determine the current response segment. This represents a normal segment; where δ represents the hallucination threshold.

[0045] Preferably, step 4 further includes the following sub-steps:

[0046] Step 41: Combine the initial response text marked with hallucination fragments, the retrieval information set SIC, the current demand q, and the demand index structure. The enhanced prompts are integrated with the rewritten prompts to form the third RAG model. The third RAG model generates corrected fragments for each hallucination fragment in the current preliminary response text based on the enhanced prompts.

[0047] Step 42: Calculate the hallucination fragments using the cosine similarity function. and corresponding corrected fragments Vector semantic similarity ;like Then use the current corrected fragment. Replace hallucination fragments ω represents the first similarity threshold; otherwise, return to step 41.

[0048] Step 43: When all the illusion fragments in the initial response text are replaced with the corresponding corrected fragments, this is recorded as the corrected text. The corrected text is then output to the user as the final response text for the current requirement q.

[0049] Preferably, step 43 is replaced by step 43': In step 43', when all the illusion fragments in the initial response text are replaced with the corresponding corrected fragments, it is recorded as the corrected text AT. The semantic similarity of each corrected fragment relative to the corrected text AT is calculated using the cosine similarity function. If the semantic similarity of all corrected fragments relative to the corrected text AT is above the second similarity threshold η, it is determined that the overall consistency verification of the current corrected text has passed, and the corrected text AT is output to the user as the final response text for the current requirement q; otherwise, return to step 41.

[0050] The beneficial effects of this invention are as follows:

[0051] (1) The retrieval enhancement generation method of the present invention can accurately identify abnormal needs, effectively utilize historical needs, reduce illusory fragments in the response text, and improve the accuracy of the final response text.

[0052] (2) The retrieval enhancement generation method of the present invention uses two-level anomaly verification and composite risk scoring to accurately filter out abnormal needs in layers. For needs that still cannot be determined after one round of two-level anomaly verification and composite risk scoring, the present invention will not blindly determine whether the needs are abnormal or not, but will use follow-up questions to allow users to supplement the current needs into new needs with more information before making a judgment.

[0053] (3) The retrieval enhancement generation method of the present invention, once it determines whether the current demand is abnormal or not, will only spend storage resources (abnormal demands will not become historical demands) and computing costs on normal demands. Therefore, for processing a large number of demands, this not only saves storage resources but also saves computing costs. Not processing abnormal demands further ensures network security and a civilized network environment.

[0054] (4) The retrieval enhancement generation method of the present invention has significantly reduced the proportion of hallucination fragments in the subsequent preliminary response text by accurately filtering out abnormal needs and not processing the abnormal needs. This improves the accuracy of the final response text.

[0055] (5) In the retrieval enhancement generation method of the present invention, as long as it is determined to be an abnormal demand, it will not become a historical demand, thus avoiding the interference of abnormal demands on the accuracy of subsequent demand index structure construction.

[0056] (6) In the retrieval enhancement generation method of the present invention, some / all of the historical requirements related to the current requirements are extracted to construct the current requirement index structure, and the dynamic drift threshold used in the process of constructing the current requirement index structure is... It is also a value that changes dynamically within a certain range based on historical and current needs. Therefore, the constructed demand index structure can better capture the demands that are truly related to the current demand in the historical demands, and more effectively use historical demands to reduce illusory fragments in the initial response text and improve the accuracy of the final response text.

[0057] (7) In S43, the present invention also performs semantic similarity calculation on the rewritten corrected text AT to further ensure that the rewritten corrected fragment does not deviate significantly from the entire corrected text AT in semantics. This also provides a backup for the correction of the illusion fragment and further ensures the accuracy of the final response text.

[0058] (8) The two-level anomaly verification and composite risk scoring of the present invention:

[0059] ① The first-level anomaly check uses simple sensitive words, length, and maximum character repetition rate to quickly determine whether the current requirement is an anomalous requirement with minimal computation. Only requirements that pass the first-level anomaly check will proceed to the second-level anomaly check. The second-level anomaly check compares the current requirement vector v with the current average index vector. The first level of anomaly detection uses semantic similarity to determine whether a requirement is abnormal. Only requirements that pass the second level of anomaly detection are included in the composite risk score calculation; otherwise, requirements that fail the second level are directly classified as abnormal. Since the first and second levels of anomaly detection only have two possible outcomes—either an abnormal requirement or further verification / calculation—they are considered hard checks. The composite risk score, on the other hand, is a flexible verification mechanism. It's a comprehensive score derived from the computational load of the first and second levels of anomaly detection. Based on this comprehensive score, it further determines which requirements that passed the first and second levels are abnormal, which are normal, and which require further user inquiry. Requirements requiring further user inquiry are not directly classified as normal or abnormal, which reflects the "flexibility" of the composite risk score.

[0060] ② The first and second levels of anomaly checks can only quickly and roughly remove obviously abnormal requirements. However, some requirements that pass the first and second levels of anomaly checks may still slip through the net. Therefore, in order to ensure that requirements with less obvious anomalies are discovered without false positives, a composite risk scoring mechanism is used as a flexible verification mechanism to further determine the anomalies of requirements that pass the two levels of anomaly checks. Requirements with a high composite risk score but not reaching the abnormal requirement range cannot be directly determined to be abnormal. Therefore, this invention asks the user to further refine and explain the current requirement to form a new requirement with richer content and more information, and then reprocesses the new requirement from the beginning. The anomalies of the basic new requirement can be directly determined by the two levels of anomaly checks and the composite risk scoring. If this is not possible, this cycle is repeated until the latest requirement contains enough information to directly determine its anomaly.

[0061] ③ If neither the first-level nor the second-level anomaly check passes, then there is no need to spend computational resources to calculate the composite risk score; that is, the present invention can determine the anomaly of the current requirement as quickly as possible with the lowest computational cost. Attached Figure Description

[0062] Figure 1 This is a flowchart of a retrieval enhancement generation method according to the present invention;

[0063] Figure 2 This is a graph comparing the response accuracy of the present invention with that of the prior art. Detailed Implementation

[0064] To make the technical solution of the present invention clearer and more explicit, the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Solutions derived by those skilled in the art through equivalent substitution and conventional reasoning of the technical features of the present invention without creative effort all fall within the protection scope of the present invention.

[0065] like Figure 1 The diagram shows a flowchart of a retrieval enhancement generation method according to the present invention, which includes the following steps:

[0066] S1: Construct the current demand index structure based on historical demands, and perform two-level anomaly checks and composite risk scoring on the current demand based on the current demand index structure. If the two-level anomaly checks pass and the composite risk score is a normal demand, proceed to S2; if any level of anomaly check fails or the composite risk score is an abnormal demand, discard the current demand.

[0067] S2. According to the current requirement and the current requirement index structure, recall the retrieval information to form a retrieval information set SIC, and then generate a preliminary response text for the current requirement based on the retrieval information set SIC;

[0068] S3. After splitting the preliminary response text into several response segments, determine the normal segments and hallucination segments in the response segments;

[0069] S4. According to the context of the hallucination segment and the retrieval information set SIC, correct the hallucination segment, and then obtain the corrected text and output it to the user.

[0070] In S1, the following sub-steps are included:

[0071] S11. After obtaining the current requirement, perform a first-level anomaly check on the current requirement q based on sensitive words, length, and maximum character repetition rate. If the first-level anomaly check of the current requirement passes, then execute S12; otherwise, discard the current requirement q and return to S11;

[0072] S12. Construct the current requirement index structure based on historical requirements;

[0073] S13. Based on the semantic similarity between the current requirement index structure and the current requirement, perform a second-level anomaly check on the current requirement. If the second-level anomaly check of the current requirement passes, then execute S14; otherwise, discard the current requirement q and return to S11;

[0074] S14. Calculate the composite risk score of the current requirement q, and take corresponding risk resistance actions according to the composite risk score. Until the risk resistance action determines that the current requirement is a normal requirement, execute S2.

[0075] In S11, the following content is also included:

[0076] The number of occurrences of sensitive words in the current requirement q is less than j times, and the character length of the current requirement q is above the first length threshold above, and the maximum proportion of repeated characters in the current requirement q is below the first repetition rate threshold below, then it is determined that the first-level anomaly check of the current requirement q passes, and S12 is executed; otherwise, it is determined that the first-level anomaly check of the current requirement q fails, the current requirement q is determined to be an abnormal requirement and discarded, and then return to S11.

[0077] In this embodiment, j = 3, = 5 characters, .

[0078] There may be multiple different repeated characters in a requirement, such as "de", "wo", etc. The proportion of each repeated character in the current requirement is also different. In this invention, the maximum proportion of repeated characters in the current requirement q is taken to compare with the first repetition rate threshold Compare them.

[0079] If a request contains an excessively high percentage of repeated characters, too few words, or multiple instances of sensitive words that shouldn't appear, it will be flagged as an abnormal request during the first level of exception checking. For example, accidental clicks could cause a high percentage of repeated characters or too few words in the request; malicious inquiries could also lead to the inclusion of multiple sensitive words in the request.

[0080] S12 also includes sub-steps S121 to S122:

[0081] S121, after encoding the current demand q into a current demand vector v based on semantics, calculate the semantic similarity of the current demand vector v relative to all historical demand vectors. H represents the set of historical demand vectors, which includes all historical demand vectors except for the current demand vector v.

[0082] In this embodiment, the SentenceTransformer semantic encoder is used to encode the requirements into a vector.

[0083] S122, Calculate the current dynamic drift threshold ,like If the current demand vector v is determined to have no semantic drift relative to all historical demand vectors, then the current demand index structure is constructed based on the set of historical demand vectors H. ;

[0084] like If the current demand vector v is determined to have undergone semantic drift relative to all historical demand vectors, then all historical demand vectors are filtered, and the current demand index structure is constructed based on the filtered historical demand vectors. .

[0085] S121 also includes the following:

[0086] ;

[0087] ;

[0088] ;

[0089] Where i and t are integers, and 1 ≤ i ≤ t, the historical demand vector set H contains a total of t historical demand vectors. This represents the t-th historical demand vector generated in chronological order. This represents the t-th historical demand vector generated in chronological order; the current demand vector v is a historical demand vector. The next demand vector; Represents the historical average demand vector; Represents the norm.

[0090] S122 also includes the following:

[0091] ;

[0092] in, This represents the current demand vector v and the historical window. The mean similarity of each historical demand vector; historical window This is a historical window for the current demand vector v, where historical demand vectors are arranged in chronological order. If there are more than x historical demand vectors preceding the current demand vector v, then the historical window... It contains x historical demand vectors, which are as follows: ,..., If the number of historical demand vectors preceding the current demand vector v is less than x, then the historical window... It contains t historical demand vectors, which are as follows: ,..., ; This represents the current demand vector v and the historical window. The standard deviation of the mean similarity of the historical demand vectors in the data; α represents the sensitivity adjustment parameter; Indicates the lower bound of semantic similarity. This indicates the upper limit of semantic similarity.

[0093] In this embodiment, α = 0.8. =0.65, =0.82, x=15.

[0094] As shown in S1, the dynamic drift threshold calculated based on the similarity of historical demand vectors... It is a constraint on the upper limit of semantic similarity. and semantic similarity lower bound The dynamic changes between them.

[0095] like Then, an approximate nearest neighbor search algorithm such as FAISS or Annoy is used to construct the current demand index structure based on all historical demand vectors in the historical demand vector set H. .

[0096] like Then, calculate the memory score of each historical demand vector relative to the current demand vector v, sort them in descending order, and select the historical demand vectors corresponding to the top k memory scores to construct the current demand index structure. k is a positive integer, and k < t:

[0097] ;

[0098] in, Representing the historical demand vector The memory score relative to the current demand vector v; This represents a weighting adjustment factor used to adjust the decay of semantic similarity over time. ; The time decay rate parameter controls the penalty for historical demand vectors that are retained in the index structure for a long time. It is set by technicians according to different scenario types. This represents the difference between the current demand vector v and the historical demand vector. The time difference between them.

[0099] In this embodiment, Take 0.76; Take 0.28.

[0100] Using approximate nearest neighbor search algorithms such as FAISS or Annoy, a current demand index structure is constructed based on k selected historical demand vectors. .

[0101] Using an approximate nearest neighbor search algorithm to construct an index structure based on several vectors is an existing technique, which will not be elaborated here.

[0102] Among all historical requests, there may be several that are strongly related to the current request. These historical requests help to further understand the current request and improve the accuracy and speed of recalling information fragments, as well as the accuracy and efficiency of generating response text, in the subsequent process of responding to the current request. Therefore, we use these historical requests that are strongly related to the current request as the index structure for the current request. The sub-step S12 details how to select from all historical requirements to construct the current requirement index structure. The historical needs.

[0103] Furthermore, the present invention is based on the current dynamic drift threshold. To filter out the indexes used to construct the current requirement index structure The historical demand vector, and the dynamic drift threshold It is not a constant value, but rather a changing value calculated based on the current demand vector v and all historical demand vectors. This is the dynamic drift threshold in this invention. upper limit of semantic similarity and semantic similarity lower bound The dynamic changes between these vectors are determined by the strength of the semantic correlation between the current demand vector v and all historical demand vectors, thus yielding the dynamic drift threshold. More flexible and better adapted to current needs, it combines the memory scores of each historical demand vector with the current demand vector v to select historical demands that are truly strongly correlated with the current demand vector v as the current demand index structure. If the dynamic drift threshold is set... Simply setting a fixed upper limit for semantic similarity This could make the filtering process too sluggish, causing the resulting index structure to miss many historical requirements that are strongly related to the current needs, resulting in inaccurate response texts. If the dynamic drift threshold is adjusted... Simply setting a fixed lower limit for semantic similarity This could make the filtering process overly sensitive, resulting in the subsequent index structure containing historical requirements that are not relevant to the current needs, thus increasing the subsequent computational burden and storage resource consumption.

[0104] S13 includes sub-steps S131 to S132:

[0105] S131, Calculate the current demand vector v and the current average index vector. semantic similarity between : ; where the current average index vector For the current requirement index structure The average value of all historical demand vectors accumulated in the data;

[0106] S132, Calculate the current demand vector v and the current demand index structure. Deviation between : ;in, Indicates the first drift threshold;

[0107] If deviation If the second-level anomaly check of the current requirement q is passed, then S14 is executed; if the deviation is... If the second-level exception check of the current requirement q fails, the current requirement q is determined to be an abnormal requirement and is discarded, and the process returns to S11.

[0108] In this embodiment, .

[0109] Current demand vector v and current average index vector The semantic similarity between them is used to reflect the relationship between the current demand vector v and the current demand index structure. The degree of semantic leap between them is inversely proportional.

[0110] S14 also includes the following:

[0111] ;

[0112] in, This represents the composite risk score of the current demand q. Indicates the first risk factor; Indicates the second risk factor; This represents the third risk factor; among which ∈[0,1], ∈[0,1] and ∈[0,1] and + + =1; This represents the sensitive word parameter in the current requirement q. If there are no sensitive words in the current requirement q, then... =0, otherwise =1; This indicates the maximum percentage of repeated characters in the current requirement q;

[0113] when If so, then the current demand q is determined to be a normal demand; This indicates the first risk threshold.

[0114] when Then, after asking the user to "please further explain the purpose of the problem, and we will do our best to provide support", we return to S11; This indicates the second risk threshold. If the user replies to the follow-up question, the user's reply is treated as a new requirement. Processing of new requirements begins from S11 and continues in a loop until the latest requirement is determined to be a normal or abnormal requirement.

[0115] when If the current requirement q is determined to be an abnormal requirement, it is discarded, and the process returns to S11.

[0116] In this embodiment, =0.6, =0.8.

[0117] If the user no longer responds to follow-up questions but instead raises new requests, the present invention will also start processing the new requests from S1.

[0118] The first level of anomaly detection uses simple criteria like sensitive words, length, and maximum character repetition rate to quickly determine if a request is abnormal with minimal computation. Only requests that pass the first level of anomaly detection will proceed to the second level of anomaly detection. The second level of anomaly detection compares the current request vector v with the current average index vector. The first level of anomaly detection uses semantic similarity to determine whether a requirement is abnormal. Only requirements that pass the second level of anomaly detection are included in the composite risk score calculation; otherwise, requirements that fail the second level are directly classified as abnormal. Since the first and second levels of anomaly detection only have two possible outcomes—either an abnormal requirement or further verification / calculation—they are considered hard checks. The composite risk score, on the other hand, is a flexible verification mechanism. It's a comprehensive score derived from the computational load of the first and second levels of anomaly detection. Based on this comprehensive score, it further determines which requirements that passed the first and second levels are abnormal, which are normal, and which require further user inquiry. Requirements requiring further user inquiry are not directly classified as normal or abnormal, which reflects the "flexibility" of the composite risk score.

[0119] The first and second levels of anomaly checks can only quickly and roughly remove obviously abnormal requirements. However, some requirements that pass the first and second levels of anomaly checks may still slip through the net. Therefore, in order to ensure that requirements with less obvious anomalies are discovered without false positives, a composite risk scoring mechanism is used as a flexible verification mechanism to further determine the anomaly status of requirements that pass the two levels of anomaly checks. Requirements with a high composite risk score but not reaching the abnormal requirement range cannot be directly determined to be abnormal. Therefore, this invention asks the user to further refine and explain the current requirement to form a new requirement with richer content and more information, and then reprocesses the new requirement starting from S1. The anomaly status of the basic new requirement can be directly determined by the two levels of anomaly checks and the composite risk scoring. If it still cannot be determined, this cycle is repeated until the latest requirement contains enough information to directly determine its anomaly status.

[0120] If neither the first-level nor the second-level anomaly check passes, then there is no need to expend computational resources to calculate the composite risk score; that is, this invention can determine the current abnormal situation of the requirement as quickly as possible with the lowest computational cost.

[0121] On the test set, the present invention achieved an accuracy of 86.7% in judging abnormal demand situations, an F1 score of 83.3%, and a balanced accuracy of 86.0%, all of which are significantly better than the prior art, greatly improving the accuracy of judging abnormal demand.

[0122] It is important to emphasize here that discarded requirements do not become historical requirements; only normal requirements or requirements that are followed up on become historical requirements. This ensures that abnormal requirements will not become historical requirements in the future, and will not interfere with the invention's judgment of abnormal situations for subsequent new requirements and the construction of the corresponding requirement index structure. Once a new requirement is generated, the current requirement vector v becomes the (t+1)th historical requirement vector, and we return to S1. The historical requirement vector set H also contains (t+1) historical requirement vectors.

[0123] Optionally, while discarding the current request q, the user can be informed that "input content that does not conform to the specifications has been detected. To ensure security, processing has been suspended."

[0124] Some unusual requests contain too many meaningless characters. If these unusual requests are processed to obtain response texts, not only will the response texts be meaningless, but the subsequent processing models will gradually overfit, reducing the accuracy of response texts for normal requests. Other unusual requests contain a lot of sensitive and offensive content. If these unusual requests are processed to obtain response texts, they will pose a threat to a civilized online environment and network security. Subsequent processing could include recalling relevant search information and using a large language model to fuse relevant search information to generate response texts.

[0125] Therefore, accurately identifying abnormal demands can not only improve the accuracy of subsequent retrieval of relevant search information fragments, but also improve the accuracy of the final generated response text.

[0126] In S2: Combine the current demand vector v and the current demand index structure. The inputs to the first RAG (Retrieval Enhancement Generation) model are the current demand vector v and the current demand index structure. After incorporating enhanced prompts from the first RAG model, a control strategy based on an interactive defense mechanism is employed to recall relevant search information, forming a Search Information Set (SIC). Then, an LLM (Large Language Model) is used to generate a preliminary response text (PRT) based on the relevant SIC. This implementation uses the LLaMA-2 model.

[0127] Based on the need for enhanced prompts, the RAG system ultimately utilizes the LLM large language model to generate response text, which is an existing technology and will not be elaborated here.

[0128] Technical personnel have verified that this invention's demand index structure, which dynamically constructs based on current needs, is effective. The filtering of abnormal requests has greatly improved the accuracy and relevance of the retrieved information, which in turn has further improved the accuracy of the initial response text.

[0129] S3 also includes the following sub-steps:

[0130] S31, Divide the preliminary response text PRT into several response segments: PRT={R1,...,R...} y ,...,R n}; where R y This represents the y-th response segment in the preliminary response text (PRT), R. n This represents the nth response segment in the preliminary response text (PRT), where y and n are both positive integers and 1 ≤ y ≤ n.

[0131] In this embodiment, the dependency syntax tree is used to segment the preliminary response text (PRT). Tools such as spaCy, Stanza, and Benepar can also be used for dependency segmentation and subject-verb-object structure extraction.

[0132] S32, perform uncertainty assessment on each response segment:

[0133] ;

[0134] in, Reply fragment Uncertainty score; ,in, Reply fragment The f-th character in Reply fragment The m-th character in the array, where f and m are both positive integers, and 1 ≤ f ≤ m; Reply fragment The character segment consisting of the first to (f-1)th characters; Indicates in an existing character field Generate characters under the condition The conditional probability.

[0135] S33, if Then determine the current response segment. For normal segments; τ represents the uncertainty threshold; if , indicating the current response segment If hallucinations are suspected, several verification questions are used to evaluate the current response fragment based on the second RAG model. Perform a hallucination assessment to determine the current response fragment. Determine if it is a hallucination fragment, and mark the hallucination fragment in the initial response text.

[0136] In this embodiment, τ=1.2.

[0137] Hallucination assessment also includes the following:

[0138] The verification question and the current response segment are respectively. As input to the second RAG model, the output of the second RAG model is used as the answer to the corresponding validation question. The answers to all corresponding validation questions are then compared with the current response fragment. Overall consistency score :

[0139] ;

[0140] Among them, A( A() represents the set of answers to the verification question. )={ },in, Let A represent the set of answers. The z-th answer in the equation is also the answer to the z-th verification question. Let A represent the set of answers. The p-th answer in the equation is also the answer to the p-th verification question, where z and p are both positive integers and 1 ≤ z ≤ p; E(·) indicates that the answer / response fragment is encoded into a vector using the SentenceTransformer semantic encoder.

[0141] The verification questions in this embodiment are pre-set by technical personnel and include three categories: The first category is fact-checking verification questions, including "Is there any factual basis for the following statement? Please provide authoritative sources to support it." and "Is this statement accurate? Can source documents be found to support it?" The second category is semantic consistency verification questions, including "Is the following content consistent with known knowledge? Please provide credible documents to corroborate it." The third category is common-sense verification questions, including "Does this statement conform to common sense and logical reasoning?" This embodiment contains a total of four verification questions.

[0142] If the overall consistency score Then determine the current response segment. This represents a hallucination segment; where δ represents the hallucination threshold. The overall consistency score is considered. Then determine the current response segment. This is a normal segment.

[0143] In this embodiment, δ=0.72.

[0144] This invention achieves an accuracy of 93.2% and an F1 score of 86.9% in identifying hallucination fragments. The balanced accuracy is 92.1%. Furthermore, after identifying and correcting hallucination fragments, it significantly improves the scores of QAFactEval and Groundedness.

[0145] There are two common reasons for the existence of hallucinatory fragments in existing technologies: ① The LLM model fabricates data and / or documents that are not in the search information when generating response text. ② Hallucinatory fragments caused by abnormal needs.

[0146] As described above, the present invention has extremely high accuracy in identifying abnormal requests. Requests identified as abnormal will no longer be processed, which directly avoids hallucination fragments caused by abnormal requests. Compared with the hallucination fragments in the responses obtained by the prior art, the present invention has significantly reduced the number of hallucination fragments in the initial response text.

[0147] S4 also includes the following:

[0148] S41, the initial response text marked with hallucination fragments, the retrieval information set SIC, the current demand q, and the demand index structure are... The enhanced prompts, which are combined with the rewritten prompts, are used to generate corrected fragments for each hallucination fragment in the current initial response text.

[0149] The third RAG model outputs existing text based on enhanced prompts, which will not be elaborated here. The rewritten prompts are pre-set by technical personnel.

[0150] In this embodiment, the rewrite prompt P is "Rewrite the sentence realistically and credibly based on the following context and search information: "each hallucination fragment", "the normal context of each hallucination fragment", "search information set SIC", "current needs", "demand index structure"".

[0151] If there are more than one hallucination fragment in the preliminary response text, then the preceding normal fragment of a hallucination fragment refers to all normal fragments between that hallucination fragment and the previous adjacent hallucination fragment; the following normal fragment of a hallucination fragment refers to all normal fragments between that hallucination fragment and the next adjacent hallucination fragment; if a hallucination fragment is the first hallucination fragment in the preliminary response text, then all normal fragments to the left of that hallucination fragment are following normal fragments; if a hallucination fragment is the last hallucination fragment in the preliminary response text, then all normal fragments to the right of that hallucination fragment are following normal fragments.

[0152] S42, calculate the semantic similarity between all corrected segments and hallucination segments:

[0153] ;

[0154] in, This refers to a fragment of a hallucination; Indicating hallucination fragments The corresponding corrected fragment; E(·) indicates that the answer / response fragment is encoded into a vector using the Sentence-BERT semantic embeddinger; Indicating hallucination fragments With correction fragments semantic similarity;

[0155] like Then use the current corrected fragment. Replace hallucination fragments ω represents the first similarity threshold; otherwise, return to S41.

[0156] In this embodiment, ω=0.75.

[0157] S43, when all the illusion fragments in the initial response text are replaced with the corresponding corrected fragments, it is recorded as the corrected text. The corrected text is then output to the user as the final response text for the current requirement q.

[0158] Optionally, replace S43 with S43':

[0159] S43', when all the hallucination fragments in the initial response text are replaced with the corresponding corrected fragments, it is denoted as the corrected text AT. The semantic similarity of all corrected fragments relative to the corrected text is calculated:

[0160] ;

[0161] in, Indicates a corrected fragment The semantic similarity between the corrected text AT and the corrected text is calculated. If the semantic similarity of all corrected fragments to the corrected text is above the second similarity threshold η, the overall consistency verification of the current corrected text is deemed to have passed, and the corrected text is output to the user as the final response text for the current requirement q; otherwise, the process returns to S41.

[0162] Optionally, if the overall consistency verification of the current requirement q fails N consecutive revisions, the Nth revision will be output as the final response text to the user, and an uncertainty label will be added to the Nth revision and fed back to the technical staff. The uncertainty label will state "The generation of this content is uncertain and requires further verification."

[0163] In this embodiment, the second similarity threshold η=0.25 and N=7.

[0164] The search enhancement generation method of this invention and the search enhancement generation method of the prior art were used to respond to 3000 requests respectively. After review by technical personnel, the response accuracy of this invention and the response accuracy of the prior art are as follows: Figure 2As shown: Curve 1 is the fitting curve for the response accuracy of the present invention, and Curve 2 is the fitting curve for the response accuracy of the prior art. It can be seen that the response accuracy of the present invention is much higher than that of the prior art, and the response accuracy of the present invention is more stable, while the response accuracy of the prior art fluctuates greatly. That is, the present invention can effectively reduce the hallucinatory segments in the final response text, thereby significantly improving the accuracy of the final response text and ensuring the stability of the response accuracy.

[0165] The retrieval enhancement generation method of the present invention can accurately identify abnormal requests, effectively utilize historical requests, reduce illusory fragments in the response text, and improve the accuracy of the final response text.

[0166] The retrieval enhancement generation method of the present invention uses two-level anomaly verification and composite risk scoring to accurately filter out abnormal needs in a hierarchical manner. For needs that still cannot be determined after one round of two-level anomaly verification and composite risk scoring, the present invention will not blindly determine whether the need is abnormal or not, but will use follow-up questions to allow users to supplement the current need into a new need with more information before making a judgment.

[0167] The retrieval enhancement generation method of this invention, once determining whether a current request is abnormal, will only allocate storage resources (abnormal requests will not become historical requests) and computational costs to normal requests. Therefore, for processing a large number of requests, this not only saves storage resources but also reduces computational costs. By not further processing abnormal requests, it also further ensures network security and a civilized network environment.

[0168] The retrieval enhancement generation method of the present invention significantly reduces the proportion of hallucinatory fragments in the subsequent preliminary response text by accurately filtering out abnormal requests and not processing them further (during the processing of abnormal requests, the retrieved retrieval information is also inaccurate, and the content of hallucinatory fragments in the obtained response text is also extremely high; while the present invention only processes normal requests, and the accuracy and relevance of the retrieved retrieval information are also greatly increased, so the proportion of hallucinatory fragments in the subsequent preliminary response text will naturally decrease significantly), thus improving the accuracy of the final response text.

[0169] In the retrieval enhancement generation method of the present invention, once a request is determined to be abnormal, it will not become a historical request, thus avoiding the interference of abnormal requests with the accuracy of subsequent request index structure construction.

[0170] In the retrieval enhancement generation method of this invention, some / all of the historical requirements related to the current requirement are extracted to construct the current requirement index structure, and the dynamic drift threshold used in the process of constructing the current requirement index structure is... It is also a value that changes dynamically within a certain range based on historical and current needs. Therefore, the constructed demand index structure can better capture the demands that are truly related to the current demand in the historical demands, and more effectively use historical demands to reduce illusory fragments in the initial response text and improve the accuracy of the final response text.

[0171] For the initial response text, this invention first uses uncertainty assessment to determine which segments might be potential hallucination segments, and then uses hallucination assessment to further determine which of the potential hallucination segments are genuine hallucination segments, thus improving the accuracy of hallucination judgment. Furthermore, based on the normal segments in the context of the current initial response text and the recalled search information, the hallucination segments are rewritten, reducing the proportion of hallucination segments in the final response text and improving the accuracy of the final response text.

[0172] In S43', the present invention also performs semantic similarity calculation on the rewritten corrected text AT to further ensure that the rewritten corrected fragment does not deviate significantly from the entire corrected text AT in semantics. This also provides a backup for the correction of the hallucination fragment, further ensuring the accuracy of the final response text.

[0173] The technologies, shapes, and structures not described in detail in this invention are all well-known technologies. It should also be noted that the above are merely preferred embodiments of this invention and are not intended to limit the scope of the invention. The components or steps in the embodiments of this invention can be decomposed and / or recombined, and these decompositions and / or recombinations should be considered equivalent solutions to this application and should all fall within the protection scope of this invention.

Claims

1. A retrieval enhancement generation method, characterized in that, Includes the following steps: Step 1: Perform first-level anomaly detection based on sensitive words, length, and maximum character repetition rate in the current requirements; After the first level of anomaly check passes, the current requirement index structure is built based on historical requirements, and the current requirement is subjected to a second level of anomaly check based on the semantic similarity between the current requirement index structure and the current requirement. After the second-level anomaly check passes, a composite risk score is performed. If the composite risk score indicates normal demand, proceed to step 2; otherwise, discard the current demand. Step 2: Based on the current needs and the current needs index structure, retrieve search information to form a search information set, and then generate a preliminary response text for the current needs based on the search information set; Step 3: After dividing the initial response text into several response segments, identify the normal segments and hallucination segments within the response segments; Step 4: Based on the context of the hallucination fragment and the retrieved information set, the hallucination fragment is corrected, and the corrected text is obtained and output to the user; In step 1, the specific content of the composite risk score is as follows: ; in, This represents the composite risk score of the current demand q. Indicates the first risk factor; Indicates the second risk factor; This represents the third risk factor; among which ∈[0,1], ∈[0,1] and ∈[0,1] and + + =1; This represents the sensitive word parameter in the current requirement q. If there are no sensitive words in the current requirement q, then... =0, otherwise =1; This represents the maximum percentage of repeated characters in the current requirement q; based on semantics, the current requirement q is encoded into a current requirement vector v. This indicates the deviation between the current demand vector v and the current demand index structure; when If so, then the current demand q is determined to be a normal demand; Indicates the first risk threshold; when Then ask the user follow-up questions and return to step 1; Indicates the second risk threshold; If a user replies to a follow-up question, the user's reply is treated as a new requirement and this process continues until the latest requirement is determined to be a normal or abnormal requirement. when If so, the current requirement q is determined to be an abnormal requirement; Discard the abnormal request and return to step 1.

2. The retrieval enhancement generation method according to claim 1, characterized in that, In step 1, the specific content of the first-level anomaly check is as follows: The number of times the sensitive words appear in the current requirement q is less than j, and the character length of the current requirement q is within the first length threshold. The above, and the maximum percentage of repeated characters in the current requirement q, is based on the first repetition rate threshold. If the first level of exception check for the current requirement q is passed, then the first level of exception check for the current requirement q is considered to have failed and the current requirement q is considered to be an abnormal requirement.

3. The retrieval enhancement generation method according to claim 1, characterized in that, In step 1, the specific details of constructing the current demand index structure based on historical demands are as follows: Step 11: After encoding the current demand q into a current demand vector v based on semantics, calculate the semantic similarity of the current demand vector v relative to all historical demand vectors. H represents the set of historical demand vectors, which includes all historical demand vectors except for the current demand vector v. Step 12, if , If the dynamic drift threshold is set, it is determined that the current demand vector v has not undergone semantic drift relative to all historical demand vectors. Then, the current demand index structure is constructed based on the set of historical demand vectors H. ;like If the current demand vector v has undergone semantic drift relative to all historical demand vectors, then the memory scores of each historical demand vector relative to the current demand vector v are calculated and sorted in descending order. The historical demand vectors corresponding to the top k memory scores are then selected to construct the current demand index structure. k is a positive integer, and k < t: ； ； in, Representing the historical demand vector The memory score relative to the current demand vector v; Indicates the weighting adjustment factor; This represents the time decay rate parameter; This represents the difference between the current demand vector v and the historical demand vector. The time difference between them; i and t are integers, and 1≤i≤t, the historical demand vector set H contains a total of t historical demand vectors, This represents the i-th historical demand vector generated in chronological order, where the current demand vector v is a historical demand vector. The next demand vector.

4. The retrieval enhancement generation method according to claim 2, characterized in that, In step 1, the specific content of the second-level anomaly check is as follows: Step 11': Calculate the current demand vector v and the current average index vector using the cosine function. semantic similarity between ; where the current average index vector For the current requirement index structure The average value of all historical demand vectors accumulated in the data; Step 12': Calculate the current demand vector v and the current demand index structure. Deviation between : If the deviation If the deviation is found to be within acceptable limits, then the second-level anomaly check for the current requirement q is deemed to have passed; if the deviation is within acceptable limits... If the second-level exception check for the current requirement q fails, then the current requirement q is an exception requirement. This represents the first drift threshold.

5. The retrieval enhancement generation method according to claim 3, characterized in that, Step 11 also includes the following specific details: ; ;in, Represents the historical average demand vector; Represents the norm.

6. The retrieval enhancement generation method according to claim 3, characterized in that, Step 12 also includes the following specific details: ； in, This represents the current demand vector v and the historical window. The mean similarity of each historical demand vector; historical window It consists of x historical demand vectors preceding the current demand vector v; This represents the current demand vector v and the historical window. The standard deviation of the mean similarity of the historical demand vectors in the data; α represents the sensitivity adjustment parameter; Indicates the lower bound of semantic similarity. Indicates the upper limit of semantic similarity; like Then, an approximate nearest neighbor search algorithm is used to construct the current demand index structure based on all historical demand vectors in the historical demand vector set H. .

7. The retrieval enhancement generation method according to claim 1, characterized in that, Step 3 also includes the following sub-steps: Step 31: Divide the preliminary response text (PRT) into several response segments: PRT={R1,...,R y ,...,R n }; where 1≤y≤n, R y This represents the y-th response segment in the preliminary response text (PRT). Step 32: Perform uncertainty assessment on each response segment: ; in, Reply fragment Uncertainty score; ,in, Reply fragment The f-th character in the sequence, 1≤f≤m; Reply fragment The character segment consisting of the first to (f-1)th characters; Indicates in an existing character field Generate characters under the condition The conditional probability; Step 33, if Then determine the current response segment. For normal segments; τ represents the uncertainty threshold; if , indicating the current response segment If hallucinations are possible, several verification questions are used to evaluate the current response fragment based on the second RAG model. Perform a hallucination assessment to determine the current response fragment. Is it a hallucination fragment? Hallucination assessment also includes the following: comparing the verification question with the current response fragment. As input to the second RAG model, the output of the second RAG model is used as the answer to the corresponding validation question. The answers to all corresponding validation questions are then compared with the current response fragment. Overall consistency score : ； Among them, A( A() represents the set of answers to the verification question. )={ },in, Let A represent the set of answers. The z-th answer in ) is also the answer to the z-th verification question, and 1≤z≤p; E(·) represents encoding into a vector; If the overall consistency score Then determine the current response segment. This is a hallucination fragment; if the overall consistency score is... Then determine the current response segment. The segment represents a normal segment; where δ represents the hallucination threshold.

8. The retrieval enhancement generation method according to claim 1, characterized in that, Step 4 also includes the following sub-steps: Step 41: Combine the initial response text marked with hallucination fragments, the retrieval information set SIC, the current demand q, and the demand index structure. The enhanced prompts are integrated with the rewritten prompts to form the third RAG model. The third RAG model generates corrected fragments for each hallucination fragment in the current preliminary response text based on the enhanced prompts. Step 42: Calculate the hallucination fragments using the cosine similarity function. and corresponding corrected fragments Vector semantic similarity ;like Then use the current corrected fragment. Replace hallucination fragments ω represents the first similarity threshold; otherwise, return to step 41. Step 43: When all the illusion fragments in the initial response text are replaced with the corresponding corrected fragments, this is recorded as the corrected text. The corrected text is then output to the user as the final response text for the current requirement q.

9. The retrieval enhancement generation method according to claim 8, characterized in that, Replace step 43 with step 43': In step 43', when all the illusion fragments in the initial response text are replaced with the corresponding corrected fragments, denoted as the corrected text AT, the semantic similarity of each corrected fragment relative to the corrected text AT is calculated using the cosine similarity function. If the semantic similarity of all corrected fragments relative to the corrected text AT is above the second similarity threshold η, then the overall consistency verification of the current corrected text is deemed to have passed, and the corrected text AT is output to the user as the final response text for the current requirement q; otherwise, return to step 41.

Images