A retrieval augmentation generation method, device, apparatus and storage medium
By dynamically adjusting the evidence threshold and conducting multi-dimensional evaluation, the access problem of the retrieval enhancement generation system under different risk levels has been solved, improving the accuracy and credibility of the generated results and achieving a balance between the system's security and usability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING DIGITAL CHINA CLOUD COMPUTING CO LTD
- Filing Date
- 2026-03-05
- Publication Date
- 2026-07-21
Smart Images
Figure CN121786069B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of search enhancement generation technology, and in particular to a search enhancement generation method, apparatus, device and storage medium. Background Technology
[0002] Retrieval-Augmented Generation (RAG) systems, as a technology to suppress the illusion of generative AI (Artificial Intelligence) and improve the accuracy and credibility of content generation, have been widely applied in various risk-level scenarios such as medical diagnosis, legal consultation, and general information question answering.
[0003] Currently, mainstream RAG systems all adopt preset and fixed evidence admission standards. The industry's conventional implementation is a static control mechanism, which includes determining the validity of evidence through static similarity thresholds and screening candidate evidence using Top-K truncation. This type of mechanism is uniformly applied to the query evidence processing flow in different scenarios.
[0004] This static evidence access control mechanism, which is applicable to all scenarios, cannot take into account the differentiated processing needs of queries with different risk levels, and is prone to causing the following problems: First, in high-risk scenarios such as medical diagnosis and legal consultation, existing static admission standards are often not strict enough, which can easily lead to the adoption of some misleading evidence that is highly similar but lacks authority or timeliness, thus inducing hallucinations. Second, in low-risk or general information question-and-answer scenarios, if high static admission thresholds are deliberately set to adapt to high-risk scenarios, the standards may be too conservative, filtering out a large amount of valid evidence that is slightly less relevant but still has practical reference value, resulting in an excessively high system rejection rate and unnecessary refusal to answer questions. Summary of the Invention
[0005] This application provides a retrieval enhancement generation method, apparatus, device, and storage medium that can achieve a balance between high-risk and low-risk scenarios, improving the system's adaptability in different environments.
[0006] To achieve the above objectives, this application adopts the following technical solution: Firstly, this application provides a retrieval enhancement generation method, including: Get the query request; Based on the query domain and query sensitivity corresponding to the query request, the query risk value is determined, and based on the number of entities corresponding to the query request and the number of relationships between entities, the query complexity is determined. Based on the query risk value and the query complexity, a dynamic evidence threshold is determined; Obtain the quality values of multiple pieces of evidence corresponding to the query request; Evidence with a quality value greater than the dynamic evidence threshold is considered as the evidence set. Based on the aforementioned set of evidence, content generation is controlled.
[0007] Optionally, the quality value of evidence may be determined by the following methods: The authority of the source of the evidence, the time of its publication, and the semantic relevance of the evidence to the query request; The quality value of the evidence is determined based on the authority of its source, the time of its publication, and the semantic relevance of the evidence to the query request.
[0008] Optionally, controlling content generation based on the evidence set includes: Calculate the model sub-uncertainty value and the quality sub-uncertainty value of the evidence in the evidence set; The uncertainty value of the evidence set is determined based on the model sub-uncertainty value and the mass sub-uncertainty value. Content generation is controlled based on the uncertainty value of the evidence set.
[0009] Optionally, based on the uncertainty value of the evidence set, content generation is controlled, including: If the uncertainty value of the evidence set is greater than or equal to the first uncertainty threshold, then content generation is rejected. If the uncertainty value of the evidence set is less than the first uncertainty threshold, then content generation is performed.
[0010] Optionally, content generation can be performed, including: If the uncertainty value of the evidence set is greater than the second uncertainty threshold, the generation strategy is downgraded, and content is generated according to the downgraded strategy. If the uncertainty value of the evidence set is less than or equal to the second uncertainty threshold, then content generation is performed according to the initial generation strategy; Wherein, the second uncertainty threshold is less than the first uncertainty threshold.
[0011] Optionally, the method further includes: Get feedback on the generated content.
[0012] Optionally, the method further includes: Based on the feedback information, the weights of query risk value and query complexity are adjusted during the process of determining the dynamic evidence threshold.
[0013] Secondly, this application provides a retrieval enhancement generation apparatus, comprising: The retrieval module is used to retrieve query requests; The processing module is used to determine the query risk value based on the query domain and query sensitivity corresponding to the query request, determine the query complexity based on the number of entities corresponding to the query request and the number of relationships between entities, and determine the dynamic evidence threshold based on the query risk value and the query complexity. The acquisition module is also used to acquire the quality values of multiple pieces of evidence corresponding to the query request; The processing module is also used to collect evidence with a quality value greater than the dynamic evidence threshold as an evidence set; The generation module is used to control content generation based on the evidence set.
[0014] Thirdly, this application provides a computing device, including a memory and a processor; The memory stores one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the computing device performs the method as described in any one of the first aspects.
[0015] Fourthly, this application provides a computer-readable storage medium for storing a computer program for performing the method as described in any one of the first aspects.
[0016] As can be seen from the above technical solution, this application has at least the following beneficial effects: In this application, by combining the query domain and query sensitivity to determine the query risk value, and combining it with the query complexity calculated based on entity-related features, the evidence admission threshold is dynamically adjusted, achieving scenario-based adaptation of evidence screening standards. In high-risk scenarios such as medical diagnosis and legal consultation, the evidence admission threshold is automatically raised to block the adoption of misleading evidence such as low authority or outdated evidence from the source, reducing the risk of generating illusions and ensuring content security. In low-risk or general information question-and-answer scenarios, the threshold is appropriately relaxed to avoid over-filtering of valid evidence, reduce the system's rejection rate, improve service availability, and successfully balance the needs of different scenarios.
[0017] Furthermore, by comprehensively considering the source authority, publication time (timeliness), and semantic relevance of the evidence to the query, the quality value is determined, breaking through the limitations of traditional single semantic similarity assessment. This ensures that the selected evidence is not only relevant but also authoritative and fresh, providing high-quality support for the generated content and fundamentally improving the accuracy and credibility of the generated results.
[0018] Furthermore, by quantifying the uncertainty of the model sub-uncertainty and the uncertainty of the evidence quality sub-uncertainty, a comprehensive uncertainty assessment system is constructed. Based on the assessment results, a tiered generation control is implemented: generation is rejected when there is high uncertainty, downgraded when there is medium uncertainty, and normal when there is low uncertainty. This makes the system's generation behavior more transparent and controllable. It avoids the risks of ignorant and reckless statements, and provides valuable and cautious answers through the downgraded generation mechanism, reducing the experience loss caused by simply rejecting the system and increasing user trust in it.
[0019] Finally, by collecting user feedback on the generated content, the weights of query risk and query complexity in the dynamic threshold calculation are adjusted in reverse, forming a closed-loop optimization mechanism. This enables the system to continuously iterate based on feedback from actual usage scenarios, constantly adapting to real needs, and gradually improving the accuracy of evidence screening, generation accuracy, and scenario adaptability over long-term use.
[0020] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description
[0021] Figure 1 A flowchart illustrating a retrieval enhancement generation method provided in this application embodiment; Figure 2 A schematic diagram of a search enhancement generation device provided in an embodiment of this application; Figure 3 This is a schematic diagram of a computing device provided in an embodiment of this application. Detailed Implementation
[0022] The terms "first," "second," and "third," etc., used in this application specification and accompanying drawings are used to distinguish different objects, not to limit a specific order.
[0023] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0024] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the related technologies is given first: Retrieval-enhanced generation (RAG) systems are a technology system that integrates external knowledge retrieval with large language model generation capabilities. By retrieving relevant external evidence before generating content, it provides factual basis for the model, thereby effectively suppressing the illusion problem of generative AI and improving the accuracy and credibility of generated content. It is widely used in scenarios such as medical diagnosis, legal consultation, and general information question answering.
[0025] The static evidence admission standards used in existing search enhancement generation systems have obvious adaptation defects and cannot meet the differentiated needs of application scenarios with different risk levels. In high-risk scenarios such as medical diagnosis and legal consultation, the rigor of static standards is insufficient, which can easily lead to the adoption of misleading evidence such as low authority or outdated evidence, thereby causing the illusion of generation and creating security risks. In low-risk or general information question-and-answer scenarios, if high standard static thresholds are deliberately set to adapt to high-risk scenarios, it will over-filter valid evidence that is slightly less relevant but still has reference value, resulting in an excessively high rejection rate, unnecessary refusal to answer, and a serious reduction in service availability.
[0026] The main reason for this problem is that the design logic of the static evidence admission standard is out of touch with the actual application needs: on the one hand, it takes universality across all scenarios as its core, without considering the difference between the priority of evidence authority and timeliness in high-risk scenarios and the emphasis on evidence coverage in low-risk scenarios. The fixed rules cannot achieve scenario-based adaptation. On the other hand, traditional static standards rely heavily on single semantic similarity to evaluate evidence, ignoring key quality dimensions and lacking a mechanism for dynamic adjustment based on query features or usage feedback. In addition, the generation control lacks refined uncertainty quantification and intermediate state strategies, ultimately making it difficult for the system to balance the main contradiction between security and usability.
[0027] In view of this, embodiments of this application provide a retrieval enhancement generation method, which can be executed by a processing device. The processing device can be a terminal or a server. Terminals include, but are not limited to, smartphones, tablets, laptops, personal digital assistants, or smart wearable devices. The server can be a cloud server, such as a central server in a central cloud computing cluster or an edge server in an edge cloud computing cluster. Alternatively, the server can be a server in a local data center. A local data center refers to a data center directly controlled by the user.
[0028] To address the major contradiction in existing Search Enhanced Generation (RAG) systems where static evidence admission standards cannot simultaneously address the security and usability issues across different risk levels, this application first determines dynamic evidence admission thresholds by quantifying query risk and complexity. Then, it filters high-quality evidence through multi-dimensional evaluation, establishes an uncertainty quantification system to achieve tiered generation control, and finally optimizes threshold calculation parameters based on user feedback. This forms a closed-loop mechanism of perception, filtering, generation, and optimization, thereby strengthening security in high-risk scenarios and improving service availability in low-risk scenarios, while balancing the accuracy and controllability of generated content with the system's adaptability.
[0029] To make the technical solution of this application clearer and easier to understand, a search enhancement generation method provided by an embodiment of this application will be described below with reference to the accompanying drawings. Figure 1 As shown, this figure is a flowchart of a retrieval enhancement generation method provided in an embodiment of this application. The method includes: S201, Processing device obtains query request.
[0030] A query request is an instruction or question that a user initiates to the system in order to obtain specific information or solve a specific problem. It is presented in the form of natural language text used in everyday communication. It not only clearly contains the user's core needs and the target information they want to obtain, but also implies relevant application scenario characteristics. It serves as the basic input for all subsequent processing steps of the system, such as query analysis, evidence retrieval, and content generation. It is also the starting point for the system to understand the user's intent and provide accurate services.
[0031] The processing device receives user commands or questions in natural language, containing specific information needs, through pre-defined interactive interfaces (such as web page input boxes, APP interfaces, API interfaces, etc.). The purpose of this step is to transform the user's unstructured needs into input data that the system can process. This provides the foundation for subsequent processes such as risk and complexity quantification, dynamic threshold calculation, evidence retrieval and evaluation. It serves as a bridge connecting user needs and system services, ensuring that the system can accurately capture user intent and initiate subsequent adaptive processing logic.
[0032] S202. The processing device determines the query risk value based on the query domain and query sensitivity corresponding to the query request, and determines the query complexity based on the number of entities corresponding to the query request and the number of relationships between entities.
[0033] The query domain refers to the professional field or application scenario to which the query request belongs. It is one of the bases for judging the risk level of the query. Categories include medical, legal, financial, and general information Q&A. Different domains have significantly different requirements for the accuracy and security of information. For example, the medical field requires extremely high rigor, while general scenarios are relatively more lenient.
[0034] Query sensitivity refers to the degree to which the information or issues involved in a query request affect the user's rights and security. It is directly related to the sensitive entities in the query. The more sensitive entities there are and the greater the impact, the higher the query sensitivity.
[0035] The query risk value is a numerical result obtained by quantifying the risks and sensitivity of the query domain. It is an indicator that reflects potential security risks in the query processing process. The higher the value, the more stringent the evidence screening and risk control are required for the query.
[0036] An entity refers to the main object involved in a query request, which can be a person, thing, event, concept, etc., and is the basic unit that constitutes the semantics of the query.
[0037] Inter-entity relationships refer to the logical connections between different entities in a query request, reflecting the semantic complexity of the query.
[0038] Query complexity is a quantitative indicator calculated based on the number of entities and the number of relationships between entities in a query. It is used to characterize the complexity of the query semantics. The more entities there are and the more complex the relationships are, the higher the query complexity is, and the higher the requirements for evidence coverage and logical relevance are.
[0039] After receiving a user's query request, the processing equipment performs structured analysis and quantitative evaluation of the query from two dimensions. On the one hand, by identifying the professional field to which the query belongs (such as medical or legal) and the sensitivity of the query (such as whether it is related to the user's health or property safety), a comprehensive calculation is made to obtain the query risk value and determine the security control level required for the query. On the other hand, by statistically analyzing the number of entities contained in the query and parsing the relationship logic between these entities (the number of relationships between entities), the query complexity is determined, and the query's requirements for evidence coverage and logical integrity are judged.
[0040] The expression for calculating the risk value is:
[0041] in, This indicates a query for risk values. Indicates the weighting coefficient. This represents the risk benchmark value for the query domain. It is a preset basic risk score corresponding to the domain to which the query belongs (such as medical or legal). For example, it is set to 0.8 for the medical domain and 0.2 for general Q&A. This indicates the number of sensitive entities in the query. This represents the sensitivity score of the j-th sensitive entity. This indicates an exponential transformation of the sensitivity score of a single sensitive entity, amplifying the impact of highly sensitive entities on the overall risk.
[0042] The expression for calculating query complexity is:
[0043] in, Indicates the query complexity. This represents the first weighting coefficient. This indicates the total number of entities in the query. This represents the second weighting coefficient. This indicates the number of relationships between entities in the query. This indicates taking the larger value between 1 and the total number of entities, avoiding the error of a denominator of 0 when the number of entities is 0.
[0044] The purpose of this step is to transform the user's unstructured query into computable quantitative features (risk value, complexity), providing a basis for subsequent dynamic adjustment of evidence admission thresholds and accurate screening of evidence, and ensuring that the system can perform adaptive processing based on the specific features of the query.
[0045] S203. The processing device determines the dynamic evidence threshold based on the query risk value and query complexity.
[0046] The dynamic evidence threshold is an evidence admission threshold that is adjusted in real time based on the query risk value and query complexity, and is not a fixed value. Only when the quality value of the evidence is higher than the dynamic evidence threshold will it be adopted by the system for subsequent content generation; the dynamic evidence threshold is dynamically adapted to changes in query characteristics, rather than a fixed standard applicable to all scenarios.
[0047] After acquiring the query risk value and query complexity, the processing device combines these two quantitative characteristics and uses a preset algorithm model to calculate the dynamic evidence threshold corresponding to the current query. The query risk value reflects the security control level of the query, while the query complexity reflects the query's requirements for evidence coverage and logical integrity.
[0048] The expression for calculating the dynamic evidence threshold is:
[0049] in, Indicates the dynamic evidence threshold. Indicates the basic threshold. Indicates the weight of risk adjustment. Indicates the risk benchmark value. The weights represent the complexity adjustment. This represents the baseline value for complexity. This represents the timeliness correction factor, which is a correction coefficient based on the timeliness requirements of the query domain. For example, it can be set to 1.2 for the time-sensitive medical field and 1.0 for general scenarios.
[0050] Specifically, high-risk and highly complex queries (such as medical issues involving multiple entities related to disease and medication) will be matched with a higher dynamic evidence threshold. Only evidence that meets high standards in terms of authority, timeliness, and relevance will be included in the system's adoption scope. On the other hand, low-risk and low-complexity queries (such as general life knowledge questions) have a relatively lenient evidence threshold, and more valuable evidence can enter the subsequent processing flow.
[0051] The purpose of this step is to ensure that the evidence screening criteria are accurately matched with the actual needs of the current query. This can both avoid security risks caused by low-quality evidence in high-risk scenarios and prevent effective evidence from being over-filtered in low-risk scenarios, thus providing a basis for dynamic adjustment for subsequent accurate evidence screening.
[0052] S204. The processing device obtains the quality values of multiple pieces of evidence corresponding to the query request.
[0053] The quality score of evidence is a quantitative result of the comprehensive value of a single piece of evidence. It is calculated by integrating three dimensions: the authority of the source of the evidence, the timeliness of its release, and the semantic relevance to the query. The higher the score, the better the quality of the evidence and the stronger its support for the generated content.
[0054] The quality of evidence is determined in the following ways: First, consider the authority of the source of the evidence obtained by the processing device, the time of the evidence's release, and the semantic relevance of the evidence to the query request.
[0055] The authority of the source refers to the credibility and professional level of the entity publishing the evidence or the source of the information, and is one of the dimensions for assessing the quality of evidence. Authoritative sources include official institutions (such as industry regulatory agencies), top academic journals, professional databases, and official websites of well-known professional organizations, while non-authoritative sources may include personal blogs, ordinary social media posts, and content from non-professional platforms.
[0056] Publication time refers to the specific point in time when evidence is officially released or updated, and it serves as a basis for measuring the timeliness of evidence. In time-sensitive fields such as medicine, law, and finance, the more recent the publication date of evidence, the higher its reference value; while outdated evidence may lose its validity due to knowledge updates, policy adjustments, etc.
[0057] Semantic relevance refers to the degree to which the information contained in evidence matches the needs of a query request, reflecting the close semantic connection between the two. It is quantified by methods such as calculating the text vector similarity between the query and the evidence. Higher relevance indicates a stronger direct support from the evidence in answering the query question.
[0058] After retrieving multiple candidate evidences related to the query request, the processing device extracts information for each piece of evidence from three dimensions: first, the authority of the source of the evidence (to determine whether the publishing entity is professional and credible); second, the publication time of the evidence (to determine whether the information is fresh and valid); and third, the semantic relevance of the evidence to the query request (to determine whether the information meets the user's needs).
[0059] The purpose of this step is to collect evaluation indicators for subsequent calculation of evidence quality values. By collecting information from multiple dimensions, we can ensure that the evaluation of evidence quality is comprehensive and objective, and avoid misjudgment of low-quality evidence or omission of high-quality evidence due to a single-dimensional judgment.
[0060] Then, the processing device determines the quality value of the evidence based on the authority of its source, the time of its publication, and the semantic relevance of the evidence to the query request.
[0061] The formula for calculating the quality value of evidence is:
[0062] in, Indicates the quality value of the evidence. Weight coefficients representing semantic relevance. This represents the semantic relevance score between the evidence and the query request. The weighting coefficients represent the authority of the source. The score indicates the authority of the source of the evidence. The weighting coefficient represents the timeliness. This represents the half-life parameter. Indicates time difference, This indicates the time-related decay term.
[0063] S205. The processing device will use evidence with a quality value greater than the dynamic evidence threshold as the evidence set.
[0064] An evidence set refers to the combination of evidence that is ultimately adopted by the system after being filtered by a dynamic evidence threshold and used to support the generation of subsequent content. It is an information set composed of multiple valid pieces of evidence with quality values higher than the dynamic evidence threshold, and serves as the main factual basis for the generated content.
[0065] The processing device compares the quality value of each candidate piece of evidence with the dynamic evidence threshold corresponding to the current query, which was calculated earlier, and retains only the evidence whose quality value exceeds the dynamic evidence threshold. It then integrates these qualified pieces of evidence into a set of evidence.
[0066] In simple terms, this step filters candidate evidence. Evidence that meets the quality standards is included in the set, while evidence that does not meet the standards is excluded. The final evidence set not only meets the scenario-specific requirements of the current query for evidence quality (strict screening in high-risk scenarios and moderate relaxation in low-risk scenarios), but also ensures that the evidence in the set has sufficient authority, timeliness, and relevance, providing reliable factual support for the subsequent generation of accurate and credible content.
[0067] S206. The processing device controls content generation based on the evidence set.
[0068] Specifically, the processing device calculates the sub-uncertainty value of the model and the quality sub-uncertainty value of the evidence in the evidence set.
[0069] The model sub-uncertainty value is a quantitative result of the confidence risk of the large language model when generating content. It is calculated based on the prediction probability of tokens (text units) during the content generation process and reflects the uncertainty of the model about the generated content. The higher the value, the less confidence the model has in its own output content.
[0070] The quality sub-uncertainty value is a quantitative result of the quality risk of the evidence set. It is calculated based on the overall quality of the evidence set (such as the degree of conflict between evidence, information coverage, etc.) and reflects whether the evidence set can reliably and sufficiently support the query. The higher the value, the greater the quality risk of the evidence set.
[0071] The processing device assesses the potential risks of the generated content from two dimensions: on the one hand, it calculates the model sub-uncertainty value to measure the confidence of the large language model itself in the content to be generated; on the other hand, it calculates the quality sub-uncertainty value to measure the reliability and sufficiency of the current evidence set in supporting the query requirements.
[0072] The expression for calculating the model sub-uncertainty value is:
[0073] in, This represents the model's sub-uncertainty value. The token length represents the number of text units (such as words or characters) contained in the text generated by the model. This represents the predicted probability (value 0~1) when the model generates each token.
[0074] The expression for calculating the uncertainty value of the mass is:
[0075] in, This represents the uncertainty value of the mass. Indicates the first coefficient. This represents the average quality value of the evidence set, that is, the average quality value of all the evidence in the set, reflecting the overall quality level of the evidence. Indicates the second coefficient. This indicates the coverage of query information, quantifying the degree to which the evidence set covers the query requirements, with a value ranging from 0 to 1 (1 represents complete coverage of the query requirements).
[0076] The purpose of this step is to comprehensively assess the risk level of the generated content through two-dimensional uncertainty quantification, providing a basis for subsequent strategies of normal generation, downgraded generation, or rejection of generation, and avoiding risk misjudgment caused by relying solely on a single-dimensional assessment.
[0077] Then, the processing device determines the uncertainty value of the evidence set based on the model sub-uncertainty value and the quality sub-uncertainty value.
[0078] The uncertainty value of the evidence set is the final risk quantification result obtained by combining the uncertainty values of the model sub-uncertainty value and the quality sub-uncertainty value. It reflects the overall risk level of the entire process of evidence support and model generation, and serves as the basis for subsequent selection of generation strategies.
[0079] The expression for calculating the uncertainty value of the evidence set is:
[0080] in, This represents the uncertainty value of the evidence set. Used to adjust the proportion of mass uncertainty in the final result.
[0081] Finally, based on the uncertainty value of the evidence set, the content generation is controlled.
[0082] Specifically, if the uncertainty value of the evidence set is greater than or equal to the first uncertainty threshold, content generation is rejected; if the uncertainty value of the evidence set is less than the first uncertainty threshold, content generation is performed.
[0083] The first uncertainty threshold is a pre-set risk threshold used by the system to determine whether the risk of the generated content has exceeded an acceptable range. When the uncertainty value of the evidence set reaches or exceeds this first uncertainty threshold, it means that the evidence is insufficient or the model has extremely low confidence, and continuing to generate content will have a high risk of illusion or error.
[0084] When the uncertainty value of the evidence set is greater than or equal to the first uncertainty threshold, the system will directly refuse to generate an answer. This situation typically occurs in high-risk scenarios, such as when there are serious conflicts in the evidence during medical diagnosis, or when the model has absolutely no confidence in the generated content. In such cases, forcibly generating an answer could be misleading or even harmful. Refusing to generate an answer is to prevent unfounded statements and is the last line of defense for ensuring system security.
[0085] When the uncertainty value of the evidence set is less than the first uncertainty threshold, the system will enter the content generation stage. This indicates that the current evidence support and model confidence are within an acceptable risk range, capable of providing users with a relatively reliable answer. However, this does not mean that the final answer will be output directly; further adjustments will be made based on the specific range of the uncertainty value, deciding whether to generate normally or in a downgraded manner.
[0086] The purpose of this rule is to make the system's generation behavior transparent and controllable through clear quantitative standards, and to establish a clear decision boundary between security and availability.
[0087] If the uncertainty value of the evidence set is greater than the second uncertainty threshold, the generation strategy is downgraded and the content is generated according to the downgraded strategy; if the uncertainty value of the evidence set is less than or equal to the second uncertainty threshold, the content is generated according to the initial generation strategy; wherein, the second uncertainty threshold is less than the first uncertainty threshold.
[0088] The second uncertainty threshold is a risk threshold preset by the system that is lower than the first uncertainty threshold. It is used to further subdivide the risk level within the allowed range of generation, so as to achieve more refined generation strategy control.
[0089] The initial generation strategy is the default generation method of the system in low-risk scenarios. It usually outputs a complete and clear answer directly, along with evidence as a basis, which is the most ideal generation mode.
[0090] The generation strategy after downgrading is a cautious generation approach adopted in medium-risk scenarios. Common practices include: outputting answers with confidence statements (such as conclusions based on existing evidence and for reference only), providing only partial information, and clearly indicating the limitations of the evidence. The purpose is to alert users to potential risks and avoid misleading them.
[0091] This passage describes a refined risk control rule under the premise of allowing generation. It divides risk into two ranges, low risk and medium risk, using a second uncertainty threshold to match different generation strategies: When the uncertainty value of the evidence set exceeds the second uncertainty threshold but falls below the first uncertainty threshold, the system triggers a downgraded generation strategy. This typically occurs in scenarios where the evidence supports an answer but lacks sufficient coverage or the model has moderate confidence, such as when multiple pieces of evidence exist for a general knowledge question but have slightly conflicting details. Downgraded generation avoids the harshness of direct rejection while clearly indicating the risk to the user, allowing them to understand the reliability of the information.
[0092] When the uncertainty value of the evidence set is less than or equal to the second uncertainty threshold, the system executes the initial generation strategy. This indicates that the evidence set is of high quality, the model has sufficient confidence, and the risk of the generated content is at a minimum. Therefore, it can directly output a clear and definite answer, along with complete evidence citations, providing users with the highest quality information service.
[0093] The purpose of this rule is to make the system's generation behavior no longer a black-and-white binary choice, but to achieve a more delicate balance between security and availability through a three-level strategy of initial generation, downgraded generation, and rejection of generation.
[0094] The method also includes: The processing device receives feedback information regarding the generated content.
[0095] Feedback information refers to various evaluations, corrections, or suggestions submitted by users through the system's preset interactive interface after receiving content generated by the system. These feedbacks take many forms, including quantitative feedback such as scoring and star ratings for the usefulness and accuracy of the generated content, as well as textual feedback pointing out errors, supplementing key information, and offering optimization suggestions. They also include qualitative evaluations of the clarity, completeness, and usability of the responses. This feedback directly reflects users' true feelings and differences in needs regarding the generated content, serving as the basis for the system to perceive its own service quality, identify existing problems, and conduct subsequent optimization and iteration.
[0096] The processing device will proactively collect user feedback on the generated content through preset interaction channels.
[0097] The purpose of this step is to break away from the one-way service model where generation ends immediately, and to incorporate users' real experiences and evaluations into the system's iterative process. By analyzing feedback, the system can dynamically adjust subsequent parameter configurations, such as the weight coefficient of the dynamic evidence threshold and the critical value of the uncertainty threshold, thereby continuously optimizing the entire search enhancement generation system and constantly improving its ability to understand user needs and the accuracy of content generation.
[0098] The method also includes: Based on the feedback, the weights of query risk value and query complexity were adjusted during the process of determining the dynamic evidence threshold.
[0099] Weight refers to the coefficients configured for query risk value and query complexity during the calculation of dynamic evidence threshold. These coefficients determine the degree of influence of the two quantitative features on the final result of dynamic evidence threshold. The higher the weight value, the greater the influence of the corresponding feature on the threshold. Moreover, the weight will be dynamically adjusted according to user feedback in actual application, rather than remaining fixed.
[0100] Based on user feedback regarding the generated content, the processing device adjusts the weights of query risk values and query complexity used to determine the dynamic evidence threshold. Simply put, if user feedback indicates that unreasonable query risk value weighting leads to overly strict or lenient evidence screening, or that inappropriate query complexity weighting results in insufficient adaptability of the evidence threshold for high-complexity queries, the processing device will accordingly increase or decrease the weights of the corresponding features, making the calculation logic of the dynamic evidence threshold more aligned with actual usage needs.
[0101] This adjustment allows the criteria for evidence screening to be continuously optimized based on user feedback, thereby improving the accuracy and reliability of subsequent content generation and achieving a closed-loop iteration of the entire system from information processing to service optimization.
[0102] Based on the above description, this application has the following beneficial effects: In this application, by combining the query domain and query sensitivity to determine the query risk value, and combining it with the query complexity calculated based on entity-related features, the evidence admission threshold is dynamically adjusted, achieving scenario-based adaptation of evidence screening standards. In high-risk scenarios such as medical diagnosis and legal consultation, the evidence admission threshold is automatically raised to block the adoption of misleading evidence such as low authority or outdated evidence from the source, reducing the risk of generating illusions and ensuring content security. In low-risk or general information question-and-answer scenarios, the threshold is appropriately relaxed to avoid over-filtering of valid evidence, reduce the system's rejection rate, improve service availability, and successfully balance the needs of different scenarios.
[0103] Furthermore, by comprehensively considering the source authority, publication time (timeliness), and semantic relevance of the evidence to the query, the quality value is determined, breaking through the limitations of traditional single semantic similarity assessment. This ensures that the selected evidence is not only relevant but also authoritative and fresh, providing high-quality support for the generated content and fundamentally improving the accuracy and credibility of the generated results.
[0104] Furthermore, by quantifying the uncertainty of the model sub-uncertainty and the uncertainty of the evidence quality sub-uncertainty, a comprehensive uncertainty assessment system is constructed. Based on the assessment results, a tiered generation control is implemented: generation is rejected when there is high uncertainty, downgraded when there is medium uncertainty, and normal when there is low uncertainty. This makes the system's generation behavior more transparent and controllable. It avoids the risks of ignorant and reckless statements, and provides valuable and cautious answers through the downgraded generation mechanism, reducing the experience loss caused by simply rejecting the system and increasing user trust in it.
[0105] Finally, by collecting user feedback on the generated content, the weights of query risk and query complexity in the dynamic threshold calculation are adjusted in reverse, forming a closed-loop optimization mechanism. This enables the system to continuously iterate based on feedback from actual usage scenarios, constantly adapting to real needs, and gradually improving the accuracy of evidence screening, generation accuracy, and scenario adaptability over long-term use.
[0106] The above text combined Figure 1 The retrieval enhancement generation method provided in the embodiments of this application has been described in detail. The apparatus and equipment provided in the embodiments of this application will be described below with reference to the accompanying drawings.
[0107] like Figure 2 As shown in the figure, this is a schematic diagram of a search enhancement generation device provided in an embodiment of this application. The device includes: Module 301 is used to obtain query requests; Processing module 302 is used to determine a query risk value based on the query domain and query sensitivity corresponding to the query request, determine the query complexity based on the number of entities corresponding to the query request and the number of relationships between entities, and determine a dynamic evidence threshold based on the query risk value and the query complexity. The acquisition module 301 is also used to acquire the quality values of multiple pieces of evidence corresponding to the query request; Processing module 302 is further configured to include evidence with a quality value greater than the dynamic evidence threshold as an evidence set; The generation module 303 is used to control content generation based on the evidence set.
[0108] Optionally, module 301 is used to obtain the source authority of the evidence, the publication time of the evidence, and the semantic relevance of the evidence to the query request; The processing module 302 is specifically used to determine the quality value of the evidence based on the authority of the source of the evidence, the publication time of the evidence, and the semantic relevance of the evidence to the query request.
[0109] Optionally, the generation module 303 is specifically used to calculate the model sub-uncertainty value and the quality sub-uncertainty value of the evidence in the evidence set; determine the uncertainty value of the evidence set based on the model sub-uncertainty value and the quality sub-uncertainty value; and control the content generation based on the uncertainty value of the evidence set.
[0110] Optionally, the generation module 303 is specifically used to refuse to generate content if the uncertainty value of the evidence set is greater than or equal to a first uncertainty threshold. If the uncertainty value of the evidence set is less than the first uncertainty threshold, then content generation is performed.
[0111] Optionally, the generation module 303 is specifically used to downgrade the generation strategy if the uncertainty value of the evidence set is greater than the second uncertainty threshold, and generate content according to the downgraded strategy. If the uncertainty value of the evidence set is less than or equal to the second uncertainty threshold, then content generation is performed according to the initial generation strategy; Wherein, the second uncertainty threshold is less than the first uncertainty threshold.
[0112] Optionally, the acquisition module 301 is also used to acquire feedback information regarding the generated content.
[0113] Optionally, the processing module 302 is also used to adjust the weights of query risk value and query complexity in the process of determining the dynamic evidence threshold based on the feedback information.
[0114] The retrieval enhancement generation apparatus according to the embodiments of this application can correspond to the execution of the method described in the embodiments of this application, and the other operations and / or functions of each module / unit of the retrieval enhancement generation apparatus are respectively for implementing Figure 1 For the sake of brevity, the corresponding processes of each method in the illustrated embodiments will not be described in detail here.
[0115] This application also provides a computing device. For example... Figure 3 As shown in the figure, this is a schematic diagram of a computing device provided in an embodiment of this application. The computing device 700 includes a bus 701, a processor 702, a communication interface 703, and a memory 704. The processor 702, the memory 704, and the communication interface 703 communicate with each other via the bus 701.
[0116] The 701 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0117] The processor 702 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).
[0118] The communication interface 703 is used for external communication.
[0119] Memory 704 may include volatile memory, such as random access memory (RAM). Memory 704 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).
[0120] The memory 704 stores executable code, which the processor 702 executes to perform the aforementioned retrieval enhancement generation method.
[0121] Specifically, in achieving Figure 2 In the case of the illustrated embodiment, and Figure 2 When the modules or units of the retrieval enhancement generation device described in the embodiments are implemented by software, the following steps are performed: Figure 2 The software or program code required for the functions of each module / unit can be partially or wholly stored in memory 704. Processor 702 executes the program code corresponding to each unit stored in memory 704 and performs the aforementioned search enhancement generation method.
[0122] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the aforementioned retrieval enhancement generation method.
[0123] This application also provides a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.
[0124] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means.
[0125] When the computer program product is executed by a computer, the computer performs any of the aforementioned search enhancement generation methods. The computer program product can be a software installation package; when any of the aforementioned search enhancement generation methods is required, the computer program product can be downloaded and executed on the computer.
[0126] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.
[0127] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered within the scope of protection of this application.
Claims
1. A retrieval enhancement generation method, characterized in that, The method includes: Get the query request; Based on the query domain and query sensitivity corresponding to the query request, the query risk value is determined, and based on the number of entities corresponding to the query request and the number of relationships between entities, the query complexity is determined. The expression for calculating the risk value is: in, This indicates a query for risk values. Indicates the weighting coefficient. This represents the risk benchmark value for the query domain. This indicates the number of sensitive entities in the query. This represents the sensitivity score of the j-th sensitive entity. This indicates an exponential transformation of the sensitivity score of a single sensitive entity, amplifying the impact of highly sensitive entities on the overall risk; The expression for calculating query complexity is: in, Indicates the query complexity. This represents the first weighting coefficient. This indicates the total number of entities in the query. This represents the second weighting coefficient. This indicates the number of relationships between entities in the query. This indicates that the larger value between 1 and the total number of entities is taken to avoid the error of a denominator of 0 when the number of entities is 0; Based on the query risk value and the query complexity, a dynamic evidence threshold is determined; The expression for calculating the dynamic evidence threshold is: in, Indicates the dynamic evidence threshold. Indicates the basic threshold. Indicates the weight of risk adjustment. Indicates the risk benchmark value. The weights represent the complexity adjustment. This represents the baseline value for complexity. Indicates the timeliness correction factor; Obtain the quality values of multiple pieces of evidence corresponding to the query request; Evidence with a quality value greater than the dynamic evidence threshold is considered as the evidence set. Based on the aforementioned set of evidence, content generation is controlled; The control of content generation based on the evidence set includes: Calculate the model sub-uncertainty value and the quality sub-uncertainty value of the evidence in the evidence set; the model sub-uncertainty value is a quantitative result of the confidence risk when the large language model generates content itself, calculated based on the prediction probability of Token during the content generation process of the model; the quality sub-uncertainty value is a quantitative result of the quality risk of the evidence set, calculated based on the overall quality of the evidence set. The uncertainty value of the evidence set is determined based on the model sub-uncertainty value and the mass sub-uncertainty value. Content generation is controlled based on the uncertainty value of the evidence set.
2. The method according to claim 1, characterized in that, The quality of evidence is determined by the following methods: The authority of the source of the evidence, the time of its publication, and the semantic relevance of the evidence to the query request; The quality value of the evidence is determined based on the authority of its source, the time of its publication, and the semantic relevance of the evidence to the query request.
3. The method according to claim 1, characterized in that, Based on the uncertainty value of the evidence set, content generation is controlled, including: If the uncertainty value of the evidence set is greater than or equal to the first uncertainty threshold, then content generation is rejected. If the uncertainty value of the evidence set is less than the first uncertainty threshold, then content generation is performed.
4. The method according to claim 3, characterized in that, Content generation includes: If the uncertainty value of the evidence set is greater than the second uncertainty threshold, the generation strategy is downgraded, and content is generated according to the downgraded strategy. If the uncertainty value of the evidence set is less than or equal to the second uncertainty threshold, then content generation is performed according to the initial generation strategy; Wherein, the second uncertainty threshold is less than the first uncertainty threshold.
5. The method according to claim 1, characterized in that, The method further includes: Get feedback on the generated content.
6. The method according to claim 5, characterized in that, The method further includes: Based on the feedback information, the weights of query risk value and query complexity are adjusted during the process of determining the dynamic evidence threshold.
7. A retrieval enhancement generation device, characterized in that, The device includes: The retrieval module is used to retrieve query requests; The processing module is used to determine the query risk value based on the query domain and query sensitivity corresponding to the query request, and to determine the query complexity based on the number of entities corresponding to the query request and the number of relationships between entities; the expression for calculating the query risk value is: in, This indicates a query for risk values. Indicates the weighting coefficient. This represents the risk benchmark value for the query domain. This indicates the number of sensitive entities in the query. This represents the sensitivity score of the j-th sensitive entity. This indicates an exponential transformation of the sensitivity score of a single sensitive entity, amplifying the impact of highly sensitive entities on the overall risk; The expression for calculating query complexity is: in, Indicates the query complexity. This represents the first weighting coefficient. This indicates the total number of entities in the query. This represents the second weighting coefficient. This indicates the number of relationships between entities in the query. This indicates that the larger value between 1 and the total number of entities is taken to avoid the error of a denominator of 0 when the number of entities is 0; Based on the query risk value and the query complexity, a dynamic evidence threshold is determined; the calculation expression for the dynamic evidence threshold is: in, Indicates the dynamic evidence threshold. Indicates the basic threshold. Indicates the weight of risk adjustment. Indicates the risk benchmark value. The weights represent the complexity adjustment. This represents the baseline value for complexity. Indicates the timeliness correction factor; The acquisition module is also used to acquire the quality values of multiple pieces of evidence corresponding to the query request; The processing module is also used to collect evidence with a quality value greater than the dynamic evidence threshold as an evidence set; A generation module is configured to control content generation based on the evidence set; the control of content generation based on the evidence set includes: Calculate the model sub-uncertainty value and the quality sub-uncertainty value of the evidence in the evidence set; The model sub-uncertainty value is a quantitative result of the confidence risk when the large language model generates content itself, based on the prediction probability calculation of Token during the content generation process. The quality sub-uncertainty value is a quantitative result of the quality risk of the evidence set, based on the overall quality calculation of the evidence set. The uncertainty value of the evidence set is determined based on the model sub-uncertainty value and the mass sub-uncertainty value. Content generation is controlled based on the uncertainty value of the evidence set.
8. A computing device, characterized in that, Including memory and processor; The memory stores one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the computing device performs the method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method as described in any one of claims 1 to 6.