A method for determining the necessity of RAG in large model retrieval enhancement generation
By calculating the factual tendency score of user questions and determining whether to call RAG to assist in answering, the problems of resource waste and model confusion in the use of RAG are solved, and the answering speed and accuracy of large models are improved.
Patent Information
- Application Number
- CN202411870255.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-18
- Publication Date
- 2025-09-16
- Estimated Expiration
- 2044-12-18
AI Technical Summary
In the existing technology, the use of RAG may introduce unnecessary complexity and waste of resources in some cases, and may increase model confusion, especially when user questions do not require an external knowledge base, affecting the answer speed and accuracy of large models.
By obtaining user question information, calculating its fact tendency score, judging whether it is necessary to call RAG for auxiliary answering, using the trained general model BERT to determine the fact tendency score, and combining it with historical question information, deciding whether to call RAG for auxiliary answering based on the score difference.
Effectively identify whether a question requires RAG assistance, reduce resource waste, improve reasoning speed, avoid model confusion, and improve the accuracy and efficiency of answers.
Smart Images
Figure CN119760080B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of large model retrieval enhancement generation, and in particular to a method for judging the necessity of RAG in large model retrieval enhancement generation. Background Art
[0002] Retrieval-augmented generation (RAG) is a technology applied to knowledge content retrieval. It combines vector databases and large language models, aiming to enhance the generative capabilities of large models through retrieval technology. However, in some cases, the use of RAG can introduce unnecessary complexity and waste resources. Many RAG studies and practices have shown that retrieval does not always benefit the final generated results. When the model's own parameterized knowledge is sufficient to answer relevant questions, excessive retrieval wastes resources and may increase model confusion.
[0003] In actual user scenarios, the current common method is to calculate the similarity between the user's input text and the vectors in the knowledge vector database, and then concatenate the original text corresponding to the vector with the highest similarity that exceeds the preset threshold with the user's input text through the prompt word project, and send it to the large model for final answer generation.
[0004] In this case, no matter what the input is, the RAG system will be called directly, which will directly affect the answering speed of the large model. Moreover, the user's questions do not necessarily need to call the external knowledge base, which will have a counterproductive effect. Summary of the Invention
[0005] The technical problem to be solved by the present invention is to address the deficiencies of the existing technology and specifically provide a method for determining the necessity of RAG in large model retrieval enhancement generation, which is as follows:
[0006] 1) In the first aspect, the present invention provides a method for determining the necessity of RAG in large-scale model retrieval enhancement generation. The specific technical solution is as follows:
[0007] Obtain information about current issues raised by users in social networking sites or social media comment sections or conversation logs;
[0008] Determining a fact tendency score corresponding to the user based on the current question information;
[0009] Based on the difference between the factual propensity score and the preset propensity score, it is determined whether it is necessary to call RAG to assist in answering the current question information.
[0010] The beneficial effects of the RAG necessity judgment method in large model retrieval enhancement generation provided by the present invention are as follows:
[0011] By calculating the factual tendency score of the current question information, it is possible to effectively identify whether the current question information needs to reference external knowledge using RAG. By judging the necessity of RAG, unnecessary resource waste can be reduced, the reasoning speed can be improved, and the confusion of the model caused by the knowledge introduced by RAG can be avoided.
[0012] Based on the above solution, the present invention can also be improved as follows.
[0013] Furthermore, the process of determining the fact tendency score corresponding to the user is specifically as follows:
[0014] The current question information is processed by the trained general model BERT to obtain the current fact tendency score corresponding to the current question information, and the historical fact tendency scores corresponding to all historical question information of the same user before the current question information are called. Based on the current weight and the historical weight, the current fact tendency score and the fact tendency score after weighted summation of the historical fact tendency scores are determined.
[0015] Furthermore, the training set of the general model BERT includes:
[0016] The collected questions are cleaned, and the text cleaning includes: removing special characters in the text, processing HTML tags, and converting uppercase and lowercase letters.
[0017] Furthermore, based on the difference between the factual propensity score and the preset propensity score, the process of determining whether to call RAG to assist in answering the current question information is as follows:
[0018] When the difference between the factual propensity score and the preset propensity score is greater than zero, RAG is called to assist in answering the current question information;
[0019] When the difference between the factual propensity score and the preset propensity score is less than zero, the current question information is answered directly through the base large model.
[0020] 2) In a second aspect, the present invention further provides a system for determining the necessity of RAG in large-scale model retrieval enhancement generation. The specific technical solution is as follows:
[0021] The acquisition module is used to: obtain current question information raised by users in social networking sites or social media comment areas or conversation logs;
[0022] The determination module is used to: determine the fact tendency score corresponding to the user based on the current question information;
[0023] The judgment module is used to determine whether it is necessary to call RAG to assist in answering the current question information based on the difference between the fact tendency score and the preset tendency score.
[0024] Based on the above solution, the present invention can also be improved as follows.
[0025] Furthermore, the process of determining the fact tendency score corresponding to the user is specifically as follows:
[0026] The current question information is processed by the trained general model BERT to obtain the current fact tendency score corresponding to the current question information, and the historical fact tendency scores corresponding to all historical question information of the same user before the current question information are called. Based on the current weight and the historical weight, the current fact tendency score and the fact tendency score after weighted summation of the historical fact tendency scores are determined.
[0027] Furthermore, the training set of the general model BERT includes:
[0028] The collected questions are cleaned, and the text cleaning includes: removing special characters in the text, processing HTML tags, and converting uppercase and lowercase letters.
[0029] Furthermore, based on the difference between the factual propensity score and the preset propensity score, the process of determining whether to call RAG to assist in answering the current question information is as follows:
[0030] When the difference between the factual propensity score and the preset propensity score is greater than zero, RAG is called to assist in answering the current question information;
[0031] When the difference between the factual propensity score and the preset propensity score is less than zero, the current question information is answered directly through the base large model.
[0032] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, wherein the processor is coupled to a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above methods.
[0033] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor to enable a computer to implement any of the above methods.
[0034] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:
[0036] Figure 1 Schematic diagram of a flow chart of a method for determining the necessity of RAG in large model retrieval enhancement generation according to an embodiment of the present invention;
[0037] Figure 2 This is a structural framework diagram of an electronic device. DETAILED DESCRIPTION
[0038] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.
[0039] like Figure 1 As shown, a method for determining the necessity of RAG in large model retrieval enhancement generation according to an embodiment of the present invention includes the following steps:
[0040] S1, obtains information about current issues raised by users in social networking sites or social media comment areas or conversation logs;
[0041] S2, determining a fact tendency score corresponding to the user based on the current question information;
[0042] S3: Determine whether it is necessary to call RAG to assist in answering the current question information based on the difference between the factual propensity score and the preset propensity score.
[0043] The beneficial effects of the RAG necessity judgment method in large model retrieval enhancement generation provided by the present invention are as follows:
[0044] By calculating the factual tendency score of the current question information, it is possible to effectively identify whether the current question information needs to reference external knowledge using RAG. By judging the necessity of RAG, unnecessary resource waste can be reduced, the reasoning speed can be improved, and the confusion of the model caused by the knowledge introduced by RAG can be avoided.
[0045] The current question information includes the time when the question was asked, the specific text content of the question, and the context in which the question was asked (ie, where the user asked the question on a social networking site, in a social media comment area, or in a conversation log).
[0046] The method for determining the factual propensity score corresponding to the user is as follows:
[0047] The fact tendency score is determined by the trained general model BERT. Specifically:
[0048] Obtain the current question information entered by the user, classify the current question information, and determine whether the current question information has an associated time sequence problem. An associated time sequence problem refers to all questions entered consecutively by the same user in the same input box at different times. For current question information that does not have an associated time sequence problem, directly calculate the factual propensity score. When the factual propensity score is less than the preset propensity score, the base large model processes it and obtains the corresponding response to the current question information. If the factual propensity score is greater than the preset propensity score, call RAG to assist in answering the current question information.
[0049] Confirm the problem relevance of the current problem information with associated time series problems, and confirm whether there is a target associated time series problem in the associated time series problems whose correlation with the current problem information exceeds the preset correlation. If not, calculate the fact tendency score corresponding to the current problem information, and at the same time calculate the historical fact tendency scores corresponding to all associated time series problems (historical problem information). Combine the current weight and the historical weight to determine the current fact tendency score and the fact tendency score after weighted summation of the historical fact tendency scores.
[0050] If it exists, extract the first keyword in the current question information, and extract the second keyword for at least one target-related time series question that exceeds the preset correlation degree, calculate the similarity between the first keyword and any second keyword, and determine the target-related time series question corresponding to the second keyword with the highest similarity as the calibration object, and use the historical fact tendency score corresponding to the calibration object as the fact tendency score corresponding to the current question information, and judge the size of the fact tendency score and the preset tendency score. When the difference between the fact tendency score and the preset tendency score is greater than zero, call RAG to assist in answering the current question information; when the difference between the fact tendency score and the preset tendency score is less than zero, directly answer the current question information through the base large model.
[0051] It should be noted that the fact tendency score reflects the probability of factual issues. The higher the fact tendency score, the higher the probability of factual issues.
[0052] Factual questions are questions based solely on facts, unrelated to the law, and subject to court adjudication. These questions primarily inquire about objective facts, actual conditions, or specific data, and their application is broad and covers a wide range of fields.
[0053] Definition and characteristics of factual questions:
[0054] Factual questions, also known as descriptive questions or reality questions, primarily involve inquiries about objective facts, actual conditions, or specific data. These questions do not involve legal or value judgments but are based on actual circumstances.
[0055] Application scenarios of factual questions:
[0056] Legal Field: In legal cases, factual questions usually involve the specific details of the case, the authenticity of evidence, the time and place of events, etc. For example, lawyers in court ask witnesses about the specific circumstances of the crime scene to confirm or deny certain facts.
[0057] Historical Research: Historians raise numerous factual questions during their research to uncover the truth about historical events. These questions may concern the politics, economy, and culture of a particular historical period.
[0058] News reporting: News reporting needs to accurately reflect the facts, so reporters will ask factual questions to obtain accurate information, such as the time, place, and people involved in the event.
[0059] Scientific research: In scientific research, factual questions are used to explore specific situations such as natural phenomena and experimental data. Scientists use these questions and answer them to test hypotheses, establish theories, and promote scientific progress.
[0060] Market research: In market research, factual questions are used to collect specific data about consumer behavior, market demand, etc., to help companies understand market dynamics and develop appropriate marketing strategies.
[0061] It can be seen from these application scenarios that the core of factual questions lies in their objectivity and being based on actual situations. They are widely used in various fields that require accurate information.
[0062] In addition, RAG refers to Retrieval-augmented Generation, which is one of the most popular cutting-edge technologies for large models.
[0063] Retrieval-augmented generative models combine language models and information retrieval techniques. Specifically, when the model needs to generate text or answer questions, it first retrieves relevant information from a large collection of documents and then uses this retrieved information to guide text generation, thereby improving the quality and accuracy of predictions.
[0064] Furthermore, the process of determining the fact tendency score corresponding to the user is specifically as follows:
[0065] The current question information is processed by the trained general model BERT to obtain the current fact tendency score corresponding to the current question information, and the historical fact tendency scores corresponding to all historical question information of the same user before the current question information are called. Based on the current weight and the historical weight, the current fact tendency score and the fact tendency score after weighted summation of the historical fact tendency scores are determined.
[0066] Furthermore, the training set of the general model BERT includes:
[0067] The collected questions are cleaned, and the text cleaning includes: removing special characters in the text, processing HTML tags, and converting uppercase and lowercase letters.
[0068] Furthermore, based on the difference between the factual propensity score and the preset propensity score, the process of determining whether to call RAG to assist in answering the current question information is as follows:
[0069] When the difference between the factual propensity score and the preset propensity score is greater than zero, RAG is called to assist in answering the current question information;
[0070] When the difference between the factual propensity score and the preset propensity score is less than zero, the current question information is answered directly through the base large model.
[0071] Example 1, (1) Training a factual question preference recognition model
[0072] 1. Data Collection and Preprocessing
[0073] Questions are collected from conversation logs, social media comments, online forums, and other channels, and then text is cleaned. Specifically: Text cleaning removes irrelevant information such as special characters, HTML tags, and meaningless characters. Case conversion converts all letters in the text to uppercase or lowercase to eliminate case differences.
[0074] 2. Labeling
[0075] Manually label each question to indicate whether it is a factual question.
[0076] 3. Model Training
[0077] (1) Based on the general model BERT, a fully connected layer with a dimension of 2 and a softmax layer are added, and the labeled question data is used for training to obtain a binary classification model. The probability value of factual questions can be obtained through softmax, which is between 0 and 1.
[0078] (2) Receiving user input
[0079] (3) Obtaining user fact tendency scores
[0080] When the user asks a question, all the questions recorded before this question are fed into the trained classification model for inference, and the probability of preference for factual questions is output. The calculation method is as follows:
[0081]
[0082] in:
[0083] The tendency score for factual questions obtained on behalf of historical questions;
[0084] cls(k) represents the factual question propensity score obtained for the current question;
[0085] α is a smoothing coefficient (between 0 and 1) that determines the degree of influence of new and old problems on the current results. The lower α is, the greater the influence of historical problems is. In the experiment, α = 0.85;
[0086] n is the total number of historical questions of the user;
[0087] cls() is the factual question propensity score obtained by the factual question prediction model for a certain question;
[0088] i represents the i-th question in the historical questions;
[0089] k stands for the current question.
[0090] Based on this, the fact score of the user's current question can be obtained.
[0091] In practice, the historical scores of users are saved locally based on their IDs, so that the acceleration function can be directly called during subsequent calculations and replaced with the latest calculation results.
[0092] Decision whether to call RAG:
[0093] When a user enters a question, we call the factual question preference recognition model and combine it with historical questions to obtain the propensity score of the current question being a factual question. If it is greater than the preset threshold, we believe that this question needs to be answered through RAG. Otherwise, we directly call the capabilities of the base model to answer it.
[0094] Here, the preset threshold is set to 0.5.
[0095] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the scope of protection of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.
[0096] The present invention also provides a RAG necessity judgment system in large model retrieval enhancement generation, and the specific technical solution is as follows:
[0097] The acquisition module is used to: obtain current question information raised by users in social networking sites or social media comment areas or conversation logs;
[0098] The determination module is used to: determine the fact tendency score corresponding to the user based on the current question information;
[0099] The judgment module is used to determine whether it is necessary to call RAG to assist in answering the current question information based on the difference between the fact tendency score and the preset tendency score.
[0100] It should be noted that the beneficial effects of the RAG necessity judgment system in the large model retrieval enhancement generation provided by the above embodiment are the same as the beneficial effects of the RAG necessity judgment method in the large model retrieval enhancement generation provided by the above embodiment, and will not be repeated here. In addition, when the system provided by the above embodiment realizes its functions, it only uses the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided by the above embodiment belong to the same concept, and their specific implementation process is detailed in the method embodiment, which will not be repeated here.
[0101] like Figure 2 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, which is coupled to a memory 310. The memory 310 stores at least one computer program 330. The at least one computer program 330 is loaded and executed by the processor 320 to enable the electronic device 300 to implement any of the above methods. Specifically:
[0102] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements the RAG necessity judgment method in the large model retrieval enhancement generation provided by the above embodiment. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 300 may also include other components for realizing device functions, which will not be described in detail here.
[0103] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor to enable a computer to implement any of the above methods.
[0104] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, or the like.
[0105] In an exemplary embodiment, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform any of the above methods.
[0106] It should be noted that the terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects and to define a specific order or precedence. Where appropriate, the order used for similar objects may be interchanged, such that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.
[0107] Those skilled in the art will appreciate that the present invention may be implemented as a system, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the present invention may be implemented in the form of a computer program product embodied in one or more computer-readable media containing computer-readable program code.
[0108] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.
[0109] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.
Claims
1. A method for determining the necessity of RAG in large model retrieval enhancement generation, characterized in that: include: Obtain information about current issues raised by users in social networking sites or social media comment sections or conversation logs; Determining a fact tendency score corresponding to the user based on the current question information; Determine whether to call RAG to assist in answering the current question information based on the difference between the factual propensity score and the preset propensity score; The method for determining the factual propensity score corresponding to the user is as follows: The fact tendency score is determined by the trained general model BERT, specifically: Obtain the current question information entered by the user, classify the current question information, and determine whether the current question information has an associated time sequence problem. An associated time sequence problem refers to all questions entered continuously by the same user in the same input box at different times. For current question information that does not have an associated time sequence problem, directly calculate the factual propensity score. When the factual propensity score is less than the preset propensity score, the base large model processes it and obtains the corresponding response to the current question information. If the factual propensity score is greater than the preset propensity score, call RAG to assist in answering the current question information. Confirm the relevance of the current problem information with associated time series problems, and confirm whether there is a target associated time series problem in the associated time series problems whose relevance to the current problem information exceeds the preset relevance. If not, calculate the fact tendency score corresponding to the current problem information, and simultaneously calculate the historical fact tendency scores corresponding to all associated time series problems, that is, historical problem information. Combine the current weight and the historical weight to determine the fact tendency score of the current fact tendency score and the fact tendency score after weighted summation of the historical fact tendency scores; If it exists, extract the first keyword in the current question information, and extract the second keyword for at least one target-related time series question that exceeds the preset correlation degree, calculate the similarity between the first keyword and any second keyword, and determine the target-related time series question corresponding to the second keyword with the highest similarity as the calibration object, and use the historical fact tendency score corresponding to the calibration object as the fact tendency score corresponding to the current question information, and judge the size of the fact tendency score and the preset tendency score. When the difference between the fact tendency score and the preset tendency score is greater than zero, call RAG to assist in answering the current question information; when the difference between the fact tendency score and the preset tendency score is less than zero, directly answer the current question information through the base large model.
2. The method for determining the necessity of RAG in large model retrieval enhancement generation according to claim 1, characterized in that: The training set of the general model BERT includes: The collected questions are cleaned, and the text cleaning includes: removing special characters in the text, processing HTML tags, and converting uppercase and lowercase letters.
3. A system for determining the necessity of RAG in large model retrieval enhancement generation, using the method for determining the necessity of RAG in large model retrieval enhancement generation according to claim 1, characterized in that: The system comprises: The acquisition module is used to: obtain current question information raised by users in social networking sites or social media comment areas or conversation logs; The determination module is used to: determine the fact tendency score corresponding to the user based on the current question information; The judgment module is used to determine whether it is necessary to call RAG to assist in answering the current question information based on the difference between the fact tendency score and the preset tendency score.
4. The RAG necessity judgment system in large model retrieval enhancement generation according to claim 3 is characterized in that: The training set of the general model BERT includes: The collected questions are cleaned, and the text cleaning includes: removing special characters in the text, processing HTML tags, and converting uppercase and lowercase letters.
5. An electronic device, characterized in that: The electronic device includes a processor coupled to a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the method according to claim 1 or 2.
6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable a computer to implement the method according to claim 1 or 2.