RAG necessity judgment method in large model retrieval enhancement generation

Abstract

The invention discloses an RAG necessity judgment method in large model retrieval enhancement generation, and relates to the technical field of large model retrieval enhancement generation, and the method comprises the steps: obtaining current question information proposed by a user in a social network site or a social media comment area or a dialogue log; for the current question information, determining a fact tendency score corresponding to the user; and according to the difference between the fact tendency score and a preset tendency score, determining whether to call the RAG to assist in answering the current question information. According to the method, the fact tendency score of the current problem information is calculated, so that whether the current problem information needs to quote external knowledge in an RAG mode or not can be effectively recognized, unnecessary resource waste is reduced by judging the necessity of the RAG, the reasoning speed is increased, and the disorder of a model caused by the knowledge introduced by the RAG is avoided.

Description

Background Art

[0002] Retrieval Augmented Generation (RAG) is a technology applied in the field of knowledge content retrieval. It combines vector databases and large language models, aiming to enhance the generation capabilities of large models through retrieval technology. However, in some cases, the use of RAG may introduce unnecessary complexity and waste of resources. Many RAG studies and practices have shown that retrieval is not always conducive to the final generated results. When the parameterized knowledge of the model itself is sufficient to answer relevant questions, excessive retrieval will cause a waste of resources and may increase the confusion of the model.

[0003] In the actual user usage scenario, the current common method is to calculate the similarity between the user's input text and the vectors in the knowledge vector database, and then concatenate the original text corresponding to the vector with the highest similarity that exceeds the preset threshold with the user's input text through the prompt word project, and send it to the large model for final answer generation.

[0004] In this case, no matter what the input is, the RAG system will be called directly, which will directly affect the answering speed of the large model. In addition, the user's questions do not necessarily need to call the external knowledge base, which will have a counterproductive effect. Summary of the invention

[0005] The technical problem to be solved by the present invention is to address the deficiencies of the prior art and specifically provide a method for determining the necessity of RAG in large model retrieval enhancement generation, which is as follows:

[0006] 1) In the first aspect, the present invention provides a method for determining the necessity of RAG in large model retrieval enhancement generation, and the specific technical solution is as follows:

[0007] Obtain information about current issues raised by users in social networking sites or social media comment sections or conversation logs;

[0008] Determine the fact tendency score corresponding to the user according to the current question information;

[0009] Based on the difference between the factual propensity score and the preset propensity score, determine whether it is necessary to call RAG to assist in answering the current question information.

[0010] The beneficial effects of the RAG necessity judgment method in large model retrieval enhancement generation provided by the present invention are as follows:

[0011] By calculating the factual tendency score of the current problem information, it is possible to effectively identify whether the current problem information needs to reference external knowledge in the form of RAG. By judging the necessity of RAG, unnecessary resource waste can be reduced, the reasoning speed can be improved, and the confusion of the model can be avoided due to the knowledge introduced by RAG.

[0012] Based on the above solution, the present invention can also be improved as follows.

[0013] Further, the process of determining the fact tendency score corresponding to the user is specifically as follows:

[0014] The current question information is processed by the trained general model BERT to obtain the current fact tendency score corresponding to the current question information, and the historical fact tendency scores corresponding to all historical question information of the same user before the current question information are called. Based on the current weight and the historical weight, the current fact tendency score and the fact tendency score after weighted summation of the historical fact tendency scores are determined.

[0015] Furthermore, the training set of the general model BERT includes:

[0016] The collected questions are cleaned, and the text cleaning includes: removing special characters in the text, HTML tag processing, and case conversion processing.

[0017] Further, according to the difference between the factual propensity score and the preset propensity score, the process of determining whether it is necessary to call RAG to assist in answering the current question information is specifically as follows:

[0018] When the difference between the factual propensity score and the preset propensity score is greater than zero, RAG is called to assist in answering the current question information;

[0019] When the difference between the factual propensity score and the preset propensity score is less than zero, the current question information is directly answered through the base large model.

[0020] 2) In the second aspect, the present invention also provides a RAG necessity judgment system in large model retrieval enhancement generation, and the specific technical solution is as follows:

[0021] The acquisition module is used to: obtain current question information raised by users in a social networking site or social media comment area or conversation log;

[0022] The determination module is used to: determine the fact tendency score corresponding to the user according to the current question information;

[0023] The judgment module is used to determine whether it is necessary to call RAG to assist in answering the current question information based on the difference between the factual propensity score and the preset propensity score.

[0024] Based on the above solution, the present invention can also be improved as follows.

[0025] Further, the process of determining the fact tendency score corresponding to the user is specifically as follows:

[0026] The current question information is processed by the trained general model BERT to obtain the current fact tendency score corresponding to the current question information, and the historical fact tendency scores corresponding to all historical question information of the same user before the current question information are called. Based on the current weight and the historical weight, the current fact tendency score and the fact tendency score after weighted summation of the historical fact tendency scores are determined.

[0027] Furthermore, the training set of the general model BERT includes:

[0028] The collected questions are cleaned, and the text cleaning includes: removing special characters in the text, HTML tag processing, and case conversion processing.

[0029] Further, according to the difference between the factual propensity score and the preset propensity score, the process of determining whether it is necessary to call RAG to assist in answering the current question information is specifically as follows:

[0030] When the difference between the factual propensity score and the preset propensity score is greater than zero, RAG is called to assist in answering the current question information;

[0031] When the difference between the factual propensity score and the preset propensity score is less than zero, the current question information is directly answered through the base large model.

[0032] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, wherein the processor is coupled to a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above methods.

[0033] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor so that a computer implements any of the above methods.

[0034] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0036] Figure 1 A schematic flow chart of a method for determining the necessity of RAG in large model retrieval enhancement generation according to an embodiment of the present invention;

[0037] Figure 2 The figure is a structural frame diagram of an electronic device. DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0039] like Figure 1 As shown, a method for determining the necessity of RAG in large model retrieval enhancement generation according to an embodiment of the present invention includes the following steps:

[0040] S1, obtaining information about current issues raised by users in social networking sites or social media comment areas or conversation logs;

[0041] S2, determining a fact tendency score corresponding to the user with respect to the current question information;

[0042] S3, determining whether it is necessary to call RAG to assist in answering the current question information based on the difference between the factual propensity score and the preset propensity score.

[0043] The beneficial effects of the RAG necessity judgment method in large model retrieval enhancement generation provided by the present invention are as follows:

[0044] By calculating the factual tendency score of the current problem information, it is possible to effectively identify whether the current problem information needs to reference external knowledge in the form of RAG. By judging the necessity of RAG, unnecessary resource waste can be reduced, the reasoning speed can be improved, and the confusion of the model can be avoided due to the knowledge introduced by RAG.

[0045] The current question information includes the time when the question is asked, the specific text content of the question, and the context in which the question is asked (ie, where the user asked the question on a social networking site or in a social media comment area or conversation log).

[0046] The method for determining the factual propensity score corresponding to the user is:

[0047] The fact tendency score is determined by the trained general model BERT. Specifically:

[0048] Get the current question information input by the user, classify the current question information, and determine whether the current question information has an associated timing problem. An associated timing problem refers to all questions that the same user enters in the same input box at different times. Directly calculate the factual propensity score for the current question information that does not have an associated timing problem, and when the factual propensity score is less than the preset propensity score, the base model processes it and obtains the corresponding response to the current question information. If the factual propensity score is greater than the preset propensity score, call RAG to assist in answering the current question information.

[0049] Confirm the problem relevance of the current problem information with associated timing problems, and confirm whether there is a target associated timing problem in the associated timing problems whose correlation with the current problem information exceeds the preset correlation. If not, calculate the fact tendency score corresponding to the current problem information, and at the same time calculate the historical fact tendency scores corresponding to all associated timing problems (historical problem information). Combine the current weight and historical weight to determine the current fact tendency score and the fact tendency score after weighted summation of the historical fact tendency scores.

[0050] If it exists, extract the first keyword in the current question information, and extract the second keyword for at least one target-related time series question that exceeds the preset correlation degree, calculate the similarity between the first keyword and any second keyword, determine the target-related time series question corresponding to the second keyword with the highest similarity as the calibration object, use the historical fact tendency score corresponding to the calibration object as the fact tendency score corresponding to the current question information, judge the size of the fact tendency score and the preset tendency score, and when the difference between the fact tendency score and the preset tendency score is greater than zero, call RAG to assist in answering the current question information; when the difference between the fact tendency score and the preset tendency score is less than zero, directly answer the current question information through the base large model.

[0051] It should be noted that the factual tendency score reflects the probability of factual issues. The higher the factual tendency score, the higher the probability of factual issues.

[0052] Factual issues refer to issues that are based entirely on facts, have nothing to do with the law, and must be decided by the court. These issues mainly involve inquiries about objective facts, actual conditions, or specific data. They have a wide range of applications and cover many fields.

[0053] Definition and Characteristics of Factual Questions:

[0054] Factual questions are also called descriptive questions or reality questions, which mainly involve inquiries about objective facts, actual conditions or specific data. These questions do not involve legal judgments or value judgments, but are based on actual situations.

[0055] Application scenarios of factual questions:

[0056] Legal field: In legal cases, factual issues usually involve the specific details of the case, the authenticity of evidence, the time and place of the incident, etc. For example, lawyers ask witnesses in court about the specific circumstances of the case scene to confirm or deny certain facts.

[0057] Historical research: Historians raise a lot of factual questions in the process of research to restore the truth of historical events. These questions may involve politics, economy, culture and other aspects of a certain historical period.

[0058] News reporting: News reporting needs to accurately reflect the facts, so reporters will ask factual questions to obtain accurate information, including the time, place, and people involved in an event.

[0059] Scientific research: In scientific research, factual questions are used to explore specific situations such as natural phenomena and experimental data. Scientists test hypotheses, establish theories, and promote scientific progress by asking and answering these questions.

[0060] Market research: In market research, factual questions are used to collect specific data about consumer behavior, market demand, etc., to help companies understand market dynamics and develop appropriate marketing strategies.

[0061] It can be seen from these application scenarios that the core of factual issues lies in their objectivity and being based on actual situations. They are widely used in various fields that require accurate information.

[0062] In addition, RAG refers to Retrieval-augmented Generation, which is one of the most popular cutting-edge technologies for large models.

[0063] Retrieval-augmented generative models combine language models and information retrieval techniques. Specifically, when the model needs to generate text or answer questions, it first retrieves relevant information from a large collection of documents, and then uses the retrieved information to guide text generation, thereby improving the quality and accuracy of predictions.

[0064] Further, the process of determining the fact tendency score corresponding to the user is specifically as follows:

[0065] The current question information is processed by the trained general model BERT to obtain the current fact tendency score corresponding to the current question information, and the historical fact tendency scores corresponding to all historical question information of the same user before the current question information are called. Based on the current weight and the historical weight, the current fact tendency score and the fact tendency score after weighted summation of the historical fact tendency scores are determined.

[0066] Furthermore, the training set of the general model BERT includes:

[0067] The collected questions are cleaned, and the text cleaning includes: removing special characters in the text, HTML tag processing, and case conversion processing.

[0068] Further, according to the difference between the factual propensity score and the preset propensity score, the process of determining whether it is necessary to call RAG to assist in answering the current question information is specifically as follows:

[0069] When the difference between the factual propensity score and the preset propensity score is greater than zero, RAG is called to assist in answering the current question information;

[0070] When the difference between the factual propensity score and the preset propensity score is less than zero, the current question information is directly answered through the base large model.

[0071] Example 1, (1) Training a factual question preference recognition model

[0072] 1. Data collection and preprocessing

[0073] Collect questions from conversation logs, social media comments, online forums and other channels to perform text cleaning. Specific: Text cleaning: remove special characters, HTML tags, meaningless characters and other irrelevant information in the text. Case conversion: convert the letters in the text to uppercase or lowercase to eliminate the difference in uppercase and lowercase.

[0074] 2. Labeling

[0075] Manually label each question to indicate whether it is a factual question.

[0076] 3. Model training

[0077] (1) Based on the general model BERT, a fully connected layer with a dimension of 2 and a softmax layer are added, and the labeled question data is used for training to obtain a binary classification model. The probability value of factual questions can be obtained through softmax, which is between 0 and 1.

[0078] (2) Receiving user input

[0079] (3) Obtaining user fact tendency scores

[0080] When the user asks a question, all the questions recorded before this question are sent to the above-mentioned trained classification model for reasoning, and the probability of preference for factual questions is output. The calculation method is as follows:

[0081]

[0082] in:

[0083] The factual question propensity score obtained on behalf of historical questions;

[0084] cls(k) represents the factual question tendency score obtained by the current question;

[0085] α is a smoothing coefficient (between 0 and 1), which determines the degree of influence of new and old problems on the current results. The lower α is, the greater the influence of historical problems is. In the experiment, α = 0.85;

[0086] n is the total number of historical questions of the user;

[0087] cls() is the factual question tendency score obtained by the factual question prediction model for a certain question;

[0088] i represents the i-th question in the historical questions;

[0089] k stands for the current question.

[0090] Based on this, the fact score of the user's current question can be obtained.

[0091] In practice, the historical scores of users are saved locally according to their IDs, so that they can be directly called for acceleration in subsequent calculations and replaced with the latest calculation results.

[0092] Decision whether to call RAG:

[0093] When a user inputs a question, we call the factual question preference recognition model and combine it with historical questions to obtain the propensity score of the current question being a factual question. If it is greater than the preset threshold, we believe that this question needs to be answered through RAG. Otherwise, we directly call the capabilities of the base large model to answer it.

[0094] Here, the preset threshold is set to 0.5.

[0095] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0096] The present invention also provides a RAG necessity judgment system in large model retrieval enhancement generation, and the specific technical solution is as follows:

[0097] The acquisition module is used to: obtain current question information raised by users in a social networking site or social media comment area or conversation log;

[0098] The determination module is used to: determine the fact tendency score corresponding to the user according to the current question information;

[0099] The judgment module is used to determine whether it is necessary to call RAG to assist in answering the current question information based on the difference between the factual propensity score and the preset propensity score.

[0100] It should be noted that the beneficial effects of the RAG necessity judgment system in the large model retrieval enhancement generation provided by the above embodiment are the same as the beneficial effects of the RAG necessity judgment method in the large model retrieval enhancement generation provided by the above embodiment, which will not be repeated here. In addition, when the system provided by the above embodiment realizes its functions, it only takes the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the system is divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided by the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0101] like Figure 2 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, the processor 320 is coupled to a memory 310, and the memory 310 stores at least one computer program 330, and the at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 implements any of the above methods, specifically:

[0102] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements a RAG necessity judgment method in a large model retrieval enhancement generation provided in the above embodiment. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 300 may also include other components for realizing device functions, which will not be repeated here.

[0103] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements any one of the above methods.

[0104] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0105] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above methods.

[0106] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0107] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0108] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0109] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for determining the necessity of RAG in large model retrieval enhancement generation, characterized in that: include: Obtain information about current issues raised by users in social networking sites or social media comment sections or conversation logs; Determine the fact tendency score corresponding to the user according to the current question information; Based on the difference between the factual propensity score and the preset propensity score, determine whether it is necessary to call RAG to assist in answering the current question information.

2. The method for determining the necessity of RAG in large model retrieval enhancement generation according to claim 1, characterized in that: The process of determining the fact tendency score corresponding to the user is specifically as follows: The current question information is processed by the trained general model BERT to obtain the current fact tendency score corresponding to the current question information, and the historical fact tendency scores corresponding to all historical question information of the same user before the current question information are called. Based on the current weight and the historical weight, the current fact tendency score and the fact tendency score after weighted summation of the historical fact tendency scores are determined.

3. The method for determining the necessity of RAG in large model retrieval enhancement generation according to claim 2, characterized in that: The training set of the general model BERT includes: The collected questions are cleaned, and the text cleaning includes: removing special characters in the text, HTML tag processing, and case conversion processing.

4. The method for determining the necessity of RAG in large model retrieval enhancement generation according to claim 1, characterized in that: The process of determining whether to call RAG to assist in answering the current question information according to the difference between the factual propensity score and the preset propensity score is as follows: When the difference between the factual propensity score and the preset propensity score is greater than zero, RAG is called to assist in answering the current question information; When the difference between the factual propensity score and the preset propensity score is less than zero, the current question information is directly answered through the base large model.

5. A RAG necessity judgment system in large model retrieval enhancement generation, characterized in that: include: The acquisition module is used to: obtain current question information raised by users in a social networking site or social media comment area or conversation log; The determination module is used to: determine the fact tendency score corresponding to the user according to the current question information; The judgment module is used to determine whether it is necessary to call RAG to assist in answering the current question information based on the difference between the factual propensity score and the preset propensity score.

6. The RAG necessity judgment system in large model retrieval enhancement generation according to claim 5 is characterized in that: The process of determining the fact tendency score corresponding to the user is specifically as follows: The current question information is processed by the trained general model BERT to obtain the current fact tendency score corresponding to the current question information, and the historical fact tendency scores corresponding to all historical question information of the same user before the current question information are called. Based on the current weight and the historical weight, the current fact tendency score and the fact tendency score after weighted summation of the historical fact tendency scores are determined.

7. The RAG necessity judgment system in large model retrieval enhancement generation according to claim 6 is characterized in that: The training set of the general model BERT includes: The collected questions are cleaned, and the text cleaning includes: removing special characters in the text, HTML tag processing, and case conversion processing.

8. The RAG necessity judgment system in large model retrieval enhancement generation according to claim 5 is characterized in that: The process of determining whether to call RAG to assist in answering the current question information according to the difference between the factual propensity score and the preset propensity score is as follows: When the difference between the factual propensity score and the preset propensity score is greater than zero, RAG is called to assist in answering the current question information; When the difference between the factual propensity score and the preset propensity score is less than zero, the current question information is directly answered through the base large model.

9. An electronic device, characterized in that: The electronic device comprises a processor, the processor is coupled to a memory, at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements the method according to any one of claims 1 to 4.

Images