Retrieval enhancement generation method and device, equipment, storage medium and program product

Abstract

The invention discloses a retrieval enhancement generation method and device, equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence, the method comprises the following steps: making a decision on query information input by a user through a preset reinforcement learning model to obtain a decision result, the reinforcement learning model being constructed and obtained based on a strategy gradient method and a PPO algorithm; then storing the vectorized data converted from the query information into a preset knowledge base, wherein the knowledge base is used for relevance retrieval of the vectorized data; and finally, based on the decision result and the knowledge base, determining an answer result corresponding to the query information through a large language model. According to the method, when the user inputs the query information, the decision is made through the reinforcement learning model, whether to access the knowledge base or not is determined according to the decision result, and dialogue history control in continuous multi-round dialogues is optimized, so that tokens of the dialogues transmitted into a large language model are reduced, the accuracy of answers is improved, and the illusion problem is avoided.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a retrieval enhancement generation method, device, equipment, storage medium and program product. Background Art

[0002] Automatic question answering is a technology that uses computer systems to answer natural language questions raised by users. It can obtain knowledge from data of various types, formats and sources, and generate corresponding answers based on the user's questions. The existing question answering system based on Retrieval Augmented Generation (RAG) consists of two components: a retrieval model and an answer generation system based on a large language model (LLM). The retrieval model obtains the context related to the user's query; then, the query and the retrieved context are input into the LLM, and the answer is generated using appropriate prompts. The LLM has acquired the ability of contextual question answering through extensive training, and Retrieval Augmented Generation (RAG) further enables the chatbot to answer questions in the field of communication.

[0003] However, in the existing question-answering system, multiple rounds of dialogues are usually conducted through RAG during a question-answering process. The dialogue history needs to be maintained and transmitted to LLM, which increases the input tokens and leads to decreased answer accuracy or hallucination problems. Summary of the invention

[0004] The main purpose of this application is to provide a retrieval enhancement generation method, device, equipment, storage medium and program product, which aims to solve the technical problem that the existing question-answering system usually conducts multiple rounds of dialogue to pass the dialogue history to LLM, resulting in an increase in input tags, resulting in a decrease in answer accuracy or hallucinations.

[0005] To achieve the above objectives, the present application proposes a search enhancement generation method, the method comprising:

[0006] A decision is made on the query information input by the user through a preset reinforcement learning model to obtain a decision result, wherein the reinforcement learning model is constructed based on a policy gradient method and a PPO algorithm;

[0007] The vectorized data converted from the query information is stored in a preset knowledge base, where the knowledge base is used for correlation retrieval of the vectorized data;

[0008] Based on the decision result and the knowledge base, an answer result corresponding to the query information is determined through a large language model.

[0009] In one embodiment, the step of determining the answer result corresponding to the query information by using a large language model based on the decision result and the knowledge base includes:

[0010] When the decision result is failure to obtain, inputting the query information into the large language model, and outputting an answer result corresponding to the query information;

[0011] When the decision result that cannot be obtained is not received, a cosine similarity algorithm is used to perform a relevance search in the knowledge base according to the query information to obtain a context document;

[0012] The context document and the query information are input into the large language model, and an answer result corresponding to the query information is output.

[0013] In one embodiment, after the step of determining the answer result corresponding to the query information by using a large language model based on the decision result and the knowledge base, the step further includes:

[0014] Obtaining the evaluation result of the user based on the feedback of the answer result;

[0015] Using a preset supervised training model, the answer result, the evaluation result and the decision result are used as training parameters;

[0016] The training parameters are input into the reinforcement learning model for supervised training using reflection propagation.

[0017] In one embodiment, the step of inputting the training parameters into the reinforcement learning model for supervised training by using reflection propagation includes:

[0018] Determine the target vector based on the output layer and linear layer of the preset BERT model;

[0019] Determining a loss result of the target vector according to the training parameters and a preset loss function;

[0020] The loss result is input into the reinforcement learning model to adjust the model parameters by using reflection propagation.

[0021] In one embodiment, the step of making a decision on the query information input by the user through a preset reinforcement learning model to obtain a decision result includes:

[0022] Making decisions on the query information input by the user through the model decision formula of the reinforcement learning model to determine multiple current strategies;

[0023] Based on the PPO algorithm, the multiple current strategies are searched and evaluated according to the knowledge base to determine the decision result of the query information.

[0024] In one embodiment, the step of storing the vectorized data converted from the query information in a preset knowledge base includes:

[0025] Performing natural language processing on the query information to obtain multiple segmented documents;

[0026] Performing vectorization processing on the plurality of segmented documents through a vector embedding model to obtain vectorized data of the query information;

[0027] The vectorized data is stored in a preset knowledge base.

[0028] In addition, to achieve the above purpose, the present application also proposes a search enhancement generation device, the device comprising:

[0029] A decision result module is used to make a decision on the query information input by the user through a preset reinforcement learning model to obtain a decision result. The reinforcement learning model is constructed based on the policy gradient method and the PPO algorithm;

[0030] A knowledge base module, used to store the vectorized data converted from the query information into a preset knowledge base, where the knowledge base is used for correlation retrieval of the vectorized data;

[0031] The retrieval answer module is used to determine the answer result corresponding to the query information through a large language model based on the decision result and the knowledge base.

[0032] In addition, to achieve the above-mentioned purpose, the present application also proposes a retrieval enhancement generation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the retrieval enhancement generation method as described above.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the retrieval enhancement generation method described above are implemented.

[0034] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the retrieval enhancement generation method described above are implemented.

[0035] One or more technical solutions proposed in this application have at least the following technical effects: this application first makes a decision on the query information input by the user through a preset reinforcement learning model to obtain a decision result, and the reinforcement learning model is constructed based on the policy gradient method and the PPO algorithm; then the vectorized data converted from the query information is stored in a preset knowledge base, and the knowledge base is used for correlation retrieval of vectorized data; finally, based on the decision result and the knowledge base, the answer result corresponding to the query information is determined through a large language model. Because this application will first make a decision through a reinforcement learning model when the user inputs query information, and decide whether to access the knowledge base based on the decision result, optimize the dialogue history control in multiple consecutive rounds of dialogue to reduce the tokens passed into the large language model, thereby improving the accuracy of the answer and avoiding the hallucination problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0038] Figure 1 The overall logic flow chart provided for the search enhancement generation method of this application;

[0039] Figure 2 A schematic diagram of a process flow provided for the first embodiment of the search enhancement generation method of the present application;

[0040] Figure 3 A schematic diagram of a flow chart provided for the second embodiment of the search enhancement generation method of the present application;

[0041] Figure 4 A schematic diagram of the module structure of a search enhancement generation device according to an embodiment of the present application;

[0042] Figure 5 A schematic diagram of the device structure of the hardware operating environment involved in the retrieval enhancement generation method in the embodiment of the present application.

[0043] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0044] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0045] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0046] Since the existing question-answering system (i.e., RAG system) usually conducts multiple rounds of dialogue in a question-answering process, it has the following shortcomings:

[0047] 1. The actual question parameters passed into the large language model depend on the context of the retrieval model output, so if the retrieval model fails to obtain the relevant context, the generated answer may be incorrect.

[0048] 2. During a question-and-answer process, multiple rounds of conversations are usually conducted through RAG, and the conversation history needs to be maintained and passed to LLM, resulting in an increase in input tokens, which in turn leads to a decrease in answer accuracy or hallucinations.

[0049] 3. Since multi-round conversations need to maintain conversation histories, the number of context input tokens will increase during the conversation. As a result, the time required for each call increases with the number of conversation rounds or the amount of queries. The time cost and computing resource cost required for large model reasoning will increase significantly.

[0050] In response to the above-mentioned shortcomings, this application proposes a retrieval enhanced generation method, which mainly utilizes a large language model (LLM) and retrieval enhanced generation technology (RAG) to build a chatbot for answering data questions in the communication field, and combines a reinforcement learning model to achieve efficient use of the knowledge base of the large language model and the retrieval model, thereby improving the overall question and answer quality, retrieval efficiency and response speed.

[0051] Specifically, Figure 1 As shown, Figure 1 The overall logical flow chart provided for the retrieval enhancement generation method of this application. First, a reinforcement learning model based on the policy gradient method is constructed, which is applied to pre-process the user queries passed into the RAG system before calling the large language model, and generate calling decisions for the knowledge base and dialogue history, thereby optimizing the calling input of the large language model. Based on the answer results of the large language model, the reinforcement learning model is supervised and trained using reflection propagation to optimize the decision of the reinforcement learning model. Through this solution, the problems of resource consumption and low efficiency caused by repeated retrieval of interactive texts by the general question-and-answer mechanism can be solved. At the same time, the control of dialogue history in consecutive rounds of dialogue can also be optimized, thereby reducing the number of input tokens and avoiding accuracy degradation and hallucination problems.

[0052] It should be noted that the execution subject of this embodiment can be an electronic device with information decision-making, vectorization and model question-answering functions, such as a personal computer, a chat robot, etc., or an electronic device capable of realizing the above functions, a search enhancement generation device (referred to as a generation device) that executes the search enhancement generation method of this application, etc., and this embodiment does not limit this. The following takes the generation device as an example to illustrate this embodiment and the following embodiments.

[0053] Based on this, the present application embodiment provides a search enhancement generation method, referring to Figure 2 , Figure 2 A flowchart diagram of the first embodiment of the retrieval enhancement generation method of the present application is provided.

[0054] In this embodiment, the search enhancement generation method includes steps S10 to S30:

[0055] Step S10: Make a decision on the query information input by the user through a preset reinforcement learning model to obtain a decision result, wherein the reinforcement learning model is constructed based on a policy gradient method and a PPO algorithm.

[0056] It should be noted that the query information is the question sentence or expression input by the user to the generating device. The generating device will analyze, process and retrieve the query information and provide corresponding results.

[0057] It should be noted that the reinforcement learning model, also known as the reward model (RM) or discriminant model, is a module used to learn and make decisions on the query information input by the user. It is mainly used to determine the retrieval status of the query information in the knowledge base, so as to guide the large language model to decide whether to access the knowledge base according to the actual situation.

[0058] Among them, the knowledge base is a vector database used to store and manage vectorized data of query information input by users, supporting efficient similarity search and retrieval operations.

[0059] Policy Gradient Methods is an algorithm in reinforcement learning that determines how to adjust the policy to improve performance by calculating the gradient of the policy. Policy Gradient Methods estimates the gradient of the policy and then updates the policy parameters according to the direction of the gradient, so that the policy gradually develops in a better direction to determine whether the real request passed to the large language model contains context.

[0060] The PPO algorithm, or Proximal Policy Optimization algorithm, is used to update policies to maximize expected returns. The PPO algorithm can be used to reversely train reinforcement learning models and perform small parameter updates near the old policy to improve the decision-making performance of the trained reinforcement learning model.

[0061] It should be noted that the decision result is the result of the reinforcement learning model making a decision based on the query information to determine whether the actual request transmitted to the large language model contains context, such as whether the context is available or not.

[0062] like Figure 1 As shown in the figure, in the traditional RAG system, the knowledge base directly acts on the large language model. This embodiment adds a part of reinforcement learning rewards, pre-places the reinforcement reward model in the input link of the user query, and performs decision preprocessing on the input, which directly affects the input of the final call to the large language model, so as to reduce the tokens passed in the dialogue and improve the accuracy of the answer.

[0063] In this implementation, by prepending the enhanced reward model to the input stage of the user query, the problem of strong correlation between the existing RAG question-answering system and the knowledge base can be solved, so that it can be decided whether to access the knowledge base based on actual conditions and reduce the tokens passed in the conversation, thereby improving the accuracy of the answer and reducing the computing resources and computing time.

[0064] In a feasible implementation, step S10 of this embodiment may include the steps of: making a decision on the query information input by the user through the model decision formula of the reinforcement learning model to determine multiple current strategies; based on the PPO algorithm, searching and evaluating the multiple current strategies according to the knowledge base to determine the decision result of the query information.

[0065] It should be noted that the model decision formula is a mathematical expression of the reinforcement learning model's decision on the query information, so as to inform the large language model of the decision result.

[0066] Through the model decision formula, the reinforcement learning model can process the query information given by the user and generate decisions or prediction results.

[0067] Specifically, for ease of understanding, such as Figure 1 As shown, the reinforcement learning model of this embodiment makes decisions on user queries based on the Policy Gradient Methods, and notifies the RAG system and the supervised training model of the decision results. If the result of the policy function output is 0, the decision result is [Fetch] (i.e., get), if it is 1, the decision result is [NoFetch] (i.e., fail to get), and the specific model decision formula is:

[0068]

[0069] in, The performance function represents the gradient of the performance function J with respect to the policy parameter θ. The performance function usually measures the quality of the policy, which is the cumulative reward here. t=0∑T-1 is the summation symbol, which represents the accumulation of all time steps from the initial time step t=0 to the final time step T-1. represents the log-likelihood gradient of the policy, which measures the sensitivity of the policy to the probability of taking a specific action in a given state. πθ (s t , a t )] is the action value function of the reinforcement learning model in the St state. π θ (a t |s t ) is the policy function, i.e., the reinforcement learning model itself. τ～πθ It is the expectation of the model on all trajectories, that is, the sum of all calculations based on the motion trajectory formula from t = 0 to T-1.

[0070] In actual scenarios, this formula means calculating the policy gradient of each action step through the experience gained in the real environment (i.e., trajectory τ), and then adjusting the policy parameters θ based on these gradients, with the goal of maximizing the expected cumulative reward in the future. The PPO algorithm can limit the amplitude of policy updates to prevent large fluctuations in the learning process, ensure the stability of learning, and try to find the optimal policy relative to the old policy in each iteration, i.e., "proximal" optimization.

[0071] Specifically, when the reinforcement learning model receives a user query, it makes a decision on the query information input by the user through a decision formula and determines multiple current strategies; then it decides whether to retrieve information from the knowledge base (i.e., take [Fetch] or [No Fetch] actions) based on the current strategy. That is, according to the PPO algorithm, the performance under different strategies (i.e., accuracy of answers, resource consumption, etc.) will be evaluated, and the parameters in RM will be adjusted through the above formula to make it more inclined to select strategies that can efficiently utilize resources and improve the quality of answers. This not only improves the intelligence level of question and answer, but also reduces resource waste and ensures the efficient operation of the question and answer system.

[0072] In this implementation, user queries are input into a reinforcement learning model built based on a policy gradient method for parallel processing, thereby improving efficiency. At the same time, the reinforcement learning model is decoupled and deployed independently, and can be upgraded separately at any time without affecting the original RAG system. Furthermore, the reinforcement learning model acts on the query input of a large language model, thereby improving the accuracy of the query input of the large language model.

[0073] Step S20: storing the vectorized data converted from the query information in a preset knowledge base, where the knowledge base is used for correlation retrieval of the vectorized data.

[0074] It should be noted that vectorized data refers to data that represents query information in the form of vectors. By converting originally complex query information with multiple attributes into vectors, large language models can be used to perform contextual retrieval in knowledge bases.

[0075] It should be noted that the knowledge base is a vector database (such as Milvus, Embedding DB, etc.) used to store and manage vectorized data. The database supports efficient similarity search and retrieval operations.

[0076] Specifically, the query information may be represented as vectorized data through a bag-of-words model, or may be represented as vectorized data through a vector embedding model, which is not limited in this embodiment.

[0077] In this implementation, query information is vectorized to complete the construction of a knowledge base or knowledge expansion.

[0078] In a feasible implementation, step S20 of this embodiment may include the steps of: performing natural language processing on the query information to obtain multiple segmented documents; performing vectorization processing on the multiple segmented documents through a vector embedding model to obtain vectorized data of the query information; and storing the vectorized data in a preset knowledge base.

[0079] It should be noted that the vector embedding model converts the segmented documents processed by query information into the form of embedding vectors to represent their features, and is a data storage facility specifically used to store these embedding vectors in order to efficiently manage and retrieve these vectors.

[0080] The Embedding Model may be a vectorized model based on a pre-trained model (such as m3e, bge, etc.). The main content of the knowledge base includes text paragraphs formed by multiple segmented documents and vector representations formed by converting the segmented documents using the Embedding Model.

[0081] For ease of understanding, for example Figure 1As shown, after receiving the query information input by the user, the query information can be uploaded to the natural language processing module for document uploading and parsing. Usually, the long text D of the query information is segmented into Np paragraphs {P1, P2, ...} (i.e., multiple segmented documents), and the number of characters contained in each paragraph does not exceed a threshold value Lmax, which is related to the hardware conditions of the current computing environment. Then, the multiple segmented documents are vectorized, specifically, each paragraph is converted into a vector representation using a vector model (such as an Embedding Model), and the vectorized data is stored in the Embedding DB, at which point the knowledge base construction or knowledge expansion is completed.

[0082] In this implementation, by building a knowledge base, vectors related to query information can be efficiently managed and retrieved.

[0083] Step S30: Based on the decision result and the knowledge base, determine the answer result corresponding to the query information through a large language model.

[0084] It should be noted that a large language model (LLM) refers to a language model with a large number of parameters and powerful language understanding and generation capabilities.

[0085] Specifically, according to the above decision results, it can be determined whether the large language model retrieves the context information corresponding to the query information from the knowledge base; if so, the corresponding context information is retrieved from the knowledge base and output to generate the answer result; if not, the large language model directly answers the result based on the query information. This reduces the situation where the large language model blindly calls the knowledge base and causes too many tokens.

[0086] This implementation can solve the problem that the traditional question-answering model fails to pay attention to whether the input tokens themselves called by the large language model are correct, resulting in low quality of large language model calls.

[0087] In a feasible implementation, step S30 of this embodiment may include the steps of: when the decision result is that the query information cannot be obtained, inputting the query information into the large language model, and outputting the answer result corresponding to the query information; when the decision result of the failure to obtain is not received, performing a relevance search in the knowledge base according to the query information through a cosine similarity algorithm to obtain a context document; inputting the context document and the query information into the large language model, and outputting the answer result corresponding to the query information.

[0088] It should be noted that failure to obtain (i.e., the decision result of [No Fetch]) is the result of being unable to query the context from the knowledge base.

[0089] The traditional RAG system will always associate the contextual relationship of multiple rounds of questions and answers, which will lead to OOD (out of distribution range) or hallucination problems. The reinforcement learning model can establish a decision for the user's original input, thereby improving the accuracy of the prompt input and solving the OOD or hallucination problem. At the same time, the traditional RAG system will pass the context of the knowledge base query into the large model when answering. As the number of question and answer rounds increases, the input tokens data will also increase, which will affect the question and answer efficiency of the large model and increase resource consumption. The decision of the reinforcement learning model added by this embodiment can reduce the input of tokens data, thereby improving efficiency.

[0090] It should be noted that the cosine similarity algorithm is an algorithm used to measure the similarity between the vectorized data corresponding to the query information and each vector in the knowledge base.

[0091] Specifically, the cosine similarity algorithm of the relevance score (formula as follows) can be used to calculate the relevance score (similarity score) of each document in the knowledge base, and the top n relevant answer documents are screened. The specific formula is as follows:

[0092]

[0093] Among them, A is the vector corresponding to the user's query information, B is the vector representation of each document in the knowledge base, and A i , B i Represents the values ​​of vectors A and B in the i-th dimension respectively. “.” represents the dot product of the vectors, i.e., the sum of the products of the values ​​of the corresponding positions of the two vectors. |A| and |B| represent the modulus or length of the vectors, i.e., the Euclidean norm, calculated as the square root of the sum of the squares of the vector components. n is the dimension of the vector, which is 512 here.

[0094] Specifically, Figure 1 As shown in the figure, according to the decision result of the reinforcement learning model, the "prompt input" generates the final question request to the large language model. At the same time, after the "prompt input" receives the processing result of the "vector model", it performs the following steps according to the decision result:

[0095] If the reinforcement learning model selects [No Fetch], the user's query information is directly put into the large language model LLM for direct answering and generating corresponding results.

[0096] If the reinforcement learning model selects [Fetch], or there are abnormal situations such as no decision result is found or the decision result times out, the original question is rewritten and the intent is identified, and the relevant context documents are retrieved from the knowledge base. At this time, the cosine similarity algorithm can be selected as the context filtering algorithm for the query information, and the filtered documents are combined with the "prompt input" and put into the large language model to answer the question and generate an answer.

[0097] In this implementation, considering that the traditional RAG processing mechanism has the step of filtering the context of the query information, by using the reinforcement learning type to participate in the pre-retrieval work, the problem of not re-retrieving the knowledge base context in subsequent queries is effectively avoided, reducing the need for repeated retrieval, thereby saving resources and costs, while maintaining the accuracy of the answer. It also solves the problem that the document question and answer based on the knowledge base under the OOD problem is not accurate, or may produce "hallucinations". Using the reinforcement learning model, it can accurately determine whether the question is within the field, and can accurately decide whether to use the knowledge base information for question and answer.

[0098] In the technical solution provided in this embodiment, after receiving the query information input by the user, a decision is made on it through the front-end reinforcement learning model to obtain the decision result. The query information is then converted into vectorized data and stored in a preset knowledge base. At this time, "prompt input" generates a final question request to the large language model. If the reinforcement learning model selects [No Fetch], the user's query information is directly placed in the large language model LLM for direct answering and generating corresponding results. If the reinforcement learning model selects [Fetch], or there are abnormal situations such as no decision result is found or the decision result times out, the original question is rewritten, intent is recognized, and other steps are performed, and the relevant context documents are retrieved from the knowledge base, and the large language model is placed to answer the question and generate an answer. Because this embodiment will first make a decision through the reinforcement learning model when the user enters the query information, and decide whether to access the knowledge base based on the decision result, optimize the dialogue history control in multiple consecutive rounds of dialogue, so as to reduce the tokens of the dialogue passed into the large language model, thereby improving the accuracy of the answer and avoiding the hallucination problem.

[0099] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 and Figure 3 , Figure 3 A flowchart diagram is provided for the second embodiment of the retrieval enhancement generation method of the present application.

[0100] After step S30 in this example, the search enhancement generation method further includes steps S41 to S43:

[0101] Step S41: Obtain the evaluation result of the user based on the answer result feedback.

[0102] For example, Figure 1 As shown, after the large language model gives an answer, the user interacts with the answer and generates an evaluation result, which can then be fed back to the reinforcement reward learning model.

[0103] Step S42: Using a preset supervised training model, the answer result, the evaluation result and the decision result are used as training parameters.

[0104] It should be noted that the supervised training model is a model that uses the PPO algorithm to supervise the decision-making ability of the reinforcement learning model.

[0105] Step S43: using reflection propagation to input the training parameters into the reinforcement learning model for supervised training.

[0106] It should be noted that reflection propagation is a method of updating the parameters of the reinforcement learning model using backpropagation technology.

[0107] Specifically, for ease of understanding, such as Figure 1 As shown in the figure, the supervised training module combines the decision results of the reinforcement learning model with the answer results of the large language model, and uses reflection propagation to perform reflection training on the reinforcement learning model. The advantage of this is that the actual output of the large language model, the user's evaluation results, and the decision results of the reinforcement learning model can be used as training parameters to adjust the parameters of the gradient decision function of the reinforcement learning model, thereby improving the accuracy of the decision.

[0108] In this implementation, the accuracy of the decision-making of the reinforcement learning model can be further improved through reflection training.

[0109] In a feasible implementation, step S43 of this example includes the steps of: determining the target vector according to the output layer and linear layer of the preset BERT model; determining the loss result of the target vector according to the training parameters and the preset loss function; and using reflection propagation to input the loss result into the reinforcement learning model to adjust the model parameters.

[0110] It should be noted that the BERT model is a natural language processing model based on the Transformer architecture. The preset loss function is used to calculate the loss result to reflect the formula for training the reinforcement learning model.

[0111] Specifically, we can first use the BERT model, take the [CLS] token (i.e., the output layer) of the model as the output, and add a linear probe (linear layer) to get a vector (i.e., the target vector, the final dimension is 2).

[0112] Then use the following formula to perform softmax and cross entropy on the vector (combine the probability distribution of the Softmax layer output, and then calculate the Cross Entropy loss with the true label to perform optimization training) to calculate a loss result, and use reflection propagation to train the reinforcement learning model.

[0113] L = -ylog(p) - (1 - y)log(1 - p);

[0114] Among them, L represents the output value of the loss function, which is used to measure the error degree of the model prediction. The smaller it is, the closer the prediction is to the correct answer. y is the actual label, which indicates the action that the model should take (in this case, y = 0 represents [Fetch], which needs to be retrieved from the knowledge base; y = 1 represents [No Fetch], which does not require retrieval and can be answered directly). p is the positive class probability predicted by the reinforcement learning model, that is, the probability of taking a certain action. If p is close to 1, the model strongly recommends taking [Fetch], otherwise it tends to [No Fetch].

[0115] By calculating this loss function, we can clearly know whether the model's decision is "good" or "bad" in a given situation, and then use optimization methods such as gradient descent to adjust the parameters of the reinforcement learning model RM, gradually approaching the optimal strategy. This process ensures that the model can not only intelligently determine whether to use the knowledge base, but also continuously optimize its decision logic in long-term learning, reduce resource waste, and improve user experience.

[0116] For example, to facilitate understanding, if a user asks "How to set up a 5G package", RM predicts that using a specific document in the knowledge base can greatly improve the accuracy of the answer by analyzing historical data and current policies (assuming that p is very close to 0 at this time, corresponding to [Fetch]), so it chooses to retrieve relevant information from the knowledge base and generate a detailed answer in combination with LLM. On the contrary, for a general question such as "Where is your company's office", RM may predict that it is sufficient to answer directly with LLM (assuming that p is close to 1 at this time, corresponding to [No Fetch]), avoiding unnecessary knowledge base retrieval, thereby saving computing resources.

[0117] In this implementation, by calculating this loss function, it is possible to clarify whether the model's decision is "good" or "bad" in a given situation, and then use optimization methods such as gradient descent to adjust the parameters of the reinforcement learning model RM, gradually approach the optimal strategy, and continuously optimize the decision logic of the reinforcement learning model.

[0118] In the technical solution provided in this embodiment, by continuously optimizing the decision logic of the reinforcement learning model, the intelligence and accuracy of the question-answering system in the communication field can be improved, so that it can more effectively process user queries and provide more accurate and comprehensive answers, thereby providing users with better experience and services.

[0119] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the search enhancement generation method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0120] This application also provides a search enhancement generation device, please refer to Figure 4 , Figure 4 This is a schematic diagram of the module structure of the search enhancement generation device in an embodiment of the present application; the search enhancement generation device includes:

[0121] A decision result module 401 is used to make a decision on the query information input by the user through a preset reinforcement learning model to obtain a decision result, wherein the reinforcement learning model is constructed based on a policy gradient method and a PPO algorithm;

[0122] A knowledge base module 402 is used to store the vectorized data converted from the query information in a preset knowledge base, where the knowledge base is used for correlation retrieval of the vectorized data;

[0123] The retrieval answer module 403 is used to determine the answer result corresponding to the query information through a large language model based on the decision result and the knowledge base.

[0124] In this embodiment, the retrieval answer module 403 is also used to input the query information into the large language model and output the answer result corresponding to the query information when the decision result is that the query information cannot be obtained; when the decision result of the query information cannot be obtained is not received, a relevance search is performed in the knowledge base according to the query information through the cosine similarity algorithm to obtain a context document; the context document and the query information are input into the large language model, and the answer result corresponding to the query information is output.

[0125] Furthermore, the retrieval enhancement generation device also includes a supervised training module, which is used to obtain the evaluation results of the user based on the answer result feedback; through a preset supervised training model, the answer result, the evaluation result and the decision result are used as training parameters; and the training parameters are input into the reinforcement learning model for supervised training using reflection propagation.

[0126] Furthermore, the supervised training module is also used to determine the target vector according to the output layer and linear layer of the preset BERT model; determine the loss result of the target vector according to the training parameters and the preset loss function; and use reflection propagation to input the loss result into the reinforcement learning model to adjust the model parameters.

[0127] Furthermore, the decision result module 401 is also used to make decisions on the query information input by the user through the model decision formula of the reinforcement learning model to determine multiple current strategies; based on the PPO algorithm, the multiple current strategies are retrieved and evaluated according to the knowledge base to determine the decision result of the query information.

[0128] Furthermore, the knowledge base module 402 is also used to perform natural language processing on the query information to obtain multiple segmented documents; vectorize the multiple segmented documents through a vector embedding model to obtain vectorized data of the query information; and store the vectorized data in a preset knowledge base.

[0129] Other embodiments or specific implementations of the retrieval enhancement generation device of the present application can refer to the above-mentioned method embodiments, which will not be repeated here.

[0130] The search enhancement generation device provided by the present application adopts the search enhancement generation method in the above embodiment, which can solve the technical problem that the existing question-answering system usually conducts multiple rounds of dialogue to pass the dialogue history to LLM, resulting in an increase in input tags, resulting in a decrease in answer accuracy or hallucinations. Compared with the prior art, the beneficial effects of the search enhancement generation device provided by the present application are the same as the beneficial effects of the search enhancement generation method provided by the above embodiment, and the other technical features in the search enhancement generation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0131] The present application provides a retrieval enhancement generation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the retrieval enhancement generation method in the above-mentioned embodiment one.

[0132] Reference below Figure 5 , Figure 5The device structure diagram of the hardware operating environment involved in the search enhancement generation method in the embodiment of the present application shows a schematic diagram of the structure of the search enhancement generation device suitable for implementing the embodiment of the present application. The search enhancement generation device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The retrieval enhancement generation device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0133] like Figure 5 As shown, the retrieval enhancement generation device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the retrieval enhancement generation device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the search enhancement generation device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a search enhancement generation device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0134] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0135] The search enhancement generation device provided by the present application adopts the search enhancement generation method in the above embodiment, which can solve the technical problem that the existing question-answering system usually conducts multiple rounds of dialogue to pass the dialogue history to LLM, resulting in an increase in input tags, resulting in a decrease in answer accuracy or hallucinations. Compared with the prior art, the beneficial effects of the search enhancement generation device provided by the present application are the same as the beneficial effects of the search enhancement generation method provided by the above embodiment, and the other technical features in the search enhancement generation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0136] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0137] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0138] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the search enhancement generation method in the above-mentioned embodiment.

[0139] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0140] The computer-readable storage medium may be included in the retrieval enhancement generation device; or may exist independently without being assembled into the retrieval enhancement generation device.

[0141] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the retrieval enhancement generation device, the retrieval enhancement generation device: makes a decision on the query information input by the user through a preset reinforcement learning model to obtain a decision result, and the reinforcement learning model is constructed based on the policy gradient method and the PPO algorithm; stores the vectorized data converted from the query information in a preset knowledge base, and the knowledge base is used for relevance retrieval of vectorized data; based on the decision result and the knowledge base, determines the answer result corresponding to the query information through a large language model.

[0142] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0143] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] The modules involved in the embodiments described in the present application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0145] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned search enhancement generation method, and can solve the technical problem that the existing question-answering system usually conducts multiple rounds of dialogues to pass the dialogue history to the LLM, resulting in an increase in input tags, resulting in a decrease in answer accuracy or hallucinations. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the search enhancement generation method provided by the above-mentioned embodiment, and will not be repeated here.

[0146] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned retrieval enhancement generation method when executed by a processor.

[0147] The computer program product provided by the present application can solve the technical problem that the existing question-answering system usually conducts multiple rounds of dialogue to pass the dialogue history to the LLM, resulting in an increase in input tags, which leads to a decrease in answer accuracy or hallucinations. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the retrieval enhancement generation method provided by the above embodiment, which will not be repeated here.

[0148] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A search enhancement generation method, characterized in that: The method includes: A decision is made on the query information input by the user through a preset reinforcement learning model to obtain a decision result, wherein the reinforcement learning model is constructed based on a policy gradient method and a PPO algorithm; The vectorized data converted from the query information is stored in a preset knowledge base, where the knowledge base is used for correlation retrieval of the vectorized data; Based on the decision result and the knowledge base, an answer result corresponding to the query information is determined through a large language model.

2. The method according to claim 1, characterized in that The step of determining the answer result corresponding to the query information through a large language model based on the decision result and the knowledge base includes: When the decision result is failure to obtain, inputting the query information into the large language model, and outputting an answer result corresponding to the query information; When the decision result that cannot be obtained is not received, a cosine similarity algorithm is used to perform a relevance search in the knowledge base according to the query information to obtain a context document; The context document and the query information are input into the large language model, and an answer result corresponding to the query information is output.

3. The method according to claim 1, characterized in that After the step of determining the answer result corresponding to the query information through a large language model based on the decision result and the knowledge base, the method further includes: Obtaining the evaluation result of the user based on the feedback of the answer result; Using a preset supervised training model, the answer result, the evaluation result and the decision result are used as training parameters; The training parameters are input into the reinforcement learning model for supervised training using reflection propagation.

4. The method according to claim 3, characterized in that The step of inputting the training parameters into the reinforcement learning model for supervised training by using reflection propagation includes: Determine the target vector based on the output layer and linear layer of the preset BERT model; Determining a loss result of the target vector according to the training parameters and a preset loss function; The loss result is input into the reinforcement learning model to adjust the model parameters by using reflection propagation.

5. The method according to claim 1, characterized in that The step of making a decision on the query information input by the user through a preset reinforcement learning model to obtain a decision result includes: Making decisions on the query information input by the user through the model decision formula of the reinforcement learning model to determine multiple current strategies; Based on the PPO algorithm, the multiple current strategies are searched and evaluated according to the knowledge base to determine the decision result of the query information.

6. The method according to any one of claims 1 to 5, characterized in that The step of storing the vectorized data converted from the query information in a preset knowledge base comprises: Performing natural language processing on the query information to obtain multiple segmented documents; Performing vectorization processing on the plurality of segmented documents through a vector embedding model to obtain vectorized data of the query information; The vectorized data is stored in a preset knowledge base.

7. A search enhancement generation device, characterized in that: The device comprises: A decision result module is used to make a decision on the query information input by the user through a preset reinforcement learning model to obtain a decision result. The reinforcement learning model is constructed based on the policy gradient method and the PPO algorithm; A knowledge base module, used to store the vectorized data converted from the query information into a preset knowledge base, where the knowledge base is used for correlation retrieval of the vectorized data; The retrieval answer module is used to determine the answer result corresponding to the query information through a large language model based on the decision result and the knowledge base.

8. A search enhancement generation device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the retrieval enhancement generation method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the retrieval enhancement generation method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the retrieval enhancement generation method according to any one of claims 1 to 6 are implemented.

Images