Search enhancement generation method, system and equipment based on process reward optimization and storage medium

Abstract

The invention relates to a process reward optimization-based search enhancement generation method, system and device and a storage medium, and the method comprises the steps of disassembling a query problem input by a user to obtain a sub-query problem of a current sub-query; retrieving a sub-query problem of the current sub-query in a preset database to obtain a plurality of recall documents; calculating a result reward of the current sub-query; and when the result reward of the current sub-query is greater than a preset reward threshold value, repeatedly executing acquisition of the intermediate answer of the sub-query question of the current sub-query, calculation of the result reward, correction judgment and disassembly of the query question until a preset query termination condition is met, and outputting a target answer of the query question. According to the scheme provided by the invention, the question answer generated in the middle can be verified and optimized by disassembling the query question, calculating the result award corresponding to the query question and optimizing the query process based on the award, so that the situation that wrong answers are accumulated to the subsequent query process is reduced, and the query accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a search enhancement generation method, system, device and storage medium based on process reward optimization. Background Technology

[0002] Retrieval-enhanced generation techniques improve the accuracy and traceability of answers by equipping large language models with external knowledge bases, enabling the generation process to be based on retrieved relevant information. This technique typically involves two processing stages: the retrieval stage converts the user query into a vector and performs semantic searches in a pre-built knowledge base to obtain relevant text fragments; the enhancement generation stage concatenates the retrieval results with the original query, using this as additional context input to the large language model to generate the final answer. This approach effectively mitigates the illusion problem caused by the model's reliance on internal parameterized knowledge and is suitable for answering factual descriptive questions.

[0003] However, in complex problem scenarios, large model search is prone to errors, especially when multiple sub-problems that need to be linked or logically combined are involved. The retrieval accuracy will decrease as the problem complexity increases, and the lack of intermediate verification links means that the unidirectional retrieval-generation path will cause the incorrect answers in the preceding steps to directly affect the subsequent reasoning steps, resulting in a significant decrease in the quality of the final generated answer. Summary of the Invention

[0004] To address or partially address the problems existing in related technologies, this application provides a search enhancement generation method, system, device, and storage medium based on process reward optimization.

[0005] The first aspect of this application provides a search enhancement generation method based on process reward optimization, comprising: decomposing a query question input by a user to obtain a subquery question of the current subquery; retrieving the subquery question of the current subquery from a preset database to obtain several recall documents; calculating the result reward of the current subquery based on the intermediate answer of the subquery question obtained from the recall documents, the subquery question, and the recall documents; if the result reward of the current subquery is less than a preset reward threshold, correcting the current subquery until the result reward of the current subquery is greater than the preset reward threshold; when the result reward of the current subquery is greater than the preset reward threshold, further decomposing the query question based on the intermediate answer of the current subquery to obtain the subquery question of the next subquery, which is used as the subquery question of the current subquery; repeatedly executing the acquisition of the intermediate answer of the subquery question of the current query, the calculation of the result reward, and the correction judgment, as well as the decomposition of the query question, until a preset query termination condition is met, and outputting the target answer of the query question; during the query process, constructing a target optimization function containing the result reward of each step of the subquery, with the goal of maximizing the expected reward of each step of the subquery, and optimizing the intermediate answer of each step of the subquery.

[0006] In conjunction with the first aspect, in one possible implementation of the first aspect, the objective optimization function is:

[0007] in, This is the average of the sum of rewards for all subqueries. Indicates the solution The maximum value; To follow the strategy The search yields all possible paths to the target answer, and the average is taken. The reward for the result obtained by executing the subquery at time t+k is the reward for the result of the current subquery. This indicates the search state at that moment, including the current reasoning context, historical subqueries, and intermediate answers; T is the termination time of the query process for the query question. As a discount factor, , where k represents the difference in steps from the current time t to the future time t+k.

[0008] In conjunction with the first aspect, in one possible implementation of the first aspect, the correction of the current subquery includes: determining the search path corresponding to the intermediate answer of the current subquery as an erroneous path, and after pruning its execution path, re-searching to obtain the intermediate answer of the current subquery; or, re-searching the current subquery question in a preset database, and searching for the intermediate answer of the current subquery based on several newly obtained recall documents.

[0009] In conjunction with the first aspect, in one possible implementation of the first aspect, the outcome reward includes a relevance reward and a confidence reward, wherein the relevance reward is calculated according to the following formula:

[0010] In the formula, For the aforementioned relevance reward, For the subquery problem in the current subquery, The retrieved documents are those obtained from the current subquery. The confidence reward is calculated using the following formula:

[0011] In the formula, The confidence level is the reward. ( ) is the entropy function. In the state The probability distribution of the output.

[0012] In conjunction with the first aspect, in one possible implementation of the first aspect, the preset query termination condition includes: each sub-query question obtained from the breakdown of the query question has an intermediate answer, and the result reward of each sub-query is greater than the preset reward threshold; or, the number of sub-query steps reaches a preset maximum value.

[0013] In conjunction with the first aspect, one possible implementation of the first aspect also includes: if the reward of the result of the current subquery is greater than a preset reward threshold, the search path corresponding to the intermediate answer of the current subquery is determined as the dominant search path.

[0014] In conjunction with the first aspect, in one possible implementation of the first aspect, the step of retrieving the subquery question of the current subquery from a preset database to obtain a plurality of recall documents includes: vectorizing the subquery question to obtain a first query vector; retrieving a plurality of second query vectors from the preset vector database whose similarity to the first query vector is greater than a preset similarity threshold; sorting the second query vectors by similarity, and outputting a preset number of recall documents after format conversion.

[0015] The second aspect of this application provides a search-enhanced generation system based on process reward optimization, comprising: The large model is used to break down the user-input query into subquery questions for each step of the subquery. It then obtains intermediate answers to the subquery questions of the current subquery based on the recall documents returned by the retrieval enhancement module. If the reward for the current subquery is greater than a preset reward threshold, it proceeds to break down the subquery questions for the next step; or if the reward for the current subquery is less than a preset reward threshold, it corrects the intermediate answers of the current subquery. Finally, when a preset query termination condition is met, it outputs the target answer to the query question based on the intermediate answers of each step of the subquery. During the query process, a target optimization function is constructed, incorporating the reward for each step of the subquery, to optimize the intermediate answers of each step of the subquery by maximizing the expected reward of each step. The retrieval enhancement module is used to perform retrieval in a preset database based on the subquery question fed back by the large model, and return several recall documents related to the subquery question to the large model; The reward calculation module is used to calculate whether the reward for each subquery step, combined with the subquery question of that subquery step, its corresponding recall document and intermediate answer, is greater than the preset reward threshold, and then feeds it back to the large model.

[0016] A third aspect of this application provides an electronic device, comprising: Processor; and A memory that stores executable code, which, when executed by the processor, causes the processor to perform the method described above.

[0017] A fourth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0018] The technical solution provided in this application may include the following beneficial effects: This application discloses a search enhancement generation method, system, device, and storage medium based on process reward optimization. The method includes: decomposing a user-input query question to obtain a subquery question of the current subquery; retrieving the subquery question of the current subquery from a preset database to obtain several recall documents; calculating the result reward of the current subquery based on the intermediate answer of the subquery question obtained from the recall documents, the subquery question itself, and the recall documents; if the result reward of the current subquery is less than a preset reward threshold, correcting the current subquery until the result reward of the current subquery is greater than the preset reward threshold; when the result reward of the current subquery is greater than the preset reward threshold, further decomposing the query question based on the intermediate answer of the current subquery to obtain the subquery question of the next subquery, which is then used as the subquery question of the current subquery; repeatedly executing the acquisition of the intermediate answer of the subquery question of the current subquery, the calculation of the result reward, the correction judgment, and the decomposition of the query question until a preset query termination condition is met, and outputting the target answer of the query question; during the query process, constructing a target optimization function containing the result rewards of each step of the subquery, with the goal of maximizing the expected reward of each step of the subquery, and optimizing the intermediate answer of each step of the subquery. This approach breaks down the query question, calculates the corresponding reward for the query question, and optimizes the query process based on the reward. This allows for the verification and optimization of the intermediate query answers, reducing the accumulation of incorrect answers in subsequent queries and improving query accuracy.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0020] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0021] Figure 1 This is a flowchart illustrating the search enhancement generation method based on process reward optimization as shown in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of a search-enhanced generation system based on process reward optimization, as shown in an embodiment of this application. Figure 3 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application; Figure 4 This is a schematic block diagram of the structure of a computer storage medium provided in an embodiment of this application. Detailed Implementation

[0022] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0023] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0024] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0025] The core idea of ​​Retrieval-Augmented Generation (RAG) is to equip the Large Language Model (LLM) with an "external knowledge base," enabling its generation process to be based on retrieved relevant information, thereby improving the accuracy, timeliness, and traceability of answers. Its basic principle can be divided into two key stages: Retrieval Stage: When a user submits a query, RAG first performs a semantic search on the query within a massive, pre-built knowledge base (such as company documents, Wikipedia, professional databases, etc.). Documents in the knowledge base are converted into high-dimensional vectors (embedding vectors). The system quickly retrieves the most relevant text fragments by calculating the similarity between the query vector and all document vectors. Augmentation Stage: The retrieved relevant text fragments are used as additional context, concatenated with the user's original query to form an enhanced suggestion, which is then input into the LLM. The LLM's task is to generate the final answer based on this suggestion containing "evidence." This is equivalent to allowing the model to "open the book" and consult specified reference materials before answering the question, effectively reducing the "illusion" problem caused by the model relying on its own internal parameterized knowledge.

[0026] However, in complex problem scenarios, large model search is not likely to yield accurate results. When the thought process involves multiple sub-problems that need to be linked or logically combined, the quality of RAG's answers drops significantly, mainly for the following reasons: First, retrieval accuracy decreases with increasing question complexity: RAG's retrieval steps are parallel and based on semantic similarity. For a complex, compound question, its overall semantics may not accurately match a single document or a set of documents containing all necessary sub-answers. More likely, the system will retrieve partial or incomplete information for each sub-question. For example, for the question "Compare the strategic differences between Company A in market X last year and Company B in market Y this year," the retrieval engine may find documents about Company A, market X, Company B, and market Y separately, but it is extremely difficult to find a text that directly makes this specific comparison. Second, there is a lack of global verification and iterative correction mechanisms: The standard RAG process is a one-way "retrieval-generation" process, lacking an intermediate step to verify the consistency, completeness, and matching degree of multiple retrieved information blocks. When sub-questions are interdependent, an incorrect answer to a previous sub-question will directly affect all subsequent reasoning. For example, in a scenario where a user queries "Who starred in the first English-language film directed by the director of the movie 'Parasite'?", the system first correctly decomposes the subquery "Who is the director of 'Parasite'?" and obtains the answer "Bong Joon-ho". However, in subsequent steps, the system incorrectly generates the subquery "What is Bong Joon-ho's most famous film?" instead of the target subquery "What was the first English-language film directed by Bong Joon-ho?", resulting in the retrieval of irrelevant documents such as "Memories of Murder", and thus outputting an incorrect intermediate answer. This error further contaminates the subsequent reasoning process, causing the system to generate the subquery "Who starred in 'Memories of Murder'?" based on the incorrect information, ultimately outputting a target answer that does not conform to the facts.

[0027] To address the aforementioned issues, this application provides a search enhancement generation method based on process reward optimization. This method can decompose the query question, calculate the process reward corresponding to the query question, and optimize the query process based on the reward. It can also verify and optimize the intermediate query answers to reduce the accumulation of incorrect answers in subsequent query processes, thereby improving query efficiency and accuracy.

[0028] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0029] Figure 1 This is a flowchart illustrating the search enhancement generation method based on process reward optimization as shown in the embodiments of this application.

[0030] See Figure 1 A search-enhanced generation method based on process reward optimization includes: S110: Decompose the query problem entered by the user to obtain the subquery problem of the current subquery.

[0031] Specifically, the user inputs a query question into the preset LLM. The LLM, by reading the input query question, can break it down. In this embodiment of the invention, the user-input question is usually complex, containing several sub-query questions, and these sub-query questions are dependent on each other. The entire query process of the LLM for the query question can include several sub-query steps, and the final target answer is output by obtaining the sub-query results corresponding to each sub-query step. After reading the input sub-query question, the LLM breaks down the first sub-query question that needs to be answered, which is taken as the current sub-query question. For example, if the query question is "Who is the lead actor in the first English film directed by the director of the movie 'Parasite'?", after breakdown, the current sub-query question can be obtained as "Who is the director of 'Parasite'?".

[0032] S120: Search the subquery question of the current subquery in the preset database to obtain several recall documents.

[0033] Specifically, the pre-defined database refers to an external knowledge base that stores a large amount of structured knowledge to supplement the factual knowledge of the LLM. By inputting the current subquery into the pre-defined database for retrieval, recall documents related to the subquery question are obtained.

[0034] In one possible implementation, the subquery question of the current subquery is retrieved in a preset database to obtain a number of recall documents, including: vectorizing the subquery question to obtain a first query vector; retrieving a number of second query vectors from the preset vector database whose similarity to the first query vector is greater than a preset similarity threshold; sorting the second query vectors by similarity, and outputting a preset number of recall documents after format conversion.

[0035] Specifically, upon receiving a subquery, it is first vectorized into a first query vector. Then, a search is performed in a pre-defined vector database to obtain several second query vectors sorted by similarity that have a similarity greater than a pre-defined similarity threshold to the first query vector. For example, by converting the subquery into a first query vector, Top-K relevant text blocks (i.e., recalled documents, such as 5-20 documents / paragraphs) can be retrieved from the pre-defined database using maximum inner product search and cosine similarity. After format conversion, the LLM receives a pre-defined number of recalled documents.

[0036] S130: Calculate the result reward for the current subquery based on the intermediate answer to the subquery question obtained from the recall document search, the subquery question, and the recall document.

[0037] Specifically, the LLM searches the recall document to obtain an intermediate answer to the subquery question. After obtaining the intermediate answer, it combines the intermediate answer, the subquery question, and the recall document to calculate the result reward for the current subquery.

[0038] In one possible implementation, the outcome reward includes a relevance reward and a confidence reward, wherein the relevance reward is calculated according to the following formula:

[0039] In the formula, For relevance rewards, For the subquery problem in the current subquery, The retrieved documents are those obtained from the current subquery. The confidence reward is calculated using the following formula:

[0040] In the formula, For confidence level reward, ( ) is the entropy function. In the state The probability distribution of the output.

[0041] Specifically, the relevance reward is used to measure the semantic relevance between the subquery question and the recalled documents. A higher relevance reward indicates that the retrieved recalled documents are highly relevant to the subquery question. This is a function for calculating the semantic similarity between the subquery question qt and the recalled document dt, for example, using the cosine similarity of the text.

[0042] Confidence rewards, by penalizing low-confidence inferences through prediction entropy, address the "illusion" and "overconfidence" problems of LLM in parametric memory models. In information theory, entropy... Used to measure a probability distribution The uncertainty lies in the distribution. A uniform distribution (i.e., the one with the least uncertainty in prediction) has the highest entropy. In this embodiment, for LLM in token generation... Predicted probability at time That is, the probability that the model predicts the next token is v, and its negative entropy is defined as:

[0043] In the confidence reward formula, In the state The probability distribution of the output, i.e., the model in the state. Below is the predicted probability distribution for the next token. This indicates punishment; low confidence represents entropy. The confidence reward is very large. Very small; while high confidence represents entropy. Very small, thus confidence reward It is close to zero, or a tiny negative number.

[0044] If the model is very certain about the intermediate answer (low entropy): The model is penalized slightly or without penalty to encourage reliance on internal knowledge and save retrieval costs; if the model is highly uncertain about the answer (high entropy): The negative reward is very large. The model receives a strong negative reward. This negative reward signal is transmitted back to the previous action that led to this high-entropy state, namely the "choose not to retrieve" action, through the temporal difference (TD) update mechanism of reinforcement learning, thereby encouraging the model to choose the "retrieve" action when uncertain.

[0045] The confidence reward based on negative entropy forces the model to learn "self-censorship." That is, when it finds that its internal knowledge is insufficient to confidently generate an answer, it tends to choose "retrieval" as the next action, because choosing "not to retrieve" incurs a strong penalty. This effectively transforms the probabilistic uncertainty signal built into LLM into a retrieval gating signal in the RAG decision-making process, thereby improving the scientific rigor and accuracy of RAG decision-making.

[0046] Traditional LLM only optimizes the negative log-likelihood or the final reward, lacking direct constraints on the output distribution entropy, leading to a tendency for the model to produce an overly sharp erroneous confidence distribution (a high-confidence illusion). This invention introduces a distribution entropy regularization term into the policy optimization objective, causing the model to directly penalize low-entropy erroneous outputs during training. This forces the model to maintain higher entropy or actively invoke external knowledge when evidence is insufficient, systematically improving calibration and factual consistency from the reward mechanism level, significantly outperforming traditional RAG methods and improving the accuracy of query answers.

[0047] S140: If the reward of the current subquery is less than the preset reward threshold, then the current subquery is corrected until the reward of the current subquery is greater than the preset reward threshold.

[0048] Specifically, the result reward for this subquery step is calculated based on the relevance reward and the confidence reward. The result is compared with a preset reward threshold to determine whether to accept the intermediate answer of the subquery. For example, if the preset reward threshold is 0, then... They consider it a positive reward and acknowledge the search results; if If this is considered a negative reward, the intermediate answer needs to be corrected and regenerated. The threshold of 0 can also be adjusted according to the actual application needs; it is a hyperparameter in the model application.

[0049] In one possible implementation, correcting the current subquery includes: identifying the search path corresponding to the intermediate answer of the current subquery as an incorrect path, and after pruning its execution path, re-searching to obtain the intermediate answer of the current subquery; or, re-searching the current subquery question in a preset database, and searching for the intermediate answer of the current subquery based on several newly obtained recall documents.

[0050] Specifically, the search path refers to the decision-making, retrieval, and reasoning processes corresponding to the query step during the generation of intermediate answers. When the reward for the result of the query step is less than a preset reward threshold (i.e., a negative reward), the LLM will identify the search path corresponding to the intermediate answer of the current subquery as an erroneous path. The model can remove or reduce the priority of the erroneous path and its derivative paths in subsequent search processes, thereby avoiding re-exploring known invalid search directions. Then, the model returns to the decision position of the previous step and re-retrieve using different retrieval strategies to obtain a new intermediate answer; or, it can re-retrieve the current subquery question from the preset database. This re-retrieval can use different retrieval parameters, keyword combinations, and other different retrieval strategies to obtain new recalled documents, and then re-search based on the new recalled documents to obtain the intermediate answer of the current subquery.

[0051] In this embodiment of the invention, a result reward is calculated for the intermediate answer of each subquery step through result reward. When the result is a negative reward, the intermediate answer is corrected in a timely manner, which avoids the contamination of the subsequent reasoning process by the intermediate erroneous result, thereby improving the accuracy and reliability of the target answer.

[0052] S150: When the reward of the current subquery is greater than the preset reward threshold, the query problem is further decomposed by combining the intermediate answer of the current subquery to obtain the subquery problem of the next subquery, which is then used as the subquery problem of the current subquery.

[0053] Specifically, when the reward of the current subquery result is greater than the preset reward threshold, which is a positive incentive, the intermediate answer obtained by the current subquery is considered to meet expectations. The LLM uses the intermediate answer of the current query as new context information to further decompose the query problem and obtain the subquery problem of the next subquery, which is then used as the subquery problem of the current subquery.

[0054] In one possible implementation, if the reward of the current subquery is greater than a preset reward threshold, the search path corresponding to the intermediate answer of the current subquery is determined as the dominant search path.

[0055] Specifically, the dominant search path is defined as follows: during the search process, if the reward corresponding to the intermediate answer of the current subquery is greater than the preset reward threshold (i.e., a positive reward), then the action corresponding to the intermediate answer will be generated, including the subquery question, the retrieved recall document, and the reasoning process for generating the intermediate answer, etc., and marked as a high-priority search path. This allows subsequent queries to prioritize the information from the dominant search path, thereby improving overall search efficiency and accuracy.

[0056] S160: Repeatedly execute the acquisition of intermediate answers to the subquery question of the current subquery, the calculation of result rewards and the correction judgment, as well as the decomposition of the query question, until the preset query termination condition is met, and output the target answer of the query question; during the query process, construct a target optimization function containing the result rewards of each step of the subquery, with the goal of maximizing the expected reward of each step of the subquery, and optimize the intermediate answer of each step of the subquery.

[0057] Specifically, for each subquery step, the process repeatedly executes document retrieval, intermediate answer search, result reward calculation and correction judgment, and processing of the subquery question to obtain the next subquery by combining the intermediate answer from the previous step, until the preset query termination condition is met, and the final target answer of the query question is output. Throughout the entire query process of the model, a target optimization function containing the result reward of each subquery step is constructed, with the goal of maximizing the expected reward of each subquery step, to optimize the intermediate answer of each subquery step.

[0058] In one possible implementation, the objective optimization function is:

[0059] in, This is the average of the sum of rewards for all subqueries. Indicates the solution The maximum value; To follow the strategy The search yields all possible paths to the target answer, and the average is taken. The reward for the result obtained by executing the subquery at time t+k is the reward for the result of the current subquery. This represents the search state at this moment, including the current reasoning context, historical subqueries, and intermediate answers; T is the termination time of the query process for the query question. As a discount factor, , where k represents the difference in steps from the current time t to the future time t+k.

[0060] Specifically, cumulative summation ( () indicates that from the current time until the end time The results are summed up. γ is used to measure the importance of future rewards relative to current rewards. When γ is close to 1, the large language model will pay more attention to long-term rewards; when γ is close to 0, the large language model will pay more attention to immediate rewards. For example, γ can be 0.7 to balance long-term rewards and immediate rewards. Indicates the passage of time Increase, future rewards will be The "discount" on the product encourages the model to complete inference as quickly as possible, i.e., to pursue an efficient path, while also reflecting the uncertainty of future rewards. Expectation ( This indicates that the model is following the policy. In this case, the choice of each state is probabilistic, so It is the average of the total rewards for all possible paths, the objective. This is the solution to the best prediction under uncertainty.

[0061] The objective optimization function can predict the cumulative future reward obtained by following a certain policy, i.e., from the current state. Initially, the expected cumulative process reward is obtained by following policy π until the query process terminates at time T. Policy π controls how the model generates subqueries, selects recall documents, and obtains intermediate answers based on the recall documents in each query step. Policy π can be a deterministic policy, i.e., always choosing the same action given a state, such as the policy gradient algorithm in reinforcement learning; or it can be a stochastic policy, i.e., choosing different actions with a certain probability given a state, by solving... The maximum value of π can be found from all possible search strategies to maximize the average cumulative reward in the future. By calculating the expected value of each query step, different strategies π can be evaluated to iteratively adjust the model's strategy π and select the optimal search strategy π, thereby maximizing the cumulative reward action of the entire query process. This avoids the model getting stuck in local optima when generating intermediate answers, thus generating a more accurate target answer.

[0062] Traditional LLM relies solely on a binary 0 / 1 signal provided by the correctness of the final answer or a single-outcome reward model (ORM) score, resulting in a linear or even exponential increase in credit allocation error with the number of inference steps T, leading to extremely high training variance. This invention, by quantifying and optimizing each intermediate inference step, achieves credit allocation accurate to a single step, significantly reducing the variance of the advantage function estimation, greatly improving sample efficiency and long-chain complex inference, solving the problems of traditional sparse rewards in multi-step tasks, and improving the efficiency and accuracy of target answer generation.

[0063] In one possible implementation, the preset query termination conditions include: each subquery obtained from the breakdown of the query question has an intermediate answer, and the result reward of each subquery is greater than a preset reward threshold; or, the number of subquery steps reaches a preset maximum value.

[0064] Specifically, when each subquery question has obtained a corresponding intermediate answer and the reward for each step of the subquery is greater than the preset reward threshold, the model stops searching and outputs the target answer; or, when the number of steps in the subquery has reached the preset maximum value, usually 3 to 6 steps, the model stops searching and outputs the target answer.

[0065] The implementation of the present invention will be illustrated below with an example. For example, a user inputs a query: "Who starred in the first English-language film directed by the director of the movie 'Parasite'?". First, the LLM breaks down the query into a subquery: "Who directed 'Parasite'?". Then, it searches a pre-defined database using this subquery to obtain several recall documents. From these documents, it searches for the intermediate answer to the subquery: "Bong Joon-ho," and calculates the reward for the current subquery. If the reward is positive, the next subquery is executed. The model then combines the intermediate answer "Bong Joon-ho," the query, and the recall documents to further break down the query into a subquery: "What was the first English-language film directed by Bong Joon-ho?". After searching for and obtaining recall documents, it finds the intermediate answer to this subquery: "Okja." The reward for this subquery is calculated; if it is negative, the subquery is immediately corrected. The corrected subquery generates a new intermediate answer: "Snowpiercer," which is now a positive reward, and the next subquery is executed. Repeat the process of obtaining the subquery results of the current subquery, calculating the result reward, and making correction judgments. When the intermediate answer "Chris Evans" is obtained, the model determines that all subquery questions have obtained valid intermediate answers, and the result rewards of each step are greater than the preset reward threshold, thus satisfying the preset query termination condition. Then, the target answer of the original query question is output: "Chris Evans".

[0066] This application discloses a search enhancement generation method based on process reward optimization, comprising: decomposing a user-input query question to obtain a subquery question of the current subquery; retrieving the subquery question of the current subquery from a preset database to obtain several recall documents; calculating the result reward of the current subquery based on the intermediate answer of the subquery question obtained from the recall documents, the subquery question itself, and the recall documents; if the result reward of the current subquery is less than a preset reward threshold, correcting the current subquery until the result reward of the current subquery is greater than the preset reward threshold; when the result reward of the current subquery is greater than the preset reward threshold, further decomposing the query question based on the intermediate answer of the current subquery to obtain a subquery question of the next subquery, which is then used as the subquery question of the current query; repeatedly performing the retrieval, result reward calculation, and correction judgment of the subquery question of the current query, as well as the decomposition of the query question, until a preset query termination condition is met, and outputting the target answer of the query question; during the query process, constructing a target optimization function containing the result rewards of each step of the subquery, with the goal of maximizing the expected reward of each step of the subquery, and optimizing the intermediate answer of each step of the subquery. This approach breaks down the query question, calculates the corresponding reward for the query question, and optimizes the query process based on the reward. This allows for the verification and optimization of the intermediate query answers, reducing the accumulation of incorrect answers in subsequent queries and improving query efficiency and accuracy.

[0067] Corresponding to the aforementioned application function implementation method embodiments, this application also provides a search enhancement generation system based on process reward optimization, an electronic device, and corresponding embodiments.

[0068] Figure 2 This is a schematic diagram of the structure of a search-enhanced generation system based on process reward optimization, as shown in an embodiment of this application.

[0069] See Figure 2 A search enhancement generation system 200 based on process reward optimization includes a large model 210, a retrieval enhancement module 220, and a reward calculation module 230. The system includes: The large model 210 is used to decompose the query question input by the user into subquery questions for each step, and obtain the intermediate answer of the subquery question of the current subquery based on the recall document search returned by the retrieval enhancement module; when the result reward of the current subquery is greater than a preset reward threshold, the model proceeds to decompose the subquery question of the next subquery, or when the result reward of the current subquery is less than a preset reward threshold, the model corrects the intermediate answer of the current subquery; and when a preset query termination condition is met, the model outputs the target answer of the query question based on the intermediate answers of each step subquery; during the query process, a target optimization function containing the result rewards of each step subquery is constructed to optimize the intermediate answer of each step subquery with the goal of maximizing the expected reward of each step subquery. The retrieval enhancement module 220 is used to perform a retrieval in a preset database based on the subquery question fed back by the large model, and return several recall documents related to the subquery question to the large model; The reward calculation module 230 is used to calculate whether the reward of each step subquery, combined with the subquery question of that step subquery, its corresponding recall document and intermediate answer, is greater than the preset reward threshold, and then feeds it back to the large model.

[0070] Regarding the system in the above embodiments, the specific ways in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0071] For further details regarding the implementation of the above technical solutions by each module in the search enhancement generation system based on process reward optimization, please refer to the description in the search enhancement generation method based on process reward optimization provided in the above embodiments of the invention, which will not be repeated here.

[0072] Based on the above-described search enhancement generation method based on process reward optimization, this invention also provides a search enhancement generation processing device based on process reward optimization, as shown in the schematic diagram below. Figure 3 As shown, the process reward-based search enhancement generation device 300 includes a processor 301 and a memory 302 coupled to the processor 301. The memory 302 stores a computer program, which, when executed by the processor 301, causes the processor 301 to perform the steps of the process reward-based search enhancement generation method in the above embodiments.

[0073] For further details regarding the implementation of the above-mentioned technical solution by the processor 301 in the search enhancement generation processing device based on process reward optimization, please refer to the description in the search enhancement generation method based on process reward optimization provided in the above-mentioned embodiments of the invention, which will not be repeated here.

[0074] The processor 301 can also be called a CPU (Central Processing Unit). The processor 301 may be an integrated circuit chip with signal processing capabilities. The processor 301 may also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor may be a microprocessor, or the processor 301 may be any conventional processor.

[0075] This invention also provides a computer-readable storage medium, the structure of which is illustrated in the following diagram: Figure 4 As shown, the storage medium 400 stores a readable computer program 401. This computer program 401 can be stored in the storage medium as a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks or optical disks, ROM (Read-Only Memory), RAM (Random Access Memory), or terminal devices such as computers, servers, mobile phones, and tablets.

[0076] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or modules, and may be electrical, mechanical, or other forms.

[0077] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0078] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0079] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0080] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0081] The technical solutions provided in this application have been described in detail above. Specific examples have been used in this application to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0082] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0083] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0084] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0085] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0086] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A search-enhanced generation method based on process reward optimization, characterized in that, include: The query input by the user is broken down to obtain the subquery of the current subquery; The subquery question of the current subquery is retrieved from the preset database to obtain several recall documents; The result reward for the current subquery is calculated based on the intermediate answer to the subquery question obtained from the recall document search, the subquery question, and the recall document. If the reward of the current subquery is less than the preset reward threshold, the current subquery will be corrected until the reward of the current subquery is greater than the preset reward threshold. If the reward of the current subquery is greater than the preset reward threshold, the query problem is further broken down by combining the intermediate answer of the current subquery to obtain the subquery problem of the next subquery, which is then used as the subquery problem of the current subquery. Repeatedly execute the process of obtaining intermediate answers to the subquery question of the current subquery, calculating result rewards and making correction judgments, as well as decomposing the query question, until the preset query termination condition is met, and output the target answer of the query question; During the query process, a target optimization function is constructed that includes the reward for the result of each subquery step. The goal is to optimize the intermediate answer of each subquery step by maximizing the expected reward of each subquery step.

2. The method according to claim 1, characterized in that, The objective optimization function is: in, This is the average of the sum of rewards for all subqueries. Indicates the solution The maximum value; To follow the strategy The search yields all possible paths to the target answer, and the average is taken. The reward for the result obtained by executing the subquery at time t+k is the reward for the result of the current subquery. This indicates the search state at that moment, including the current reasoning context, historical subqueries, and intermediate answers; T is the termination time of the query process for the query question. As a discount factor, , where k represents the difference in steps from the current time t to the future time t+k.

3. The method according to claim 1, characterized in that, The correction of the current subquery includes: The search path corresponding to the intermediate answer of the current subquery is identified as an incorrect path. After pruning the execution path, the search is re-examined to obtain the intermediate answer of the current subquery. Alternatively, the current subquery question can be retrieved again in the preset database, and based on the newly obtained recall documents, an intermediate answer to the current subquery can be found.

4. The method according to claim 2, characterized in that, The reward for the outcome includes a relevance reward and a confidence reward, wherein... The correlation reward is calculated using the following formula: In the formula, For the aforementioned relevance reward, For the subquery problem in the current subquery, The retrieved documents are those obtained from the current subquery. The confidence reward is calculated using the following formula: In the formula, The confidence level is the reward. ( ) is the entropy function. In the state The probability distribution of the output.

5. The method according to claim 1, characterized in that, The preset query termination conditions include: Each sub-query obtained from the breakdown of the query problem has an intermediate answer, and the reward of each sub-query result is greater than the preset reward threshold. Alternatively, the number of subquery steps reaches the preset maximum value.

6. The method according to claim 1, characterized in that, Also includes: If the reward for the current subquery is greater than the preset reward threshold, the search path corresponding to the intermediate answer of the current subquery will be determined as the dominant search path.

7. The method according to claim 1, characterized in that, The step involves retrieving the subquery question of the current subquery from a preset database to obtain several recall documents, including: The subquery problem is vectorized to obtain the first query vector; Several second query vectors with a similarity greater than a preset similarity threshold to the first query vector are retrieved from the preset vector database; The second query vector is sorted by similarity, and after format conversion, a preset number of the recalled documents are output.

8. A search-enhanced generation system based on process reward optimization, characterized in that, include: The large model is used to break down the query question input by the user into subquery questions of each step, and to obtain the intermediate answer of the subquery question of the current subquery based on the recall document search returned by the retrieval enhancement module; If the reward for the current subquery is greater than a preset reward threshold, the subquery problem is broken down to proceed to the next subquery. Alternatively, if the reward for the current subquery is less than a preset reward threshold, the intermediate answer of the current subquery is corrected. Furthermore, when a preset query termination condition is met, the target answer of the query problem is output based on the intermediate answers of each subquery step. During the query process, a target optimization function containing the reward for each subquery step is constructed to optimize the intermediate answer of each subquery step by maximizing the expected reward of each subquery step. The retrieval enhancement module is used to perform retrieval in a preset database based on the subquery question fed back by the large model, and return several recall documents related to the subquery question to the large model; The reward calculation module is used to calculate whether the reward for each subquery step, combined with the subquery question of that subquery step, its corresponding recall document and intermediate answer, is greater than the preset reward threshold, and then feeds it back to the large model.

9. An electronic device, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-7.

Images