Hybrid retrieval enhanced reward model training method and system
By optimizing the generation strategy of the large language model through hybrid retrieval mode, temperature control, and reinforcement learning, the problem of insufficient accuracy of LLM-generated content is solved, achieving high-quality generation and evaluation accuracy in professional fields, and enhancing its application value in professional fields.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-07
- Publication Date
- 2026-04-03
AI Technical Summary
Existing Large Language Models (LLMs) often exhibit an illusion phenomenon when generating professional content, due to the limitations of pre-training data and the randomness of generation strategies. They fail to accurately capture the implicit business rules and judgment logic of professional fields, resulting in insufficient accuracy of generated content and limiting their application value in professional fields.
A hybrid retrieval mode is adopted to obtain relevant documents from the knowledge base. Combined with preset temperature parameters to control iterative generation, a reward model is trained based on relevance evaluation ranking and target loss function, and the generation strategy of the large language model is optimized through reinforcement learning PPO algorithm to ensure that the generated knowledge conforms to professional standards and factual accuracy.
It improves the generation quality of large language models in professional fields, solves the problems of inaccurate retrieval, illusion in generation, and unreliable evaluation, and significantly enhances its application value in professional fields.
Smart Images
Figure CN121786473A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing technology, and in particular to a method and system for training a reward model with hybrid retrieval enhancement. Background Technology
[0002] With the widespread application of natural language processing (NLP) technology in professional services, Large Language Models (LLMs) have become a core support for scenarios such as intelligent question answering and professional consultation. Especially in the insurance sector, LLMs need to generate accurate responses based on massive amounts of policy terms and claims rules to meet users' efficient business consultation needs. However, when generating professional content, LLMs often exhibit a "hallucination" phenomenon—inconsistent with actual business rules—due to limitations in the professional knowledge within the pre-training data or the randomness of the generation strategy. Improving the factual accuracy and professional adaptability of LLM-generated content has become a significant challenge restricting its in-depth application in professional fields.
[0003] To alleviate the inaccuracy issue of LLM-generated content, related technologies employ a Retrieval-Enhanced Generation (RAG) framework. This framework builds a professional knowledge base and uses keyword or vector retrieval modes to extract factual knowledge from the knowledge base to supplement the implicit knowledge within the LLM. External retrieval information is then used to correct generation biases. The Retrieval-Enhanced Generation (RAG) framework refers to a natural language processing framework that integrates information retrieval and LLM generation capabilities. The traditional core process is "retrieval-reordering-generation." By retrieving factual knowledge from the corpus to supplement the implicit knowledge within the LLM parameters, it balances the factual accuracy and contextual coherence of the generated content.
[0004] However, the relevant technologies rely solely on the mechanical matching of retrieval algorithms, which cannot accurately capture the implicit business rules and judgment logic in professional fields. Ultimately, they cannot guide LLM to generate content that conforms to insurance business standards and has both factual accuracy and professional rigor, which seriously limits the application value of LLM in professional fields. Summary of the Invention
[0005] This invention provides a hybrid retrieval-enhanced reward model training method and system, which can solve the defects of inaccurate retrieval knowledge and illusion of generated knowledge in the prior art. By providing accurate reward signals for the RLHF framework, it improves the response quality of LLM in professional fields.
[0006] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions: Firstly, a hybrid retrieval-enhanced reward model training method is provided. This method includes: for an input user query, retrieving relevant documents from a knowledge base using a hybrid retrieval mode; filtering the relevant documents to obtain N retrieval knowledge items, where the hybrid retrieval mode includes keyword retrieval and vector retrieval; controlling an initial large language model to iteratively generate N generated knowledge items based on a preset temperature parameter; responding to the relevance evaluation results of each input retrieval knowledge item and each generated knowledge item with the user query, sorting the N retrieval knowledge items and N generated knowledge items from largest to smallest according to the relevance evaluation results to obtain a preference ranking result; training a reward model using a target loss function based on the preference ranking result to obtain a trained reward model; wherein the target loss function introduces a dynamically weighted hybrid retrieval score, and the hybrid retrieval score is a weighted sum of keyword retrieval scores and vector retrieval scores; and iteratively optimizing the generation strategy of the initial large language model using a reinforcement learning PPO algorithm, so that the fluctuation range of the reward score of the generated knowledge items of the initial large language model is less than a preset range for consecutive preset rounds, thereby obtaining an optimized large language model.
[0007] The beneficial effects of this invention are as follows: The method provided by this invention completes the process of acquiring retrieval knowledge through hybrid retrieval, generating knowledge through temperature control, ranking based on relevance evaluation, training a reward model by fusing retrieval scores, and optimizing a large language model using the PPO algorithm. It specifically addresses the core shortcomings of existing technologies in combining retrieval enhancement generation with reinforcement learning: First, the hybrid retrieval mode can accurately match query terms through keyword retrieval on the one hand, and capture deep semantic connections through vector retrieval on the other. The combination of these two approaches ensures that the selected N retrieval knowledge points not only fit the literal requirements of the query but also conform to the deep semantic intent. Second, the preset temperature parameter controls the LLM iterative generation, avoiding… Traditional generation methods suffer from insufficient diversity, leading to repetition or excessive randomness and creating illusions. This invention ensures that the generated knowledge (N pieces) possess both quantity and fundamental quality. Furthermore, preference ranking based on relevance assessment provides explicit supervisory signals for Resource Management (RM), addressing the lack of effective ranking data in existing RM systems. The objective loss function introduces dynamically weighted hybrid retrieval scores, ensuring that RM scoring considers both ranking priority and the factual reliability of the retrieved knowledge, avoiding evaluation bias caused by RM relying solely on textual features. Finally, the PPO algorithm iteratively optimizes the LLM generation strategy, aiming for a stable reward score for RM, ensuring that the knowledge generated by the LLM consistently meets professional requirements. In summary, the method provided by this invention improves the quality of LLM generation, effectively solving the problems of inaccurate retrieval, illusions in generation, and unreliable evaluation in traditional methods, significantly enhancing the application value of LLM in professional fields.
[0008] In one possible implementation of the first aspect, the step of retrieving relevant documents from a knowledge base using a hybrid retrieval mode for the input user query, and filtering the relevant documents to obtain N retrieval knowledge includes: preprocessing the documents in the knowledge base, using a pre-trained sentence embedding model to convert each document into an embedding vector of a preset dimension, and storing the embedding vector in a vector database; extracting keywords from the user query and determining the keyword retrieval score between the user query and the documents in the vector database; inputting the user query into the pre-trained sentence embedding model to generate a query embedding vector corresponding to the user query, determining the semantic similarity between the query embedding vector and the embedding vectors of each document in the vector database, and using the semantic similarity as the vector retrieval score; constructing an attention layer, inputting the query embedding vector, keyword retrieval score, and vector retrieval score of the user query into the attention layer, and dynamically learning weight coefficients through an attention mechanism, wherein the weight coefficients are greater than or equal to 0 and less than or equal to 1; calculating a hybrid retrieval score based on the weight coefficients; sorting the documents in the vector database in descending order according to the hybrid retrieval score, selecting the top N documents, and determining the N documents as the N retrieval knowledge, where N is a positive integer greater than 1.
[0009] The method provided by this invention uses a pre-trained sentence embedding model to convert documents into embedding vectors. Compared with traditional simple encoding methods, it can more accurately capture the semantic information of documents, laying the foundation for the accuracy of subsequent vector retrieval. Furthermore, the dynamic learning weight coefficients of the attention layer can adaptively adjust the weights of the two retrieval methods according to the query type. Finally, the top N documents are selected in descending order of the mixed retrieval score to ensure that the final retrieved retrieval knowledge is the optimal result under the dual dimensions of term matching and semantic association. This provides high-quality factual knowledge support for subsequent preference ranking and RM training, reducing subsequent process deviations caused by poor retrieval knowledge quality.
[0010] In one possible implementation of the first aspect, the step of controlling the initial large language model to perform N iterations of generating knowledge based on a preset temperature parameter to obtain N generated knowledge includes: setting the preset temperature parameter of the initial large language model; constructing structured prompt words, wherein the structured prompt words contain the core needs guidance of the user query, the content format requirements of the generated knowledge, and professional domain constraints; concatenating the user query and the structured prompt words into initial input text; inputting the initial input text into the initial large language model, performing N iterations of generation tasks, and obtaining N generated knowledge.
[0011] The method provided by this invention addresses the issues of traditional LLM (Local Mastery Management) knowledge generation deviating from requirements and exhibiting unstable quality by combining structured prompts with temperature parameter control. Structured prompts explicitly guide users to their core query needs, content format requirements, and professional domain constraints, directly guiding the initial LLM to focus on the core query and preventing content deviation or formatting issues. Preset temperature parameters balance the diversity and accuracy of generated knowledge, preventing content duplication due to excessively low temperatures and illusions due to excessively high temperatures. This combination allows the initial LLM to directly obtain N generated knowledge items that conform to professional standards and possess differentiation during N iterations of generation tasks without additional screening. This provides qualified samples for subsequent ranking and comparison with retrieved knowledge, improving the efficiency and quality of the generation process.
[0012] In one possible implementation of the first aspect, the step of training a reward model using a target loss function based on the preference ranking result to obtain a trained reward model includes: determining multiple associated sample pairs from the preference ranking result, each associated sample pair including a first knowledge sample and a second knowledge sample, wherein the priority of the first knowledge sample is greater than or equal to that of the second knowledge sample; determining a mixed retrieval score for each first knowledge sample and the second knowledge sample; and training the reward model based on the target loss function, using each associated sample pair and the mixed retrieval score of each first knowledge sample and the second knowledge sample as training data to obtain a trained reward model. The formula for determining the mixed retrieval score is as follows: ; in, For mixed retrieval scores, Keyword search score, For vector retrieval scores, These are the weighting coefficients; The method provided by this invention trains a knowledge management system (RM) by extracting associated sample pairs and combining them with mixed retrieval scores. It determines the associated sample pairs of the first knowledge sample and the second knowledge sample from the preference ranking results, providing clear pairwise comparative supervision signals for the RM and avoiding evaluation bias caused by single text scoring. At the same time, it calculates a mixed retrieval score for each sample and incorporates it into the training data, so that the RM learns not only by referring to the ranking priority, but also by combining the factual reliability of the retrieved knowledge. This avoids the problem that the RM relies solely on text semantics and ignores the quality of the knowledge source. The RM trained in this invention can more accurately quantify the quality of knowledge and provide a reliable reward signal for subsequent LLM optimization.
[0013] In one possible implementation of the first aspect, the target loss function Loss is: ; Where E represents the expected calculation, For the sigmoid function, The predicted reward value for the first knowledge sample. The predicted reward value for the second knowledge sample. The mixed retrieval score of the first knowledge sample. The mixed retrieval score is the score of the second knowledge sample.
[0014] The method provided by this invention offers a target loss function comprising two parts. The former quantifies the scoring differences of the Retrieval Management (RM) on high- and low-priority samples using the sigmoid function, ensuring that the RM can effectively distinguish between superior and inferior samples. The latter introduces mixed retrieval score differences, matching the scoring differences of the RM with the differences in retrieval reliability of the samples, thus avoiding the RM from misjudging samples with low factual accuracy but high ranking. The combination of these two parts guides the RM to simultaneously consider priority judgment and factual reliability assessment, significantly reducing RM scoring bias and providing a more accurate basis for value judgment in LLM optimization.
[0015] In one possible implementation of the first aspect, the iterative optimization of the generation strategy of the initial large language model based on the trained reward model using the reinforcement learning PPO algorithm includes: constructing a dual-model architecture for the PPO algorithm, using the initial large language model as an Actor strategy model, the Actor strategy model being used to receive user queries and corresponding N retrieval knowledge items, and to generate candidate knowledge sequences based on the user queries and the N retrieval knowledge items; using the trained reward model as a Critic value model, the Critic value model being used to evaluate the reward value of the candidate knowledge sequences generated by the Actor strategy model; and executing an iterative optimization process according to preset optimization parameters until the fluctuation range of the reward score of the knowledge generated by the initial large language model is less than a preset range for a consecutive preset number of rounds, thereby obtaining an optimized large language model.
[0016] The method provided by this invention constructs a PPO dual-model architecture and sets preset optimization parameters to limit the division of labor between the Actor policy model (generating candidate knowledge) and the Critic value model (evaluating reward value), thus avoiding inefficient optimization caused by functional confusion. The preset optimization parameters ensure that LLM optimization aims at stable reward scores, avoiding excessive iteration and wasting resources. At the same time, KL divergence constraints control the policy update amplitude of the Actor model, preventing policy mutations from causing fluctuations in generation quality. This can efficiently guide the LLM generation strategy to iterate towards higher reward scores, ultimately obtaining a stable optimized large language model.
[0017] In one possible implementation of the first aspect, the iterative optimization process for any round includes: concatenating the user query and the corresponding N retrieved knowledge items into input text; inputting the input text into the Actor strategy model, which outputs one candidate knowledge sequence based on a preset generation strategy; inputting the candidate knowledge sequence into the Critic value model, which outputs an evaluation result, the evaluation result including an actual reward value reflecting the actual quality of the candidate knowledge sequence and a predicted reward value reflecting the expected quality of the candidate knowledge sequence; calculating an advantage function value based on the difference between the actual reward value and the predicted reward value, the advantage function value being used to quantify the value of the actual quality of the candidate knowledge sequence exceeding the expected quality; constructing the strategy gradient loss function of the Actor strategy model based on the advantage function value and a preset KL divergence constraint threshold, and updating the parameters of the Actor strategy model through a backpropagation algorithm.
[0018] The method provided by this invention refines the iterative process, concatenating the user query and N retrieved knowledge points into input text. This allows the Actor model to reference factual knowledge when generating candidate knowledge, avoiding a disconnect from reality. The Critic model outputs the actual reward value and the predicted reward value, and calculates the advantage function value by the difference between the two, accurately quantifying the value of candidate knowledge exceeding expectations and providing a clear direction for policy updates. The method combines preset KL divergence constraints to construct a policy gradient loss, ensuring that the Actor model parameter updates follow the high-value direction without deviating too much from professional standards. The precise design of each round of the process allows the LLM generation strategy to be gradually optimized under factual constraints and value guidance, accelerating model convergence.
[0019] Secondly, the present invention provides a hybrid retrieval-enhanced reward model training system, the system comprising: a knowledge retrieval module, used to retrieve relevant documents from a knowledge base using a hybrid retrieval mode in response to an input user query, and to filter the relevant documents to obtain N retrieval knowledge items, wherein the hybrid retrieval mode includes keyword retrieval and vector retrieval; a knowledge generation module, used to control an initial large language model to iterate and generate N generated knowledge items based on a preset temperature parameter; and a preference ranking module, used to, in response to the relevance evaluation results of each input retrieval knowledge item and each generated knowledge item with the user query, rank the N retrieval knowledge items according to the relevance evaluation results. The generated knowledge is sorted from largest to smallest to obtain a preference ranking result. A model training module is used to train a reward model based on the preference ranking result using a target loss function, resulting in a trained reward model. The target loss function incorporates a dynamically weighted mixed retrieval score, which is a weighted sum of keyword retrieval scores and vector retrieval scores. A model optimization module is used to iteratively optimize the generation strategy of the initial large language model using a reinforcement learning PPO algorithm, based on the trained reward model, so that the fluctuation range of the reward score of the generated knowledge in the initial large language model is less than a preset range for consecutive preset rounds, resulting in an optimized large language model.
[0020] Thirdly, an electronic device is provided, the electronic device including a memory and one or more processors; the memory is coupled to the processors; wherein the memory stores computer program code, the computer program code including computer instructions, which, when executed by the processor, cause the electronic device to perform the method as described in any implementation of the first aspect.
[0021] Fourthly, a computer-readable storage medium is provided, including computer instructions that, when executed on an electronic device, cause the electronic device to perform a method as described in any implementation of the first aspect.
[0022] Fifthly, a computer program product is provided that, when run on a computer, causes the computer to perform the method in any implementation of the first aspect.
[0023] Understandably, the beneficial effects achieved by the system of the second aspect, the electronic device of the third aspect, the computer-readable storage medium of the fourth aspect, and the computer program product of the fifth aspect provided above can be referred to with reference to the beneficial effects of the first aspect and any of its possible design embodiments, which will not be repeated here. Attached Figure Description
[0024] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a hybrid retrieval enhancement reward model training method provided in an embodiment of the present invention; Figure 3 A flowchart illustrating another hybrid retrieval enhancement reward model training method provided in an embodiment of the present invention; Figure 4 A schematic diagram illustrating the process of a hybrid retrieval-enhanced reward model training method provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a training system provided in an embodiment of the present invention. Detailed Implementation
[0025] The technical solutions of the embodiments of the present invention will be described below with reference to the accompanying drawings. In the description of the present invention, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can represent A or B. The "or" in the present invention is merely a description of the relationship between the related objects, indicating that three relationships can exist. For example, A or B can represent: A alone, A and B simultaneously, and B alone. A and B can be singular or plural. Furthermore, in the description of the present invention, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items.
[0026] Furthermore, to facilitate a clear description of the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.
[0027] In this embodiment of the invention, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this embodiment of the invention should not be construed as superior or more advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner for ease of understanding.
[0028] With the widespread application of natural language processing (NLP) technology in professional services, Large Language Models (LLMs) have become a core support for scenarios such as intelligent question answering and professional consultation. Especially in the insurance sector, LLMs need to generate accurate responses based on massive amounts of policy terms and claims rules to meet users' efficient business consultation needs. However, when generating professional content, LLMs often exhibit a "hallucination" phenomenon—inconsistent with actual business rules—due to limitations in the professional knowledge within the pre-training data or the randomness of the generation strategy. Improving the factual accuracy and professional adaptability of LLM-generated content has become a significant challenge restricting its in-depth application in professional fields.
[0029] To alleviate the inaccuracy issue of LLM-generated content, related technologies employ a Retrieval-Enhanced Generation (RAG) framework. This framework builds a professional knowledge base and uses keyword or vector retrieval modes to extract factual knowledge from the knowledge base to supplement the implicit knowledge within the LLM. External retrieval information is then used to correct generation biases. The Retrieval-Enhanced Generation (RAG) framework refers to a natural language processing framework that integrates information retrieval and LLM generation capabilities. The traditional core process is "retrieval-reordering-generation." By retrieving factual knowledge from the corpus to supplement the implicit knowledge within the LLM parameters, it balances the factual accuracy and contextual coherence of the generated content.
[0030] However, the relevant technologies rely solely on the mechanical matching of retrieval algorithms, which cannot accurately capture the implicit business rules and judgment logic in professional fields. Ultimately, they cannot guide LLM to generate content that conforms to insurance business standards and has both factual accuracy and professional rigor, which seriously limits the application value of LLM in professional fields.
[0031] In view of this, embodiments of the present invention provide a hybrid retrieval-enhanced reward model training method and system. The method includes: for an input user query, retrieving relevant documents from a knowledge base using a hybrid retrieval mode, filtering the relevant documents to obtain N retrieval knowledge, wherein the hybrid retrieval mode includes keyword retrieval and vector retrieval; controlling an initial large language model to iterate and generate N generated knowledge based on a preset temperature parameter, thereby obtaining N generated knowledge; responding to the relevance evaluation results of each input retrieval knowledge and each generated knowledge with the user query, sorting the N retrieval knowledge and N generated knowledge from largest to smallest according to the relevance evaluation results, thereby obtaining a preference ranking result; training a reward model using a target loss function based on the preference ranking result, thereby obtaining a trained reward model; wherein the target loss function introduces a dynamically weighted hybrid retrieval score, and the hybrid retrieval score is a weighted sum of keyword retrieval score and vector retrieval score; based on the trained reward model, iteratively optimizing the generation strategy of the initial large language model using a reinforcement learning PPO algorithm, so that the fluctuation range of the reward score of the generated knowledge of the initial large language model is less than a preset range for a consecutive preset number of rounds, thereby obtaining an optimized large language model.
[0032] The method provided by this invention is a complete process of acquiring retrieval knowledge through hybrid retrieval, generating knowledge through temperature control, ranking based on relevance evaluation, training a reward model by fusing retrieval scores, and optimizing a large language model using the PPO algorithm. It specifically addresses the core shortcomings of existing technologies that combine retrieval-enhanced generation with reinforcement learning: First, the hybrid retrieval mode can accurately match query terms through keyword retrieval and capture deep semantic connections through vector retrieval. The combination of these two ensures that the selected N retrieval knowledge items not only meet the literal requirements of the query but also conform to the deep semantic intent. Second, a preset temperature parameter controls the iterative generation of LLM, avoiding the problems of insufficient diversity leading to repetition or excessive randomness and creating illusions in traditional generation processes, ensuring that the N generated knowledge items have both quantity and basic quality. Furthermore, preference ranking based on relevance evaluation provides clear supervision signals for RM, solving the problem of existing RM lacking effective ranking data. The objective loss function introduces dynamically weighted hybrid retrieval scores, ensuring that RM scoring considers both ranking priority and the factual reliability of the retrieval knowledge, avoiding evaluation bias caused by RM relying solely on textual features. Finally, the PPO algorithm iteratively optimizes the LLM generation strategy, aiming for a stable reward score for RM, ensuring that the knowledge ultimately generated by LLM continuously meets professional needs. In summary, the method provided by this invention can improve the quality of LLM generation, effectively solve the problems of inaccurate retrieval, illusion in generation, and unreliable evaluation in traditional methods, and significantly enhance the application value of LLM in professional fields.
[0033] In some embodiments, a hybrid retrieval-enhanced reward model training method provided by the present invention can be executed by a hybrid retrieval-enhanced reward model training system 100 (hereinafter referred to as training system 100).
[0034] As an example, the training system 100 can be any electronic device 200 with data processing capabilities, such as a general-purpose computer, personal computer, laptop computer, switch, or tablet computer. The specific implementation of the training system 100 is not limited here.
[0035] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention is shown. The electronic device 200 includes a processor 210, a memory 220, and a communication interface 230.
[0036] Processor 210 may include one or more processing cores. Processor 210 connects to various parts within electronic device 200 using various interfaces and lines, and performs various functions and processes data of electronic device 200 by running or executing instructions, programs, code sets, or instruction sets stored in memory 220, and by calling data stored in memory 220. Optionally, processor 210 may be implemented using at least one of the following hardware forms: Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA).
[0037] The memory 220 may include random access memory (RAI) or read-only memory (ROI). Optionally, the memory 220 may include non-transitory computer-readable storage ledger. The memory 220 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 220 may include a program storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as video acquisition, feature extraction, and process detection), and instructions for implementing the various method embodiments described above.
[0038] Communication interface 230 is used to communicate with other devices, equipment or communication networks, such as data storage devices, image processing devices or Ethernet, wireless access network (RAN), wireless local area network (WLAN), etc.
[0039] In terms of physical implementation, the aforementioned devices (such as processor 210, memory 220, and communication interface 230) can each be devices within the same device (such as a laptop computer). Alternatively, at least two of these devices can be located within the same device, i.e., as different devices within the same device, similar to the deployment of devices or components in a distributed system.
[0040] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 200. In other embodiments of the present invention, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0041] The following description, in conjunction with the accompanying drawings, illustrates a hybrid retrieval enhancement reward model training method provided by an embodiment of the present invention.
[0042] Figure 2 This is a flowchart illustrating a hybrid retrieval-enhanced reward model training method provided in an embodiment of the present invention. Optionally, this method can be... Figure 1 The illustrated electronic device 200 performs this operation. The method may include the following steps: S1. For the input user query, a hybrid retrieval mode is used to retrieve relevant documents from the knowledge base, and the relevant documents are filtered to obtain N search knowledge. The hybrid retrieval mode includes keyword retrieval and vector retrieval. In one possible implementation, S1 above includes: preprocessing the documents in the knowledge base, using a pre-trained sentence embedding model to convert each document into an embedding vector of a preset dimension, and storing the embedding vector in a vector database; extracting keywords from the user query, and determining the keyword retrieval score between the user query and the documents in the vector database; inputting the user query into the pre-trained sentence embedding model to generate a query embedding vector corresponding to the user query, determining the semantic similarity between the query embedding vector and the embedding vectors of each document in the vector database, and using the semantic similarity as the vector retrieval score; constructing an attention layer, inputting the query embedding vector, keyword retrieval score, and vector retrieval score of the user query into the attention layer, and dynamically learning weight coefficients through an attention mechanism, wherein the weight coefficients are greater than or equal to 0 and less than or equal to 1; calculating a mixed retrieval score based on the weight coefficients; sorting the documents in the vector database in descending order according to the mixed retrieval score, selecting the top N documents, and determining the N documents as the N retrieval knowledge points, where N is a positive integer greater than 1.
[0043] The method provided by this invention uses a pre-trained sentence embedding model to convert documents into embedding vectors. Compared with traditional simple encoding methods, it can more accurately capture the semantic information of documents, laying the foundation for the accuracy of subsequent vector retrieval. Furthermore, the dynamic learning weight coefficients of the attention layer can adaptively adjust the weights of the two retrieval methods according to the query type. Finally, the top N documents are selected in descending order of the mixed retrieval score to ensure that the final retrieved retrieval knowledge is the optimal result under the dual dimensions of term matching and semantic association. This provides high-quality factual knowledge support for subsequent preference ranking and RM training, reducing subsequent process deviations caused by poor retrieval knowledge quality.
[0044] S2. Based on preset temperature parameters, control the initial large language model to perform N iterations to generate the user query, resulting in N generated knowledge.
[0045] In one possible implementation, S2 above includes: setting preset temperature parameters for the initial large language model; constructing structured prompts, which include guidance on the core needs of the user query, content format requirements for generated knowledge, and professional domain constraints; concatenating the user query with the structured prompts to form initial input text; inputting the initial input text into the initial large language model, performing N iterations of generation tasks, and obtaining N generated knowledge.
[0046] In one example, multimodal data, such as images and videos, can be introduced during the knowledge generation and retrieval stages to enrich the content of the knowledge base. For instance, in the insurance field, image recognition technology can be used to identify information on insurance policies as part of the retrieved knowledge.
[0047] The method provided by this invention addresses the issues of traditional LLM (Local Mastery Management) knowledge generation deviating from requirements and exhibiting unstable quality by combining structured prompts with temperature parameter control. Structured prompts explicitly guide users to their core query needs, content format requirements, and professional domain constraints, directly guiding the initial LLM to focus on the core query and preventing content deviation or formatting issues. Preset temperature parameters balance the diversity and accuracy of generated knowledge, preventing content duplication due to excessively low temperatures and illusions due to excessively high temperatures. This combination allows the initial LLM to directly obtain N generated knowledge items that conform to professional standards and possess differentiation during N iterations of generation tasks without additional screening. This provides qualified samples for subsequent ranking and comparison with retrieved knowledge, improving the efficiency and quality of the generation process.
[0048] S3. In response to the relevance evaluation results of each retrieved knowledge and each generated knowledge with the user query, sort the N retrieved knowledge and N generated knowledge from largest to smallest according to the relevance evaluation results to obtain the preference ranking result; Specifically, the relevance assessment results of each retrieved knowledge and each generated knowledge to the user query were obtained through manual annotation by insurance experts. These experts ranked the retrieved and generated knowledge based on their business experience. The annotation format was: Retrieval Knowledge 2 > Generated Knowledge 1 > Retrieval Knowledge 1 = Generated Knowledge 2, indicating that Retrieval Knowledge 2 is more relevant to the query than Generated Knowledge 1, and Generated Knowledge 1 is more relevant to the query than both Retrieval Knowledge 1 and Generated Knowledge 2.
[0049] It should be understood that the above is merely an illustrative example. The method provided in this embodiment of the invention can also employ other methods to sort the N retrieved knowledge items and N generated knowledge items from largest to smallest to obtain a preferred sorting result. In one example, a rule engine can be introduced to automatically sort the retrieved knowledge items and generated knowledge items according to preset rules. For example, sorting can be based on factors such as the source of the knowledge (e.g., an authoritative database), generation time, and user feedback.
[0050] S4. Based on the preference ranking results, a reward model is trained using a target loss function to obtain a trained reward model; wherein, the target loss function introduces a dynamically weighted mixed retrieval score, and the mixed retrieval score is a weighted sum of keyword retrieval score and vector retrieval score; In one possible implementation, see Figure 3 The above S4 includes: S41. Determine multiple associated sample pairs from the preference ranking results. Each associated sample pair includes a first knowledge sample and a second knowledge sample, wherein the priority of the first knowledge sample is greater than or equal to that of the second knowledge sample. S42. Determine the mixed retrieval score for each first knowledge sample and second knowledge sample; The formula for determining the mixed retrieval score is as follows: ; in, For mixed retrieval scores, Keyword search score, For vector retrieval scores, These are the weighting coefficients; It should be noted that keyword retrieval scores are calculated using algorithms such as TF-IDF / BM25, while vector retrieval scores are calculated using similarity algorithms such as cosine similarity, Euclidean distance, and edit distance.
[0051] The method provided by this invention trains a knowledge management system (RM) by extracting associated sample pairs and combining them with mixed retrieval scores. It determines the associated sample pairs of the first knowledge sample and the second knowledge sample from the preference ranking results, providing clear pairwise comparative supervision signals for the RM and avoiding evaluation bias caused by single text scoring. At the same time, it calculates a mixed retrieval score for each sample and incorporates it into the training data, so that the RM learns not only by referring to the ranking priority, but also by combining the factual reliability of the retrieved knowledge. This avoids the problem that the RM relies solely on text semantics and ignores the quality of the knowledge source. The RM trained in this invention can more accurately quantify the quality of knowledge and provide a reliable reward signal for subsequent LLM optimization.
[0052] S43. Based on the target loss function, the reward model is trained using each associated sample pair and the mixed retrieval score of each first knowledge sample and second knowledge sample as training data to obtain the trained reward model. The target loss function Loss is: ; Where E represents the expected calculation, For the sigmoid function, The predicted reward value for the first knowledge sample. The predicted reward value for the second knowledge sample. The mixed retrieval score of the first knowledge sample. The mixed retrieval score is the score of the second knowledge sample.
[0053] The method provided by this invention offers a target loss function comprising two parts. The former quantifies the scoring differences of the Retrieval Management (RM) on high- and low-priority samples using the sigmoid function, ensuring that the RM can effectively distinguish between superior and inferior samples. The latter introduces mixed retrieval score differences, matching the scoring differences of the RM with the differences in retrieval reliability of the samples, thus avoiding the RM from misjudging samples with low factual accuracy but high ranking. The combination of these two parts guides the RM to simultaneously consider priority judgment and factual reliability assessment, significantly reducing RM scoring bias and providing a more accurate basis for value judgment in LLM optimization.
[0054] S5. Based on the completed training reward model, the generation strategy of the initial large language model is iteratively optimized using the reinforcement learning PPO algorithm, so that the fluctuation range of the reward score of the generated knowledge of the initial large language model is less than a preset range for a consecutive preset number of rounds, thereby obtaining an optimized large language model.
[0055] In some embodiments, S5 above includes: constructing a dual-model architecture for the PPO algorithm, using the initial large language model as the Actor strategy model, the Actor strategy model being used to receive user queries and corresponding N retrieval knowledge items, and to generate candidate knowledge sequences based on the user queries and the N retrieval knowledge items; using the trained reward model as the Critic value model, the Critic value model being used to evaluate the reward value of the candidate knowledge sequences generated by the Actor strategy model; and executing an iterative optimization process according to preset optimization parameters until the fluctuation range of the reward score of the knowledge generated by the initial large language model is less than a preset range for a consecutive preset number of rounds, thereby obtaining an optimized large language model.
[0056] The method provided by this invention constructs a PPO dual-model architecture and sets preset optimization parameters to limit the division of labor between the Actor policy model (generating candidate knowledge) and the Critic value model (evaluating reward value), thus avoiding inefficient optimization caused by functional confusion. The preset optimization parameters ensure that LLM optimization aims at stable reward scores, avoiding excessive iteration and wasting resources. At the same time, KL divergence constraints control the policy update amplitude of the Actor model, preventing policy mutations from causing fluctuations in generation quality. This can efficiently guide the LLM generation strategy to iterate towards higher reward scores, ultimately obtaining a stable optimized large language model.
[0057] Furthermore, the iterative optimization process for any round includes: concatenating the user query and the corresponding N retrieved knowledge points into input text; inputting the input text into the Actor strategy model, which outputs one candidate knowledge sequence based on a preset generation strategy; inputting the candidate knowledge sequence into the Critic value model, which outputs an evaluation result, including an actual reward value reflecting the actual quality of the candidate knowledge sequence and a predicted reward value reflecting the expected quality of the candidate knowledge sequence; calculating the advantage function value by subtracting the actual reward value from the predicted reward value, which quantifies the value of the actual quality of the candidate knowledge sequence exceeding the expected quality; constructing the strategy gradient loss function of the Actor strategy model based on the advantage function value and a preset KL divergence constraint threshold, and updating the parameters of the Actor strategy model through a backpropagation algorithm.
[0058] The method provided by this invention refines the iterative process, concatenating the user query and N retrieved knowledge points into input text. This allows the Actor model to reference factual knowledge when generating candidate knowledge, avoiding a disconnect from reality. The Critic model outputs the actual reward value and the predicted reward value, and calculates the advantage function value by the difference between the two, accurately quantifying the value of candidate knowledge exceeding expectations and providing a clear direction for policy updates. The method combines preset KL divergence constraints to construct a policy gradient loss, ensuring that the Actor model parameter updates follow the high-value direction without deviating too much from professional standards. The precise design of each round of the process allows the LLM generation strategy to be gradually optimized under factual constraints and value guidance, accelerating model convergence.
[0059] It should be noted that, in addition to the PPO algorithm, other reinforcement learning algorithms, such as A2C (advantage actor-critic) and TRPO (trust region policy optimization), can also be used to optimize the training process of the reward model.
[0060] As shown in S1-S5, the method provided by this invention provides a complete process of acquiring retrieval knowledge through hybrid retrieval, generating knowledge through temperature control, ranking based on relevance evaluation, training a reward model by fusing retrieval scores, and optimizing a large language model using the PPO algorithm. This specifically addresses the core shortcomings of existing technologies that combine retrieval enhancement generation with reinforcement learning: First, the hybrid retrieval mode can accurately match query terms through keyword retrieval on the one hand, and capture deep semantic connections through vector retrieval on the other. The combination of these two approaches ensures that the selected N retrieval knowledge points not only fit the literal requirements of the query but also conform to the deep semantic intent. Second, preset temperature parameters control the LLM iterative generation, avoiding... This invention avoids the problems of insufficient diversity leading to repetition or excessive randomness and creating illusions in traditional generation processes, ensuring that N generated knowledge items possess both quantity and basic quality. Furthermore, preference ranking based on relevance assessment provides clear supervisory signals for Resource Management (RM), addressing the lack of effective ranking data in existing RM systems. The objective loss function introduces dynamically weighted hybrid retrieval scores, ensuring that RM scoring considers both ranking priority and the factual reliability of the retrieved knowledge, avoiding evaluation bias caused by RM relying solely on textual features. Finally, the PPO algorithm iteratively optimizes the LLM generation strategy, aiming for a stable reward score for RM, ensuring that the knowledge generated by the LLM consistently meets professional needs. In summary, the method provided by this invention improves the quality of LLM generation, effectively solving the problems of inaccurate retrieval, illusions in generation, and unreliable evaluation in traditional methods, significantly enhancing the application value of LLM in professional fields.
[0061] To facilitate understanding of this solution, an example will be provided below. Figure 4The specific implementation process of this invention is explained below. The method provided by this invention specifically includes: Knowledge Retrieval: Documents in the knowledge base are converted into embedding vectors using models such as BERT and BGE, and stored in a database. A hybrid mode of keyword retrieval and vector retrieval is adopted. Based on the user query, relevant context is retrieved, and N relevant documents are selected as retrieval knowledge. Knowledge Generation: Given a prompt word, context is generated using LLM. By setting appropriate temperature parameters, the diversity and accuracy of the generated content are controlled, and the generation is iterated N times to obtain N generated knowledge. Expert Annotation: Insurance experts are invited to participate in the manual annotation stage, and the retrieval knowledge and generated knowledge are ranked according to their business experience. The annotation format is: Retrieval Knowledge 2 > Generated Knowledge 1 > Retrieval Knowledge 1 = Generated Knowledge 2, indicating that Retrieval Knowledge 2 is more relevant to the query than Generated Knowledge 1, and Generated Knowledge 1 is more relevant to the query than both Retrieval Knowledge 1 and Generated Knowledge 2.
[0062] Reward Model Training: The ranking of insurance expert annotations is converted into scores using the RM formula. The RM formula optimizes the ranking task, ensuring the model can accurately evaluate the quality of generated knowledge and generate more accurate reward signals. The original RM loss function formula is: The improved hybrid retrieval enhancement loss function is: in, For mixed retrieval scores, Keyword search score, For vector retrieval scores, These are weighting coefficients, which are dynamically learned through an attention mechanism (not fixed values).
[0063] Specifically, θ represents the parameters of the RM model. x represents the input prompt. y w For prompt x, the response that wins (is better) according to human preference. l For prompt x, the human-preferred, less desirable response. D is a comparison dataset annotated by humans, where each sample is a triple (x, y). w ,y l rθ(x,y) is the scalar reward calculated by the RM model for a given prompt x and response y. σ is the sigmoid function.
[0064] Reinforcement learning PPO iteration: The PPO algorithm ensures that the model can continuously iterate through policy optimization, thereby improving the accuracy and reliability of generated knowledge.
[0065] Where ϕ represents the model parameters (such as the weights and biases of a neural network), which are variables optimized by the objective function. E represents the expected value calculation. (x, y) represents a sample pair, where x is the input (such as text or an image) and y is the output (such as a label or prediction result). DϕRL represents the dataset or distribution in the Reinforcement Learning (RL) phase, which depends on the current model parameters ϕ. r(ϕ, x, y) is the reward function, used to measure the quality of the output y generated by the model parameters ϕ given input x. β is a hyperparameter used to balance the weights of different terms in the objective function; here, it controls the influence of the logarithmic ratio term. πϕRL(y∣x) is the policy in the Reinforcement Learning phase, representing the conditional probability distribution of the model generating output y given input x and parameters ϕ. πSFT(y∣x) is the policy in the Supervised Fine-Tuning (SFT) phase, representing the conditional probability distribution of y generated by the model trained in the Supervised Learning phase. It serves as a reference distribution (such as the baseline policy) for the reinforcement learning phase. γ is a hyperparameter used to balance the weights of the first part (reinforcement learning loss) and the second part (pre-training loss). Dpretrain is the pre-training dataset used for the initial pre-training phase of the model. log(πϕRL(x)) is the log probability of the reinforcement learning policy generating the input x.
[0066] The foregoing mainly describes the solutions of the embodiments of the present invention from a methodological perspective. It is understood that, in order to achieve the above-mentioned functions, the training system 100 includes at least one of the hardware structures and software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of the present invention.
[0067] In this embodiment of the invention, the training system 100 can be divided into functional units according to the above method example. For example, the training system 100 can be divided into functional units corresponding to various functions, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this embodiment of the invention is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.
[0068] For example, Figure 5 This diagram illustrates the hardware structure of a training system according to an embodiment of the present invention. The training system 100 includes: a knowledge retrieval module 110, used to retrieve relevant documents from a knowledge base using a hybrid retrieval mode in response to an input user query, and to filter the relevant documents to obtain N retrieval knowledge items; the hybrid retrieval mode includes keyword retrieval and vector retrieval; a knowledge generation module 120, used to control an initial large language model to iterate and generate N generated knowledge items based on a preset temperature parameter, resulting in N iterations; and a preference ranking module 130, used to, in response to the relevance evaluation results of each input retrieval knowledge item and each generated knowledge item with the user query, rank the N retrieval knowledge items and N generated knowledge items from largest to smallest according to the relevance evaluation results. The algorithm sorts the rows to obtain a preference ranking result. A model training module 140 trains a reward model based on the preference ranking result using a target loss function, resulting in a trained reward model. The target loss function incorporates a dynamically weighted mixed retrieval score, which is a weighted sum of keyword retrieval scores and vector retrieval scores. A model optimization module 150 iteratively optimizes the generation strategy of the initial large language model using a reinforcement learning PPO algorithm, based on the trained reward model, so that the fluctuation range of the reward score for generated knowledge in the initial large language model is less than a preset range for consecutive preset rounds, resulting in an optimized large language model.
[0069] It should be understood that specific descriptions of the above-mentioned optional methods can be found in the foregoing method embodiments, and will not be repeated here. Furthermore, explanations of any of the training systems 100 provided above, as well as descriptions of their beneficial effects, can be found in the corresponding method embodiments described above, and will not be repeated here.
[0070] This invention also provides a computer-readable storage medium storing at least one computer instruction, which is loaded and executed by a processor to implement the methods of the various embodiments described above. Explanations of the relevant content and descriptions of the beneficial effects of any of the computer-readable storage media provided above can be found in the corresponding embodiments described above, and will not be repeated here.
[0071] This invention also provides a chip. This chip integrates a control circuit for implementing the functions of the training system 100 described above, and one or more ports. Optionally, the functions supported by this chip are as described above, and will not be repeated here.
[0072] Those skilled in the art will understand that the program for implementing all or part of the steps of the above embodiments, which can be executed by a program instructing related hardware, can be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a random access memory, etc. The processing unit or processor mentioned above can be a central processing unit, a general-purpose processor, an application-specific integrated circuit (ASIC), a microprocessor (DSP), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.
[0073] This invention also provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform any of the methods described in the above embodiments. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this invention is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD), etc.
[0074] It should be noted that the devices for storing computer instructions or computer programs provided in the embodiments of the present invention, such as, but not limited to, the aforementioned memory, computer-readable storage medium, and communication chip, are all non-transitory. Those skilled in the art should recognize that the functions described in the embodiments of the present invention in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable storage medium or transmitted as one or more instructions or code on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium accessible to general-purpose or special-purpose computers.
[0075] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
Claims
1. A method for training a reward model with hybrid retrieval enhancement, characterized in that, The method includes: For the input user query, a hybrid retrieval mode is used to retrieve relevant documents from the knowledge base, and the relevant documents are filtered to obtain N search knowledge. The hybrid retrieval mode includes keyword retrieval and vector retrieval. Based on preset temperature parameters, the initial large language model is controlled to perform N iterations to generate N generated knowledge items for the user query. In response to the relevance evaluation results of each retrieved knowledge and each generated knowledge with the user query, the N retrieved knowledge and N generated knowledge are sorted from largest to smallest according to the relevance evaluation results to obtain the preference ranking result; Based on the preference ranking results, a reward model is trained using a target loss function to obtain the trained reward model; wherein, the target loss function introduces a dynamically weighted mixed retrieval score, and the mixed retrieval score is a weighted sum of keyword retrieval score and vector retrieval score; Based on the completed reward model, the generation strategy of the initial large language model is iteratively optimized using the reinforcement learning PPO algorithm, so that the fluctuation range of the reward score of the generated knowledge of the initial large language model is less than a preset range for a consecutive preset number of rounds, thereby obtaining an optimized large language model.
2. The method according to claim 1, characterized in that, The system employs a hybrid retrieval mode to retrieve relevant documents from the knowledge base in response to the input user query. These relevant documents are then filtered to obtain N searchable knowledge items, including: The documents in the knowledge base are preprocessed by using a pre-trained sentence embedding model to convert each document into an embedding vector of a preset dimension, and the embedding vectors are stored in a vector database. Keyword extraction is performed on the user query to determine the keyword retrieval score between the user query and documents in the vector database; The user query is input into the pre-trained sentence embedding model to generate a query embedding vector corresponding to the user query. The semantic similarity between the query embedding vector and the document embedding vectors in the vector database is determined, and the semantic similarity is used as the vector retrieval score. An attention layer is constructed by inputting the query embedding vector, keyword retrieval score, and vector retrieval score of the user query into the attention layer, and dynamically learning weight coefficients through an attention mechanism. The weight coefficients are greater than or equal to 0 and less than or equal to 1. Calculate the hybrid retrieval score based on the weighting coefficients; The documents in the vector database are sorted in descending order according to the hybrid retrieval scores. The top N documents are selected and identified as the N retrieval knowledge items, where N is a positive integer greater than 1.
3. The method according to claim 2, characterized in that, The method involves controlling an initial large language model to iterate through the user query N times based on preset temperature parameters, resulting in N generated knowledge items, including: Set the preset temperature parameters for the initial large language model; Construct structured prompts, which include guidance on the core needs of user queries, content format requirements for generated knowledge, and professional domain constraints; The user query and structured prompts are concatenated to form the initial input text; The initial input text is input into the initial large language model, and N iterations of the generation task are performed to obtain N generated knowledge.
4. The method according to claim 3, characterized in that, The step of training a reward model using a target loss function based on the preference ranking results to obtain the trained reward model includes: Multiple associated sample pairs are determined from the preference ranking results. Each associated sample pair includes a first knowledge sample and a second knowledge sample, wherein the priority of the first knowledge sample is greater than or equal to that of the second knowledge sample. Determine the combined retrieval score for each first knowledge sample and second knowledge sample; Based on the target loss function, the reward model is trained using each associated sample pair and the mixed retrieval score of each first knowledge sample and second knowledge sample as training data, resulting in a trained reward model. The formula for determining the mixed retrieval score is as follows: ； in, For mixed retrieval scores, Keyword search score, For vector retrieval scores, These are the weighting coefficients.
5. The method according to claim 4, characterized in that, The target loss function Loss is: ； Where E represents the expected calculation, For the sigmoid function, The predicted reward value for the first knowledge sample. The predicted reward value for the second knowledge sample. The mixed retrieval score of the first knowledge sample. The mixed retrieval score is the score of the second knowledge sample.
6. The method according to claim 5, characterized in that, The reward model based on the completed training iteratively optimizes the generation strategy of the initial large language model using the reinforcement learning PPO algorithm, including: A dual-model architecture for the PPO algorithm is constructed, with the initial large language model as the Actor strategy model, which is used to receive user queries and corresponding N retrieval knowledge, and generate candidate knowledge sequences based on the user queries and N retrieval knowledge; and the trained reward model as the Critic value model, which is used to evaluate the reward value of the candidate knowledge sequences generated by the Actor strategy model. An iterative optimization process is executed according to preset optimization parameters until the fluctuation range of the reward score of the knowledge generated by the initial large language model is less than a preset range for a consecutive preset number of rounds, thus obtaining an optimized large language model.
7. The method according to claim 6, characterized in that, The iterative optimization process for any round includes: Concatenate the user query and the corresponding N search results into the input text; The input text is input into the Actor strategy model, and the Actor strategy model outputs a candidate knowledge sequence based on a preset generation strategy. The candidate knowledge sequence is input into the Critic value model, and the Critic value model outputs an evaluation result, which includes an actual reward value reflecting the actual quality of the candidate knowledge sequence and a predicted reward value reflecting the expected quality of the candidate knowledge sequence. Based on the actual reward value and the predicted reward value, the advantage function value is calculated by the difference. The advantage function value is used to quantify the value of the actual quality of the candidate knowledge sequence exceeding the expected quality. Based on the advantage function value and the preset KL divergence constraint threshold, the policy gradient loss function of the Actor policy model is constructed, and the parameters of the Actor policy model are updated through the backpropagation algorithm.
8. A training system for a hybrid retrieval-enhanced reward model, characterized in that, The system includes: The knowledge retrieval module is used to retrieve relevant documents from the knowledge base in response to the input user query using a hybrid retrieval mode, and to filter the relevant documents to obtain N searchable knowledge items. The hybrid retrieval mode includes keyword retrieval and vector retrieval. The knowledge generation module is used to control the initial large language model to perform N iterations to generate N knowledge based on preset temperature parameters; The preference ranking module is used to sort the N retrieved knowledge and N generated knowledge in descending order based on the relevance evaluation results of each retrieved knowledge and each generated knowledge with the user query, in response to the relevance evaluation results, to obtain the preference ranking result. The model training module is used to train the reward model based on the preference ranking results using a target loss function, thereby obtaining the trained reward model; wherein, the target loss function introduces a dynamically weighted mixed retrieval score, and the mixed retrieval score is a weighted sum of keyword retrieval score and vector retrieval score; The model optimization module is used to iteratively optimize the generation strategy of the initial large language model based on the trained reward model using the reinforcement learning PPO algorithm, so that the fluctuation range of the reward score of the generated knowledge of the initial large language model is less than a preset range for a consecutive preset number of rounds, thereby obtaining an optimized large language model.
9. An electronic device, comprising a processor; and a memory for storing processor-executable instructions; wherein, The processor is configured to execute the instructions to implement the hybrid retrieval-enhanced reward model training method as described in any one of claims 1-7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the hybrid retrieval-enhanced reward model training method as described in any one of claims 1-7.