Model training method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a model training method and device, equipment and a storage medium, and the method comprises the steps: obtaining a retrieval track of a model for retrieving a target question and a target answer finally output by the model, and determining the retrieval quality and a result-level feedback value of the retrieval track according to the target answer, and meanwhile, determining a feedback mechanism for each step in the retrieval track according to the retrieval quality, further determining a process-level feedback value of each step based on the feedback mechanism, and training the model based on the result-level feedback value and the process-level feedback value.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and more specifically, to a model training method, apparatus, device, and storage medium in the field of computer technology. Background Technology

[0002] To improve the performance of Large Language Models (LLMs) in handling complex tasks such as multi-hop question answering, LLMs employ Retrieval-Augmented Generation (RAG) to give the model "thinking-searching" capabilities, dynamically invoking retrieval tools for information supplementation and judgment during the answer generation process. However, RAG systems often use reinforcement learning methods based on the final result, only rewarding the correctness of the generated answer and ignoring the impact of the retrieval process on the answer, resulting in lower answer quality. Summary of the Invention

[0003] This specification provides a model training method, apparatus, device, and storage medium that improves retrieval efficiency and the accuracy and robustness of the model for complex question-answering tasks, while ensuring answer quality.

[0004] Firstly, embodiments of this specification provide a model training method, which includes: Input the target question into the target model, and obtain the retrieval trajectory of the target model in response to the target question and the output target answer; The retrieval quality and result-level feedback value are determined based on the target answer to determine the retrieval trajectory. Based on retrieval quality, a target feedback mechanism is determined for each step in the retrieval trajectory, and the process-level feedback value of each step is determined based on the target feedback mechanism. The target model is trained based on result-level feedback values ​​and process-level feedback values.

[0005] Secondly, embodiments of this specification provide a model training apparatus, which includes: The question input unit is used to input the target question into the target model, obtain the retrieval trajectory of the target model in response to the target question, and the output target answer; The result feedback unit is used to determine the retrieval quality and result-level feedback value of the retrieval trajectory based on the target answer; The process feedback unit is used to determine the target feedback mechanism for each step in the retrieval trajectory based on the retrieval quality, and to determine the process-level feedback value for each step based on the target feedback mechanism. The feedback training unit is used to train the target model based on result-level feedback values ​​and process-level feedback values.

[0006] Thirdly, embodiments of this specification provide a computer device, the device including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method described above.

[0007] Fourthly, embodiments of this specification provide a storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described above.

[0008] Fifthly, embodiments of this specification provide a computer program product, including: a computer program that, when executed by a processor of a computer device, enables the processor to at least implement the method as described in the first aspect.

[0009] In the embodiments of this specification, the retrieval trajectory of the model searching for the target question and the target answer output by the model are obtained. Based on the target answer, the retrieval quality and result-level feedback value of the retrieval trajectory are determined. Simultaneously, based on the retrieval quality, a feedback mechanism for each step in the retrieval trajectory is determined. Then, based on the feedback mechanism, a process-level feedback value for each step is determined. The model is trained using both the result-level and process-level feedback values. By simultaneously focusing on result-level and process-level feedback values ​​during model training, retrieval efficiency and the model's accuracy and robustness for complex question-answering tasks are improved, ensuring answer quality. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is an example diagram illustrating a multi-hop question-answering task provided in the embodiments of this specification; Figure 2 This is a system architecture diagram of a model training method provided in the embodiments of this specification; Figure 3 This is a flowchart illustrating a model training method provided in an embodiment of this specification; Figure 4 This is a flowchart illustrating a model training method provided in an embodiment of this specification; Figure 5 This is a schematic diagram illustrating an example of a model training method provided in the embodiments of this specification; Figure 6 This is a schematic diagram of the structure of a model training device provided in the embodiments of this specification; Figure 7 This is a schematic diagram of the structure of a computer device provided in the embodiments of this specification. Detailed Implementation

[0012] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0013] The model training method provided in the embodiments of this specification is applicable to multi-hop question answering tasks. Multi-hop question answering is a question-answering format that requires reasoning and integrating multiple information fragments to arrive at the correct answer. It requires the model to retrieve, extract, and integrate relevant information from text or a knowledge base through multiple logical reasoning steps to answer complex questions. Unlike single-hop question answering tasks, multi-hop question answering requires the model to perform retrieval and reasoning across multiple information fragments, with each step depending on the result of the previous step.

[0014] Please see Figure 1 , Figure 1 This is an example diagram illustrating a multi-hop question-answering task provided in the embodiments of this specification. For example... Figure 1 As shown, the question in this task is "Which manufacturer supplies the raw materials for the core hardware of product A?" To answer this question, we first need to go through step 1: find out the core hardware of product A. After searching, we find that the core hardware of product A is B. Next, we need to go through step 2: find out what raw materials are used in core hardware B. After searching, we find that the raw materials used in core hardware B are C and D. Then, we need to go through step 3: find out which manufacturers C and D are from. After searching, we find that the manufacturer of C is E and the manufacturer of D is F. Finally, we need to go through step 4: integrate the first three steps to get the answer to the question: "The core hardware of product A is B, and the raw materials used in B are C and D, where C comes from manufacturer E and D comes from manufacturer F." As mentioned above, multi-hop question-answering tasks require reasoning for accurate retrieval and ultimately, the integration of information fragments. However, purely parametric models struggle to store and associate long chains of knowledge and easily overlook intermediate information in the question, leading to errors in the retrieval step. These errors accumulate and amplify in subsequent steps, resulting in incorrect answers. Therefore, a Retrieval and Argumentation (RAG) system is used in the model to enable it to "think-search," dynamically invoking retrieval tools for information supplementation and judgment during answer generation. The RAG system can identify short text fragments in the question, reducing interference from large blocks of information. By focusing on short texts and increasing the reasoning process, the model outputs relatively accurate answers. However, the RAG system still has significant limitations. When training the model to handle question-answering tasks, it often employs reinforcement learning methods based on the final result, only rewarding the correctness of the generated answer and ignoring the impact of the retrieval process, resulting in lower answer quality.

[0015] To address the above problems, this specification provides a model training method in its embodiments. Please refer to [link / reference]. Figure 2 , Figure 2 This is a system architecture diagram of a model training method provided in the embodiments of this specification. For example... Figure 2 As shown in the embodiments of this specification, result-level feedback values ​​and process-level feedback values ​​are obtained from the model's retrieval trajectory and the output answers. The result-level feedback values ​​and process-level feedback values ​​are combined to perform reinforcement learning training on the model.

[0016] In the embodiments of this specification, the retrieval trajectory of the model searching for the target question and the target answer output by the model are obtained. Based on the target answer, the retrieval quality and result-level feedback value of the retrieval trajectory are determined. Simultaneously, based on the retrieval quality, a feedback mechanism for each step in the retrieval trajectory is determined. Then, based on the feedback mechanism, a process-level feedback value for each step is determined. The model is trained using both the result-level and process-level feedback values. By simultaneously focusing on result-level and process-level feedback values ​​during model training, retrieval efficiency and the model's accuracy and robustness for complex question-answering tasks are improved, ensuring answer quality.

[0017] based on Figure 2 The system architecture diagram shown below will be used in conjunction with... Figures 3-5 This document provides a detailed description of the model training methods provided in the embodiments.

[0018] Please see Figure 3 , Figure 3 This is a flowchart illustrating a model training method provided in an embodiment of this specification. Figure 3 As shown, the method in the embodiments of this specification may include the following steps S102-S108.

[0019] S102, Input the target question into the target model, and obtain the retrieval trajectory of the target model in response to the target question and the output target answer; Specifically, a target question is input into the target model to be trained, and the retrieval trajectory of the target model in response to the target question and the output target answer are obtained. In the embodiments of this specification, the target model is an LLM (Limited Least Model), and the target question is a multi-hop question-answering task that requires the model to perform multiple inferences and retrievals. The retrieval trajectory is the complete process of inference and retrieval performed by the target model from receiving the target question to outputting the target answer. Optionally, the target model can be configured to output the retrieval trajectory and the target answer simultaneously, or it can be configured to output only the target answer, but temporarily store the retrieval trajectory inside the model for later retrieval.

[0020] S104, Determine the retrieval quality and result-level feedback value of the retrieval trajectory based on the target answer; Specifically, the format and content of the target answer and the standard answer are compared separately. The retrieval quality and result-level feedback value of the retrieval trajectory are determined based on the similarity between the target answer and the standard answer. Format is used to determine whether the target answer contains necessary fields or whether the format of each field meets preset requirements. Format rules are pre-set to constrain the target answer, and the format feedback value of the target answer is determined according to these rules. Content is used to determine whether the target answer is accurate and complete. The content feedback value can be determined by the harmonic mean of precision and recall (F1 score) or by a semantic evaluation method based on semantic similarity. In the F1 score calculation, precision (x) is used to determine the relevance between the target answer and the standard answer, and recall (y) is used to determine the completeness of the target answer. The result-level feedback value is the reward score obtained by weighted summation of the format feedback value and the content feedback value.

[0021] In one feasible implementation, the result-level feedback value can be used to determine the retrieval quality of the retrieval trajectory. A global threshold is preset. If the result-level feedback value is greater than or equal to the preset global threshold, the retrieval quality of the retrieval trajectory is determined to be the first quality. If the result-level feedback value is less than the preset global threshold, the retrieval quality of the retrieval trajectory is determined to be the second quality. The first quality is higher than the second quality.

[0022] S106, Determine the target feedback mechanism for each step in the retrieval trajectory based on the retrieval quality, and determine the process-level feedback value for each step based on the target feedback mechanism; Specifically, the embodiments in this specification set up process-level feedback mechanisms for each step in the retrieval trajectory, but the process-level feedback mechanisms set for each step in the retrieval trajectory with different retrieval qualities are different. A target feedback mechanism is determined for each step in the retrieval trajectory based on the retrieval quality, and a process-level feedback value for each step is determined based on the target feedback mechanism. The process-level feedback value is the reward score for each step.

[0023] If the retrieval quality is of the highest quality, then the retrieval trajectory is a first-quality trajectory; if the retrieval quality is of the second highest quality, then the retrieval trajectory is a second-quality trajectory. Since the first-quality trajectory is higher than the second-quality trajectory, the first-quality trajectory is a high-quality trajectory, and the second-quality trajectory is a low-quality trajectory. The high-quality trajectory corresponds to the knowledge redundancy feedback mechanism, and the low-quality trajectory corresponds to the knowledge matching feedback mechanism. The knowledge redundancy feedback mechanism calculates the similarity between each step in the high-quality trajectory and the historical steps preceding that step, identifying repeated steps. This is used to train the model by introducing retrieval steps with information gain, improving retrieval and inference efficiency. The knowledge matching feedback mechanism matches each step in the low-quality trajectory with each step in the high-quality trajectory to reward the positive gain steps in the low-quality trajectory.

[0024] This specification describes embodiments where a target question is input into the target model multiple times. The model's retrieval trajectory in response to the target question and the output target answer are obtained each time. A trajectory set is determined based on all retrieval trajectories. When implementing the knowledge matching feedback mechanism, each step in a low-quality trajectory is matched with all steps in all high-quality trajectories within the trajectory set.

[0025] S108 trains the target model based on result-level feedback values ​​and process-level feedback values.

[0026] Specifically, each retrieval trajectory in the trajectory set corresponds to a result-level feedback value. The trajectory set is treated as a strategy group, and all result-level feedback values ​​within this group are standardized to obtain the global advantage value for each retrieval trajectory. The process-level feedback values ​​for all steps within a single retrieval trajectory are standardized to obtain the local feedback factors for each step. For each step within a single retrieval trajectory, the local feedback factors of each step are combined with the global advantage value of that retrieval trajectory to obtain the fusion advantage value for each step in that retrieval trajectory. Based on the fusion advantage of all steps across all retrieval trajectories in the trajectory set, the model parameters of the target model are iteratively updated to achieve model training, enabling the target model to generate more accurate answers with more efficient retrieval trajectories.

[0027] In the embodiments of this specification, the retrieval trajectory of the model searching for the target question and the target answer output by the model are obtained. Based on the target answer, the retrieval quality and result-level feedback value of the retrieval trajectory are determined. Simultaneously, based on the retrieval quality, a feedback mechanism for each step in the retrieval trajectory is determined. Then, based on the feedback mechanism, a process-level feedback value for each step is determined. The model is trained using both the result-level and process-level feedback values. By simultaneously focusing on result-level and process-level feedback values ​​during model training, retrieval efficiency and the model's accuracy and robustness for complex question-answering tasks are improved, ensuring answer quality.

[0028] Please see Figure 4 , Figure 4 This is a flowchart illustrating a model training method provided in an embodiment of this specification. Figure 4 As shown, the method in the embodiments of this specification may include the following steps S202-S224.

[0029] S202, Input the target question into the target model a preset number of times, obtain the retrieval trajectory of the target model in response to the target question and the output target answer each time, and determine the trajectory set based on each retrieval trajectory; Specifically, the target question is input into the target model to be trained according to a preset number of times, and the retrieval trajectory of the target model in response to the target question and the output target answer are obtained each time. Each retrieval trajectory is added to the trajectory set. The trajectory set is the training set of the model training method provided in the embodiments of this specification, which includes each retrieval trajectory and each step contained in each retrieval trajectory.

[0030] The target model is an LLM (Limited Least Model), and the target question is a multi-hop question-answering task requiring multiple inferences and retrievals by the model. The retrieval trajectory is the complete process of inference and retrieval performed by the target model from receiving the target question to outputting the target answer. Optionally, the target model can be configured to output both the retrieval trajectory and the target answer simultaneously, or it can be configured to output only the target answer while temporarily storing the retrieval trajectory internally for later retrieval. To ensure the diversity of training samples, preferably, the preset number of iterations should be set to at least two.

[0031] S204, Result-level feedback value for determining the retrieval trajectory based on the target answer; Specifically, the format and content of the target answer output by the target model are compared with the standard answer for each iteration. The result-level feedback value for each retrieval trajectory in the trajectory set is determined based on the matching degree between the target answer and the standard answer. Format feedback is used to determine whether the target answer contains necessary fields or whether the format of each field meets preset requirements. Format rules are pre-set to constrain the target answer, and the format feedback value is determined according to these rules. Content feedback is used to determine the accuracy and completeness of the target answer. The content feedback value can be determined using the harmonic mean (F1 score) based on precision and recall. Precision is used to determine the relevance between the target answer and the standard answer, and recall is used to determine the completeness of the target answer; alternatively, it can be determined using a semantic evaluation method based on semantic similarity. The result-level feedback value is a reward score obtained by weighted summation of the format feedback value and the content feedback value.

[0032] In one feasible implementation, the target answer is matched with the standard answer to the target question to obtain a matching result; the format feedback value and content feedback value of the target answer are determined based on the matching result; and the result-level feedback value of the retrieval trajectory is determined based on the format feedback value and content feedback value.

[0033] For example, assuming that the embodiments of this specification determine the content feedback value of the target answer using the F1 score, if the standard answer should include three key entities L, M, and N, and the target answer contains two key entities M and N, then M and N in the target answer are relevant to the key entities M and N in the standard answer, and the precision of the target answer is 1.0. However, compared to the standard answer, the target answer lacks the key entity L, and the recall of the target answer is 0.67. The harmonic mean of precision and recall is then calculated to obtain... Therefore, the content feedback value of the target answer is 0.80. Assuming the format feedback value of the target answer is 0.70, the first weight set for the format feedback value is 0.3, and the second weight set for the content feedback value is 0.7, then the result-level feedback value of the target answer is 0.70×0.3+0.80×0.7=0.77.

[0034] In one feasible implementation, if the format feedback value of the target answer is lower than a preset format threshold or the content feedback value of the target answer is lower than a preset content threshold, then the retrieval trajectory corresponding to the target answer is determined to be a noise trajectory, and it is removed from the trajectory set and no longer used for subsequent model training.

[0035] S206, If the result-level feedback value is greater than or equal to the preset global threshold, then the retrieval quality of the retrieval trajectory is determined to be the first quality, and the retrieval trajectory is determined to be the first quality trajectory. S208, If the result-level feedback value is less than the preset global threshold, then the retrieval quality of the retrieval trajectory is determined to be the second quality, and the retrieval trajectory is determined to be the second quality trajectory. Specifically, a preset global threshold is set. The result-level feedback values ​​corresponding to the target answers of all search trajectories in the trajectory set are compared with the preset global threshold. Based on the comparison results, the search quality of each search trajectory in the trajectory set is determined. If the result-level feedback value is greater than or equal to the preset global threshold, the search quality of the search trajectory corresponding to that result-level feedback value is determined to be of first quality, and the search trajectory is designated as a first-quality trajectory. If the result-level feedback value is less than the preset global threshold, the search quality of the search trajectory corresponding to that result-level feedback value is determined to be of second quality, and the search trajectory is designated as a second-quality trajectory. The first quality is higher than the second quality, meaning the first-quality trajectory is a high-quality trajectory, and the second-quality trajectory is a low-quality trajectory.

[0036] S210, If the retrieval quality is the first quality, then the similarity between each first step in the first quality trajectory and the corresponding historical steps is calculated to obtain the first similarity between each first step and the historical steps. Specifically, the embodiments in this specification set up process-level feedback mechanisms for each step in the retrieval trajectory, but the process-level feedback mechanisms set for each step in the retrieval trajectory with different retrieval qualities are different. Based on the retrieval quality, a target feedback mechanism is determined for each step in the retrieval trajectory. If the retrieval quality is first quality, a knowledge redundancy feedback mechanism is applied to each first step in the first quality trajectory. The similarity between each first step and its corresponding historical steps is calculated sequentially to obtain the first similarity between each first step and the historical steps. The knowledge redundancy feedback mechanism is used to identify repeated steps in the first quality trajectory, thereby training the model to introduce retrieval steps with information gain and improving retrieval or reasoning efficiency.

[0037] Optionally, similarity calculation can be based on the Jaccard similarity calculation method, which obtains the similarity between two sets by calculating the ratio of the intersection size to the union size. In the embodiments of this specification, the current document set retrieved in the current first step is obtained, the historical document set retrieved in the previous historical steps is obtained, and then the document intersection and document union of the current document set and the historical document set are obtained. The first similarity between the first step and the historical steps is determined by the document intersection and document union.

[0038] For example, suppose the documents retrieved in the first step include a, b, c, and d, then the current document set is {a, b, c, d}; the documents retrieved in previous steps include b, c, e, and f, then the historical document set includes {b, c, e, f}. The intersection of the current document set and the historical document set is {b, c}, and the union is {a, b, c, d, e, f}. The first similarity is 2 / 6 = 0.33.

[0039] S212, Determine the process-level feedback value of each first step based on the first similarity; Specifically, the process-level feedback value for each first step is determined based on the first similarity. This process-level feedback value is the reward signal used for model training. A higher first similarity indicates that the first step is closer to a redundant step, resulting in a smaller reward; conversely, a lower first similarity indicates that the first step is closer to a gain step, resulting in a larger reward. Therefore, the first similarity is negatively correlated with the process-level feedback value. For example, if the first similarity is 0.33, the process-level feedback value for the first step is 0.67; if the first similarity is 0.67, the process-level feedback value for the first step is 0.33.

[0040] S214, If the retrieval quality is the second quality, then match each second step in the second quality trajectory with each first step in the trajectory set to obtain the first step corresponding to each second step; Specifically, if the retrieval quality is second quality, a knowledge matching feedback mechanism is applied to each second step in the second quality trajectory. This mechanism matches each second step in the second quality trajectory with each first step in all first quality trajectories in the trajectory set, identifying the first step most similar to each second step. The knowledge matching feedback mechanism is a method that matches each step in a low-quality trajectory with each step in a high-quality trajectory, rewarding positive gain steps in the low-quality trajectory.

[0041] In one feasible implementation, if the retrieval quality is the second quality, then extreme value bipartite graph matching is performed on each second step in the second quality trajectory and each first step in each first quality trajectory in the trajectory set; based on the extreme value bipartite graph matching, the first step that satisfies the extreme value alignment relationship with each second step is determined.

[0042] Extreme value bipartite graph matching is a graph theory algorithm used to find pairings between two sets that satisfy an extreme value alignment relation. The extreme value alignment relation is defined as follows: during the pairing process, each second step is matched with all first steps to determine a matching pattern such that each second step matches at most one first step, and each first step matches at most one second step, and the similarity of all paired step pairs reaches a maximum value.

[0043] For example, assuming the set of second steps in the current second quality trajectory is {g, h, i}, and the set of first steps determined based on the first steps of all first quality trajectories in the trajectory set is {l, m, n, r, s}, similarity calculations show that step g has a similarity of 0.40 with step l, 0.80 with step m, 0.33 with step n, 0.52 with step r, and 0.60 with step s. The similarity between steps h and l is 0.66, between h and m is 0.35, between h and n is 0.28, between h and r is 0.50, and between h and s is 0.20. Similarly, the similarity between steps i and l is 0.45, between i and m is 0.50, between i and n is 0.72, between i and r is 0.20, and between i and s is 0.55. Using the Kuhn-Munkres algorithm, extreme value bipartite graph matching is performed on the above similarities to determine the extreme value alignment relationships: the second step g corresponds to the first step m, the second step h corresponds to the first step l, and the second step i corresponds to the first step n.

[0044] S216, Based on the second similarity between each second step and the corresponding first step, determine the process-level feedback value of each second step; Specifically, the process-level feedback value for each second step is determined based on the second similarity between each second step and the corresponding first step. Optionally, the second similarity can be directly determined as the process-level feedback value. Taking the example in step S214, the process-level feedback value for second step g is 0.80, the process-level feedback value for second step h is 0.66, and the process-level feedback value for second step i is 0.72.

[0045] In one feasible implementation, a similarity threshold is preset. If the second similarity between the second step and the corresponding first step in the determined extreme value alignment relationship is less than the preset similarity threshold, the second step is removed and no longer used for subsequent model training.

[0046] S218, standardize the result-level feedback values ​​of each search trajectory in the trajectory set to obtain the global advantage value of each search trajectory; Specifically, the result-level feedback values ​​of each search trajectory in the trajectory set are standardized to obtain a global advantage value for each search trajectory. Standardization is performed to make the result-level feedback values ​​comparable global advantage values. The specific process of standardization involves calculating the mean and standard deviation of the result-level feedback values ​​of all search trajectories in the trajectory set. The mean is used to calculate the arithmetic mean of all result-level feedback values, and the standard deviation is used to measure the dispersion among the result-level feedback values. Then, for a single search trajectory, its result-level feedback value is subtracted from the mean, and finally divided by the standard deviation to obtain a comparable global advantage value.

[0047] For example, suppose there are 5 retrieval trajectories in the trajectory set. The result-level feedback value of retrieval trajectory 1 is 0.85, that of retrieval trajectory 2 is 0.60, that of retrieval trajectory 3 is 0.75, that of retrieval trajectory 4 is 0.45, and that of retrieval trajectory 5 is 0.90. The mean and standard deviation of all result-level feedback values ​​are calculated, yielding a mean of 0.71 and a standard deviation of 0.17. Standardization is then applied to each retrieval trajectory, resulting in a global advantage value of 0.82 for retrieval trajectory 1, -0.65 for retrieval trajectory 2, 0.24 for retrieval trajectory 3, -1.53 ​​for retrieval trajectory 4, and 1.12 for retrieval trajectory 5.

[0048] S220, standardize the process-level feedback values ​​of each step in each retrieval trajectory to obtain the local feedback factors of each step; Specifically, within the set of process-level feedback values ​​for all steps in each retrieval trajectory, the process-level feedback values ​​for each step are standardized to obtain the local feedback factor for each step. The standardization process is detailed in step S218 and will not be repeated here.

[0049] S222, based on the preset feedback shaping formula, the global advantage value and local feedback factors are combined to obtain the fusion advantage value of each step; Specifically, a feedback shaping formula is preset, and based on this preset formula, the global advantage value and local feedback factors are combined to obtain the fused advantage value for each step. The preset feedback shaping formula in this embodiment is as follows: ,in, The fused advantage value is obtained by fusing the global advantage value and the local feedback factor. This is the global advantage value of the i-th retrieval trajectory in the trajectory set, used to reflect the overall performance of the entire retrieval trajectory; Let be the local feedback factor for the j-th step in the i-th retrieval trajectory, used to represent the local performance of a single step; This is the fusion intensity coefficient, used to control the degree of influence of local feedback factors on the global advantage value; It is a sign function, according to The sign is returned as (+1), 0, or (-1) to ensure that the adjustment direction of the local feedback factor is consistent with the global advantage value.

[0050] when When the global advantage value is greater than 0, the global advantage value is positive. The preset feedback shaping formula is 1. If the local feedback factor >0, then > The global advantage value is amplified, that is, it encourages the behavior of the j-th step in the i-th retrieval trajectory corresponding to the local feedback factor; if the local feedback factor <0, then < The global advantage value is weakened, that is, the behavior of the j-th step in the i-th retrieval trajectory corresponding to the local feedback factor is penalized.

[0051] when When =0, The default value is 0, and the preset feedback shaping formula is: Local feedback factors will not affect the global advantage value.

[0052] when When the global advantage is less than 0, the global advantage value is negative. The preset feedback shaping formula is (-1). If the local feedback factor >0, then > As the negative value decreases, the global advantage is mitigated, meaning that even if the global advantage of the i-th retrieval trajectory is at a disadvantage, the action in the j-th step will still be rewarded; if the local feedback factor... <0, then < As the negative value increases, the global advantage value intensifies, meaning that the action of this step exacerbates the global disadvantage of the retrieval trajectory in which it is located.

[0053] S224, based on the fusion advantage values ​​and the loss function of the target model, the model parameters of the target model are iteratively updated.

[0054] Specifically, each fusion advantage value is input into the loss function of the target model. Based on the loss function, the policy loss of the target model under the fusion advantage value is calculated. The model parameters of the target model are then iteratively updated using gradients based on the policy loss. The model training method provided in this specification is a policy gradient-based reinforcement learning algorithm. By limiting the magnitude of policy updates, it ensures that each update of the target model is within an acceptable range, thus maintaining the stability and convergence of model training. The loss function used in this method is the policy loss function, which measures the difference between the policy performance of the current step and the expected performance. This guides the model to adjust its parameters, enabling the target model to select steps that yield more feedback, thereby generating more accurate answers with a more efficient retrieval trajectory.

[0055] In the embodiments of this specification, the same target question is input into the model multiple times to obtain multiple retrieval trajectories and output target answers in response to the target question. All retrieval trajectories are added to a trajectory set, and the model is trained using this trajectory set. First, the result-level feedback value of the retrieval trajectory is determined based on the target answer. The target answer directly displays the model's retrieval results. The result-level feedback value is determined based on the content and format of the target answer, improving the comprehensiveness and accuracy of the numerical determination. Based on the result-level feedback value, the retrieval trajectories are classified into first-quality trajectories and second-quality trajectories to determine the process feedback mechanism for each retrieval trajectory. The process feedback mechanism includes a knowledge redundancy feedback mechanism and a knowledge matching feedback mechanism. The knowledge redundancy feedback mechanism dynamically detects and penalizes repeated retrieval behavior within the same retrieval trajectory, significantly reducing invalid and repeated retrieval behavior. The knowledge matching reward mechanism identifies partially correct steps in low-quality trajectories and rewards them. The process-level feedback value for each step in each retrieval trajectory is obtained according to the process feedback mechanism. The result-level feedback value and process-level feedback value are standardized. The obtained global advantage value and local feedback factor are combined to obtain a fusion advantage value, which is used to update the model parameters. By simultaneously monitoring both outcome-level and process-level feedback values ​​during model training, retrieval efficiency and the model's accuracy and robustness in complex question-answering tasks are improved, ensuring answer quality. Updating model parameters using reinforcement learning algorithms guarantees the stability and convergence of model training.

[0056] Please see Figure 5 , Figure 5 This is a flowchart illustrating a model training method provided in an embodiment of this specification. Figure 5 As shown, the method in the embodiments of this specification may include the following steps S302-S306.

[0057] S302, Determine the loss function of the target model based on the pruning strategy formula or the information divergence formula; Specifically, the loss function of the target model is determined based on the pruning policy formula or the Kullback-Leibler Divergence (KL) formula. The model training method provided in the embodiments of this specification is based on the policy gradient method in reinforcement learning. Pruning policy is a core capability of reinforcement learning. By limiting the magnitude of policy updates to ensure update stability, the pruning policy formula can be determined as the loss function of the target model. In reinforcement learning, the KL formula is used to measure the difference between the new policy and the old policy, ensuring that the new policy does not deviate from the old policy. The KL formula can also be determined as the loss function of the target model.

[0058] The trimming strategy formula is: ,in, This represents the ratio of the probability that the new strategy selects the j-th step in the i-th retrieval trajectory to the probability that the old strategy selects that step. ; These are pre-defined pruning parameters used to control the maximum magnitude of the strategy update. The lower bound of the clipping boundary. This is the upper bound of the pruning boundary, which is the maximum allowable range of variation in the probability ratio between the old and new strategies. Let be the fusion advantage value of the j-th step in the i-th retrieval trajectory; Here is the clipping function, used to limit the probability ratio to a certain value. Internally, to prevent excessive changes from a single update; The function is a conservative update strategy used to update... and Choose a smaller update magnitude; E refers to the expected value of multiple steps.

[0059] The KL formula is: ,in, Represents probability distribution Compared to KL divergence, This refers to the probability of choosing each step in the old strategy. E refers to the probability of choosing each step in the new strategy; E refers to the expected value of multiple steps.

[0060] It should be noted that, in addition to the pruning strategy formula and the KL formula, other loss functions or constraint methods for strategy updates can also be used to determine the loss function, and the embodiments in this specification do not limit this.

[0061] S304, Substitute the fused advantage value into the loss function to obtain the policy loss value of the target model; In one feasible implementation, a predetermined number of steps are sequentially retrieved from the retrieved trajectories in the trajectory set, and the fusion advantage value of all the retrieved steps is then calculated. Substitute the parameters into the loss function and update them using the parameters before. Calculate the probability of the old policy selecting the current step, and update the parameters using the iteratively updated parameters. Calculate the probability of the new strategy choosing the current step, combine it with the fusion advantage value of the current step to obtain the strategy loss value of the target model for each step, and then obtain the expected value of each strategy loss value.

[0062] S306, iteratively update the model parameters of the target model based on the policy loss value.

[0063] Specifically, the gradient of the expected value of the policy loss with respect to the model parameters of the target model is calculated using the backpropagation algorithm, and the model parameters are updated in the opposite direction of the gradient. After the parameters are updated, the steps already obtained in step S304 are removed from the trajectory set, and step S304 is returned to be executed until the model parameters are updated based on all steps of all retrieved trajectories in the trajectory set, thus obtaining the trained target model.

[0064] In the embodiments of this specification, the loss function of the model is determined by pruning strategy and KL divergence to limit the magnitude of each model update, ensure the stability and convergence of model training, improve the inference efficiency of the trained model, enable it to have good transferability, and be applicable to multiple question answering scenarios to handle a variety of complex question answering tasks.

[0065] based on Figure 2 The system architecture diagram will be presented below, in conjunction with... Figure 6 This specification provides a detailed description of the model training apparatus provided in the embodiments. It should be noted that... Figure 6 The model training device described herein is used to execute the instructions in this manual. Figures 3-5 The methods shown in the embodiments are illustrated for ease of explanation, showing only the parts related to the embodiments of this specification. For specific technical details not disclosed, please refer to this specification. Figures 3-5 The example shown.

[0066] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a model training device provided in an embodiment of this specification. Figure 6 As shown, the model training device 1 in this embodiment may include: a problem input unit 11, a result feedback unit 12, a process feedback unit 13, and a feedback training unit 14.

[0067] Question input unit 11 is used to input the target question into the target model, obtain the retrieval trajectory of the target model in response to the target question and the output target answer; Result feedback unit 12 is used to determine the retrieval quality and result-level feedback value of the retrieval trajectory based on the target answer; The process feedback unit 13 is used to determine the target feedback mechanism for each step in the retrieval trajectory based on the retrieval quality, and to determine the process-level feedback value of each step based on the target feedback mechanism. Feedback training unit 14 is used to train the target model based on result-level feedback values ​​and process-level feedback values.

[0068] Optionally, the question input unit 11 is specifically used to input the target question into the target model a preset number of times, and obtain the retrieval trajectory of the target model in response to the target question each time and the target answer output. The trajectory set is determined based on each retrieval trajectory.

[0069] Optionally, the result feedback unit 12 is specifically used to determine the result-level feedback value based on the target answer to determine the retrieval trajectory; If the result-level feedback value is greater than or equal to the preset global threshold, then the retrieval quality of the retrieval trajectory is determined to be the first quality, and the retrieval trajectory is determined to be the first quality trajectory. If the result-level feedback value is less than the preset global threshold, the retrieval quality of the retrieval trajectory is determined to be the second quality, and the retrieval trajectory is determined to be the second quality trajectory.

[0070] Optionally, the result feedback unit 12 is specifically used to match the target answer with the standard answer of the target question to obtain the matching result; The format feedback value and content feedback value of the target answer are determined based on the matching results; The result-level feedback value is determined based on the format feedback value and the content feedback value.

[0071] Optionally, the process feedback unit 13 is specifically used to calculate the similarity between each first step in the first quality trajectory and the corresponding historical steps if the retrieval quality is the first quality, so as to obtain the first similarity between each first step and the historical steps. The process-level feedback value for each first step is determined based on the first similarity.

[0072] Optionally, the process feedback unit 13 is specifically used to match each second step in the second quality trajectory with each first step in the trajectory set if the retrieval quality is the second quality, so as to obtain the first step corresponding to each second step; Based on the second similarity between each second step and the corresponding first step, the process-level feedback value of each second step is determined.

[0073] Optionally, the process feedback unit 13 is specifically used to perform extreme value bipartite graph matching on each second step in the second quality trajectory and each first step in each first quality trajectory in the trajectory set if the retrieval quality is the second quality. The first step is determined based on extreme value bipartite graph matching, which satisfies the extreme value alignment relationship with each of the second steps.

[0074] Optionally, the feedback training unit 14 is specifically used to standardize the result-level feedback values ​​of each retrieved trajectory in the trajectory set to obtain the global advantage value of each retrieved trajectory. The process-level feedback values ​​of each step in each retrieval trajectory are standardized to obtain the local feedback factors of each step. Based on the preset feedback shaping formula, the global advantage value and local feedback factors are combined to obtain the fusion advantage value of each step; The model parameters of the target model are iteratively updated based on the fusion advantage values ​​and the loss function of the target model.

[0075] Optionally, the feedback training unit 14 is specifically used to determine the loss function of the target model based on the pruning strategy formula or the information divergence formula; Substituting the fusion advantage value into the loss function yields the policy loss value of the target model; The model parameters of the target model are iteratively updated based on the policy loss value.

[0076] In this embodiment of the specification, the same target question is input into the model multiple times to obtain multiple retrieval trajectories and output target answers in response to the target question. All retrieval trajectories are added to a trajectory set, and the model is trained using this trajectory set. First, the result-level feedback value of the retrieval trajectory is determined based on the target answer. The target answer directly displays the model's retrieval results. The result-level feedback value is determined based on the content and format of the target answer, improving the comprehensiveness and accuracy of the numerical determination. Based on the result-level feedback value, the retrieval trajectories are classified into first-quality trajectories and second-quality trajectories to determine the process feedback mechanism for each retrieval trajectory. The process feedback mechanism includes a knowledge redundancy feedback mechanism and a knowledge matching feedback mechanism. The knowledge redundancy feedback mechanism dynamically detects and penalizes repeated retrieval behavior within the same retrieval trajectory, significantly reducing invalid and repeated retrieval behavior. The knowledge matching reward mechanism identifies partially correct steps in low-quality trajectories and rewards them. The process-level feedback value for each step in each retrieval trajectory is obtained according to the process feedback mechanism. The result-level feedback value and process-level feedback value are standardized. The obtained global advantage value and local feedback factor are combined to obtain a fusion advantage value, which is used to update the model parameters. By simultaneously focusing on both outcome-level and process-level feedback values ​​during model training, retrieval efficiency and the model's accuracy and robustness for complex question-answering tasks are improved, ensuring answer quality. The loss function of the model is determined based on pruning strategies and KL divergence in reinforcement learning algorithms to limit the magnitude of each model update, ensuring the stability and convergence of model training. This improves the inference efficiency of the trained model, giving it good transferability and enabling it to be applied to multiple question-answering scenarios and handle various complex question-answering tasks.

[0077] It should be noted that the model training device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the model training method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the model training device and the model training method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0078] The embodiment numbers in this specification are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0079] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a computer device provided in the embodiments of this specification.

[0080] For example, such as Figure 7 As shown, the computer device 700 includes a processor 701 and a memory 702, wherein the processor 701 is electrically connected to the memory 702.

[0081] Processor 701 is the control center of computer device 700 and may include one or more processing cores. Processor 701 connects to various parts of the computer device using various interfaces and lines. By running or calling computer programs stored in memory 702, and by calling data stored in memory 702, it executes various functions of the computer device and processes data, thereby providing overall control of computer device 700. Optionally, processor 701 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 701 may integrate one or more of the following: CPU, Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user page, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 701 and may be implemented separately using a communication chip.

[0082] The memory 702 can be used to store software programs and modules. The processor 701 executes various functional applications and data processing by running the computer programs and modules stored in the memory 702. The memory 702 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, computer programs required for at least one function, etc.; the data storage area may store data created based on the use of the computer device 700, etc.

[0083] Furthermore, memory 702 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory 702 may also include a memory controller to provide processor 701 with access to memory 702.

[0084] In a first feasible embodiment of the embodiments of this specification, the processor 701 in the computer device 700 loads the instructions corresponding to the processes of one or more computer programs into the memory 702 according to the following steps, and the processor 701 runs the computer programs stored in the memory 702 to realize various functions, as follows: Input the target question into the target model, and obtain the retrieval trajectory of the target model in response to the target question and the output target answer; The retrieval quality and result-level feedback value are determined based on the target answer to determine the retrieval trajectory. Based on retrieval quality, a target feedback mechanism is determined for each step in the retrieval trajectory, and the process-level feedback value of each step is determined based on the target feedback mechanism. The target model is trained based on result-level feedback values ​​and process-level feedback values.

[0085] Optionally, when processor 701 inputs the target question into the target model, obtains the retrieval trajectory of the target model in response to the target question, and outputs the target answer, it specifically performs the following: Input the target question into the target model a preset number of times, and obtain the retrieval trajectory of the target model in response to the target question and the output target answer each time; The trajectory set is determined based on each retrieval trajectory.

[0086] Optionally, when processor 701 executes the retrieval quality and result-level feedback values ​​based on determining the retrieval trajectory according to the target answer, it specifically performs the following: Result-level feedback values ​​are determined based on the target answer to determine the retrieval trajectory; If the result-level feedback value is greater than or equal to the preset global threshold, then the retrieval quality of the retrieval trajectory is determined to be the first quality, and the retrieval trajectory is determined to be the first quality trajectory. If the result-level feedback value is less than the preset global threshold, the retrieval quality of the retrieval trajectory is determined to be the second quality, and the retrieval trajectory is determined to be the second quality trajectory.

[0087] Optionally, when the processor 701 executes the result-level feedback value based on the target answer to determine the retrieval trajectory, it specifically performs the following: The target answer is matched with the standard answer to the target question to obtain the matching result; The format feedback value and content feedback value of the target answer are determined based on the matching results; The result-level feedback value is determined based on the format feedback value and the content feedback value.

[0088] Optionally, when the processor 701 executes the target feedback mechanism for each step in the retrieval trajectory based on retrieval quality, and determines the process-level feedback value for each step based on the target feedback mechanism, it specifically performs the following: If the retrieval quality is the first quality, then the similarity between each first step in the first quality trajectory and the corresponding historical steps is calculated to obtain the first similarity between each first step and the historical steps. The process-level feedback value for each first step is determined based on the first similarity.

[0089] Optionally, when the processor 701 executes the target feedback mechanism for each step in the retrieval trajectory based on retrieval quality, and determines the process-level feedback value for each step based on the target feedback mechanism, it specifically performs the following: If the retrieval quality is the second quality, then each second step in the second quality trajectory is matched with each first step in the first quality trajectory in the trajectory set to obtain the first step corresponding to each second step; Based on the second similarity between each second step and the corresponding first step, the process-level feedback value of each second step is determined.

[0090] Optionally, when the processor 701 executes the step of matching each second step in the second quality trajectory with each first step in the trajectory set to obtain the first step corresponding to each second step, if the search quality is the second quality, the processor 701 specifically performs the following: If the retrieval quality is the second quality, then perform extreme value bipartite graph matching on each second step in the second quality trajectory and each first step in each first quality trajectory in the trajectory set; The first step is determined based on extreme value bipartite graph matching, which satisfies the extreme value alignment relationship with each of the second steps.

[0091] Optionally, when the processor 701 trains the target model based on the result-level feedback values ​​and the process-level feedback values, it specifically performs the following: The result-level feedback values ​​of each search trajectory in the trajectory set are standardized to obtain the global advantage value of each search trajectory; The process-level feedback values ​​of each step in each retrieval trajectory are standardized to obtain the local feedback factors of each step. Based on the preset feedback shaping formula, the global advantage value and local feedback factors are combined to obtain the fusion advantage value of each step; The model parameters of the target model are iteratively updated based on the fusion advantage values ​​and the loss function of the target model.

[0092] Optionally, when processor 701 iteratively updates the model parameters of the target model based on the loss function of each fusion advantage value and the target model, it specifically performs the following: The loss function of the target model is determined based on the pruning strategy formula or the information divergence formula; Substituting the fusion advantage value into the loss function yields the policy loss value of the target model; The model parameters of the target model are iteratively updated based on the policy loss value.

[0093] In this embodiment of the specification, the same target question is input into the model multiple times to obtain multiple retrieval trajectories and output target answers in response to the target question. All retrieval trajectories are added to a trajectory set, and the model is trained using this trajectory set. First, the result-level feedback value of the retrieval trajectory is determined based on the target answer. The target answer directly displays the model's retrieval results. The result-level feedback value is determined based on the content and format of the target answer, improving the comprehensiveness and accuracy of the numerical determination. Based on the result-level feedback value, the retrieval trajectories are classified into first-quality trajectories and second-quality trajectories to determine the process feedback mechanism for each retrieval trajectory. The process feedback mechanism includes a knowledge redundancy feedback mechanism and a knowledge matching feedback mechanism. The knowledge redundancy feedback mechanism dynamically detects and penalizes repeated retrieval behavior within the same retrieval trajectory, significantly reducing invalid and repeated retrieval behavior. The knowledge matching reward mechanism identifies partially correct steps in low-quality trajectories and rewards them. The process-level feedback value for each step in each retrieval trajectory is obtained according to the process feedback mechanism. The result-level feedback value and process-level feedback value are standardized. The obtained global advantage value and local feedback factor are combined to obtain a fusion advantage value, which is used to update the model parameters. By simultaneously focusing on both outcome-level and process-level feedback values ​​during model training, retrieval efficiency and the model's accuracy and robustness for complex question-answering tasks are improved, ensuring answer quality. The loss function of the model is determined based on pruning strategies and KL divergence in reinforcement learning algorithms to limit the magnitude of each model update, ensuring the stability and convergence of model training. This improves the inference efficiency of the trained model, giving it good transferability and enabling it to be applied to multiple question-answering scenarios and handle various complex question-answering tasks.

[0094] It should be understood that the apparatus provided in the embodiments of this specification is used to execute the above-described model training method, and therefore can achieve the same effect as the above-described implementation method.

[0095] When using integrated units, the device may include a processing module and a storage module. When applied to a computer device, the processing module can be used to control and manage the operations of the computer device. The storage module can be used to support the computer device in executing relevant program code, etc.

[0096] The processing module may be a processor or a controller, which can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure herein. The processor may also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc., and the storage module may be a memory.

[0097] In addition, the device provided in the embodiments of this specification may specifically be a chip, component or module. The chip may include a connected processor and a memory. The memory is used to store instructions. When the processor calls and executes the instructions, the chip can execute a model training method provided in the above embodiments.

[0098] This specification also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the aforementioned method steps to implement the model training method provided in the above embodiments.

[0099] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement a model training method provided in the above embodiment.

[0100] In this embodiment, the device, computer-readable storage medium, computer program product, or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0101] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0102] In the embodiments provided in this specification, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0103] The above description is merely a specific embodiment of this specification, but the scope of protection of this specification is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this specification should be included within the scope of protection of this specification. Therefore, the scope of protection of this specification should be determined by the scope of the claims.

Claims

1. A model training method, the method comprising: Input a target question into the target model, and obtain the retrieval trajectory of the target model in response to the target question and the output target answer; The retrieval quality and result-level feedback value of the retrieval trajectory are determined based on the target answer; Based on the retrieval quality, a target feedback mechanism is determined for each step in the retrieval trajectory, and a process-level feedback value for each step is determined based on the target feedback mechanism. The target model is trained based on the result-level feedback value and the process-level feedback value.

2. The method according to claim 1, wherein inputting a target question into the target model and obtaining the retrieval trajectory of the target model in response to the target question and the output target answer comprises: The target question is input into the target model a preset number of times, and the retrieval trajectory of the target model in response to the target question and the output target answer are obtained each time. The trajectory set is determined based on each of the aforementioned retrieval trajectories.

3. The method according to claim 2, wherein determining the retrieval quality and result-level feedback value of the retrieval trajectory based on the target answer includes: Based on the target answer, determine the result-level feedback value of the retrieval trajectory; If the result-level feedback value is greater than or equal to a preset global threshold, then the retrieval quality of the retrieval trajectory is determined to be the first quality, and the retrieval trajectory is determined to be the first quality trajectory. If the result-level feedback value is less than a preset global threshold, then the retrieval quality of the retrieval trajectory is determined to be the second quality, and the retrieval trajectory is determined to be the second quality trajectory.

4. The method according to claim 3, wherein determining the result-level feedback value of the retrieval trajectory based on the target answer includes: The target answer is matched with the standard answer to the target question to obtain the matching result; Based on the matching results, determine the format feedback value and content feedback value of the target answer; The result-level feedback value of the retrieval trajectory is determined based on the format feedback value and the content feedback value.

5. The method according to claim 3, wherein determining a target feedback mechanism for each step in the retrieval trajectory based on the retrieval quality, and determining a process-level feedback value for each step based on the target feedback mechanism, comprises: If the retrieval quality is the first quality, then the similarity between each first step in the first quality trajectory and the corresponding historical step is calculated to obtain the first similarity between each first step and the historical step. The process-level feedback value for each of the first steps is determined based on the first similarity.

6. The method according to claim 5, further comprising: If the retrieval quality is the second quality, then each second step in the second quality trajectory is matched with each first step in the first quality trajectory in the trajectory set to obtain the first step corresponding to each second step; Based on the second similarity between each second step and the corresponding first step, the process-level feedback value of each second step is determined.

7. The method according to claim 6, wherein if the retrieval quality is a second quality, then matching each second step in the second quality trajectory with each first step in the trajectory set to obtain the first step corresponding to each second step includes: If the retrieval quality is the second quality, then extreme value bipartite graph matching is performed on each second step in the second quality trajectory and each first step in each first quality trajectory in the trajectory set; Based on the extreme value bipartite graph matching, the first step is determined to satisfy the extreme value alignment relationship with each of the second steps.

8. The method according to claim 2, wherein training the target model based on the result-level feedback value and the process-level feedback value comprises: The result-level feedback values ​​of each search trajectory in the trajectory set are standardized to obtain the global advantage value of each search trajectory; The process-level feedback values ​​of each step in each retrieval trajectory are standardized to obtain the local feedback factors of each step. The global advantage value and the local feedback factor are combined based on the preset feedback shaping formula to obtain the fusion advantage value of each step. The model parameters of the target model are iteratively updated based on the fusion advantage values ​​and the loss function of the target model.

9. The method according to claim 8, wherein the iterative update of the model parameters of the target model based on each of the fusion advantage values ​​and the loss function of the target model comprises: The loss function of the target model is determined based on the pruning strategy formula or the information divergence formula; Substituting the fusion advantage value into the loss function yields the policy loss value of the target model; The model parameters of the target model are iteratively updated based on the strategy loss value.

10. A model training apparatus, the apparatus comprising: The question input unit is used to input a target question into the target model, obtain the retrieval trajectory of the target model in response to the target question, and the output target answer. The result feedback unit is used to determine the retrieval quality and result-level feedback value of the retrieval trajectory based on the target answer; A process feedback unit is used to determine a target feedback mechanism for each step in the retrieval trajectory based on the retrieval quality, and to determine the process-level feedback value for each step based on the target feedback mechanism. The feedback training unit is used to train the target model based on the result-level feedback value and the process-level feedback value.

11. A computer device, comprising: Processor and memory; The memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method as described in any one of claims 1 to 9.

12. A storage medium storing a computer program that, when executed by a processor, implements the steps of the method as claimed in any one of claims 1 to 9.

13. A computer program product comprising: A computer program, when executed by a processor of a computer device, causes the processor to perform the steps of the method as described in any one of claims 1 to 9.

Images