A large model training method and device
By using multiple evaluation models to assess the relevance and consistency between preset questions and corpus fragments during large-scale model training, and generating a comprehensive reward score, the problem of inconsistent model output answers in closed-set question answering scenarios is solved, achieving high-quality answers and corpus consistency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING ACAD OF ARTIFICIAL INTELLLIGENCE
- Filing Date
- 2026-01-29
- Publication Date
- 2026-06-05
AI Technical Summary
In closed-set question-answering scenarios, existing large-scale model training methods struggle to guarantee the consistency and quality of the model's output answers with the original corpus data. This is especially true in reinforcement learning algorithms, where reward scores cannot effectively measure the relevance and consistency between the predicted answers and corpus segments.
By inputting preset questions, predicted answers, and corpus fragments into multiple evaluation models, consistency scores, relevance scores, and quality scores are generated. A first reward score is then generated by combining these scores, and the model to be trained is trained based on this score to ensure the consistency and quality of the model's output answers with the corpus fragments.
It effectively improves the consistency and quality of the model's output answer and the corpus fragments, ensures the accuracy and rationality of the predicted answer in closed-set question-answering scenarios, and reduces bias and randomness in the model training process.
Smart Images
Figure CN122154774A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of model training technology, and in particular to a method and apparatus for training large models. Background Technology
[0002] Currently, many large-scale models employ reinforcement learning algorithms for training. Reward scores inform the model whether the current training is sufficient, prompting the model to update its parameters towards higher rewards. However, in closed-set question-answering scenarios, the model's training data strictly comes from a fixed corpus (e.g., domain-specific data), and the model's output answers cannot exceed the original corpus. This necessitates that the reward score not only ensure the quality of the model's output answers but also maintain consistency with the original corpus data. Summary of the Invention
[0003] This invention provides a training method and apparatus for a large model. By inputting a preset question, the predicted answer output by the model to be trained based on the preset question, and the corresponding corpus fragments into multiple evaluation models, a first reward score associated with the corpus fragments can be obtained. The model to be trained is then further trained based on this first reward score, effectively ensuring the consistency between the model's output answer and the corpus fragments. Furthermore, since the first reward score also references the preset question and the predicted answer, training the model based on the first reward score also ensures the quality of the model's output answer.
[0004] This invention provides a training method and apparatus for a large model applied to a closed-set question-answering scenario, comprising the following steps: obtaining a model to be trained and the predicted answer output by the model to be trained based on a preset question; wherein the model to be trained is a neural network model trained using a reinforcement learning algorithm; inputting the preset question, the predicted answer, and the corpus fragment corresponding to the preset question into multiple evaluation models respectively, and outputting a first reward score corresponding to each evaluation model; and training the model to be trained based on each of the first reward scores.
[0005] Optionally, the first reward score includes: the consistency score between the predicted answer and the corpus segment, the relevance score between the predicted answer and the preset question, and the quality score of the predicted answer; The process involves inputting the preset question, the predicted answer, and the corpus fragment corresponding to the preset question into multiple evaluation models, and outputting a first reward score for each evaluation model. This includes: inputting the preset question, the standard answer corresponding to the preset question, and the corpus fragment corresponding to the standard answer into multiple evaluation models, and outputting the consistency score, the relevance score, and the quality score respectively; and generating the first reward score based on the consistency score, the relevance score, and the quality score.
[0006] Optionally, generating the first reward score based on the consistency score, the correlation score, and the quality score includes: determining the weights corresponding to the consistency score, the correlation score, and the quality score respectively; and using the weights to weight the consistency score, the correlation score, and the quality score to obtain the first reward score.
[0007] Optionally, training the model to be trained based on each of the first reward scores includes: removing outliers from the plurality of first reward scores to obtain a plurality of target first reward scores; using the median of the plurality of target first reward scores as the second reward score, or using the average of the plurality of target first reward scores as the second reward score; and training the model to be trained using the second reward scores.
[0008] Optionally, training the model to be trained using the second reward score includes: substituting the second reward score into the reinforcement learning training of the model to be trained, so that the model to be trained performs the next round of reinforcement learning based on the second reward score.
[0009] Optionally, when the reinforcement learning algorithm uses the proximal policy optimization algorithm as the core algorithm, the reward model in the model to be trained is trained using the second reward score; or, When the reinforcement learning algorithm uses the group relative policy optimization algorithm as the core algorithm, the preference scores in the model to be trained are trained using the second reward score.
[0010] The present invention also provides a training device for a large model applied to a closed-set question-answering scenario, comprising the following modules: The acquisition module is used to acquire the model to be trained and the predicted answer output by the model to be trained based on a preset question; wherein, the model to be trained is a neural network model trained using reinforcement learning; The prediction module is used to input the preset question, the predicted answer, and the corpus fragment corresponding to the preset question into multiple evaluation models, and output the first reward score corresponding to each evaluation model. The training module is used to obtain a second reward score based on each of the first reward scores, and to use the second reward score to train the model.
[0011] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the training method for a large model as described in any of the above.
[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method for a large model as described above.
[0013] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the training method for a large model as described above.
[0014] This invention provides a training method and apparatus for a large model. By inputting a preset question, the predicted answer output by the model to be trained based on the preset question, and the corresponding corpus fragment into multiple evaluation models, a first reward score associated with the corpus fragment can be obtained. The model to be trained is then further trained based on this first reward score, effectively ensuring the consistency between the model's output answer and the corpus fragment. Furthermore, since the first reward score also references the preset question and the predicted answer, training the model based on the first reward score also ensures the quality of the model's output answer. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0016] Figure 1 This is one of the flowcharts illustrating the training method for large models provided by this invention.
[0017] Figure 2 This is a schematic diagram of the process for generating the first reward score in step 102 provided by the present invention.
[0018] Figure 3This is a schematic diagram of the process of training the model to be trained according to the first reward score in step 103 of the present invention.
[0019] Figure 4 This is a schematic diagram of the structure of the training device for large models provided by the present invention.
[0020] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0022] Figure 1 This is one of the flowcharts illustrating a large model training method provided by the present invention, such as... Figure 1 As shown, the method includes the following: Step 101: Obtain the model to be trained and the predicted answer output by the model to be trained based on the preset question; wherein, the model to be trained is a neural network model trained using a reinforcement learning algorithm; In this embodiment of the invention, a preset question needs to be input into the model to be trained first, and the predicted answer returned by the model to be trained needs to be obtained before the large model can be optimized based on the correlation and accuracy between the predicted answer and the preset question and the corpus fragment.
[0023] Step 102: Input the preset question, the predicted answer, and the corresponding corpus fragments into multiple evaluation models, and output the first reward score corresponding to each evaluation model. The evaluation model refers to currently open-source large language models, such as the ChatGPT model, Doubao model, and Wenxin Yiyan model. It should be noted that a single evaluation model is susceptible to model bias, output fluctuations, and task mismatch when evaluating predicted answers, leading to unstable reward scores or low accuracy. Therefore, in this embodiment of the invention, multiple evaluation models are used for simultaneous evaluation, and subsequent models are trained based on the first reward score obtained from each evaluation model. This effectively reduces bias and the randomness of the first reward score, improving its stability and robustness.
[0024] In an optional embodiment of the present invention, the reward score includes: a consistency score between the predicted answer and the corpus segment, a relevance score between the predicted answer and the preset question, and a quality score between the predicted answer, the preset question, and the corpus segment; step 102 may specifically include: inputting the preset question, the standard answer corresponding to the preset question, and the corpus segment corresponding to the standard answer into multiple evaluation models respectively, and outputting consistency score, relevance score, and quality score respectively; generating a first reward score based on the consistency score, relevance score, and quality score.
[0025] In existing methods for awarding reward scores using evaluation models, the output of the evaluation model only considers the preset question and the predicted answer, and evaluates the correctness of the predicted answer by judging whether it is consistent with the preset question. However, the embodiments of the present invention are aimed at closed-set question-answering scenarios. In this scenario, the predicted answer has the requirement of being closed-set, that is, the predicted answer cannot exceed the range of the original corpus data and can only be obtained from the original corpus fragments. Therefore, in the embodiments of the present invention, the corpus fragments corresponding to the preset question are specifically added to the input of the evaluation model. This allows the consistency score obtained to reflect the consistency relationship between the predicted answer and the corpus fragments, thereby meeting the model requirements in closed-set question-answering scenarios.
[0026] Specifically, the consistency score reflects whether the predicted answer is inferred from the corpus fragments, such as whether the core information of the predicted answer completely matches the corpus fragments, whether the predicted answer can be logically deduced based on the corpus fragments, and whether there is any additional information in the predicted answer that exceeds the scope of the corpus fragments; the relevance score reflects whether the predicted answer is a correct and complete response to the preset question, such as whether the predicted answer accurately responds to the core requirements of the preset question, whether the predicted answer fully covers the key points of the question, and whether the logic of the predicted answer is clear; the quality score reflects whether the preset question is clear, whether the corpus fragments can provide the necessary information, and whether the predicted answer is correctly extracted.
[0027] In an optional embodiment, when the first reward score includes a consistency score C1 corresponding to the predicted answer and the corpus segment, a relevance score C2 corresponding to the predicted answer and the preset question, and a quality score C3 corresponding to the predicted answer, the preset question, and the corpus segment, the process of generating the first reward score can be as follows: Figure 2 As shown, it includes: Step 201: Input the preset question, the standard answer corresponding to the preset question, and the corpus fragment corresponding to the standard answer into multiple evaluation models respectively, and output the consistency score C1, the relevance score C2, and the quality score C3 respectively. For example, the total score for consistency score C1, relevance score C2, and quality score C3 can all be set to 5 points. Specifically, for consistency score C1, 5 points represents complete consistency, rigorous reasoning, and no redundant or missing information; 4 points represents basic consistency, no deviation in core information, and only differences in expression; 3 points represents partial consistency, complete core information, but with minor deviations in non-critical information; 2 points represents weak relevance, deviations in key information, and weak reasoning; and 1 point represents complete irrelevance or exceeding the scope of the corpus, with no valid reasoning basis. For relevance score C2: 5 points represents a complete response to the core issue, complete information, rigorous logic, and no redundancy; 4 points represents an accurate response to the core request, relatively complete information, and only minor non-critical redundancy; 3 points represents a basic response to the core request, clear core information, but with some missing details or minor redundancy; 2 points represents a deviation from some core requests, incomplete information, and inability to fully answer the question; and 1 point represents complete irrelevance and no effective response to the question. For the quality score C3, 5 points represent that the pre-set question is clear and unambiguous, the information in the corpus fragment is sufficient and accurate, and the predicted answer is completely accurate; 4 points represent that the pre-set question is basically clear, the information in the corpus fragment is relatively sufficient, the predicted answer is basically accurate, and there are no key errors; 3 points represent that the pre-set question has slight ambiguity but does not affect understanding, the key information in the corpus fragment is complete, and the predicted answer is not obviously wrong; 2 points represent that the pre-set question has obvious ambiguity or the key information in the corpus fragment is missing or the predicted answer has slight errors; 1 point represents that the pre-set question is seriously ambiguous and cannot be understood or the corpus fragment cannot support the answer to the question or the predicted answer is seriously wrong.
[0028] Step 202: Determine the weights corresponding to the consistency score C1, the relevance score C2, and the quality score C3 respectively; For example, the weight W1 of consistency score C1 can be set to 0.4, the weight W2 of relevance score C2 can be set to 0.3, and the weight W3 of quality score C3 can be set to 0.3. It should be noted that when targeting application scenarios such as finance and healthcare where the consistency of the corpus is extremely important, the weight W1 of C1 can be appropriately increased to 0.5-0.6. In scenarios where the completeness of the question response is important, the weight W2 of C2 can be appropriately increased. This invention does not impose any specific limitations on this.
[0029] Step 203: Using weights, weight the consistency score, correlation score, and quality score to obtain the first reward score.
[0030] For example, the weights W1, W2, and W3 corresponding to the consistency score C1, relevance score C2, and quality score C3, respectively, can be configured randomly or according to the degree of emphasis on these scores. For instance, in a closed-set question-answering scenario, where consistency is highly valued, the weight corresponding to the consistency score C1 will be larger. However, in ordinary application scenarios, where the model can freely predict answers, the weight corresponding to the consistency score C1 can be set smaller, even close to 0. This invention does not impose specific limitations on this.
[0031] Step 103: Train the model to be trained according to each first reward score.
[0032] It is understandable that since multiple evaluation models have obtained unique and corresponding first reward scores, it is necessary to uniformly calculate each first reward score to obtain a unique reward score before it can be used for training subsequent models. Therefore, in an optional embodiment of the present invention, the process of step 103 is as follows: Figure 3 As shown, it specifically includes: Step 301: Remove outliers from multiple first reward scores to obtain multiple target first reward scores; In this step, a deviation threshold can be preset, and the average of multiple first reward scores can be calculated. Then, by determining whether the difference between each first reward score and the average is greater than the preset deviation threshold, it can be determined whether the first reward score is an outlier. For example, the deviation threshold can be 2σ, that is, first reward scores that deviate significantly from the average are removed to obtain the target first reward score.
[0033] Step 302: Use the median of the multiple target first reward scores as the second reward score, or use the average of the multiple target first reward scores as the second reward score; Step 303: Train the model to be trained using the second reward score.
[0034] It should be noted that only reinforcement learning algorithms involve reward scores. Therefore, the large-scale model training method provided in this invention can only be implemented for models trained using reinforcement learning algorithms. Specifically, the second reward score can be substituted into the reinforcement learning training of the model to be trained, so that the model can perform the next round of reinforcement learning based on the second reward score.
[0035] In one optional embodiment, when the reinforcement learning algorithm uses the proximal policy optimization algorithm as the core algorithm, the reward model in the model to be trained is trained using the second reward score; when the reinforcement learning algorithm uses the group relative policy optimization algorithm as the core algorithm, the preference score in the model to be trained is trained using the second reward score.
[0036] Specifically, when applied to the PPO-RLHF model, the second reward score can be used as the core reward signal and directly input into the reward model of the PPO training framework, providing explicit feedback for policy network updates and guiding the model to generate high-quality answers that conform to closed-set constraints. When applied to the GRPO model, the second reward score can be used as a preference score during the GRPO training process, assisting GRPO in achieving constrained policy optimization and strengthening the preference guidance for closed-set compliant answers. When applied to the DPO model, the second reward score can be used as a preference score during the DPO training process, providing DPO with preference data that does not require additional annotation and driving the model to learn output patterns that conform to corpus constraints. In addition, it can also be applied to the training of reinforcement learning models with other reward feedback through rapid integration via interface adaptation. The second reward score provided in this embodiment of the invention has strong stability and high interpretability, strictly ensuring that the answer behavior does not exceed the range of the closed-set corpus.
[0037] In a further optional embodiment, for the trained model, after processing a predetermined amount of question-and-answer data, a certain percentage (e.g., 10%) of the data can be automatically extracted for manual review, and calibration can be performed based on the manual review scores to ensure the long-term stability of the large model. Alternatively, the weights of evaluation models with significant deviations from the manual scores can be reduced to improve the overall evaluation system.
[0038] In summary, the large-scale model training method provided in this invention, by inputting a preset question, the predicted answer output by the model to be trained based on the preset question, and the corresponding corpus fragment into multiple evaluation models, can obtain a first reward score associated with the corpus fragment. Furthermore, training the model to be trained based on this first reward score effectively ensures the consistency between the model's output answer and the corpus fragment. Moreover, since the first reward score also references the preset question and the predicted answer, training the model to be trained based on the first reward score also ensures the quality of the model's output answer.
[0039] The training apparatus for large models provided by the present invention will be described below. The training apparatus for large models described below can be referred to in correspondence with the training method for large models described above. The large model is applied to closed-set question answering scenarios.
[0040] like Figure 4As shown, the training device 400 for large models provided in this embodiment of the invention includes: The acquisition module 401 is used to acquire the model to be trained and the predicted answer output by the model to be trained based on a preset question; wherein, the model to be trained is a neural network model trained using reinforcement learning; The prediction module 402 is used to input the preset question, the predicted answer, and the corpus fragment corresponding to the preset question into multiple evaluation models respectively, and output the first reward score corresponding to each evaluation model. Training module 403 is used to obtain a second reward score based on each of the first reward scores, and to use the second reward score to train the model.
[0041] In an optional embodiment of the present invention, the first reward score includes: a consistency score between the predicted answer and the corpus segment, a relevance score between the predicted answer and the preset question, and a quality score between the predicted answer, the preset question, and the corpus segment; the prediction module 402 is further configured to input the preset question, the standard answer corresponding to the preset question, and the corpus segment corresponding to the standard answer into multiple evaluation models respectively, and output the consistency score, the relevance score, and the quality score respectively; and generate the first reward score based on the consistency score, the relevance score, and the quality score.
[0042] In an optional embodiment of the present invention, the prediction module 402 is further configured to determine the weights corresponding to the consistency score, the correlation score, and the quality score respectively; and to use the weights to weight the consistency score, the correlation score, and the quality score to obtain the first reward score.
[0043] In an optional embodiment of the present invention, the training module 403 is further configured to: remove outliers from a plurality of first reward scores to obtain a plurality of target first reward scores; use the median of the plurality of target first reward scores as the second reward score, or use the average of the plurality of target first reward scores as the second reward score; and train the model to be trained using the second reward score.
[0044] In an optional embodiment of the present invention, the training module 403 is further configured to substitute the second reward score into the reinforcement learning training of the model to be trained, so that the model to be trained performs the next round of reinforcement learning based on the second reward score.
[0045] In an optional embodiment of the present invention, when the reinforcement learning algorithm uses the proximal policy optimization algorithm as the core algorithm, the reward model in the model to be trained is trained using the second reward score; or, when the reinforcement learning algorithm uses the group relative policy optimization algorithm as the core algorithm, the preference score in the model to be trained is trained using the second reward score.
[0046] In summary, the training apparatus for large models provided in this invention, by inputting a preset question, the predicted answer output by the model to be trained based on the preset question, and the corresponding corpus fragment into multiple evaluation models, can obtain a first reward score associated with the corpus fragment. Furthermore, training the model to be trained based on this first reward score effectively ensures the consistency between the model's output answer and the corpus fragment. Moreover, since the first reward score also references the preset question and the predicted answer, training the model to be trained based on the first reward score also ensures the quality of the model's output answer.
[0047] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a training method for a large model. This method includes: acquiring a model to be trained and the predicted answer output by the model to be trained based on a preset question; wherein the model to be trained is a neural network model trained using a reinforcement learning algorithm; inputting the preset question, the predicted answer, and the corpus fragment corresponding to the preset question into multiple evaluation models respectively, and outputting a first reward score corresponding to each evaluation model; and training the model to be trained based on each of the first reward scores.
[0048] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0049] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the training method for the large model provided by the above methods. The method includes: obtaining a model to be trained and a predicted answer output by the model to be trained based on a preset question; wherein the model to be trained is a neural network model trained using a reinforcement learning algorithm; inputting the preset question, the predicted answer, and the corpus fragment corresponding to the preset question into multiple evaluation models respectively, and outputting a first reward score corresponding to each evaluation model; and training the model to be trained based on each of the first reward scores.
[0050] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for training a large model provided by the above methods. The method includes: acquiring a model to be trained and a predicted answer output by the model to be trained based on a preset question; wherein the model to be trained is a neural network model trained using a reinforcement learning algorithm; inputting the preset question, the predicted answer, and a corpus fragment corresponding to the preset question into multiple evaluation models respectively, and outputting a first reward score corresponding to each evaluation model; and training the model to be trained based on each of the first reward scores.
[0051] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0052] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A training method for a large model, characterized in that, The large model is applied to closed-set question answering scenarios, including: Obtain the model to be trained and the predicted answer output by the model to be trained based on a preset question; wherein, the model to be trained is a neural network model trained using a reinforcement learning algorithm; The preset question, the predicted answer, and the corpus fragment corresponding to the preset question are respectively input into multiple evaluation models, and the first reward score corresponding to each evaluation model is output. The model to be trained is trained based on each of the first reward scores.
2. The training method for a large model according to claim 1, characterized in that, The first reward score includes: the consistency score between the predicted answer and the corpus segment, the relevance score between the predicted answer and the preset question, and the quality score corresponding to the predicted answer, the preset question, and the corpus segment; The preset question, the predicted answer, and the corresponding corpus fragment are input into multiple evaluation models, and the first reward score corresponding to each evaluation model is output, including: The preset question, the standard answer corresponding to the preset question, and the corpus fragment corresponding to the standard answer are respectively input into multiple evaluation models, and the consistency score, the relevance score, and the quality score are respectively output. The first reward score is generated based on the consistency score, the correlation score, and the quality score.
3. The training method for a large model according to claim 2, characterized in that, The step of generating the first reward score based on the consistency score, the correlation score, and the quality score includes: The weights corresponding to the consistency score, the correlation score, and the quality score are determined respectively. The consistency score, the correlation score, and the quality score are weighted using the weights to obtain the first reward score.
4. The training method for a large model according to claim 1, characterized in that, The step of training the model to be trained based on each of the first reward scores includes: Outliers are removed from the multiple first reward scores to obtain multiple target first reward scores; The second reward score is the median of the multiple target first reward scores, or the second reward score is the average of the multiple target first reward scores. The model to be trained is trained using the second reward score.
5. The training method for a large model according to claim 4, characterized in that, The step of training the model to be trained using the second reward score includes: The second reward score is substituted into the reinforcement learning training of the model to be trained, so that the model to be trained can perform the next round of reinforcement learning based on the second reward score.
6. The training method for a large model according to claim 1, characterized in that, When the reinforcement learning algorithm uses the proximal policy optimization algorithm as the core algorithm, the reward model in the model to be trained is trained using the second reward score; or, When the reinforcement learning algorithm uses the group relative policy optimization algorithm as the core algorithm, the preference scores in the model to be trained are trained using the second reward score.
7. A training device for large models, characterized in that, The large model is applied to closed-set question answering scenarios, including: The acquisition module is used to acquire the model to be trained and the predicted answer output by the model to be trained based on a preset question; wherein, the model to be trained is a neural network model trained using reinforcement learning; The prediction module is used to input the preset question, the predicted answer, and the corpus fragment corresponding to the preset question into multiple evaluation models, and output the first reward score corresponding to each evaluation model. The training module is used to obtain a second reward score based on each of the first reward scores, and to use the second reward score to train the model.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the training method for the large model as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method for the large model as described in any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method for the large model as described in any one of claims 1 to 6.