Large model training method and device, computer equipment and storage medium

Abstract

The invention belongs to the technical field of model training, and particularly relates to a large model training method and device based on reinforcement learning framework implicit coupling, computer equipment and a storage medium, and the method comprises the following steps: for an input query, generating a plurality of candidate responses by using a current strategy model, and verifying each candidate response, obtaining a corresponding reward signal; calculating a reward agent value of each candidate response based on a probability difference between the current policy model and the reference policy model; aiming at the reward agent values of all candidate responses, constructing a baseline-removed scoring function value for each candidate response; and mapping a scoring function value into a prediction probability, taking a reward signal as a supervision label, constructing a loss function adopting supervised learning, updating parameters of a current strategy model through the loss function, and implicitly coupling strategy optimization and value evaluation. According to the method, the calculation cost and the video memory overhead are remarkably reduced, and the training stability is remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of model training technology, and in particular relates to a method, apparatus, computer equipment and storage medium for training large models based on implicit coupling of a reinforcement learning framework. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, Large Language Models (LLMs) have achieved significant breakthroughs in the field of natural language processing. Especially in complex logical tasks such as mathematical reasoning and code generation, the reasoning ability of large models has been greatly enhanced by expanding test-time compute and utilizing Chain of Thought (CoT) techniques. To further enhance the problem-solving capabilities of models, Reinforcement Learning with Verifiable Rewards (RLVR) has become a mainstream technological paradigm. RLVR uses objective results (such as the correctness of answers to mathematical problems or whether code passes test cases) as reward signals to guide the policy model in self-optimization.

[0003] Current RLVR methods are mainly divided into two categories: value-model-based methods and value-model-free methods. The first category is represented by Proximal Policy Optimization (PPO) and its variants. These methods estimate state value or advantage function by training an explicit value model (Critic), thus guiding the updating of the policy model (Actor). While this method can provide relatively stable training signals, its drawbacks are also significant: the introduction of a value model leads to extremely high computational overhead and memory consumption, and the training process is complex, involving the interaction of multiple models, reducing overall training efficiency. The second category is represented by value-model-free methods, such as Group Relative Policy Optimization (GRPO). These methods abandon explicit value models and instead utilize Monte Carlo sampling to estimate advantage by calculating the relative reward of a set of outputs. Although this method reduces model complexity and computational resource requirements, it often exhibits extremely high variance when facing the typical "sparse reward" problem in RLVR scenarios (i.e., obtaining a unique reward signal only after generating a complete long response). Such high variance estimation can easily lead to the collapse of the dominance function, which in turn causes instability in the training process and ultimately limits the upper limit of model performance.

[0004] Based on the above, existing large model training methods based on reinforcement learning have fallen into a dilemma: either they need to bear high computational costs to maintain an explicit value model, such as PPO; or they face challenges such as large gradient update variance and unstable training after removing the value model, such as GRPO.

[0005] Therefore, designing a method that can maintain low resource consumption while providing stable and efficient gradient update signals like the Actor-Critic architecture to effectively solve the policy optimization problem under sparse rewards is a key technical challenge that urgently needs to be addressed in the field of large-scale reinforcement learning. Summary of the Invention

[0006] The purpose of this invention is to provide a method, apparatus, computer device, and storage medium for training large models based on implicit coupling within a reinforcement learning framework. This aims to address the problems of high computational resource consumption and complex training processes in existing value-based methods, as well as the issues of large gradient estimation variance and unstable training in sparse reward scenarios in existing value-free model-based methods.

[0007] The present invention is implemented as follows: a method for training large models based on implicit coupling within a reinforcement learning framework is provided, comprising the following steps: For the input query, multiple candidate responses are generated using the current policy model, and each candidate response is verified to obtain the corresponding reward signal; Based on the probability difference between the current strategy model and the reference strategy model, the reward proxy value of each candidate response is calculated; For the reward proxy values ​​of all said candidate responses, construct a baseline-free scoring function value for each said candidate response; and The scoring function value is mapped to the predicted probability, and the reward signal is used as the supervision label to construct a loss function using supervised learning. The parameters of the current policy model are updated through the loss function to implicitly couple policy optimization and value evaluation.

[0008] In some implementations, the formula for calculating the reward agent value is as follows: , in, For the input query, The candidate response, For the current strategy model, The reference strategy model is as follows. The probability of generating the candidate response for the current policy model. The probability of generating the candidate response for the reference policy model. This is the preset regularization coefficient.

[0009] In some implementations, the formula for calculating the scoring function value is as follows: , in, For the input query, The candidate response, For the current strategy model, The reward agent value, The total number of candidate responses.

[0010] In some implementations, the loss function is a binary cross-entropy loss function, expressed as follows: , in, For the input query, The candidate response, For the current strategy model, The total number of candidate responses. The reward signal, The value of the scoring function. The confidence level for predicting the candidate response as correct by the current strategy model.

[0011] In some implementations, the reference policy model freezes its parameters after pre-training, and the parameters remain unchanged during training.

[0012] In some implementations, the reward signal is a binarized signal or a continuous real-valued signal.

[0013] In some implementations, the reward signal is obtained through one of the following methods: automatic validator judgment, manual annotation, multi-model voting consistency judgment, or automatic scoring based on rules or knowledge base.

[0014] Another embodiment of the present invention provides a large model training device based on implicit coupling of a reinforcement learning framework, comprising: The acquisition and verification unit is used to generate multiple candidate responses for the input query using the current strategy model, and to verify each candidate response to obtain the corresponding reward signal. The reward proxy value calculation unit is used to calculate the reward proxy value of each candidate response based on the probability difference between the current strategy model and the reference strategy model. A scoring function value construction unit is configured to construct a baseline-free scoring function value for each candidate response based on the reward proxy value for all candidate responses; and The model update unit is used to map the scoring function value to the predicted probability, and use the reward signal as a supervision label to construct a loss function using supervised learning. The loss function is used to update the parameters of the current policy model, thereby implicitly coupling policy optimization and value evaluation.

[0015] Another embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the large model training method based on implicit coupling of reinforcement learning framework as described in any of the above embodiments.

[0016] Another embodiment of the present invention provides a storage medium storing a computer program that can be executed to implement the steps of the large model training method based on implicit coupling of reinforcement learning framework as described in any of the above embodiments.

[0017] The present invention provides a method, apparatus, computer device, and storage medium for training large models based on implicit coupling within a reinforcement learning framework. First, the input query is input to the current policy model, and multiple samples are used to generate candidate responses. For each candidate sample, a reward signal is obtained through a verifiable mechanism. Then, a reward surrogate value is calculated based on the probability difference between the current policy model and a reference policy model. Next, a baseline-free scoring function is constructed based on the reward surrogate value. Finally, the scoring function value is mapped to a predicted probability, and a supervised learning loss function is constructed using the reward signal as a supervision label. The parameters of the current policy model are updated through the loss function, implicitly coupling policy optimization and value evaluation. This invention significantly reduces computational costs and memory overhead, and through the design of the reward surrogate value and the baseline-free scoring function, effectively reduces the gradient estimation variance in sparse reward scenarios, significantly improving training stability. Attached Figure Description

[0018] Figure 1 This is a flowchart of a large model training method based on implicit coupling of a reinforcement learning framework provided in an embodiment of the present invention. Figure 2 This is a structural block diagram of a large model training device based on implicit coupling of a reinforcement learning framework, provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the invention, and should not be construed as limiting the invention. Furthermore, it should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0020] In the description of this invention, it should be understood that the terms "length", "width", "upper", "lower", "left", "right", "horizontal", "top", "bottom", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0021] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0022] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection, an electrical connection, or a connection that allows for communication; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0023] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0024] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the invention. Furthermore, reference numerals and / or letters may be repeated in different examples; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. In addition, examples of various specific processes and materials are provided in this invention, but those skilled in the art will recognize the application of other processes and / or the use of other materials.

[0025] This invention provides a method for training large models based on implicit coupling within a reinforcement learning framework, referencing... Figure 1 This includes the following steps: S1. For the input query, generate multiple candidate responses using the current strategy model, and verify each candidate response to obtain the corresponding reward signal; S2. Based on the probability difference between the current strategy model and the reference strategy model, calculate the reward proxy value for each candidate response; S3. For the reward proxy values ​​of all the candidate responses, construct a baseline-free scoring function value for each candidate response; and S4. Map the scoring function value to the predicted probability, and use the reward signal as the supervision label to construct a loss function using supervised learning. Update the parameters of the current policy model through the loss function to implicitly couple policy optimization and value evaluation.

[0026] First, candidate responses are generated and reward signals are obtained, where a candidate response can be represented as... If it is the first Each candidate response can be represented as For each input query (Query), which is input into the current policy model to be trained. In the middle. The current strategy model. It is a big oracle model that generates multiple different candidate responses through sampling, for example, G different candidate responses, forming a set. Subsequently, a task-related, automatically executed validator is used to evaluate each candidate response in the set. Verification is performed. For example, in mathematical reasoning tasks, the verifier compares the model's output answer with the standard answer; in code generation tasks, the verifier runs pre-defined test cases. Based on the verification results, a reward signal corresponding to the candidate response is obtained. This reward signal is used to objectively indicate whether the candidate response is correct or not, or its quality. In some implementations, the reward signal is a binary reward signal. , where 1 represents a correct candidate response and 0 represents an incorrect candidate response.

[0027] Next, the reward surcharge is calculated. To quantify the behavioral bias of the current policy model, the reward surcharge for each candidate response needs to be calculated. First, a reference policy model needs to be defined. The reference strategy model Compared with the current strategy model The architecture is the same, and its parameters are derived from the current policy model. A certain historical state, such as the current policy model before initialization. Or the previous checkpoint, and remain unchanged during this training process. Then, for each candidate response... Calculate the current strategy model respectively and reference strategy model Generate this candidate response conditional probability and Then, based on conditional probability... and The reward agent value is calculated from the logarithmic ratio. Based on the reward agent value It can measure the current strategy model Compared to the reference strategy model In generating this candidate response The degree of "preference" on [the platform].

[0028] Then, a baseline-free scoring function is constructed. Directly using the reward surrogate value for optimization may introduce noise due to the overall bias within the batch. To more fairly evaluate the relative strength of each candidate response within the batch, the common baseline needs to be removed. This step uses a leave-one-out approach for each candidate response. Construct a baseline-free scoring function value Specifically, for the first For each candidate response, its scoring function value It equals its own reward surrogate value minus the arithmetic mean of the reward surrogate values ​​of all other candidate responses in the same batch. This operation allows the scoring function value to more accurately reflect the relative merits of individual candidate responses, effectively reducing the variance of subsequent gradient estimation.

[0029] Finally, construct the loss function and update the model. Then, use the score function value obtained in the previous step... This is mapped to a predicted probability through a function (such as the sigmoid function), and this predicted probability can be understood as the current policy model. The confidence level of a candidate response is used to determine if it is correct. Next, the obtained reward signal is used as a supervision label, and together with the predicted probability, a supervised learning loss function is constructed. This loss function could be, for example, a binary cross-entropy loss function. Finally, by optimizing this loss function, the gradient is backpropagated to update the parameters of the current policy model. Thus, the model achieves efficient coupling between policy optimization (i.e., tending to produce the correct candidate response) and value evaluation (i.e., correctly distinguishing the quality of candidate responses) by minimizing the supervision loss and implicitly optimizing both.

[0030] In the large model training method provided in this invention, the reinforcement learning task is reconstructed into a supervised learning task training framework, breaking the traditional reinforcement learning paradigm. Verifiable reward signals are directly used as supervision labels to construct a loss function for training the current policy model. This method effectively solves the problems of large gradient update variance and unstable training in sparse reward scenarios. Furthermore, an implicit model-coupled parameter update mechanism is adopted. Through a single loss function, the model simultaneously improves the policy generation probability and calibrates the reward scoring function in a single gradient update, achieving implicit coupling and efficient training.

[0031] In some specific embodiments of this application, the formula for calculating the reward agent value is as follows: , in, For the input query, The candidate response, For the current strategy model, The reference strategy model is as follows. The probability of generating the candidate response for the current policy model. The probability of generating the candidate response for the reference policy model. The preset regularization coefficients and update magnitudes prevent the current policy model from deviating too quickly from the reference policy model, thus stabilizing the training process. For example, 0.05 or 0.1.

[0032] The essence of this reward proxy value is the offset between the current policy model and the reference policy model in the logarithmic probability space. If a candidate response... If the probability of being sampled by the current strategy model is greater than the probability of being sampled by the reference strategy model (i.e., the ratio is greater than 1 and the logarithm is positive), then the current strategy model is considered to have made a larger adjustment to this behavior, and the reward agent value will be higher; otherwise, it will be lower.

[0033] In some specific embodiments of this application, the calculation formula for the scoring function value is as follows: , in, For the input query, The candidate response, For the current strategy model, The reward agent value, The total number of candidate responses.

[0034] The calculation formula calculates a baseline for each candidate response individually, which is the average of the reward proxy values ​​for all other candidate responses within the same batch. This calculation eliminates the impact of overall batch fluctuations, allowing the scoring function value to more accurately reflect the relative advantage of the candidate response compared to other candidate responses in the same batch. This significantly reduces the variance of gradient estimation and improves training stability, especially in reward-sparse scenarios.

[0035] In some specific embodiments of this application, the loss function is a binary cross-entropy loss function, expressed as follows: , in, For the input query, The candidate response, For the current strategy model, The total number of candidate responses. The reward signal, The value of the scoring function. The confidence level for predicting the candidate response as correct by the current policy model. If it is the first The scoring function value of each candidate response can be expressed as: .

[0036] The loss function is a binary classification loss used in supervised learning. The model's goal is to make the predicted probabilities as close as possible to the true supervised labels. By optimizing this loss function, the parameters of the current policy model are simultaneously driven to produce higher score function values ​​(and thus higher predicted probabilities) for correct candidate responses, and lower score function values ​​for incorrect candidate responses, thus implicitly coupling policy optimization and value evaluation.

[0037] In some specific embodiments of this application, the reference policy model freezes its parameters after pre-training, and the parameters remain unchanged during training.

[0038] In this embodiment, before initiating the large-scale model training method of this application, a base model is obtained through large-scale pre-training (e.g., self-supervised learning on massive text data). After initiating the large-scale model training method of this application, the pre-trained base model is used as a reference policy model, and all its parameters are immediately frozen. This means that in all subsequent forward computations and gradient backpropagation, the parameters of the reference policy model will not be updated, always maintaining its initial pre-trained state. In this way, a stable behavioral probability baseline can be provided for the current policy model, enabling the current policy model to be effectively regularized, preventing the current policy model from collapsing or deviating excessively into meaningless regions during optimization, and ensuring the stability of training.

[0039] In some specific embodiments of this application, the reward signal is a binarized signal or a continuous real-valued signal.

[0040] Binary signals typically take values ​​of 0 or 1. For example, in solving mathematical problems, the reward signal is 1 if the final answer is completely correct, and 0 otherwise; in code generation, the reward signal is 1 if the generated code passes all preset test cases, and 0 otherwise. This form is suitable for tasks where the results have clear right or wrong boundaries. Continuous real-valued signals, on the other hand, can more accurately characterize the gradient differences in response quality. For example, in code generation, the reward signal can be set as a decimal within the interval [0, 1] based on the proportion of test cases passed; in text generation tasks, the reward signal can be based on the quality score judged by the model. Using continuous real-valued signals provides the model with richer learning signals.

[0041] In some specific embodiments of this application, the reward signal is obtained through one of the following methods: automatic validator judgment, manual annotation, multi-model voting consistency judgment, and automatic scoring based on rules or knowledge base.

[0042] Automated validator judgment refers to the use of programmatic and rule-based tools to automatically determine whether an equation is true. Examples include running code to check if it passes test cases, calling a mathematical engine to verify the validity of an equation, or using rule templates to match key information.

[0043] Manual annotation refers to the scoring of candidate responses by human experts based on subjective or objective criteria. This method yields high-quality reward signals, but it is also costly.

[0044] Multi-model voting consensus evaluation involves using multiple different pre-trained models to evaluate the same candidate response, and then combining the results based on the majority principle or the degree of consistency to arrive at a final reward. This helps reduce the bias of a single model.

[0045] Rule-based or knowledge-based automatic scoring refers to the construction of a detailed scoring rule base or query structured knowledge base (such as a fact knowledge graph) for a specific domain, and the automatic, multi-dimensional scoring of candidate responses for accuracy, security, information integrity, etc.

[0046] In some implementations, the reward signal can be a single reward signal that is finally fully output, or a step-by-step reward signal that is output gradually (e.g., reward signals are given per token or per reasoning step).

[0047] The verification process is provided below: Dataset: To verify the effectiveness of the proposed method, the DeepScaleR dataset was used as the training corpus throughout the experiments. This dataset contains approximately 40,000 high-quality mathematical question-answer pairs. During the evaluation phase, the MATH500 benchmark set, a leading standard in mathematical reasoning, was used for testing. The base model and Qwen2.5-3B were selected to verify the versatility of the proposed method across models with different parameter scales.

[0048] Baseline: To highlight the technical advantages of the method in this invention, two mainstream representative algorithms for large model alignment based on reinforcement learning were selected as baselines for comparison: ① PPO: A classic policy optimization algorithm based on a value model, which estimates advantage through an explicit value network. ② GRPO: A relative policy optimization algorithm without a value model, which estimates intra-group relative advantage using Monte Carlo sampling.

[0049] Hardware and hyperparameters: All experiments were conducted on an NVIDIA A100 GPU. Inference was performed using VLLM, and the model was trained using the Verl framework.

[0050] Metric: pass@k was used as the primary evaluation metric, representing the probability that at least one of the k candidate answers is correct. To obtain stable evaluation results, 32 candidate responses were sampled for each question on the MATH500 dataset, and the value of pass@k was calculated using unbiased estimation.

[0051] The following experimental results were obtained: Method or model Pass@1 Pass@2 Pass@4 Pass@8 Pass@16 Pass@32 Basic Model 56.66 67.64 75.75 81.91 86.87 90.60 PPO 65.48 71.94 77.51 82.14 86.02 89.20 GRPO 66.92 73.34 78.60 83.01 86.60 89.00 This method 67.31 74.75 80.60 85.03 88.22 90.40 Table 1 As shown in Table 1 above, the method of this invention achieved an accuracy of 67.31% in the most critical pass@1 metric, representing a 1.83% improvement over PPO (65.48%) and a 0.39% improvement over GRPO (66.92%). Under various sampling settings from pass@2 to pass@32, the method of this invention maintained its leading advantage over PPO and GRPO, demonstrating that in small-parameter models, implicitly coupled training strategies can more effectively utilize reward signals to improve inference accuracy. Compared to the stagnation or decline in performance of PPO and GRPO at high k values ​​(such as pass@32), the method of this invention maintains extremely high accuracy at high k values ​​on the 7B model, indicating that the method effectively maintains the policy entropy during training, avoids premature convergence, and achieves a good balance between exploration and exploitation.

[0052] This invention provides a large model training device based on implicit coupling within a reinforcement learning framework, with reference to... Figure 2 ,include: The acquisition and verification unit 100 is used to generate multiple candidate responses for the input query using the current strategy model, and to verify each candidate response to obtain the corresponding reward signal. The reward proxy value calculation unit 200 is used to calculate the reward proxy value of each candidate response based on the probability difference between the current strategy model and the reference strategy model. The scoring function value construction unit 300 is configured to construct a baseline-free scoring function value for each candidate response based on the reward proxy value for all candidate responses; and The model update unit 400 is used to map the scoring function value to the predicted probability, and to construct a loss function using supervised learning with the reward signal as the supervision label. The loss function is used to update the parameters of the current policy model, thereby implicitly coupling policy optimization and value evaluation.

[0053] In some specific embodiments of this application, the formula for calculating the reward agent value is as follows: , in, For the input query, The candidate response, For the current strategy model, The reference strategy model is as follows. The probability of generating the candidate response for the current policy model. The probability of generating the candidate response for the reference policy model. This is the preset regularization coefficient.

[0054] In some specific embodiments of this application, the calculation formula for the scoring function value is as follows: , in, For the input query, The candidate response, For the current strategy model, The reward agent value, The total number of candidate responses.

[0055] In some specific embodiments of this application, the loss function is a binary cross-entropy loss function, expressed as follows: , in, For the input query, The candidate response, For the current strategy model, The total number of candidate responses. The reward signal, The value of the scoring function. The confidence level for predicting the candidate response as correct by the current strategy model.

[0056] In some specific embodiments of this application, the reference policy model freezes its parameters after pre-training, and the parameters remain unchanged during training.

[0057] In some specific embodiments of this application, the reward signal is a binarized signal or a continuous real-valued signal.

[0058] In some specific embodiments of this application, the reward signal is obtained through one of the following methods: automatic validator judgment, manual annotation, multi-model voting consistency judgment, and automatic scoring based on rules or knowledge base.

[0059] The specific implementation of the large model training device based on implicit coupling of reinforcement learning framework provided by the present invention can be referred to the specific implementation of the large model training method based on implicit coupling of reinforcement learning framework described above, and will not be repeated here.

[0060] This invention provides a computer device including a processor, which executes a computer program stored in a memory to implement the steps of the large model training method based on implicit coupling of a reinforcement learning framework as described above.

[0061] The present invention also provides a storage medium storing a computer program (instructions) thereon, which, when executed by a processor, implements the steps of the large model training method based on implicit coupling of a reinforcement learning framework as described above.

[0062] For example, a computer program can be divided into one or more modules, one or more of which are stored in memory and executed by a processor to perform the present invention. One or more modules can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a computer device. For example, the computer program can be divided into the steps of the large model training method based on implicit coupling of a reinforcement learning framework provided in the above-described method embodiments.

[0063] Those skilled in the art will understand that the above description of the computer device is merely an example and does not constitute a limitation on the computer device. It may include more or fewer components than described above, or a combination of certain components, or different components, such as input / output devices, network access devices, buses, etc.

[0064] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the computer device, connecting various parts of the computer device via various interfaces and lines.

[0065] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as interface display function, interface interaction function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as map interface, selection interface, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0066] If the modules / units integrated into the computer device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, an electrical signal, and a software distribution medium, etc.

[0067] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A large model training method based on an implicit coupling of a reinforcement learning framework, characterized in that, The method comprises the following steps: For an input query, a plurality of candidate responses are generated using a current policy model, and each candidate response is verified to obtain a corresponding reward signal; Based on the probability difference between the current policy model and a reference policy model, a reward proxy value of each candidate response is calculated; For the reward proxy values of all candidate responses, a score function value without baseline is constructed for each candidate response; And The score function value is mapped to a predicted probability, and a loss function using supervised learning is constructed by taking the reward signal as a supervised label, and the parameters of the current policy model are updated through the loss function to implicitly couple policy optimization and value evaluation.

2. The large model training method based on the reinforcement learning framework implicit coupling according to claim 1, characterized in that, The calculation formula of the reward proxy value is as follows: ， wherein, is the input query, is the candidate response, is the current policy model, is the reference policy model, is the probability of the candidate response generated by the current policy model, is the probability of the candidate response generated by the reference policy model, is a preset regularization coefficient.

3. The large model training method based on the reinforcement learning framework implicit coupling according to claim 1, characterized in that, The calculation formula of the score function value is as follows: ， wherein, is the input query, is the candidate response, is the current policy model, is the reward agent value, is the total number of candidate responses.

4. The large model training method based on the reinforcement learning framework implicit coupling according to claim 1, characterized in that, The loss function is a binary cross-entropy loss function, and the expression is as follows: ， wherein, for the input query, for the candidate response, for the current policy model, for the total number of candidate responses, for the reward signal, for the score function value, for the confidence of the current policy model predicting the candidate response to be correct.

5. The large model training method based on the reinforcement learning framework implicit coupling according to claim 1, characterized in that, The reference policy model freezes the parameters after pre-training, and the parameters remain unchanged during the training process.

6. The large model training method based on the reinforcement learning framework implicit coupling according to claim 1, characterized in that, The reward signal is a binary signal or a continuous real value signal.

7. The large model training method based on the reinforcement learning framework implicit coupling according to claim 1, characterized in that, The reward signal is obtained by one of the following methods: automatic verifier judgment, manual annotation, multi-model voting consistency judgment, automatic scoring based on rules or knowledge base.

8. A large model training device based on an implicit coupling of a reinforcement learning framework, characterized by, It comprises: A collection and verification unit is configured to generate a plurality of candidate responses using a current policy model for an input query, and to verify each candidate response to obtain a corresponding reward signal; A reward proxy value calculation unit is configured to calculate a reward proxy value of each candidate response based on the probability difference between the current policy model and a reference policy model; A score function value construction unit is configured to construct a score function value without baseline for each candidate response based on the reward proxy values of all candidate responses; And A model updating unit is configured to map the score function value to a predicted probability, and construct a loss function using supervised learning by taking the reward signal as a supervised label, and update the parameters of the current policy model through the loss function to implicitly couple policy optimization and value evaluation.

9. A computer device, comprising: The storage medium stores a computer program, and the computer program can be executed to implement the steps of the large model training method based on the implicit coupling of the reinforcement learning framework in any one of claims 1 to 7.

10. A storage medium, characterized by The storage medium stores a computer program, and the computer program can be executed to implement the steps of the large model training method based on the implicit coupling of the reinforcement learning framework in any one of claims 1 to 7.

Images