Model training method, device and product

Abstract

The invention provides a model training method, equipment and a product. The method comprises the following steps: inputting input information into a first model to obtain a first output result; inputting the input information into a second model to obtain a second output result containing a second response sample, a second output probability and a second output reward value; wherein the number of parameters in the first model is smaller than the number of parameters in the second model; calculating an estimated output probability of generating a second response sample by using the first model; calculating a ratio of the estimated output probability to the second output probability to obtain an importance sampling weight; performing reward conversion processing by using the estimated output probability, the second output probability and the second output reward value to generate a reward value conversion result matched with the first model; and inputting the importance sampling weight, the reward value conversion result and the first output result into a first model, and performing training by using a preset reinforcement learning optimization algorithm.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to model training methods, equipment, and products. Background Technology

[0002] With the development of artificial intelligence (AI) technology, more and more jobs will require the assistance of artificial intelligence.

[0003] In practical applications, while large-parameter models are certainly useful, their use and training also require higher hardware and application costs. Therefore, in many scenarios, small-parameter models are needed to replace large-parameter models. To improve the performance of small-parameter models, they need to be trained. However, small-parameter models have inherent limitations in the output process of reinforcement learning. When faced with difficult or complex tasks, small-parameter models, due to their inherent limitations, may fail to output high-quality correct samples. Existing mainstream reinforcement learning methods, such as the policy gradient method (GRPO) combining proximal policy optimization (PPO) and generalized advantage estimation (GAE), have significant drawbacks in addressing the insufficient output of small-parameter models. These methods typically rely on the output capabilities of the trained model itself to explore the environment and collect experience; if the model's output capabilities are insufficient, training efficiency and effectiveness will be greatly reduced. Therefore, a solution is needed to effectively improve the training performance of small-parameter models. Summary of the Invention

[0004] This disclosure provides model training methods, equipment, and products.

[0005] According to a first aspect of this disclosure, a model training method is provided. The method specifically includes: inputting input information into a first model to obtain a first output result; inputting the input information into a second model to obtain a second output result containing a second response sample, a second output probability, and a second output reward value; wherein the number of parameters in the first model is less than the number of parameters in the second model; calculating the estimated output probability of generating the second response sample using the first model; calculating the ratio of the estimated output probability to the second output probability to obtain an importance sampling weight; performing a reward transformation process using the estimated output probability, the second output probability, and the second output reward value to generate a reward value transformation result matching the first model; and inputting the importance sampling weight, the reward value transformation result, and the first output result into the first model for training using a preset reinforcement learning optimization algorithm.

[0006] According to this technical solution, a collaborative reinforcement learning method is used to generate high-quality second response samples using a second model. These samples are then adapted to the optimization objective of the first model through importance sampling and reward value transformation. This significantly improves the performance of a small-parameter-scale, large-scale language model in complex tasks, while drastically reducing computational resource consumption. It compensates for the first model's weak exploration ability in complex tasks, ensuring high-quality training signals. With the assistance of the second model, the first model converges quickly, shortening training time. The performance of the first model approaches that of the second model. Furthermore, it overcomes the problem of poor training results caused by the first model directly imitating the second model during reinforcement learning.

[0007] According to at least one embodiment of this disclosure, a reward conversion process is performed using the estimated output probability, the second output probability, and the second output reward value to generate a reward value conversion result that matches the performance characteristics of the first model. This includes: calculating the logarithmic difference between the first logarithm of the estimated output probability and the second logarithm of the second output probability; and calculating the reward value conversion result that matches the performance characteristics of the first model using the logarithmic difference, the second output reward value, and a preset adjustment factor.

[0008] According to this technical solution, by calculating the importance sampling weights and the reward value transformation results, the efficiency and effectiveness of the first model's reinforcement learning training are significantly improved. Traditional reinforcement learning methods rely on the output of the first model itself. Due to limited computational resources, the generated samples are of low quality, and the training process is prone to failure due to distribution mismatch or unsuitable reward signals. This disclosed solution solves the bias problem introduced by heterogeneous policy data by introducing importance sampling weights and using probability ratios to weight the reward value of the second model, ensuring unbiased training. At the same time, the reward value transformation results are dynamically adapted to the performance characteristics of the first model, avoiding excessively high expectations caused by directly using the reward signal of the second model, thus optimizing training stability.

[0009] According to at least one embodiment of this disclosure, inputting input information into a first model to obtain a first output result includes: performing an output operation on the input information through the first model to generate a first response sample; determining the generation probability of the first response sample under the first model strategy as a first output probability; and calculating a first output reward value corresponding to the first response sample based on a preset reward function.

[0010] According to this technical solution, the first output result is obtained through the local output operation of the first model, serving as the foundational data for subsequent reinforcement learning training. Due to the lightweight nature of the first model, it adapts to the resource limitations of the local computing environment, requiring low computing power and ensuring the real-time performance and accessibility of training. This disclosure reduces computing costs and latency through the autonomous output of the first model, making it particularly suitable for edge devices or low-cost scenarios.

[0011] According to at least one embodiment of this disclosure, inputting input information into a second model to obtain a second output result containing a second output probability and a second output reward value includes: performing an output operation on the input information through the second model to generate a second response sample; determining the generation probability of the second response sample under the second model strategy as the second output probability; and calculating the second output reward value corresponding to the second response sample based on the same preset reward function.

[0012] According to this technical solution, high-quality second output results are generated through the output operations of the second model, providing crucial support for the reinforcement learning training of the first model. This significantly improves the training quality and efficiency of the first model. Furthermore, leveraging the high computing power of the second model on a cloud computing server, a sufficient number of high-quality second response samples are generated, providing the first model with diverse and high-value external learning data.

[0013] Calculating the estimated output probability of generating a second response sample using a first model according to at least one embodiment of this disclosure includes: constructing a target sequence using the second response sample; and calculating the estimated output probability by sequentially calculating multiple tokens in the target sequence using the first model.

[0014] According to this technical solution, the capability gap between the first and second models is precisely quantified by calculating the generation probability of the second response sample generated by the first model. The estimated output probability directly reflects the likelihood of the second response sample being achieved by the first model. When the estimated output probability is low, it indicates that the sample exceeds the current capability range of the first model, and its weight in training will be automatically reduced to prevent the first model from blindly pursuing behaviors that cannot be achieved, thereby maintaining the stability of the training process.

[0015] According to at least one embodiment of this disclosure, reinforcement learning training is performed by inputting importance sampling weights, reward value transformation results, and a first output result into a first model to be trained, including: inputting the first output result, importance sampling weights, and transformed reward value transformation results into an objective function to train the first model; calculating the expected reward of the objective function based on a reinforcement learning optimization algorithm, and determining the parameters of the first model corresponding to maximizing the expected reward.

[0016] According to at least one embodiment of this disclosure, the expected reward of the objective function is calculated based on a reinforcement learning optimization algorithm, and the parameters of the first model corresponding to maximizing the expected reward are determined, including: constructing a hybrid experience replay pool by combining a first response sample with a second response sample processed by importance sampling weights and reward value transformation; calculating the objective function of the first model based on the samples in the hybrid experience replay pool, wherein the objective function is the expected reward under the first model policy; calculating the advantage function value of the samples in the hybrid experience replay pool to estimate the gradient direction of the expected reward; calculating the policy gradient in the gradient direction using the importance sampling weights, the advantage function value, and the log probability gradient of the first output probability; and updating the parameters of the first model through an optimizer according to the policy gradient to maximize the expected reward.

[0017] According to this technical solution, the importance sampling weight is adjusted by calculating the ratio of the estimated output probability to the second output probability, ensuring the unbiasedness of the second model's sample contribution in the first model's optimization objective. For example, if the sample generated by the second model has a low probability under the first model's policy, the weighting value will amplify its reward impact to compensate for distribution bias. The reward value transformation result is generated by analyzing the probability distribution differences, combining the second output reward value and adjustment factors, to produce an actual reward value that fits the first model's capabilities, avoiding overfitting caused by the direct transfer of the second model's reward signal. In the hybrid experience replay pool, the above results are combined with the first output result, enriching the training data source. At the same time, through advantage function and policy gradient optimization, the performance bottleneck caused by insufficient sample quality and distribution mismatch in traditional methods is eliminated.

[0018] According to at least one embodiment of this disclosure, the first model is subjected to iterative reinforcement learning training; the iteration is terminated when the change in the average reward value of the first model is less than the reward threshold, or when the number of consecutive iterations N is not less than the number threshold.

[0019] According to this technical solution, the efficiency and stability of reinforcement learning training for the first model are significantly improved by establishing clear iteration termination conditions. By setting termination conditions such as the average reward value changing less than a reward threshold or the number of consecutive iterations N reaching a threshold, combined with optional early stopping mechanisms and objective function optimization, the timely termination of the training process is ensured, avoiding resource waste and performance degradation.

[0020] According to at least one embodiment of this disclosure, the first model and the second model are deployed in different computer devices.

[0021] According to this technical solution, the collaborative output of the first and second models in different computing environments overcomes the limitations of the first model. If relying solely on the output of the first model, the limited computing resources of the local computer result in low-quality samples, low training efficiency, and a tendency to get trapped in local optima. This disclosed solution deploys the second model on a high-performance cloud computing server, leveraging its powerful generation capabilities to obtain high-quality second output results, thus compensating for the insufficient output of the first model. Although the second model is used, because the second and first models are distributed, they do not consume the local computing resources of the first model. In other words, during training, the first model runs locally, reducing the overall computing cost and latency of training, making it suitable for resource-constrained scenarios.

[0022] According to a second aspect of this disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, such that the processor performs the method described in the first aspect of any embodiment of this disclosure.

[0023] According to a third aspect of this disclosure, a readable storage medium is provided, wherein executable instructions are stored therein, which, when executed by a processor, are used to implement the method described in the first aspect of any embodiment of this disclosure.

[0024] According to a fourth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described in the first aspect of any embodiment of this disclosure. Attached Figure Description

[0025] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0026] Figure 1 This is a schematic flowchart of the model training method provided in the embodiments of this disclosure.

[0027] Figure 2 This is a flowchart illustrating the output result conversion method provided in the embodiments of this disclosure.

[0028] Figure 3 This is a schematic diagram of the process for generating the first output result provided in an embodiment of the present disclosure.

[0029] Figure 4 This is a schematic flowchart illustrating the generation of a second output result provided in an embodiment of this disclosure.

[0030] Figure 5This is a flowchart illustrating a reinforcement learning method provided in an embodiment of the present disclosure.

[0031] Figure 6 A schematic diagram of the iterative optimization process provided for embodiments of this disclosure.

[0032] Figure 7 This is a schematic block diagram of a model training device according to one embodiment of the present disclosure.

[0033] Figure 8 This is a schematic block diagram of an electronic device according to one embodiment of the present disclosure. Detailed Implementation

[0034] The present disclosure will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0035] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0036] While reinforcement learning methods are already in use in current technologies, mainstream approaches (such as PPO and GRPO) rely on the model's own sampling, which struggles to address the insufficient sampling of the first model in complex tasks, leading to training stagnation or getting trapped in local optima. Reinforcement learning depends on reward signals to guide model optimization. When the first model cannot generate high-quality samples, the obtained reward signals are generally low and lack discriminative power, resulting in a poor learning signal. Furthermore, directly using samples from the second model to train the first model can lead to distribution shift issues. The first model struggles to generate responses similar to the second model, resulting in incorrect gradient update directions. Simply copying the output of the second model can cause the first model to learn behaviors beyond its capabilities. In summary, a solution is needed to improve the training performance of the first model in reinforcement learning with limited computational resources.

[0037] Figure 1 This is a schematic flowchart illustrating the model training method provided in an embodiment of this disclosure. Figure 1 The method shown includes steps 101 to 105. This method can be executed by an electronic device such as a server (local server or cloud server).

[0038] Specifically, Figure 1 The method shown includes: Step 101: Inputting the input information into the first model to obtain the first output result.

[0039] Step 102: Input the input information into the second model to obtain the second output result containing the second output probability and the second output reward value; wherein, the number of parameters in the first model is less than the number of parameters in the second model.

[0040] Step 103: Calculate the predicted output probability of generating the second response sample using the first model.

[0041] Step 104: Calculate the ratio of the estimated output probability to the second output probability to obtain the importance sampling weight.

[0042] Step 105: Using the estimated output probability, the second output probability, and the second output reward value, perform reward transformation processing to generate a reward value transformation result that matches the first model.

[0043] Step 106: Input the importance sampling weights, reward value conversion results, and first output results into the first model and train it using a preset reinforcement learning optimization algorithm.

[0044] It's important to note that the first model here has fewer parameters than the second model. In other words, the first model is a pre-trained language model with a smaller parameter size and lower computational resource requirements, such as a Transformer-based model with fewer than 1 billion parameters. The second model is a pre-trained language model with a larger parameter size and superior performance, such as a model with over 10 billion parameters. Because of the difference in parameter count, the output response quality of the second model is generally higher than that of the first model. Therefore, the reinforcement learning training process of the first model can be optimized by synergistically utilizing the sampling capabilities of both models.

[0045] Specifically, the same input information (e.g., a sequence of text representing the task context or query request) is first input into the first model and the second model respectively. Then, the first model outputs a first output result. The second model outputs a second output result, including a second response sample, a second output probability, and a second output reward value.

[0046] After obtaining the second response sample, the estimated output probability of the second response sample generated by the first model needs to be calculated. In other words, it's necessary to assess whether the first model is capable of generating response samples of the same quality as the second model. The specific calculation process will be explained in the examples below.

[0047] Subsequently, the importance sampling weight and reward value transformation result are calculated using the estimated output probability and the second output probability. The importance sampling weight is calculated as the ratio of the estimated output probability of the second response sample generated by the first model to the second output probability of the second model generating that second response sample. It is used to correct the contribution of the second response sample to the training of the first model. In other words, the importance sampling weight quantifies the relative importance of high-quality second response samples generated by the second model under the first model's strategy. When the importance sampling weight is greater than or close to 1, the relative importance of the corresponding second response sample is relatively high; when the importance sampling weight is much less than 1, the relative importance of the corresponding second response sample is relatively low. By calculating the importance sampling weight, it is ensured that the first model can learn safely and effectively from these second response samples, while avoiding training instability caused by attempting to imitate behaviors beyond its capabilities. Specifically, when the first model is capable of generating samples of similar quality (a larger importance sampling weight w value), a higher weight is given to the sample; when the sample exceeds the capabilities of the first model (a smaller importance sampling weight w value), its influence is reduced, thereby achieving reasonable utilization of the second model's output results.

[0048] To better understand the working principle of the relative importance represented by importance sampling weights, the following two specific examples are used for comparison: Now, suppose the input information is the speech "wo shi shui", and the second model (high-performance speech-to-text model) generates the accurate text "I am who". Assume the probability of the second output "I" in the second response sample is 0.82. When calculating the probability that the first model (lightweight speech-to-text model) generates the same sample, two opposite scenarios may occur: First scenario: Suppose that the predicted output probability of the first model is "I" is 0.65.

[0049] The importance sampling weight is calculated as 0.65 / 0.82 ≈ 0.79. At this point, the weight is close to 1, indicating that although the first model has a small parameter scale, it already possesses the ability to accurately recognize this short speech. This means that the sample "I" has high relative importance for the training of the first model, and in subsequent training, this sample "I" should be assigned a larger weight. This will effectively guide the first model to improve its speech recognition accuracy.

[0050] The second scenario: Suppose that the predicted output probability of the first model is 0.16 for "I".

[0051] The importance sampling weight is calculated as 0.16 / 0.82 ≈ 0.20. At this point, the weight is much less than 1, indicating that the first model may have limitations due to its own capabilities (e.g., poor model parameters) or interference in the speech (e.g., speech in a noisy environment or with a heavy accent). The accurate recognition result of this sample "I" far exceeds the current capability range of the first model. Directly using this sample "I" for training would lead to overfitting and recognition errors. Therefore, this scheme assigns a smaller importance sampling weight to this sample "I," thus reducing its relative importance. During the reward value conversion process, the difference between the estimated output probability of the first model generating the second response sample and the second distribution probability of the second model generating the same second response sample is used as the basis for adjusting the reward signal. This difference reflects the gap in sample generation capabilities between the first and second models. In other words, the first model, due to its smaller parameter size, has limited ability to generate high-quality response samples, while the second model can generate high-quality response samples with a higher probability. The reward value conversion result is obtained by analyzing the probability difference and the second output reward value, adjusting the reward signal of the second model, and generating an actual reward value that matches the performance characteristics of the first model, thereby avoiding the first model blindly imitating the high-complexity behavior of the second model.

[0052] The first output result (including the first response sample, the first output reward value, and the first output probability), the importance sampling weights, and the reward value transformation result are all input into the first model to be trained. A reinforcement learning optimization algorithm (such as Proximal Policy Optimization, PPO) is used to optimize the parameters. Through iterative training, the performance of the first model is gradually improved.

[0053] As can be seen from the above-disclosed solutions, traditional reinforcement learning training of the first model relies on its own sampling capabilities. Due to the limited parameters of the first model, the quality of the generated samples is relatively low, especially in complex tasks (such as logical reasoning or long text generation), where it is prone to getting stuck in local optima or experiencing slow convergence, leading to low training efficiency and limited performance. This disclosed solution overcomes this limitation by introducing auxiliary sampling from a second model. The high-quality second output of the second model provides rich external guidance, while the mechanisms of importance sampling and reward value transformation ensure that these samples can seamlessly adapt to the optimization objective of the first model, avoiding the problem of distribution mismatch. This solution achieves a significant improvement in the performance of the first model while reducing computational resource requirements, demonstrating the advanced nature of efficient collaborative optimization.

[0054] In one or more embodiments of this disclosure, the first model and the second model run on different computer devices. By inputting the same input information into the first model and the second model respectively, a first output result and a second output result are obtained to facilitate subsequent reinforcement learning training optimization of the first model. In specific applications, the same input information (i.e., the task context or the input text sequence of the query) is input into the first model to generate a first output result. The first output result includes a first response sample output by the first model, the generation probability of the first response sample under the first model strategy (i.e., the first output probability), and the corresponding reward value calculated based on a preset reward function (i.e., the first output reward value). Simultaneously, the same input information is input into the second model to generate a second output result. The second output result includes a second response sample output by the second model, the generation probability of the second response sample under the second model strategy (i.e., the second output probability), and the corresponding reward value calculated based on the same preset reward function (i.e., the second output reward value). The first model and the second model run on different computer devices. For example, the first model can be deployed on a local computer (e.g., a desktop computer, a laptop computer, etc.) with limited computing resources, while the second model runs in a high-computing environment such as a cloud computing server.

[0055] The reason for calculating the first output of the first model and the second output of the second model separately is to fully utilize the advantages of both. The first model, with its smaller parameter size, is suitable for running on local devices, reducing computational power requirements and latency; however, its sampling capability is limited, resulting in insufficient sample quality and diversity. The second model, with its larger parameter size, runs on high-performance servers and can generate high-quality, diverse second response samples; however, its computational cost is high and it is not suitable for local deployment. By inputting the same input information into both models and obtaining their respective sampling results, resource allocation is optimized and data sources are complementary.

[0056] Based on the publicly available solutions described above, the limitations of the first model and hardware environment are overcome by co-sampling the first and second models under different computing environments. If relying solely on the output of the first model, the limited computing resources of the local computer result in low-quality first response samples, low training efficiency, and a tendency to get trapped in local optima. By deploying the second model on a high-performance cloud computing server and leveraging its powerful generation capabilities to obtain high-quality second output results, the shortcomings of the first model's output are compensated for. Although the second model is used, because the second and first models are distributed, they do not consume the local computing resources of the first model. In other words, during training, the first model runs locally, reducing the overall computing cost and latency of training, making it suitable for resource-constrained scenarios.

[0057] In one or more embodiments of this disclosure, such as Figure 2 This is a flowchart illustrating the output result conversion method provided in the embodiments of this disclosure. Figure 2 As shown, step 104, which involves using the estimated output probability, the second output probability, and the second output reward value to perform a reward conversion process and generate a reward value conversion result that matches the performance characteristics of the first model, includes: Step 1041: Calculating the logarithmic difference between the first logarithm of the estimated output probability and the second logarithm of the second output probability. Step 1042: Calculating the reward value conversion result that matches the performance characteristics of the first model using the logarithmic difference, the second output reward value, and a preset adjustment factor.

[0058] In practical applications, importance sampling weights and reward value conversion results are calculated by utilizing the estimated output probability of the first model and the second output probability of the second model to optimize the reinforcement learning training of the first model. Specifically, the importance sampling weights are obtained by calculating the ratio of the estimated output probability to the second output probability. The estimated output probability is the probability that the first model generates a second response sample under its current policy, and the second output probability is the probability that the second model generates the same second response sample under its current policy. By calculating the ratio of the two, a weighted weight (i.e., importance sampling weight) is generated to weight the reward value. This weighted weight quantifies the relative importance of the second response sample under the first model's policy, ensuring unbiased contribution from data of different policies. Specifically, the formula for calculating the importance sampling weights is: w = πsmall(a|s) / πlarge(a|s).

[0059] Here, πsmall(a|s) is the predicted output probability, i.e., the probability that the first model generates the second response sample 'a'. πlarge(a|s) is the second output probability, i.e., the probability that the second model generates the same second response sample 'a'. 's' represents the input information. It should be noted that the second response sample 'a' mentioned here refers to a sample generated by the second model but which needs to be evaluated from the perspective of the first model.

[0060] The importance sampling weight (that is, the ratio of the estimated output probability to the second output probability) is used as the weighting weight. w Used to correct for biases in the data source. When a second response sample... a When the probability of being generated by the second model is very small (πlarge(a|s) is small), but the probability of being generated by the first model is relatively large (πsmall(a|s) is large), the weighted weights... w The larger the value of the second response sample 'a', the more important it is for the first model to learn. Conversely, the smaller the value, the smaller the weight. This allows the expected reward of the second response sample collected under the second model policy to be losslessly transformed into the expected reward under the first model policy.

[0061] The generation of the reward value conversion result is based on the estimated output probability, the second output probability, and the second output reward value. By performing reward conversion processing, the reward signal of the second model is dynamically adjusted to generate an actual reward value that matches the first model. This process, through comparison of probability distributions and adaptation of reward signals, effectively eliminates the gap between the high performance of the second model and the resource limitations of the first model, enabling the second response samples of the second model to be reasonably applied to the training of the first model.

[0062] As mentioned earlier, the original reward value r(s,a) of the second model sample is generated based on its own capabilities, and its absolute value may not be suitable for the first model, which has weaker guidance capabilities. Therefore, a reward value transformation step is needed. The reward value transformation result r converted Calculate using the following formula: r converted =r(s,a)+λ⋅(logπsmall(a∣s)−logπlarge(a∣s)).

[0063] Where r(s,a) is the second output reward value, i.e., the original reward value of the second model sample. πsmall(a|s) and πlarge(a|s) are the estimated output probability and the second output probability, respectively. λ is a preset adjustment factor (e.g., set to a small positive number, such as 0.1) used to control the strength of the conversion term. In this formula, the first part r(s,a) retains the absolute quality information of the original sample. The second part λ⋅(logπsmall−logπlarge) is essentially the difference in log probabilities between the two policies for the same response sample a, reflecting the difference in the behavioral capabilities of the first and second models when generating this response sample.

[0064] When the probability of the first model generating the sample is greater than that of the second model (πsmall > πlarge), this term is positive, which enhances the original reward and encourages the first model to maintain its existing good performance. When the probability of the first model generating the sample is less than that of the second model (πsmall < πlarge), this term is negative, which weakens the original reward. This adjustment prevents the first model from becoming unstable due to excessively high targets. The adjustment factor λ effectively balances the weight between the original reward signal and the difference signal. Through this transformation, the reward value r... converted Instead of being an absolute score detached from the actual performance of the first model, it is a relative and more learnable guiding signal that dynamically adapts to the current performance characteristics of the first model, enabling it to learn more effectively and stably from the high-quality samples (i.e., the second response samples) provided by the second model. It should be noted that, in addition to using the above method to transform the second output reward value, it can also be achieved through algorithms such as setting multiple transformation factors.

[0065] Based on the aforementioned publicly available solution, it is evident that by calculating importance sampling weights and the reward value transformation results, the efficiency and effectiveness of the first model's reinforcement learning training are significantly improved. Traditional reinforcement learning methods rely on the sampling of the first model itself. Due to limited computational resources, the generated samples are of low quality, and the training process is prone to failure due to distribution mismatch or unsuitable reward signals. This publicly available solution addresses the bias problem introduced by heterogeneous policy data by introducing importance sampling weights and using probability ratios to weight the second output reward value of the second model, ensuring unbiased training. Simultaneously, the reward value transformation results dynamically adapt to the performance characteristics of the first model, avoiding excessively high expectations caused by directly using the second model's reward signal, thus optimizing training stability.

[0066] In one or more embodiments of this disclosure, such as Figure 3 This is a schematic diagram of the process for generating the first output result provided in an embodiment of this disclosure. Figure 3 As shown, step 101, which involves inputting the same input information into the first model to obtain a first output result containing the estimated output probability and a first output reward value, includes: Step 1011: Performing an output operation on the same input information through the first model to generate a first response sample. Step 1012: Determining the generation probability of the first response sample under the first model strategy as the first output probability. And, Step 1013: Calculating the first output reward value corresponding to the first response sample based on a preset reward function.

[0067] It should be noted that the first output probability is used to characterize the policy confidence of the first model over the first response sample it generates. In other words, it is the probability value of the first model generating that specific first response sample based on its parameters after the input information is fed into the first model. In an optional embodiment, the specific calculation process is as follows: the first output probability is obtained by performing an autoregressive probability calculation on the first response sample sequence, that is, calculating the product of the generation probabilities of each token in the sequence given the preceding tokens. Since the parameter size of the first model is limited, its first output probability distribution may have a high entropy value, indicating that the first model has significant limitations in its exploration ability on complex tasks. This is the fundamental reason why a second model is needed to assist in sampling.

[0068] In practical applications, input information is fed into a first model, and a first output result is obtained to support the reinforcement learning training and optimization of the first model. Specifically, the same input information (i.e., the task context or the input text sequence of the query) is input into the first model, triggering its sampling operation based on the current policy. This first output result includes a first response sample generated by the first model, which is the direct output of the first model to the input information, reflecting its current policy's generation capability. Furthermore, the first output result includes a first output probability, i.e., the probability of generating the first response sample under the first model's policy, reflecting the first model's preference for that first response sample. In addition, the response sample is evaluated based on a preset reward function, and a first output reward value is calculated. This reward value quantifies the performance quality of the first response sample in the target task, such as based on task accuracy, language fluency, or user feedback.

[0069] The sampling process involves the policy network of the first model, which can employ a lightweight Transformer architecture, containing a decoder structure of 12-24 layers, each layer equipped with a small number of attention heads (e.g., 8-16) and a simplified feedforward neural network, with the number of parameters ranging from hundreds of millions to billions. To adapt to resource-constrained environments, the first model can be optimized using techniques such as knowledge distillation, parameter sharing, or quantization. Its positional encoding mechanism can employ simpler absolute positional encoding, and the normalization layer can be simplified to a standard LayerNorm. This streamlined design allows the first model to significantly reduce computational complexity and memory consumption while maintaining basic language understanding capabilities, making it particularly suitable for deployment on edge devices or resource-constrained local environments. Based on the context of the input information, a series of candidate response samples are generated, and the optimal or diversified output is selected through a probability distribution. The first output probability is output by the softmax layer of the first model, representing the generation probability of each token or sequence in the first response sample. The preset reward function is a configurable evaluation mechanism, which may include manually labeled scores, automated metrics (such as BLEU scores), or the output of an external reward model, depending on the task requirements. The entire process is completed in the local computing environment of the first model, without requiring high computing resources.

[0070] As can be seen from the above-disclosed solution, the first output result is obtained through local sampling operations of the first model, serving as the foundational data for subsequent reinforcement learning training. Due to the lightweight nature of the first model, it adapts to the resource limitations of local computing environments, requiring minimal computing power and ensuring the real-time nature and accessibility of training. This disclosure reduces computing costs and latency through autonomous sampling of the first model, making it particularly suitable for edge devices or low-cost scenarios.

[0071] In one or more embodiments of this disclosure, Figure 4 This is a schematic diagram of the process for generating a second output result provided in an embodiment of this disclosure. Figure 4 As shown, step 102, which involves inputting the same input information into the second model to obtain a second output result containing a second response sample, a second output probability, and a second output reward value, includes: Step 1021: Performing an output operation on the same input information through the second model to generate a second response sample. Step 1022: Determining the generation probability of the second response sample under the second model strategy as the second output probability. And, Step 1023: Calculating the second output reward value corresponding to the second response sample based on the same preset reward function.

[0072] It should be noted that the second output probability represents the policy confidence of the second model in the second response sample it generates. In other words, it is the probability value of the second model generating that specific second response sample based on its model parameters after the input information is fed into the second model. In an optional embodiment, the specific calculation process is as follows: the second output probability is the product of the conditional probabilities of each token in the second response sample by the softmax output layer of the second model. Since the second model has a large parameter scale, its second output probability may exhibit a low entropy distribution, that is, assigning a high probability to high-quality samples. This can provide a reliable benchmark reference for subsequent importance sampling and reward conversion. Therefore, the second model can be used to assist in training the first model.

[0073] In practical applications, a second model is used to generate a second output result for the same input information to enhance the reinforcement learning training effect of the first model. Specifically, the same input information as the first model (i.e., the task context or the input text sequence of the query) is input into the second model, triggering its sampling operation based on the second model's current strategy. The second output result includes a second response sample generated by the second model. Due to the powerful generation capability and extensive training data of the second model, this response sample possesses higher semantic accuracy, logical coherence, and task relevance. Furthermore, the second output result includes a second output probability, i.e., the probability of generating the second response sample under the second model's strategy, reflecting the second model's preference for that sample. Additionally, the second response sample is evaluated based on the same preset reward function as the first model, calculating a second output reward value. This reward value quantifies the performance quality of the response sample, for example, through metrics such as task accuracy, language fluency, or external feedback.

[0074] The sampling operation of the second model involves its complex neural network structure and optimized policy network. This second model can employ a deep Transformer-based architecture, containing a decoder structure with dozens of layers (e.g., 48-96 layers), each layer equipped with a multi-head self-attention mechanism (e.g., 32-64 attention heads) and a feedforward neural network, with parameters ranging from tens of billions to hundreds of billions. This deep structure enables the second model to capture complex language patterns and long-range dependencies. Its positional encoding mechanisms (e.g., Rotational Positional Encoding RoPE) and advanced normalization techniques (e.g., RMSnorm) further enhance the model's ability to handle complex tasks. Diverse and high-quality candidate response samples are generated based on the context of the input information. The second output probability, output by the softmax layer of the second model, represents the generation probability of each token or sequence. A pre-defined reward function serves as a unified evaluation criterion, ensuring the comparability of the reward values ​​of the first and second models. This can include manually labeled scores, automated metrics, or the output of the reward model, depending on the task requirements. Because the second model runs in a high-computing environment such as cloud computing servers, its sampling process can fully utilize powerful computing resources and parallel processing capabilities to generate a sufficient quantity and high-quality data.

[0075] It should be noted that the main purpose of using the second model to output a second result for the same input information is to provide high-quality external guidance data for the reinforcement learning training of the first model. Due to its smaller parameter size and operation on a local computer, the first model is limited by computational resources and training data, resulting in its own sampled first output results often being of lower quality and lacking diversity and coverage of complex tasks. In contrast, the second model's second output results, due to its powerful generative capabilities, can generate more diverse, accurate, and task-relevant response samples, thus compensating for the deficiencies of the first model's output.

[0076] During training, a second model can be used to compensate for the shortcomings of the first model. First, the second response samples of the second output provide the first model with rich learning objectives, expanding its exploration space. This is particularly evident in complex tasks such as logical reasoning, long text generation, or multi-turn dialogues, significantly improving the performance of the first model. Second, the second output probabilities provide crucial probabilistic information for subsequent importance sampling and reward value transformation, facilitating adjustments to the contribution of the second model's samples and ensuring compatibility with the first model's strategy. Furthermore, the second output reward value serves as a high-standard reference, guiding the first model to optimize its policy parameters by comparing it with the first output reward value, thus accelerating training convergence. In summary, the role of the second output is to eliminate the capability gap between the first and second models, providing efficient external supervision for the first model and overcoming the training bottleneck caused by its limited local computing resources.

[0077] Based on the aforementioned publicly available scheme, it is evident that generating high-quality second output results through the output operations of the second model provides crucial support for the reinforcement learning training of the first model. This significantly improves the training quality and efficiency of the first model. Furthermore, leveraging the high computing power of the second model on the cloud computing server, a sufficient number of high-quality second response samples are generated, providing the first model with diverse and high-value external learning data.

[0078] In one or more embodiments of this disclosure, such as Figure 5 This is a flowchart illustrating a reinforcement learning method provided in an embodiment of this disclosure. Figure 5 As shown, step 105 inputs the importance sampling weights, the transformed reward value, and the first output result into the first model for training using a preset reinforcement learning optimization algorithm, including: Step 1051: Inputting the first output result, the importance sampling weights, and the transformed reward value result into the objective function to train the first model. Step 1052: Calculating the expected reward of the objective function based on the reinforcement learning optimization algorithm, and determining the parameters of the first model corresponding to maximizing the expected reward. Specifically, step 1052, calculating the expected reward of the objective function based on the reinforcement learning optimization algorithm, and determining the parameters of the first model corresponding to maximizing the expected reward, includes: Step 10521: Constructing a hybrid experience replay pool by combining the first response sample and the second response sample processed by the importance sampling weights and reward value transformation result. Step 10522: Calculating the objective function of the first model based on the samples in the hybrid experience replay pool, where the objective function is the expected reward under the first model's policy. Step 10523: Calculating the advantage function value of the samples in the hybrid experience replay pool to estimate the gradient direction of the expected reward. Step 10524: Calculate the policy gradient in the gradient direction using the importance sampling weights, the advantage function value, and the log probability gradient of the first output probability. Step 10525: Update the parameters of the first model using the optimizer based on the policy gradient to maximize the expected reward.

[0079] During reinforcement learning training, the first output result generated by the first model (including the first response sample, the first output probability, and the first output reward value), the importance sampling weight (based on the ratio of the estimated output probability and the second output probability), and the transformed reward value (the actual reward value adapted to the performance of the first model) are all input into the objective function to train the first model.

[0080] For example, the objective function formula for the first model before optimization is: .

[0081] The objective function formula, after inputting the first output result, the importance sampling weight, and the transformed reward value into the objective function, is as follows: .

[0082] in, This represents the new objective function (expected reward) for the first model. This represents the parameters of the first model (i.e., the quantities that need to be optimized during training). G represents the total number of samples collected. G There are 10 samples. q represents the input information. This represents the output sequence (response) corresponding to the i-th input. This represents the t-th token in the i-th output sequence. This represents the historical context of the first t-1 tokens in the i-th output. This represents the reward value of the first response sample generated by the first model at position t. The reward value at position t for the second response sample generated by the second model.

[0083] The objective function quantifies the expected reward of the first model's policy, and its parameters are calculated to maximize this reward using a reinforcement learning optimization algorithm (such as Proximal Policy Optimization, PPO). Through this process, the parameters of the first model are gradually adjusted to improve its performance on the target task. Specifically, in the reinforcement learning process, the first response samples and the second output results (processed with importance sampling weights and reward value transformations) are first combined to construct a hybrid experience replay pool. This replay pool integrates the first response samples from the first model itself and the second response samples from the second model's output, along with the importance sampling weights and the second output reward value transformation results. Based on the samples in the hybrid experience replay pool, the objective function of the first model is calculated, which takes the form of the expected reward under the first model's policy. Next, the dominance function value of the samples is calculated to estimate the gradient direction of the expected reward. Using the importance sampling weights, the dominance function value, and the log probability gradient of the first output probability, the policy gradient is calculated. The formula is: Policy gradient = w × A × ∇_θlogπ_small(a|s), where w is the importance sampling weight, A is the dominance function value, and θ is the model parameters. Finally, based on the calculated policy gradient, the parameters of the first model are updated by an optimizer (such as Adam), gradually approaching the optimal policy that maximizes expected reward.

[0084] It's important to note that when calculating importance sampling weights, the estimated output probability (i.e., the probability of the first model generating the second response sample) is used instead of the first output probability (i.e., the probability of the first model generating the first response sample). This is because, in the reinforcement learning framework, importance sampling is used to correct for differences in data distribution between different model policies. Specifically, it unbiasedly transforms the second response sample collected under the second model policy into an expected estimate under the first model policy. In this scheme, the second model, as the behavioral policy, can provide high-quality samples, while the first model, as the target policy, needs to be optimized. Therefore, the correction weights must be based on the probability ratio of the same response sample (i.e., the high-quality second response sample) under the two policies. The first output probability corresponds to the first response sample generated by the first model and is not directly related to the second response sample. The first response sample may be the same as the second response sample, but it may also be different. In this scheme, the estimated output probability is introduced as the probability of the first model generating the second response sample, ensuring the consistency of the object of importance sampling weight calculation.

[0085] In the aforementioned publicly available scheme, the importance sampling weights are adjusted by calculating the ratio of the estimated output probability to the second output probability to ensure the unbiasedness of the second response sample in the optimization objective of the first model. For example, if the sample generated by the second model has a low probability under the first model's policy, the weighting value will correspondingly reduce its reward impact, preventing the first model from being misled by behaviors that are difficult to achieve. The reward value transformation result is generated by analyzing the probability distribution differences, combining the second output reward value and the adjustment factor, to produce an actual reward value that fits the capabilities of the first model, avoiding overfitting caused by the direct transfer of the second model's reward signal. In the hybrid experience replay pool, the above results are combined with the first output result, enriching the training data source. At the same time, through the optimization of the advantage function and policy gradient, the performance bottleneck caused by insufficient sample quality and distribution mismatch in traditional methods is eliminated.

[0086] In one or more embodiments of this disclosure, such as Figure 6 This is a schematic diagram of the iterative optimization process provided in the embodiments of this disclosure. For example... Figure 6 As shown, the specific steps include: Step 601: Perform iterative reinforcement learning training on the first model. Step 602: Terminate the iteration when the average reward value of the first model changes less than the reward threshold, or when the number of consecutive iterations N is not less than the number threshold.

[0087] In practical applications, the first model undergoes multiple iterations of reinforcement learning training. Each iteration adjusts the neural network parameters of the first model based on previous training data and policy updates. The training process utilizes the first output from the hybrid experience replay pool, importance sampling weights, and reward value transformation results. The expected reward of the objective function is calculated using a reinforcement learning optimization algorithm (such as Proximal Policy Optimization, PPO), and the parameters are updated based on the policy gradient and the dominance function. After each iteration, the performance metrics of the first model are evaluated, particularly the average reward value (the average evaluation value of all response samples based on a preset reward function), to determine whether the training has reached convergence.

[0088] The termination criteria for iterative training are based on two main standards. First, when the average reward value of the first model changes by less than a preset reward threshold, it indicates that the model performance has stabilized and the training effect no longer significantly improves. For example, if the change in the average reward value (e.g., the difference between adjacent iterations) is less than 0.01 in several consecutive iterations, the model is considered close to its optimal state, and training is terminated. Second, if the number of consecutive iterations N is not less than a preset threshold, it indicates that the training has reached its maximum number of attempts, and iteration is terminated even if the reward has not fully converged to avoid overcomputation. For example, the threshold can be set to 100 rounds or dynamically adjusted according to task complexity. Furthermore, the termination criteria can be extended to combinations of other metrics, such as the objective function reaching its maximum value (i.e., the expected reward no longer increases) or performance metrics on the validation set (e.g., accuracy or perplexity) reaching a plateau. These conditions are implemented through real-time monitoring and evaluation mechanisms to ensure that the training process is both efficient and reliable.

[0089] To better understand the iterative training process, a specific example is provided below to illustrate the detailed process, including the following steps: First, batch data is extracted from the hybrid experience replay pool, the objective function under the first model policy is calculated, and the parameters are updated using the advantage function and policy gradient. Second, the average reward value after each iteration is recorded and compared with the result of the previous iteration to calculate the magnitude of change. If the magnitude of change is consistently lower than the reward threshold (e.g., 0.01), a termination signal is triggered. If this condition is not met, it is checked whether the number of consecutive iterations N exceeds the threshold (e.g., 100 rounds). Furthermore, auxiliary termination conditions can be introduced, such as when the gradient norm of the objective function is lower than a preset threshold, indicating that the optimization has stabilized; or when the loss function value on the validation set does not decrease significantly for 5 consecutive iterations (e.g., the change is less than 0.005), an early stopping mechanism is initiated. The above conditions can be flexibly adjusted through configuration parameters to adapt to the complexity and resource constraints of different tasks.

[0090] Based on the publicly available solutions, it is evident that by establishing explicit iteration termination conditions, the efficiency and stability of reinforcement learning training for the first model are significantly improved. By setting termination conditions such as the average reward value changing less than a reward threshold (e.g., 0.01) or the number of consecutive iterations N reaching a threshold (e.g., 100 rounds), combined with optional early stopping mechanisms and objective function optimization, the timely termination of the training process is ensured, avoiding resource waste and performance degradation.

[0091] In one or more embodiments of this disclosure, calculating the estimated output probability of generating the second response sample using the first model includes: constructing a target sequence using the second response sample; and calculating the estimated output probability by sequentially calculating multiple tokens in the target sequence using the first model.

[0092] In one alternative approach, the specific calculation process for the estimated output probability is as follows: For each token in the target sequence, based on the input information and the sequence preceding that token, calculate the conditional probability of the first model generating that token; multiply the conditional probabilities of all tokens to obtain the estimated output probability.

[0093] In practical applications, the second response sample is converted into a token sequence format suitable for processing. Specifically, the system uses a tokenizer compatible with both the first and second models to decompose the second response sample into a series of ordered tokens, forming the target sequence. For example, if the second response sample is "1+1=2, this is a basic addition operation.", after tokenization, the target sequence might be ["1","+","1","=","2",",","this","is","one","basic","of","addition","operation","calculation","。"].

[0094] It should be noted that the target sequence is used only for probability evaluation and is not used as a new input to the first model. In conventional inference, the model generates a new output based on the input; however, in this step, the first model keeps the original input unchanged and only evaluates the probability of generating the target sequence (that is, predicts the output probability).

[0095] For each token in the target sequence, the system calculates the conditional probability of the first model generating that token based on the original input information and the sequence preceding that token. Multiplying the conditional probabilities of all tokens in the target sequence yields the joint probability of the first model generating the entire second response sample, i.e., the estimated output probability.

[0096] Based on the aforementioned publicly available scheme, the capability gap between the first and second models is precisely quantified by calculating the generation probability of the second response sample by the first model. The estimated output probability directly reflects the likelihood of the second response sample being achieved by the first model. When the estimated output probability is low, it indicates that the sample exceeds the current capability range of the first model, and its weight in training will be automatically reduced to prevent the first model from blindly pursuing behaviors that cannot be achieved, thereby maintaining the stability of the training process.

[0097] To facilitate understanding, the implementation process of the technical solution of this application will be described below through specific embodiments.

[0098] Step 1: Initialization and base model preparation.

[0099] First, we prepare a large language model with a small parameter base for training (hereinafter referred to as the "first model") and a more powerful language model with a larger parameter base (hereinafter referred to as the "second model"). The first model is the target of this reinforcement learning training, while the second model serves as an auxiliary tool to provide high-quality sampling results. For example, the first model can be an open-source model with a small number of parameters, and the second model can be DeepSeek-V3 or other high-performance models.

[0100] Step 2: Strengthen the construction of the learning environment and design the objective function.

[0101] Construct a reinforcement learning training environment and design an objective function. The objective function is used to evaluate the quality of the text generated by the model. For example, it can be designed based on human feedback (such as preference annotation), task-specific evaluation metrics (such as accuracy, fluency, and relevance), or in combination with other automatic evaluation methods. The goal of the objective function is to quantify the degree of match between the model output and the desired objective.

[0102] Step 3: Second model-assisted sampling.

[0103] In each training iteration of reinforcement learning, in addition to the first model sampling and generating text, a second model is used for additional sampling. Specifically, for a given input (e.g., a question or a hint), the second model generates a series of high-quality response samples. These samples have higher quality, stronger logicality, and more accurate semantics due to the powerful generation capabilities of the second model.

[0104] Step 4: Objective function transformation and importance sampling.

[0105] The goal of reinforcement learning can be to maximize the expected reward of the trained model (i.e., the first model). However, since the sampling results of the second model are introduced, these results are based on the second model's policy (i.e., generation probability) and cannot be directly used to optimize the first model. Therefore, it is necessary to transform the expected reward of the second output of the second model (i.e., the second output reward value) into the expected reward form under the first model's policy through importance sampling.

[0106] The specific conversion process is as follows: Probability transformation (i.e., importance sampling weighting): For each sample generated by the second model, its generation probability under the second model policy (second output probability) is different from its generation probability under the first model policy (estimated output probability). The reward for the second model sample is weighted using the importance sampling ratio, i.e., estimated output probability / second output probability. This ensures that even if a sample is generated by the second model, its contribution to the objective function of the first model can be correctly measured.

[0107] Reward Value Conversion: In addition to probability conversion, a specific formula was derived to convert the actual reward value of the second model samples. This is because the second model's perception and evaluation of rewards may differ from the first model when generating samples. This conversion formula adjusts the original reward value of the second model samples to a more meaningful reward signal from the first model's perspective. This formula ensures that even if the second model generates high-quality samples, its reward signal can effectively guide the first model's learning, rather than simply "copying" the second model's behavior. The conversion formula is as follows: in, This represents the new objective function (expected reward) for the first model. This represents the parameters of the first model (i.e., the quantities that need to be optimized during training). G represents the total number of samples collected. G There are 10 samples. q represents the input information. This represents the output sequence (response) corresponding to the i-th input. This represents the t-th token in the i-th output sequence. This represents the historical context of the first t-1 tokens in the i-th output. This represents the reward value of the sample generated by the first model at position t. The reward value of the sample generated by the second model at position t.

[0108] Step 5: Strengthen the training of the first model.

[0109] All samples (including samples from the first model itself and those from the second model's auxiliary samples) after importance sampling and reward value transformation, along with their corresponding reward signals, are used together to optimize the parameters of the first model. This can be achieved using standard reinforcement learning optimization algorithms (such as PPO, GRPO, or other policy gradient methods). In this way, the first model not only learns from its own exploration but also learns efficiently from the high-quality "demonstrations" provided by the second model, thereby rapidly improving its performance on complex tasks with limited computational resources.

[0110] Step 6: Iterative Training. Repeat steps 3 through 5 to continuously iterate the training of the first model until the expected performance metrics are achieved or convergence is reached.

[0111] Based on any of the above embodiments, this disclosure also provides a model training apparatus. Figure 7 This is a schematic block diagram of a model training apparatus according to one embodiment of the present disclosure. Figure 7 As shown, the model training device includes: The first input module 71 is used to input input information into the first model to obtain the first output result.

[0112] The second input module 72 is used to input input information into the second model to obtain a second output result containing a second response sample, a second output probability, and a second output reward value; wherein the number of parameters in the first model is less than the number of parameters in the second model.

[0113] The calculation module 73 is used to calculate the estimated output probability of generating the second response sample using the first model.

[0114] The calculation module 73 is used to calculate the ratio of the estimated output probability and the second output probability to obtain the importance sampling weight used to weight the reward value.

[0115] The reward value conversion module 74 is used to perform reward conversion processing using the estimated output probability, the second output probability, and the second output reward value to generate a reward value conversion result that matches the first model.

[0116] Training module 75 is used to input the importance sampling weights, reward value conversion results and the first output results into the first model for training using a preset reinforcement learning optimization algorithm.

[0117] The reward value conversion module 74 is used to calculate the logarithmic difference between the first logarithm of the estimated output probability and the second logarithm of the second output probability; and to calculate the reward value conversion result that matches the performance characteristics of the first model using the logarithmic difference, the second output reward value and the preset adjustment factor.

[0118] The first input module 71 is used to perform output operations on the same input information through the first model to generate a first response sample; determine the generation probability of the first response sample under the first model strategy as the first output probability; and calculate the first output reward value corresponding to the first response sample based on the preset reward function.

[0119] The second input module 72 is used to perform output operations on the same input information through the second model to generate a second response sample; determine the generation probability of the second response sample under the second model strategy as the second output probability; and calculate the second output reward value corresponding to the second response sample based on the same preset reward function.

[0120] The calculation module 73 is used to construct the target sequence using the second response sample; and to obtain the estimated output probability by calculating the order of multiple tokens in the target sequence using the first model.

[0121] The training module 75 is used to input the first output result, the importance sampling weight, and the transformed reward value into the objective function to train the first model; it calculates the expected reward of the objective function based on the reinforcement learning optimization algorithm and determines the parameters of the first model corresponding to maximizing the expected reward.

[0122] Training module 75 is used to construct a hybrid experience replay pool by combining the first response samples with the second response samples processed by importance sampling weights and reward value transformation. Based on the samples in the hybrid experience replay pool, the objective function of the first model is calculated, which is the expected reward under the first model policy. The advantage function value of the samples in the hybrid experience replay pool is calculated to estimate the gradient direction of the expected reward. The policy gradient in the gradient direction is calculated using the importance sampling weights, the advantage function value, and the log probability gradient of the first output probability. According to the policy gradient, the parameters of the first model are updated by the optimizer to maximize the expected reward.

[0123] Training module 75 is used to perform iterative reinforcement learning training on the first model; the iteration is terminated when the change in the average reward value of the first model is less than the reward threshold, or when the number of consecutive iterations N is not less than the number threshold.

[0124] The first model and the second model are deployed on different computer devices.

[0125] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0126] The execution subject of the model training method in the specific embodiments of this disclosure can be an electronic device such as a server (including a local server or a cloud server).

[0127] Therefore, based on any of the above embodiments, this disclosure also provides an electronic device that can execute the model training method or knowledge sharing library construction method of any of the embodiments described above.

[0128] Figure 8 This is a schematic block diagram of an electronic device according to one embodiment of the present disclosure.

[0129] The hardware architecture of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and overall design constraints. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0130] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one connection line is used in this diagram, but this does not imply that there is only one bus or only one type of bus.

[0131] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0132] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0133] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0134] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0138] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment / mode or example, which are included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0139] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0140] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A model training method, characterized in that, The method includes: The input information is fed into the first model to obtain the first output result; The input information is input into the second model to obtain a second output result containing a second response sample, a second output probability, and a second output reward value; wherein the number of parameters in the first model is less than the number of parameters in the second model; Calculate the estimated output probability of generating the second response sample using the first model; The importance sampling weight is obtained by calculating the ratio of the estimated output probability to the second output probability. Using the estimated output probability, the second output probability, and the second output reward value, a reward conversion process is performed to generate a reward value conversion result that matches the first model; The importance sampling weights, the reward value conversion result, and the first output result are input into the first model for training using a preset reinforcement learning optimization algorithm.

2. The model training method according to claim 1, characterized in that, The step of performing reward transformation processing using the estimated output probability, the second output probability, and the second output reward value to generate a reward value transformation result matching the first model includes: Calculate the logarithmic difference between the first logarithm of the estimated output probability and the second logarithm of the second output probability; The logarithmic difference, the second output reward value, and the preset adjustment factor are used to calculate the reward value conversion result that matches the performance characteristics of the first model.

3. The model training method according to claim 1, characterized in that, The step of inputting the input information into the first model to obtain the first output result includes: The first model performs an output operation on the input information to generate a first response sample. The generation probability of the first response sample under the first model strategy is determined as the first output probability; and, The first output reward value corresponding to the first response sample calculated based on the preset reward function.

4. The model training method according to claim 3, characterized in that, The step of inputting the input information into the second model to obtain a second output result containing a second response sample, a second output probability, and a second output reward value includes: The second model performs an output operation on the input information to generate the second response sample. The generation probability of the second response sample under the second model strategy is determined as the second output probability; and, The second output reward value corresponding to the second response sample calculated based on the preset reward function.

5. The model training method according to claim 4, characterized in that, The calculation of the estimated output probability of the second response sample generated using the first model includes: Construct the target sequence using the second response sample; The estimated output probability is obtained by calculating the order of multiple tokens in the target sequence using the first model.

6. The model training method according to claim 3, characterized in that, The step of inputting the importance sampling weights, the reward value conversion result, and the first output result into the first model and training it using a preset reinforcement learning optimization algorithm includes: The first output result, the importance sampling weight, and the transformed reward value are input together into the objective function to train the first model; The expected reward of the objective function is calculated based on the reinforcement learning optimization algorithm, and the parameters of the first model corresponding to maximizing the expected reward are determined.

7. The model training method according to claim 6, characterized in that, The step of calculating the expected reward of the objective function based on the reinforcement learning optimization algorithm and determining the parameters of the first model corresponding to maximizing the expected reward includes: The first response sample and the second response sample, after being processed by the importance sampling weight and the reward value transformation result, are jointly constructed into a hybrid experience replay pool; Based on the samples in the hybrid experience replay pool, the objective function of the first model is calculated, and the objective function is the expected reward under the first model strategy. Calculate the advantage function values ​​of the samples in the hybrid experience replay pool to estimate the gradient direction of the expected reward; The policy gradient in the gradient direction is calculated using the importance sampling weight, the advantage function value, and the log probability gradient of the first output probability. Based on the policy gradient, the parameters of the first model are updated by the optimizer to maximize the expected reward.

8. The model training method according to claim 1, characterized in that, Also includes: The first model is trained using iterative reinforcement learning. The iteration is terminated when the average reward value of the first model changes by less than the reward threshold, or when the number of consecutive iterations N is not less than the number threshold.

9. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the method of any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.

Images