Model reinforcement learning method, device and equipment

Abstract

The embodiment of the invention provides a model reinforcement learning method, device and equipment. The scheme comprises the following steps: in a sampling stage, using an inference engine and adopting a to-be-trained target model to generate an output sequence for an input sequence under a first strategy parameter, recording a first probability value of each lexical element in the output sequence, and calculating a dominant value of each lexical element, in a training stage, after a training engine is used for forward propagation to obtain a second probability value of each lexical element generated by a target model under a first strategy parameter, a first ratio of the second probability value to the first probability value of each lexical element can be calculated, and then the lexical elements with the first ratios within a preset numerical range are screened out to participate in calculation of a target function; and optimizing the target function to update the parameters of the target model.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the technical field of model training, in particular to a model reinforcement learning method. One or more embodiments of the present specification also relate to a model reinforcement learning device and a computing device. BACKGROUND

[0002] Reinforcement learning, especially policy gradient methods, has become a key technology for training large models. In the on-policy reinforcement learning framework, the behavior policy used for interacting with the environment and sampling should be consistent with the target policy to be optimized. This theoretical premise is the basis for ensuring the unbiasedness of policy gradient estimation and the stability of training.

[0003] To improve training efficiency, modern reinforcement learning systems generally use a hybrid architecture in practice, which uses an independent inference engine and a training engine to work together. The inference engine is responsible for trajectory sampling, and uses its highly optimized computing kernel and cache strategy to achieve fast and low-latency data generation. The training engine is responsible for performing accurate backpropagation and model parameter updates based on the collected trajectory data. Although this division of labor and cooperation mode significantly improves the overall training efficiency, it also introduces an inherent technical contradiction: the inference engine and the training engine differ in the implementation of underlying operators. This difference leads to a slight numerical bias in the forward propagation calculation results of the inference engine and the training engine for the same initial model parameters and input sequence, i.e., the "training-inference precision difference", which is specifically manifested in that the probability values calculated by the two engines for the same token are not completely consistent. During the training iteration process, this bias will continue to accumulate and gradually magnify with the update of parameters, which essentially violates the training condition of the on-policy, causing the policy used for sampling trajectories to deviate from the policy receiving gradient updates in the numerical level. As a result, the policy gradient calculated based on the biased probability values will have errors. With the continuous expansion of the training-inference precision difference, the gradient error will continue to accumulate, eventually leading to the instability of training dynamics, which is typically characterized by the dramatic fluctuation or collapse of reward values and the abnormal explosion of gradients, causing the entire training process to fail.

[0004] Therefore, those skilled in the art are faced with the technical problem to be solved urgently: how to effectively overcome the training instability problem caused by the underlying difference between the inference and training engines when using an efficient hybrid architecture, in order to improve the robustness and success rate of reinforcement learning training. SUMMARY

[0005] Therefore, one or more embodiments of the present specification provide a model reinforcement learning method, device and equipment to solve the training instability problem caused by the difference between the inference and training engine underlying in the mixed architecture in the existing model reinforcement learning method, and improve the robustness and success rate of reinforcement learning training.

[0006] According to a first aspect of one or more embodiments of the present specification, a model reinforcement learning method is provided, comprising:

[0007] obtaining, using an inference engine, an output sequence generated by a target model under a first policy parameter for an input sequence and a first probability value of each token in the output sequence;

[0008] determining an advantage value of the each token based on the output sequence;

[0009] obtaining, using a training engine, a second probability value of the each token generated by the target model under the first policy parameter;

[0010] calculating a first ratio of the second probability value and the first probability value of the each token;

[0011] determining a value of an objective function based on the first ratio and the advantage value of the each token; wherein the each token within a preset numerical range participates in the calculation of the objective function;

[0012] adjusting parameters of the target model to obtain the target model under a second policy parameter, aiming to maximize the value of the objective function.

[0013] According to a second aspect of one or more embodiments of the present specification, a model reinforcement learning device is provided, comprising:

[0014] a sampling module configured to obtain, using an inference engine, an output sequence generated by a target model under a first policy parameter for an input sequence and a first probability value of each token in the output sequence;

[0015] an advantage calculation module configured to determine an advantage value of the each token based on the output sequence;

[0016] a forward propagation module configured to obtain, using a training engine, a second probability value of the each token generated by the target model under the first policy parameter;

[0017] a first ratio calculation module configured to calculate a first ratio of the second probability value and the first probability value of the each token;

[0018] a target function calculation module, configured to determine a value of a target function based on the first ratio of each token and the advantage value, wherein the tokens with the first ratio within a preset numerical range participate in the calculation of the target function;

[0019] a parameter updating module, configured to adjust the parameters of the target model to obtain the target model under the second policy parameters.

[0020] According to a third aspect of one or more embodiments of the present specification, a computing device is provided, comprising a memory, a processor, and computer instructions stored on the memory and executable on the processor, wherein the processor executes the computer instructions to implement the steps of the model reinforcement learning method.

[0021] At least the following beneficial effects can be achieved by one embodiment of the present specification: by using the inference engine in the sampling stage to generate an output sequence for an input sequence using the target model to be trained under the first policy parameters and record the first probability value of each token in the output sequence, and calculate the advantage value of each token, after the second probability value of each token generated by the target model under the first policy parameters is obtained by using the training engine for forward propagation in the training stage, the first ratio of the second probability value and the first probability value of each token can be calculated, and then the tokens with the first ratio within a preset numerical range are screened out to participate in the calculation of the target function, and then the target function is optimized to update the parameters of the target model. Therefore, by identifying the tokens with large differences in inference results between the inference engine and the training engine (i.e., tokens with the first ratio not within the preset numerical range) as abnormal tokens, and excluding the interference of abnormal tokens in the process of calculating the target function, the stability, robustness and success rate of the model reinforcement learning training process can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, brief descriptions will be given below for the drawings needed to be used in the embodiments or prior art descriptions. Obviously, the drawings in the following description are only some embodiments described in the present specification, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0023] Figure 1 A schematic diagram of the overall scheme of the model reinforcement learning method provided by the embodiment of the present specification is shown in the figure.

[0024] Figure 2 A schematic diagram of the model reinforcement learning method provided by the embodiment of the present specification is shown in the figure.

[0025] Figure 3 The corresponding schematic diagram of the model reinforcement learning method provided by the embodiment of the present specification is shown in the figure. Figure 2A structural schematic diagram of a model reinforcement learning device;

[0026] Figure 4 A structural block diagram of a computing device provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0027] In order to enable persons skilled in the art to better understand the technical solutions in the present specification, the technical solutions in the embodiments of the present specification will be clearly and completely described below in conjunction with the drawings in the embodiments of the present specification. Obviously, the described embodiments are only a part of the embodiments of the present specification, rather than all the embodiments. Based on the embodiments in the present specification, all other embodiments obtained by persons skilled in the art without creative labor should fall within the protection scope of the present specification.

[0028] The present specification uses specific words to describe the embodiments of the present specification. As "one embodiment", "an embodiment", and / or "some embodiments" means a certain feature, structure or characteristic related to at least one embodiment of the present specification. Therefore, it should be emphasized and noted that the "an embodiment" or "one embodiment" or "one alternative embodiment" mentioned in different positions in the present specification does not necessarily refer to the same embodiment. In addition, the skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0029] The terms used in one or more embodiments of the present specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present specification. The singular forms "a", "an", "one", "said" and "the" used in one or more embodiments of the present specification and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the present specification includes any or all possible combinations of one or more associated listed items.

[0030] The term "include", "contain" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, product or device. Without more limitations, it does not exclude the presence of other same or equivalent elements in the process, method, product or device including the elements.

[0031] Although the terms first, second, etc. can be employed in this specification to describe various information, such information should not be limited to the terms first, second, etc. These terms are only used to differentiate one set of information from another set of information. For example, a first could be termed a second, and, similarly, a second could be termed a first, without departing from the scope of the present specification. The first, second, etc. designations also do not necessarily limit a particular aspect to the field of application, but, rather, are used to differentiate one element from another. For example, a first server and a second server can each be servers of different types, and the same can hold true for a first device and a second device, first data and second data, etc.

[0032] Depending on the context, the word "if" as used herein can be interpreted to mean "when" or "while" or "in response to determining" or "in response to detecting."

[0033] In this specification, unless otherwise stated, "receiving, sending" of data is not necessarily direct receiving and sending, but can be indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving data sent by B, or A indirectly receiving data sent by B through C or other subjects. Similarly, B sending data to A can be understood as B directly sending data to A, or B indirectly sending data to A through C or other subjects. Here, C can be one subject, or two or more subjects.

[0034] In this specification, unless otherwise stated, the association relationship generated between structures can be a direct association relationship or an indirect association relationship. For example, when "A is connected with B" is described, unless it is explicitly stated that A is directly connected with B, it should be understood that A can be directly connected with B or indirectly connected with B; for another example, when "A is on B" is described, unless it is explicitly stated that A is directly on B (AB is adjacent and A is on B), it should be understood that A can be directly on B or indirectly on B (there are other elements between AB and A is on B). Similarly.

[0035] The user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present specification are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards in relevant regions, and provide corresponding operation entrances for users to choose authorization or refusal.

[0036] Large Language Model, LLM. Large Language Model refers to a deep learning model trained using a large amount of text data, enabling the model to generate natural language text or understand the meaning of language text. Large language models can provide in-depth knowledge and language production on various topics by training on large data sets. Large language models learn the patterns and structures of natural language through large-scale unsupervised training, to some extent, simulating human language cognition and generation process. Large language models use similar Transformer architecture and pre-training objectives as small models, the main difference between small models is the increase in model size, training data and computing resources. Compared with traditional Natural Language Processing (NLP) models, large language models can better understand and generate natural text, while showing certain logical thinking and reasoning ability.

[0037] With the release of OpenAI's O1 series of large language models in 2024, the focus of machine learning training for large language models has gradually shifted from supervised fine-tuning (SFT) to reinforcement learning (RL). SFT is gradually regarded as a "warm-up" stage in the training process, and its role is weakened to parameter initialization or policy guidance. Instead, RL plays an increasingly important role in improving model capabilities.

[0038] Reinforcement learning is a method of machine learning that learns how to make optimal decisions through trial and error. The basic idea of reinforcement learning is to place an agent in an environment and learn how to make optimal actions through interaction with the environment. In reinforcement learning, the agent adjusts its behavior through observation of environmental feedback to obtain the maximum reward.

[0039] In reinforcement learning, there are two basic learning methods: value function-based methods and policy-based methods (i.e., policy gradient). Value function-based methods determine the optimal policy by calculating the value of each state, while policy-based methods directly optimize the policy to obtain the maximum reward. Policy gradient algorithm directly optimizes the policy, by calculating the gradient of the reward with respect to the policy, to update the policy so that the actions taken by the agent can obtain more rewards. Unlike value function-based reinforcement learning methods, policy gradient algorithm directly optimizes the policy itself, avoiding the intermediate step of value function estimation, which makes it have unique advantages in some complex environments.

[0040] In reinforcement learning, a policy is a rule that an agent follows to select an action given a state. A policy can be deterministic, i.e., given a state, the policy deterministically selects an action; or stochastic, i.e., given a state, the policy outputs a probability distribution, and the agent randomly selects an action according to this probability distribution. Policy gradient algorithms usually deal with stochastic policies because stochastic policies can introduce an exploration mechanism to help the agent discover better action paths in the environment.

[0041] To be able to use optimization methods such as gradient descent, the policy needs to be parameterized. Typically, the policy is represented as a parameterized probability distribution, denoted as π θ (a|s), where θ is the parameter of the policy, s is the current state, and a is a possible action. The parameterization of the policy can be implemented by a function approximator such as a neural network, a linear regression model, etc. For example, a neural network can be used as the policy function, with the input being the state s and the output being the parameters of the probability distribution of the action a (such as the softmax output in a classification problem).

[0042] For a language model, the policy is the model itself (called policy model π), which, given a context (i.e., the current state), can output the probability distribution of the next wordpiece (i.e., the action). For a neural network model, the policy parameter θ is the set of all adjustable weights and biases in the policy model π.

[0043] The core idea of policy gradient algorithms is to update the policy parameters θ by computing the gradient of the reward with respect to θ, so that the actions taken by the agent can obtain more rewards. π θ represents a policy determined by the parameter θ, and the optimization process of this policy is to adjust the parameter θ to change the behavior of π θ to obtain higher rewards. Assuming that the total reward R of the agent interacting with the environment is R, we want to maximize the expected total reward J(θ) = E[R|θ]. According to the chain rule in calculus, we can get the policy gradient: This formula shows that the policy gradient can be obtained by computing the expectation of the product of the gradient of the log probability of each action with respect to the parameter θ and the total reward R. The direction of this gradient indicates how to adjust the parameter θ to increase the expected total reward J(θ).

[0044] In practical applications, due to the complexity and randomness of the environment, it is impossible to directly calculate the expectation Therefore, the expectation value is usually estimated using the Monte Carlo method. Specifically, the agent interacts with the environment multiple times, collecting multiple trajectories, each containing a sequence of states, actions, and rewards. Then, the policy gradient is estimated by sampling and averaging these trajectories. For example, suppose N independent trajectories are collected, each with a total reward R n (n = 1, 2,..., N), then the estimate of the policy gradient is: where a n and s n are the action and state sequences in the nth trajectory, respectively.

[0045] To improve efficiency, modern reinforcement learning frameworks generally use a hybrid architecture: a high-performance inference engine is used to generate trajectory data, while an independent training backend (training engine) is used for model training and updates.

[0046] The inference engine supports fast, low-latency decision-making calculations. The inference engine can interact with the environment using the current model (policy) to generate trajectories (i.e., sequences of states, actions, and rewards). During this process, the model parameters are fixed, and some optimization methods (such as caching) can be used to speed up the generation process.

[0047] The training engine provides high-performance distributed training for reinforcement learning models. The training engine can use the trajectory data collected by the inference engine to calculate gradients and update model parameters. The training engine performs backpropagation and optimization steps.

[0048] Specifically, the collaboration process between the inference engine and the training engine in a typical reinforcement learning training framework can include the following steps.

[0049] Step 1: Initialize model parameters and provide them to the training engine and the inference engine.

[0050] Step 2: In the inference phase (also known as the sampling phase), the inference engine can use the current model parameters for inference to generate a batch of trajectories (for example, a batch of output sequences generated for input sequences in a language model). At the same time, the inference engine can record the inference probability (logits) of each token in the generated output sequence.

[0051] Step 3: After collecting the trajectories, calculate the reward for each trajectory using a reward model or other mechanisms.

[0052] Step 4: In the training phase, the training engine can load the same model parameters, and then for each trajectory, the entire sequence (including the input sequence + generated output sequence) is taken as input, and a forward propagation is performed once (only the prefill stage is needed) to obtain the training probability (logits) of each position (including each token in the generated output sequence).

[0053] Step 5: In the training phase, the training engine can use the calculated training probability of each token and the trajectory data generated by the inference engine to calculate the policy gradient (for example, the PPO or GRPO algorithm can be used), and update the model parameters.

[0054] Step 6: The updated model parameters are synchronized to the inference engine for the next round of inference phase.

[0055] Repeat steps 2-6 until the predetermined number of training steps or performance requirements are met.

[0056] According to the above process, the inference engine is responsible for quickly generating data, and the training engine is responsible for accurately calculating the gradient, thereby achieving efficient and stable reinforcement training.

[0057] The task types of the inference phase and the training phase are different. In the inference phase, it is mainly a memory-intensive task, especially when dealing with long context (such as Chain-of-Thought), a large amount of KV Cache needs to be maintained. In the training phase, it is a calculation-intensive task, involving large-scale parameter updates and gradient calculations. In the inference phase, more emphasis is placed on efficiency; in the training phase, more attention is paid to accuracy.

[0058] For On-Policy reinforcement learning strategies, the model for gradient update and the model for sampling trajectories are theoretically consistent, but in the current reinforcement learning training framework, there are differences between the inference engine and the training engine. On the one hand, there are precision differences between the two, such as the inference engine may use low-precision calculations such as FP16 or INT8, while the training engine may use high-precision calculations such as FP32 / BF16, and the rounding and overflow behaviors of different precisions are different. On the other hand, the underlying operators of the two are different, such as the inference engine may use optimized operators (such as fused operators), which may have subtle differences in implementation (such as different reduction orders, different activation function approximations, etc.) from the operators of the training engine.

[0059] These differences between the inference engine and the training engine cause the "trajectory-training" mismatch. For the same sequence, the difference between the probabilities calculated by the training engine and the inference engine will gradually increase with training, leading to the model not being strictly On-Policy trained, and as the precision difference between the training and inference increases, the model may become unstable and even crash.

[0060] The training collapse phenomenon can exhibit two common signals: reward collapse, specifically, a reward value increases for a certain number of steps and then decreases, or even directly decreases to a minimum value; and gradient explosion, specifically, the model gradient value is continuously accumulated and exponentially increases during backpropagation, causing abnormal parameter updates.

[0061] In practical applications, especially in the Long-CoT scenario, training collapse phenomenon is prone to occur when relying on On-Policy policy training of large-scale MoE models. For example, for a MoE model, due to the existence of an expert selection mechanism, when the probability values of Top-K and Top-(K+1) experts are very close, a small difference in the accuracy of the training engine will cause inconsistent expert selection, thereby amplifying the difference in the final calculated probability values for the same sequence, thus making the model training instability situation more serious.

[0062] To solve the defects in the related art, in the embodiments of the present specification, a model reinforcement learning method is provided, which includes a sampling stage and a training stage. In the sampling stage, an inference engine uses a target model to be trained to generate an output sequence for an input sequence under first policy parameters and record the first probability values of each token in the output sequence, and calculate the advantage values of each token. In the training stage, after using a training engine to forward propagate and obtain the second probability values of each token generated by the target model under the first policy parameters, the first ratio of the second probability values to the first probability values of each token can be calculated, and then the tokens with the first ratio within a preset numerical range are selected to participate in the calculation of the objective function, and then the objective function is optimized to update the parameters of the target model. In this way, by identifying the tokens with large differences in inference results between the inference engine in the sampling stage and the training engine in the training stage (i.e., tokens with the first ratio not within the preset numerical range) as abnormal tokens, and excluding the interference of abnormal tokens in the process of calculating the objective function, the stability of the model reinforcement learning process can be improved.

[0063] The technical solutions provided by the embodiments of the present specification will be described in detail below with reference to the accompanying drawings.

[0064] Figure 1 The overall scheme flowchart of the model reinforcement learning method provided by the embodiments of the present specification.

[0065] As Figure 1 shown, first, in the inference stage, the inference engine can load the old policy parameters of the target model to generate an output sequence for an input sequence and record the first probability values of each token in the output sequence.

[0066] Then, a preset advantage value calculation method can be used to generate the advantage value of each token in the output sequence based on the reward value of the output sequence or the reward value of the tokens in the output sequence.

[0067] After that, in the training stage, the training engine can load the current policy parameters of the target model (the old policy parameters before the first gradient update), and calculate the second probability value of each token by forward propagation. Then, as shown in the embodiments of the present specification, abnormal tokens that do not meet the condition (i.e., tokens with a large difference between the second probability value and the first probability value) can be screened out based on the ratio of the second probability value to the first probability value of each token, and the target function and the policy gradient can be calculated based on the reward value of the normal tokens in the back propagation stage, so as to realize the update of the policy parameters.

[0068] If the preset training target is reached, the training is ended; if the preset training target is not reached, the updated policy parameters can be used as the old policy parameters of the next round of training, and the foregoing inference stage, token advantage value calculation and training stage can be repeatedly executed until the preset training target is reached.

[0069] Examples of the inference engine include but are not limited to vLLM, SGLang. Examples of the training engine can include but are not limited to Megatron-LM, DeepSpeed (FSDP), PyTorch FSDP, etc.

[0070] In actual implementation, the inference engine and the training engine can run on the same hardware, or can be distributed on different hardware. For example, the inference engine can use CPU or a dedicated inference card, and the training engine can use GPU. The inference engine and the training engine can run on the same physical device, but are logically separated.

[0071] The server for deploying the inference engine and / or the training engine can include but is not limited to any device, equipment, platform, cluster of equipment, etc. with computing and processing capabilities.

[0072] In actual application, the target model to be trained can be a large model, such as a large language model. Considering that the model parameter amount of the large model is huge, and the operation resources of the user terminal device are limited, the model reinforcement learning method provided in the embodiments of the present specification can be applied to a server, but is not limited thereto. In the case where the running resources of the user terminal device can meet the training conditions of the large model, the embodiments of the present specification can run in the user terminal device.

[0073] In the present application, a model reinforcement learning method is provided, and the present application also relates to a model reinforcement learning device and a computing device, which are described in detail one by one in the following embodiments.

[0074] Figure 2 A flowchart of a model reinforcement learning method provided by an embodiment of the present specification.

[0075] From a program perspective, the execution subject of the flowchart can be a program loaded on a server. It can be understood that the method can be executed by any device, equipment, platform, or cluster of equipment having computing and processing capabilities.

[0076] As shown in Figure 2 The flowchart can include the following steps:

[0077] Step 202: Use the inference engine to obtain an output sequence generated by a target model under a first policy parameter for an input sequence and a first probability value of each token in the output sequence.

[0078] The target model can be a model to be trained by reinforcement learning, and in the scenario of reinforcement learning based on a policy gradient algorithm, the target model can also be referred to as a policy model, and the parameters of the model can be referred to as policy parameters. In the embodiments of the present specification, the model under different policy parameter versions in the policy parameter updating process is collectively referred to as the target model.

[0079] The target model can include a large model, for example, a large language model. In the embodiments of the present specification, the large language model trained by reinforcement learning is not specifically limited, for example, it can include but is not limited to OpenAI o1 / o3 / o4 series models, DeepSeek-R1 models, Seed-Thinking models, meta series inference models, Qwen3 series models, Kimi series models, Ling models, Ring models, etc.

[0080] Further, the target model can be a pre-trained large language model, or a model obtained after supervised fine-tuning training based on a pre-trained large language model.

[0081] Step 202 specifically includes, in an inference phase (also referred to as a sampling phase), the first policy parameter of the target model can be loaded using an inference engine, so as to generate an output sequence for an input sequence based on the target model under the first policy parameter. Optionally, one output sequence or a group of output sequences can be generated for one input sequence. In actual application, the input sequence can be user query information, and the output sequence can be model reply information generated by the target model for the user query information.

[0082] In addition, in actual application, in order to update the value of the target function in the reinforcement learning process, specifically, in order to calculate the importance ratios such as the first ratio and the second ratio subsequently, the probability values of each word element inferred in each inference step in the process of generating the output sequence by the target model need to be recorded. The calculation process of the first ratio and the second ratio is shown in the relevant part below.

[0083] Step 204: Based on the output sequence, the advantage value of each word element is determined.

[0084] In the embodiments of the present specification, a method for calculating the advantage value of each word element is further provided.

[0085] Optionally, the advantage value of each word element in the output sequence can be calculated based on the reward information of the output sequence.

[0086] Specifically, one input sequence can correspond to one output sequence group; one output sequence group contains multiple intra-group output sequences; the determination of the advantage value of each word element in step 204 can specifically include: determining the reward value of each intra-group output sequence in the output sequence group; calculating the intra-group advantage value of each intra-group output sequence based on the reward value of each intra-group output sequence; and determining the intra-group advantage value of the target output sequence in the output sequence group as the advantage value of each word element in the target output sequence. The intra-group advantage value, also known as the intra-group relative advantage value, is used to represent the advantage value of an output sequence compared to the reward values of other output sequences in the output sequence group.

[0087] Further, the step of determining the reward value of each intra-group output sequence in the output sequence group can include: a preset reward value determination method can be used to determine the reward value of each intra-group output sequence in the output sequence group. The preset reward value determination method can be various.

[0088] Optionally, a preset reward value calculation rule can be used to calculate the reward value of each output sequence in the output sequence group. In actual application, in the fields of mathematics, physics, code, etc., the matching degree between the model output result and the rule can be scored. As an example, the reward value calculation rule can include: if the output sequence is consistent with the preset label sequence of the input sequence, a first reward value, such as 1, is obtained; if the output sequence is not consistent with the preset label sequence of the input sequence, a second reward value, such as -1, is obtained. Assuming that the input sequence is “10+5=”, its preset label sequence can be “15”, if the output sequence is “15”, the first reward value “1” can be obtained; if the output sequence is other values such as “50”, the second reward value “-1” can be obtained.

[0089] Optionally, a pre-trained reward model is adopted, the input sequence and the output sequence group are input into the reward model, and reward values of each output sequence in the output sequence group determined by the reward model are obtained. Further, the reward model can be a neural network model. In actual application, a lightweight reward model can be trained to score, for example, in the field of e-commerce, customer service, etc., the cost is controllable and the effect is good.

[0090] In the embodiments of the present specification, the determination method of the reward value includes but is not limited to the above examples, and a suitable reward value evaluation method can be used according to the actual application scenario, and any applicable reward value determination method in related technologies can be used.

[0091] Optionally, the in-group advantage value of each output sequence in the output sequence group is calculated based on the reward value of each output sequence in the output sequence group, which can specifically include: calculating the average value of the reward value of each output sequence in the output sequence group as the in-group reward average value of the output sequence group; and calculating the in-group advantage value of each output sequence in the output sequence group based on the reward value of each output sequence in the output sequence group and the in-group reward average value. That is, for each output sequence group, the in-group reward average value of the output sequence group can be calculated; and the in-group advantage value of each output sequence in the output sequence group can be determined based on the difference between the reward value of each output sequence in the output sequence group and the in-group reward average value.

[0092] Further, the in-group advantage value of each output sequence in the output sequence group can be calculated based on the reward value of each output sequence in the output sequence group and the in-group reward average value, which can specifically include: for any output sequence in the output sequence group, the difference between the target reward value of the any output sequence and the in-group reward average value is taken as the initial in-group advantage value of the any output sequence; and the initial in-group advantage value of the any output sequence is normalized to obtain the in-group advantage value of the any output sequence. Further, the standard deviation of the reward value of each in-group output sequence in the output sequence group can be calculated as the in-group reward standard deviation of the output sequence group; and the initial in-group advantage value of the any output sequence is divided by the in-group reward standard deviation to obtain the in-group advantage value of the any output sequence.

[0093] Alternatively, calculating the intra-group advantage value of each output sequence based on the reward value of each group's output sequence may specifically include: determining a first output sequence with the smallest advantage value and a second output sequence with the largest advantage value in the output sequence group; for any output sequence in the input sequence, dividing the first difference obtained by subtracting the first advantage value of the first output sequence from the target advantage value of the arbitrary output sequence by a second difference obtained by subtracting the first advantage value of the first output sequence from the second advantage value of the second output sequence to obtain the intra-group advantage value of the arbitrary output sequence. Optionally, the arbitrary output sequence may include either the first output sequence or the second output sequence.

[0094] In an optional embodiment, a reward value can be calculated at the end of each output sequence output during the sampling phase.

[0095] Specifically, for each input sequence x, the target model can be configured in the first policy parameter. Downsample a set of output sequences {y1,y2,…,y G Then, a preset reward value determination method can be used to score each output sequence, generating G rewards r = {r1, r2, ..., r}. G Subsequently, these reward values ​​are normalized by subtracting the group mean and dividing by the group standard deviation. This can be done in the output sequence y. i Calculate the normalized reward at the end and output the sequence y. i The advantages of all lexical units The reward is set to normalization, that is,

[0096] In an optional embodiment, the reward value can be calculated at the end of each inference step during the generation of the output sequence in the sampling phase.

[0097] Specifically, for each input sequence x, the target model can be configured in the first policy parameter. Downsample a set of output sequences {y1,y2,…,y G A preset reward value determination method can be used to score each inference step in the process of generating the output sequence (i.e., the output result of each word in the output sequence), and generate corresponding rewards. Where index(j) is the end term index of step j, K i This represents the total number of steps in the i-th output sequence. Subsequently, these rewards can be normalized using the mean and standard deviation, i.e. Furthermore, the sum of the normalized rewards in subsequent steps can be calculated as the advantage for each lexical unit, i.e.,

[0098] Alternatively, the advantage value of each token can be an advantage value determined according to a value model. Specifically, the value model can be set to calculate the advantage value through an advantage function A (θ) = Q (s, a) - V (s), where A (θ) represents the advantage under the policy parameter θ, Q (s, a) represents the probability inferred by the target model, and V (s) represents the probability inferred by the value model. More specifically, reference can be made to the calculation method of the advantage value of each token in the Proximal Policy Optimization (PPO) algorithm.

[0099] In actual application, the calculation of the advantage value of each token in the output sequence generated based on the input sequence can refer to other advantage value calculation methods in related technologies.

[0100] Step 206: obtaining, by using the training engine, the second probability value of each token generated by the target model under the first policy parameter.

[0101] In the training phase, the training engine can load the first policy parameter of the target model, and obtain the second probability value of each token in the output sequence in the forward propagation process.

[0102] As an example, assuming that the first policy parameter is θ old , the target model generates the t th token y old (θ i,t ) in the i th output sequence of the input sequence x under the current policy parameter θ old , and the advantage value is The first probability value of the t th token in the i th output sequence can be denoted as where infer represents the inference engine. The second probability value of the t th token in the i th output sequence can be denoted as where train represents the training engine.

[0103] Step 208: calculating the first ratio of the second probability value and the first probability value of each token.

[0104] Specifically, for each token, the first ratio of the second probability value and the first probability value of the token can be calculated. The first ratio can be used to reflect the difference in inference accuracy of the training engine and the inference engine for the same token under the same policy parameter (for example, the first policy parameter).

[0105] Following the above example, the first ratio of the second probability value and the first probability value of the t th token in the i th output sequence can be denoted as Based on the scheme of the embodiments of the present specification, by setting the first ratio, the problem of the training engine and the inference engine being unable to accurately infer the same token under the same policy parameter can be solved. The problem of unstable gradient update in the model training process caused.

[0106] Step 210: determining a value of an objective function based on the first ratio of each token and the advantage value; wherein the first ratio of the token within a preset numerical range participates in the calculation of the objective function.

[0107] The objective function can be a policy optimization objective function, that is, an objective function for optimizing the policy parameters of the target model. The first ratio, also known as the Train-infer Ratio, is an importance sampling ratio used to reflect the difference in inference accuracy between the training engine and the inference engine.

[0108] In the embodiments of the present specification, in fact, according to the first ratio of each token, abnormal tokens can be screened out from the tokens, so that when calculating the value of the objective function, the advantage values of these abnormal tokens can not participate in the calculation of the objective function. That is, only the advantage values of normal tokens are counted, and the gradient corresponding to the abnormal tokens is 0, which is equivalent to masking the abnormal tokens.

[0109] Specifically, the determination of the value of the objective function based on the first ratio of each token and the advantage value can specifically include: determining the weight of each token based on the first ratio of each token; if the first ratio is not within the preset numerical range, the weight of the corresponding token is 0; if the first ratio is within the preset numerical range, the weight of the corresponding token is the first ratio; and performing weighted average on the advantage values of the tokens based on the weights of the tokens to obtain the value of the objective function.

[0110] Specifically, the logic for determining the weight M(k) through the first ratio k can be as follows:

[0111]

[0112] Further, the preset numerical range can be a range greater than or equal to a first threshold and less than or equal to a second threshold, wherein the first threshold can be a number greater than zero and less than 1, and the second threshold can be a number greater than 1. The preset numerical range can be an interval [α,β], wherein α is a number greater than zero and less than 1, and β is a number greater than 1. Wherein, α and β are hyperparameters. It should be noted that although the preset numerical range is set to the closed interval [α,β] herein, in actual application, the preset numerical range can also be an open interval (α,β), or it can also be a semi-open semi-closed interval (α,β] or [α,β].

[0113] Further, the weighting average of the advantage values of the word pieces based on the weights of the word pieces can obtain the value of the target function, which can specifically include: calculating the product of the weight and the advantage value of each word piece, and averaging the products of all word pieces to obtain the value of the target function. In actual application, the calculation of the product of the weight and the advantage value of each word piece can include directly calculating the product of the weight and the advantage value of each word piece, or can include calculating the product of the weight and the optimized advantage value (for example, the first weighted advantage value and the third weighted advantage value to be introduced below) of each word piece.

[0114] Based on the scheme of the embodiments of the present specification, for the abnormal word piece identified based on the first ratio, the advantage value of the abnormal word piece does not need to participate in the calculation of the value of the target function, which is equivalent to masking the abnormal word piece with excessively large training precision, can effectively prevent the abnormal word piece from affecting the target function and the parameter update gradient, can effectively avoid the model from being affected by a small amount of abnormal word pieces in training stability and training effect, and can improve the stability and success rate of the reinforcement learning training process.

[0115] Step 212: Adjusting the parameters of the target model to obtain the target model under the second strategy parameters.

[0116] In actual application, the target function can be optimized to update the strategy parameters of the target model. For example, the first strategy parameters are updated to the second strategy parameters.

[0117] Specifically, the target function can reflect the weighted average of the advantage values. When the advantage value of a word piece is positive, the direction of reinforcement training is to increase the probability of the word piece; when the advantage value of a word piece is negative, the direction of reinforcement training is to reduce the probability of the word piece. Therefore, the process of reinforcement learning is usually aimed at maximizing the value of the target function. In actual application, the optimization process is usually to minimize the loss function, and therefore the loss function can be defined as the opposite number of the target function. Thus, the strategy parameters can be optimized by calculating the gradient.

[0118] Although one or more embodiments of the present specification provide method steps as described in the embodiments or flowcharts, it can be understood that the order of steps listed in the embodiments or flowcharts is only one of the many execution orders, and does not represent the only execution order. The order of some steps can be adjusted according to actual needs, or some steps can be omitted. When the claims involve method steps, the adjustment of the order of the steps or the parallelism between the steps is also within the protection scope of the claims.

[0119] Figure 2In the method, the inference engine is used to generate an output sequence for an input sequence under a first strategy parameter of a target model to be trained and record first probability values of each token in the output sequence, and the advantage values of each token are calculated. After the training engine is used to obtain second probability values of each token generated by forward propagation of the target model under the first strategy parameter in the training stage, the first ratio of the second probability values to the first probability values of each token can be calculated, and then the tokens with the first ratio in a preset numerical range are screened out to participate in the calculation of the target function, and then the target function is optimized to update the parameters of the target model. Therefore, by identifying the tokens with large differences in inference results of the inference engine and the training engine (i.e., the tokens with the first ratio not in the preset numerical range) as abnormal tokens, and excluding the interference of the abnormal tokens in the process of calculating the target function and the gradient, the stability, robustness and success rate of the model reinforcement learning training process can be improved.

[0120] Based on the method, Figure 2 The embodiments of the present specification also provide some improved implementations of the method, which are described below.

[0121] In one or more embodiments of the present specification, the model reinforcement learning method can further include: obtaining, using the training engine, third probability values of the tokens generated by the target model under a current strategy parameter; calculating a second ratio of the third probability values to the second probability values of the tokens; and calculating a first product of the second ratio and the advantage values of the tokens as a first weighted advantage value.

[0122] Correspondingly, the weighting average of the advantage values of the tokens based on the weights of the tokens to obtain the value of the target function can specifically include: weighting average of the first weighted advantage values of the tokens based on the weights of the tokens to obtain the value of the target function.

[0123] In the training stage, the current strategy parameter of the target model can be updated continuously, for example, from the first strategy parameter to the second strategy parameter, and then to the third strategy parameter, the fourth strategy parameter, etc. The current strategy parameter can include the first strategy parameter or the second strategy parameter, or other versions of strategy parameters other than the first strategy parameter or the second strategy parameter.

[0124] The second ratio can also be referred to as a policy ratio, which is an importance sampling ratio used to measure the change between the new and old policy parameters. During the training phase, the second ratio of the third probability value of each token under the current policy parameter to the second probability value of each token under the first policy parameter (i.e., the old policy parameter) can be continuously calculated, so as to update the value of the objective function by adjusting the first weighted advantage value. Thus, when the value of the objective function reaches a preset optimization target or the number of iterations reaches a preset number threshold, the optimized policy parameter is obtained.

[0125] Using the example above, if the advantage value of the t-th token in the i-th output sequence is The second ratio is r i,t The first weighted advantage value can be represented as Assuming that the current policy parameter is θ, the target model is y i,t (θ) for the t-th token in the i-th output sequence of the input sequence x under the current policy parameter θ, and the third probability value of the t-th token in the i-th output sequence can be denoted as where train represents a training engine. Further, the second ratio of the third probability value of the t-th token in the i-th output sequence to the second probability value can be denoted as

[0126] In one or more embodiments of the present specification, the model reinforcement learning method can further include: clipping the second ratio of each token to obtain a clipped second ratio; calculating a second product of the clipped second ratio of each token and the advantage value as a second weighted advantage value; and determining the smaller one of the first weighted advantage value and the second weighted advantage value of each token as a third weighted advantage value.

[0127] Correspondingly, the weighted average of the advantage values of each token based on the weights of each token to obtain the value of the objective function can specifically include: a weighted average of the third weighted advantage values of each token based on the weights of each token to obtain the value of the objective function.

[0128] Specifically, a clipping function can be used to clip the second ratio.

[0129] Using the example above, the second ratio corresponding to the t-th token in the i-th output sequence is denoted as r i,t The clipping process can be performed by the following formula clip(r i,t ,1-ε,1+ε) to make the clipped r i,t between 1-ε and 1+ε. Wherein, ε is a clipping parameter. In practical applications, ε is a hyperparameter of the reinforcement learning training process.

[0130] Following the example above, the advantage value of the t-th token in the i-th output sequence is denoted as The first weighted advantage value can be denoted as The second weighted advantage value is denoted as Further, the third advantage value can be determined by

[0131] In the embodiments of the present specification, by using the clipping update strategy, the second ratio (the strategy ratio parameter) is clipped, which can limit the update range of the strategy parameter, and can avoid large and unstable updates (if the ratio of the new strategy to the old strategy exceeds the range, it will be clipped to prevent excessive correction), so as to avoid the drastic change of the strategy parameter causing the model to collapse.

[0132] In practical applications, the RL algorithm itself is constantly iterating and optimizing. From the early DPO (Direct Preference Optimization) to the classic PPO (Proximal Policy Optimization), and to the new methods such as GRPO, RLOO, Reinforce++, DAPO, etc. emerging in recent years, the RL algorithm continues to optimize in terms of policy update method, stability, sample efficiency, etc. Among them, the proximal policy optimization (Proximal Policy Optimization, PPO) algorithm is a policy-based reinforcement learning algorithm, which updates the policy by optimizing an approximate objective function. Further, the group relative policy optimization (Group Relative Policy Optimization, GRPO) is a reinforcement learning algorithm, which is an improvement of PPO, aiming to be more efficient, especially in complex reasoning tasks. Unlike traditional reinforcement learning methods that rely heavily on external evaluation models (value models, value functions) to guide learning, GRPO optimizes the model by evaluating response groups that are related to each other. Specifically, by generating multiple answers to the same question and comparing them with each other, it replaces the "value model" in the traditional PPO algorithm. The GRPO algorithm can improve training efficiency, especially when dealing with reasoning tasks that require solving complex problems and long-chain thinking.

[0133] Based on the scheme provided in the embodiments of the present specification, various RL algorithms can be optimized. For ease of understanding, the following describes the optimized reinforcement learning method provided in the embodiments of the present specification by taking the GRPO algorithm as an example.

[0134] ​For example, the GRPO algorithm is improved based on the method of the embodiments of the present specification. One input sequence can correspond to one output sequence group; one output sequence group can contain multiple intra-group output sequences; the target function value can be obtained by weighted average of the advantage values of the word pieces based on the weights of the word pieces, which can specifically include: for any output sequence in the output sequence group, the sequence advantage value is obtained by weighted average calculation of the third weighted advantage values of the word pieces in the any output sequence based on the weights of the word pieces; the mean value of the sequence advantage values of the intra-group output sequences in the output sequence group is calculated as the value of the target function.

[0135] In practical applications, the target function can conform to the following formula:

[0136]

[0137] wherein,

[0138]

[0139] wherein, θ represents the current policy parameter of the target model, θ old represents the first policy parameter;

[0140] π infer represents the probability calculated in the sampling stage; π train represents the probability calculated in the training stage;

[0141] D represents a data set, and x represents an input sequence in the data set D;

[0142] G represents the number of output sequences in the output sequence group generated for the input sequence x;

[0143] y i represents the i-th output sequence generated for the input sequence x; y i,t represents the t-th word piece in the i-th output sequence generated for the input sequence x;

[0144] represents the advantage value of the t-th word piece in the i-th output sequence generated for the input sequence x.

[0145] Further, may represent that the input sequence x is sampled from: the data set D.

[0146] may represent that the output sequence in the output sequence group corresponding to the input sequence x is sampled from: the first policy parameter θ oldThe output for the input sequence x.

[0147] The probability of the t-th token in the i-th output sequence generated by the target model for the input sequence x under the first policy parameter θ old The probability of the t-th token in the i-th output sequence generated by the target model for the input sequence x under the first policy parameter θ

[0148] The probability of the t-th token in the i-th output sequence generated by the target model for the input sequence x under the first policy parameter θ old The probability of the t-th token in the i-th output sequence generated by the target model for the input sequence x under the first policy parameter θ

[0149] The probability of the t-th token in the i-th output sequence generated by the target model for the input sequence x under the first policy parameter θ

[0150] In optional embodiments of the present specification, during the reinforcement learning training of the target model, the deviation between the trained target model (or policy model) and the reference model can be further used as a constraint. The reference model can be a model trained by supervised fine-tuning (e.g., the target model before reinforcement learning training), and the parameters of the reference model are not updated during the training of the target model, which can be used as a reference for the target model.

[0151] Specifically, the KL divergence between the target model and the reference model can be calculated. By constraining the KL divergence between the target model and the reference model, the training result of the policy parameter of the target model can not only conform to the preference of the reward model (i.e., the preference of the human), but also not differ too much from the reference policy parameter, which can prevent the training process of the policy parameter from deviating too much, ensure the stability of the training process of the target model, and avoid gradient explosion.

[0152] In optional embodiments, the KL penalty term between the target model and the reference model can be added in the calculation process of the reward value.

[0153] In another optional embodiment, the KL divergence between the reference policy and the target policy can be added to the target function as a regularization. As an example, the KL divergence regularization can be calculated by the following formula:

[0154] In the case of considering the KL divergence between the target model and the reference model, taking the improvement of the GRPO algorithm based on the method of the embodiments of the present specification as an example, the target function can be further represented as:

[0155] Wherein, γ is a hyperparameter, used to represent the constraint degree of the KL divergence regularization term on the objective function.

[0156] Based on the scheme of the embodiments of the present specification, the stability of the target model training process can be further improved by using the KL divergence penalty deviation, and gradient explosion can be avoided.

[0157] The various technical features in the above embodiments can be combined in any manner as long as there is no conflict or contradiction between the features. However, due to the limited space, they are not described one by one, so any combination of the various technical features in the above embodiments also belongs to the scope disclosed by the present specification.

[0158] Based on the same idea, the present specification also provides a device corresponding to the above method.

[0159] Figure 3 The present specification provides a structure schematic diagram of a model reinforcement learning device corresponding to the above method. Figure 2

[0160] As shown in Figure 3 , the device can include:

[0161] The sampling module 302 is configured to use the inference engine to obtain an output sequence generated by a target model under first policy parameters for an input sequence and first probability values of each token in the output sequence;

[0162] The advantage calculation module 304 is configured to determine advantage values of the tokens based on the output sequence;

[0163] The forward propagation module 306 is configured to use the training engine to obtain second probability values of the tokens generated by the target model under the first policy parameters;

[0164] The first ratio calculation module 308 is configured to calculate a first ratio of the second probability values of the tokens to the first probability values;

[0165] The objective function calculation module 310 is configured to determine a value of an objective function based on the first ratio of the tokens and the advantage values, wherein the tokens within a preset numerical range participate in the calculation of the objective function;

[0166] The parameter updating module 312 is configured to adjust the parameters of the target model to obtain the target model under second policy parameters, with the goal of maximizing the value of the objective function.

[0167] Based on the device Figure 3 , the present specification also provides some specific embodiments of the method, which are described below.

[0168] ​Optionally, the preset numerical range is a range greater than or equal to a first threshold and less than or equal to a second threshold, wherein the first threshold is a number greater than zero and less than 1, and the second threshold is a number greater than 1.

[0169] Optionally, the target function calculation module 310 specifically includes:

[0170] a weight calculation unit configured to determine a weight of each token based on the first ratio of the token; if the first ratio is not within the preset numerical range, the weight of the corresponding token is 0; if the first ratio is within the preset numerical range, the weight of the corresponding token is the first ratio;

[0171] a target function calculation unit configured to perform weighted average on the advantage values of the tokens based on the weights of the tokens to obtain a value of the target function.

[0172] Optionally, the model reinforcement learning apparatus is further configured to: use a training engine to obtain third probability values of the tokens generated by the target model under current policy parameters; calculate second ratios of the third probability values and the second probability values of the tokens; and calculate first products of the second ratios and the advantage values of the tokens as first weighted advantage values. Correspondingly, the target function calculation unit is specifically configured to perform weighted average on the first weighted advantage values of the tokens based on the weights of the tokens to obtain a value of the target function.

[0173] Optionally, the model reinforcement learning apparatus is further configured to: clip the second ratios of the tokens to obtain clipped second ratios; calculate second products of the clipped second ratios and the advantage values of the tokens as second weighted advantage values; and determine a smaller one of the first weighted advantage values and the second weighted advantage values of the tokens as a third weighted advantage value. Correspondingly, the target function calculation unit is specifically configured to perform weighted average on the third weighted advantage values of the tokens based on the weights of the tokens to obtain a value of the target function.

[0174] Optionally, one input sequence corresponds to one output sequence group, and one output sequence group includes multiple intra-group output sequences. The target function calculation unit is specifically configured to, for any output sequence in the output sequence group, perform weighted average calculation on the third weighted advantage values of the tokens in the any output sequence based on the weights of the tokens to obtain a sequence advantage value, and calculate a mean value of the sequence advantage values of the intra-group output sequences in the output sequence group as a value of the target function.

[0175] Optionally, one input sequence corresponds to one output sequence group; one output sequence group contains multiple intra-group output sequences; the advantage calculation module 304 is specifically configured to: determine the reward values of the intra-group output sequences in the output sequence group; calculate the intra-group advantage values of the intra-group output sequences based on the reward values of the intra-group output sequences; and determine the intra-group advantage value of a target output sequence in the output sequence group as the advantage value of each token in the target output sequence.

[0176] Optionally, the calculation of the intra-group advantage values of the intra-group output sequences based on the reward values of the intra-group output sequences specifically includes: calculating the average value of the reward values of the intra-group output sequences as an intra-group reward average value of the output sequence group; and calculating the intra-group advantage values of the intra-group output sequences based on the reward values of the intra-group output sequences and the intra-group reward average value.

[0177] Optionally, the calculation of the intra-group advantage values of the intra-group output sequences based on the reward values of the intra-group output sequences and the intra-group reward average value specifically includes: for any output sequence in the output sequence group, taking the difference between the target reward value of the any output sequence and the intra-group reward average value as the initial intra-group advantage value of the any output sequence; calculating the standard deviation of the reward values of the intra-group output sequences in the output sequence group as an intra-group reward standard deviation of the output sequence group; and taking the quotient of the initial intra-group advantage value of the any output sequence and the intra-group reward standard deviation to obtain the intra-group advantage value of the any output sequence.

[0178] Optionally, the target function satisfies the following formula:

[0179]

[0180] wherein,

[0181]

[0182] wherein, θ represents the current policy parameter of the target model, θ old represents the first policy parameter;

[0183] π infer represents the probability calculated in the sampling stage; π train represents the probability calculated in the training stage;

[0184] D represents a data set, and x represents an input sequence in the data set D;

[0185] G represents the number of output sequences in the output sequence group generated for the input sequence x;

[0186] yi represents the i-th output sequence generated for the input sequence x; y i,t represents the t-th token in the i-th output sequence generated for the input sequence x; y

[0187] represents the advantage value of the t-th token in the i-th output sequence generated for the input sequence x; y

[0188] It can be understood that each module described above refers to a computer program or program segment, which is used to perform one or more specific functions. In addition, the division of each module described above does not mean that the actual program code must also be separated.

[0189] For the convenience of description, the above device is described as various modules or units respectively described in function. Of course, when implementing one or more of the present specification, the functions of each module or unit can be implemented in the same or more software and / or hardware, or the modules implementing the same function can be implemented by a combination of multiple sub-modules or sub-units. The above described device embodiments are only illustrative, for example, the division of the unit is only a logical function division, and actual implementation can have another division method, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0190] The above is a schematic scheme of a model reinforcement learning device of the present embodiment. It should be noted that the technical scheme of the model reinforcement learning device belongs to the same concept as the technical scheme of the model reinforcement learning method described above, and the technical scheme of the model reinforcement learning device which is not described in detail can be referred to the description of the technical scheme of the model reinforcement learning method.

[0191] Based on the same idea, the present specification embodiment also provides a device corresponding to the above method.

[0192] Figure 4 A structural block diagram of a computing device provided by the present specification embodiment.

[0193] The computing device 400 includes:

[0194] a memory 410 and a processor 420;

[0195] The memory 410 is used to store computer programs / instructions, and the processor 420 is used to execute the computer programs / instructions, which realize the steps of the model reinforcement learning method when executed by the processor 420.

[0196] In particular, the components of the computing device 400 include, but are not limited to, a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and a database 450 is used to store data.

[0197] The computing device 400 also includes an access device 440 that enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or combinations of these and / or other types of networks, such as the Internet. The access device 440 can include one or more of any type of network interface (e.g., network interface cards (NICs)) such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and / or the like.

[0198] In one embodiment of the present specification, the above-mentioned components of the computing device 400 and other components not shown in the figure can be connected to each other, for example, via a bus. It should be understood that Figure 4 the components of the computing device 400 can be connected to each other in other ways. Figure 4 The computing device structure diagram shown is only for the purpose of example, and is not a limitation on the scope of the present application. Those skilled in the art can add or replace other components as needed.

[0199] The computing device 400 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 400 can also be a mobile or stationary server.

[0200] The processor 420 implements the steps of the model reinforcement learning method when executing the computer instructions.

[0201] The above is a schematic scheme of the computing device of the embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the model reinforcement learning method described above belong to the same concept, and the details of the technical scheme of the computing device that are not described in detail can be referred to the description of the technical scheme of the model reinforcement learning method.

[0202] An embodiment of the present specification also provides a computer readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the model reinforcement learning method described above.

[0203] The above is a schematic scheme of the computer readable storage medium of the embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the model reinforcement learning method described above belong to the same concept, and the details of the technical scheme of the storage medium that are not described in detail can be referred to the description of the technical scheme of the model reinforcement learning method.

[0204] An embodiment of the present specification also provides a computer program product comprising computer program / instructions, which, when executed by a processor, implement the steps of the model reinforcement learning method described above.

[0205] The above is a schematic scheme of the computer program product of the embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the model reinforcement learning method described above belong to the same concept, and the details of the technical scheme of the computer program product that are not described in detail can be referred to the description of the technical scheme of the model reinforcement learning method.

[0206] Each of the embodiments in the present specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device and equipment embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment. The device and equipment provided by the embodiments of the present specification are corresponding to the method, so the device and equipment also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding device and equipment will not be described here.

[0207] The above described embodiments of the present description have been described. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0208] In the 1990s, it was relatively easy to distinguish whether an improvement in a technology was a hardware improvement (e.g., an improvement in the circuit structure of a diode, transistor, switch, etc.) or a software improvement (an improvement in a method flow). However, as technology has evolved, many improvements in method flows today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method flows into hardware circuits. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using hardware entity modules. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A designer programs a digital system "integrated" on a PLD by himself, without having to ask a chip manufacturer to design and manufacture a special integrated circuit chip. Moreover, instead of manually manufacturing integrated circuit chips, this programming is now mostly implemented using "logic compiler" software, which is similar to the software compiler used when developing programs, and the original code before compilation must also be written in a specific programming language, which is called a hardware description language (HDL), and there are many types of HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that it is easy to obtain a hardware circuit that implements a logical method flow by simply logically programming the method flow in the above-mentioned hardware description languages and programming it into an integrated circuit.

[0209] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to being implemented in pure computer readable program code, the controller can equally well be implemented to perform the same functions using logic gates, switches, an application specific integrated circuit, a programmable logic controller and an embedded microcontroller, etc. by means of a logical programming of the method steps. The controller can thus be considered as a hardware component, and the means comprised therein for performing the various functions can be considered as structures within the hardware component. Alternatively, the means for performing the various functions can even be considered as both a software module implementing the method and a structure within the hardware component.

[0210] The systems, apparatuses, modules or units illustrated by the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0211] For the sake of description, the above apparatuses are described in various units by functions respectively. Of course, the functions of the units can be implemented in one or more software and / or hardware in the implementation of the present application.

[0212] Those skilled in the art will understand that one or more embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0213] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps in one or more flow or blocks

[0214] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps in one or more flow or blocks

[0215] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps in one or more flow or blocks

[0216] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0217] The memory can include non-persistent memory and / or persistent memory, such as flash memory, read-only memory (ROM), and / or volatile or non-volatile random access memory (RAM), among others. The memory is an example of computer readable media.

[0218] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0219] The present application can be described in the general context of computer- executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0220] The above only describes the embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A model reinforcement learning method, comprising: obtaining, using an inference engine, an output sequence generated by a target model under a first policy parameter for an input sequence and first probability values of tokens in the output sequence; determining advantage values of the tokens based on the output sequence; obtaining, using a training engine, second probability values of the tokens generated by the target model under the first policy parameter; calculating first ratios of the second probability values and the first probability values of the tokens; determining a value of an objective function based on the first ratios and the advantage values of the tokens, wherein tokens whose first ratios are within a preset numerical range participate in calculation of the objective function; adjusting parameters of the target model to obtain the target model under a second policy parameter, aiming to maximize the value of the objective function.

2. The method of claim 1, wherein, The preset numerical range is a range greater than or equal to a first threshold and less than or equal to a second threshold, wherein the first threshold is a number greater than zero and less than 1, and the second threshold is a number greater than 1.

3. The method of claim 1, wherein the determining the value of the objective function based on the first ratios and the advantage values of the tokens comprises: determining weights of the tokens based on the first ratios of the tokens; if the first ratio is not within the preset numerical range, the weight of the corresponding token is 0; if the first ratio is within the preset numerical range, the weight of the corresponding token is the first ratio; performing weighted averaging on the advantage values of the tokens based on the weights of the tokens to obtain the value of the objective function.

4. The method of claim 3, further comprising: obtaining, using a training engine, third probability values of the tokens generated by the target model under a current policy parameter; calculating second ratios of the third probability values and the second probability values of the tokens; calculating first products of the second ratios and the advantage values of the tokens as first weighted advantage values; the performing weighted averaging on the advantage values of the tokens based on the weights of the tokens to obtain the value of the objective function comprises: performing weighted averaging on the first weighted advantage values of the tokens based on the weights of the tokens to obtain the value of the objective function.

5. The method of claim 4, further comprising: clipping the second ratios of the tokens to obtain clipped second ratios; calculating second products of the clipped second ratios and the advantage values of the tokens as second weighted advantage values; determining a smaller one of the first weighted advantage values and the second weighted advantage values of the tokens as a third weighted advantage value; the performing weighted averaging on the advantage values of the tokens based on the weights of the tokens to obtain the value of the objective function comprises: performing weighted averaging on the third weighted advantage values of the tokens based on the weights of the tokens to obtain the value of the objective function.

6. The method of claim 5, wherein, One input sequence corresponds to one output sequence group; one output sequence group contains multiple intra-group output sequences; the weighted average of the advantage values of the word units is obtained by weighting the advantage values of the word units based on the weights of the word units, and the value of the target function is obtained, specifically including: For any output sequence in the output sequence group, the sequence advantage value of the any output sequence is calculated by weighting the third weighted advantage values of the word units in the any output sequence based on the weights of the word units; The mean value of the sequence advantage values of the intra-group output sequences in the output sequence group is calculated as the value of the target function.

7. The method of claim 1, wherein, One input sequence corresponds to one output sequence group; One output sequence group contains multiple intra-group output sequences; the determination of the advantage values of the word units specifically includes: The reward values of the intra-group output sequences in the output sequence group are determined; The intra-group advantage values of the intra-group output sequences are calculated based on the reward values of the intra-group output sequences; The intra-group advantage value of the target output sequence in the output sequence group is determined as the advantage value of each word unit in the target output sequence.

8. The method of claim 7, wherein the intra-group advantage values of the intra-group output sequences are calculated based on the reward values of the intra-group output sequences, specifically including: The mean value of the reward values of the intra-group output sequences is calculated as the intra-group reward mean value of the output sequence group; The intra-group advantage values of the intra-group output sequences are calculated based on the reward values of the intra-group output sequences and the intra-group reward mean value.

9. The method of claim 8, wherein the intra-group advantage values of the intra-group output sequences are calculated based on the reward values of the intra-group output sequences and the intra-group reward mean value, specifically including: For any output sequence in the output sequence group, the initial intra-group advantage value of the any output sequence is obtained by subtracting the intra-group reward mean value from the target reward value of the any output sequence; The standard deviation of the reward values of the intra-group output sequences in the output sequence group is calculated as the intra-group reward standard deviation of the output sequence group; The intra-group advantage value of the any output sequence is obtained by dividing the initial intra-group advantage value of the any output sequence by the intra-group reward standard deviation.

10. A model reinforcement learning device, comprising: a sampling module configured to acquire, using an inference engine, an output sequence generated by a target model under first policy parameters for an input sequence and first probability values of word units in the output sequence; an advantage calculation module configured to determine, based on the output sequence, advantage values of the word units; a forward propagation module configured to acquire, using a training engine, second probability values of the word units generated by the target model under the first policy parameters; a first ratio calculation module configured to calculate first ratios of the second probability values of the word units to the first probability values; a target function calculation module configured to determine a value of a target function based on the first ratios of the word units and the advantage values; wherein the word units with the first ratios within a preset numerical range participate in the calculation of the target function. A parameter updating module is configured to adjust parameters of the target model to obtain the target model under second strategy parameters, with the aim of maximizing the value of the target function. 11.A computing device comprising: a memory and a processor; the memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, and the computer programs / instructions, when executed by the processor, implement the steps of the method in any one of claims 1 to 9.

Images