Reinforcement learning training method and device for large language model

Abstract

The embodiment of the invention provides a reinforcement learning training method and device for a large language model, and the method comprises the steps: inputting a first input text into the large language model, enabling the large language model to process the first input text based on a current strategy, and obtaining G output texts; respectively combining the G output texts with the first input text, and respectively inputting combination results into a trained reward model to obtain G reward values corresponding to the G output texts; the current strategy is updated based on a target function, the target function comprises a global penalty term, the global penalty term represents that a penalty value is applied only when the maximum value in the G reward values does not exceed a preset threshold value, and the penalty value is determined according to respective first output probabilities of the G output texts under the current strategy; therefore, the training of the large language model under the reinforcement learning framework is realized.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of artificial intelligence, and particularly relates to a reinforcement learning training method and device of a large language model. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) are increasingly widely used in natural language processing, dialogue generation, text summarization and other fields. However, the training and optimization of large language models still face many challenges, especially in the problem of policy optimization under the reinforcement learning (RL) framework. Reinforcement learning, as a method of optimizing policy by interacting with the environment, has shown great potential in improving the performance of large language models. Therefore, how to provide a training method for large language models under the reinforcement learning framework has become a problem to be solved. SUMMARY

[0003] One or more embodiments of the present specification provide a reinforcement learning training method and device of a large language model to train the large language model under the reinforcement learning framework.

[0004] According to a first aspect, a reinforcement learning training method of a large language model is provided, comprising:

[0005] inputting a first input text into a large language model, so that the large language model processes the first input text based on its current policy to obtain G output texts;

[0006] combining the G output texts with the first input text respectively, and inputting the combined results into a trained reward model respectively to obtain G reward values corresponding to the G output texts;

[0007] updating the current policy based on a target function, wherein the target function includes a global penalty term, the global penalty term represents that a penalty value is applied only when the maximum value of the G reward values does not exceed a preset threshold, and the penalty value is determined according to the first output probability of each of the G output texts under the current policy.

[0008] According to a second aspect, a reinforcement learning training device of a large language model is provided, comprising:

[0009] a first input module configured to input a first input text into a large language model, so that the large language model processes the first input text based on its current policy to obtain G output texts;

[0010] a second input module configured to combine the G output texts with the first input text respectively, and input the combination results into a trained reward model respectively to obtain G reward values corresponding to the G output texts;

[0011] an updating module configured to update the current policy based on a target function, wherein the target function comprises a global penalty term, and the global penalty term indicates that a penalty value is applied only when a maximum value in the G reward values does not exceed a preset threshold, and the penalty value is determined according to first output probabilities of the G output texts respectively under the current policy.

[0012] According to a third aspect, a computer readable storage medium is provided, which stores a computer program, and when the computer program is executed in a computer, the computer is caused to execute the method according to the first aspect.

[0013] According to a fourth aspect, a computing device is provided, which comprises a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method according to the first aspect.

[0014] The method and device for training a large language model provided by the embodiments of the present specification comprise inputting a first input text into a large language model, so that the large language model processes the first input text based on a current policy thereof to obtain G output texts; combining the G output texts with the first input text respectively, and inputting the combination results into a trained reward model respectively to obtain G reward values corresponding to the G output texts; and updating the current policy based on a target function, wherein the target function comprises a global penalty term, and the global penalty term indicates that a penalty value is applied only when a maximum value in the G reward values does not exceed a preset threshold, and the penalty value is determined according to first output probabilities of the G output texts respectively under the current policy. The training of the large language model under the reinforcement learning framework can be realized, and by setting the global penalty term, the penalty value is applied when the maximum value in the G reward values does not exceed the preset threshold, i.e., all the output texts do not match the first input text, so as to reduce the probability of the group of output texts being adopted, enhance the global consistency of the policy, and avoid the local optimization deviation. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1A The schematic diagram of the implementation framework of an embodiment disclosed in the present specification;

[0017] Figure 1B An implementation framework diagram for an embodiment disclosed in the present specification;

[0018] Figure 2 A flowchart diagram of a reinforcement learning training method for a large language model provided for an embodiment;

[0019] Figure 3 A schematic block diagram of a reinforcement learning training device for a large language model provided for an embodiment. DETAILED DESCRIPTION

[0020] The technical solutions of the embodiments of the present specification will be described in detail below with reference to the accompanying drawings.

[0021] In the present specification, a large language model (LLM) can also be referred to simply as a large model. A large language model is a natural language processing model based on deep learning technology, with a parameter order of magnitude usually reaching tens of billions to hundreds of billions or even higher, and with strong language understanding and generation capabilities. A large language model can use a Transformer architecture or its variants (such as GPT, BERT, etc.), which uses an attention mechanism to achieve global modeling of sequence data and can efficiently handle long-distance dependencies, thus performing well in natural language tasks. A large language model is pre-trained on a large-scale corpus to learn the statistical characteristics and semantic relevance of language, enabling it to have good generalization capabilities. The core capabilities of a large language model include but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Its usage methods usually include two modes: direct inference and fine-tuning. In the direct inference mode, users guide the large language model to generate specific output by designing prompts. The prompt can be a text form of task description or instruction to stimulate the semantic understanding and generation capabilities of the large language model. In the fine-tuning mode, the large language model is further trained on a small-scale dataset in a specific domain to optimize its performance on specific tasks. The powerful generalization capabilities and flexibility of a large language model make it an important tool in the field of artificial intelligence technology, providing efficient and accurate solutions for automated text generation and understanding.

[0022] In some embodiments, the large language model can also have the understanding and generation capabilities of other modalities (such as vision, audio, etc.) data, in which case the large language model can also be referred to as a multimodal large language model (MLLM). MLLMs provide a more rich and natural interactive experience by integrating text, images, sounds and other types of input and output. The core advantage of MLLMs is that they can handle and understand information from different modalities and integrate these information to complete complex tasks. For example, MLLMs can analyze a picture and generate descriptive text, or generate corresponding images according to the text description. This cross-modal understanding and generation capability makes MLLMs have wide application prospects in many fields.

[0023] It should be noted that the key technologies of the large language model can be referred to the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v16, publication time: March 11, 2025, publication link: https: / / doi.org / 10.48550 / arXiv.2303.18223). This specification does not repeat here.

[0024] The embodiments of the specification disclose a reinforcement learning training method and device of a large language model. First, the application scenario and technical concept of the method are introduced as follows:

[0025] As mentioned earlier, reinforcement learning, as a method of optimizing policy by interacting with the environment, has shown great potential in improving the performance of large language models. Therefore, how to provide a training method for large language models under the reinforcement learning framework has become a problem to be solved.

[0026] In view of this, the inventors propose a reinforcement learning training method for large language models based on robust group relative policy optimization (RGRPO). Figure 1A An implementation scenario diagram according to an embodiment disclosed by the specification is shown. In this implementation scenario, the reinforcement learning training system of the large language model includes a large language model and a trained reward model. The reward model is used to score several output texts generated by the large language model to evaluate the matching degree between each output text and its input text, that is, to apply environmental evaluation to the large language model to optimize the policy, i.e. the parameters of the large language model, to guide the large language model to generate output texts with better quality and better matching with the input texts. Each output text is a text generated by the large language model processing the input text.

[0027] In the reinforcement learning training process of the large language model, a first input text is obtained, the first input text is input into the large language model to make the large language model process the first input text based on its current policy, and G output texts are obtained, as shown in Figure 1A The G output texts include: a first output text O1, a second output text O2,..., an i-th output text Oi,..., and a G-th output text OG. i G Then, the G output texts are combined with the first input text respectively, and the combined results are input into the trained reward model respectively to obtain G reward values corresponding to the G output texts; and the current policy is updated based on a target function, wherein the target function includes a global penalty term, the global penalty term indicates that a penalty value is applied only when the maximum value in the G reward values does not exceed a preset threshold, and the penalty value is determined according to the first output probability of each of the G output texts under the current policy.

[0028] In the above process, the training of the large language model under the reinforcement learning framework can be implemented, and by setting the global penalty term, the penalty value is applied when the maximum value in the G reward values does not exceed the preset threshold, i.e., all output texts do not match the first input text, to reduce the probability of adopting this group of output texts and enhance the global consistency of the policy, thereby avoiding the case of local optimization deviation.

[0029] In some possible examples, when the target function includes the aforementioned global penalty term, it can also include a first function term and a second function term.

[0030] The first function term is determined based on G advantage values corresponding to the G output texts; and for any i-th output text in the G output texts, the determination process of the i-th advantage value corresponding to the i-th output text includes: determining the i-th advantage value corresponding to the i-th output text according to the i-th reward value corresponding to the i-th output text, an average baseline of the reward value, and a standard deviation of the reward value.

[0031] The aforementioned second function term can also be referred to as a second KL divergence term, and the second function term is determined based on a third output probability and a first probability corresponding to each of the G output texts, wherein each third output probability is obtained by processing the corresponding output text and the first input text based on the reference policy of the large language model.

[0032] As shown in Figure 1B An implementation framework schematic diagram of one embodiment disclosed in the specification is shown. As shown in Figure 1B ​As shown, "input text sampling" is performed, that is, the first input text is obtained; then, for the collected first input text, "multi-output text generation" is performed, that is, the first input text is input into the large language model, so that the large language model processes the first input text based on its current strategy to obtain G output texts; then, "reward model evaluation" is performed on the G output texts, that is, the G output texts are combined with the first input text respectively, and the combined results are input into the trained reward model respectively to obtain G reward values corresponding to the G output texts, then a penalty value is applied only when the maximum value of the G reward values does not exceed a preset threshold, and the penalty value is determined according to the respective first output probabilities of the G output texts under the current strategy, so as to obtain a global penalty term.

[0033] Then, "average baseline of reward value and standard deviation of reward value updating" is performed, that is, the average baseline of reward value and the standard deviation of reward value are determined according to a specified calculation manner; then, for any ith output text in the G output texts, the ith advantage value corresponding to the ith output text is determined according to the ith reward value corresponding to the ith output text, the average baseline of reward value and the standard deviation of reward value, that is, "advantage value calculation" is performed; then, the first function term is obtained based on the respective advantage values of the output texts.

[0034] Then, the second function term is obtained based on the respective third output probabilities and the first probabilities of the G output texts; further, the target function including the global penalty term, the first function term and the second function term is obtained; the current strategy is updated based on the target function.

[0035] The above process integrates the shear constraint, that is, the first function term, the KL divergence, that is, the second function term, and the global penalty term, to form a hierarchical strategy control system, so as to better optimize and adjust the strategy of the large language model, and significantly improve the robustness, stability and data efficiency of the strategy optimization.

[0036] The reinforcement learning training method of the large language model provided in the present specification will be described in detail below in combination with specific embodiments.

[0037] Figure 2 A flowchart of the reinforcement learning training method of the large language model in an embodiment of the present specification is shown. The method is executed by an electronic device, which can be realized by any device, apparatus, platform, device cluster, etc. with computing and processing capabilities. In the reinforcement learning training process of the large language model, as shown in Figure 2 The method includes the following steps S210-S230:

[0038] In step S210, the first input text is input into the large language model, so that the large language model processes the first input text based on its current strategy to obtain G output texts.

[0039] In some possible examples, any text obtained from a training set used for training the large language model is taken as the first input text X, and then the first input text X is input into the large language model to make the large language model process the first input text based on its current strategy, to obtain G output texts corresponding to the first input text X.

[0040] In some possible examples, the large language model is used to provide a question and answer service, and accordingly, the first input text X can be a question text, and the G output texts O are G answer texts obtained by the large language model processing the first input text based on its current strategy.

[0041] In yet some possible examples, the large language model is used to provide a translation service, and accordingly, the first input text X can be a first language text to be translated, and the G output texts O are G second language texts obtained by the large language model processing the first input text based on its current strategy, where the first language and the second language are different languages.

[0042] In yet some possible examples, the large language model is used to provide an abstract generation service, and accordingly, the first input text X can be a text to be abstracted, and the G output texts O are G abstract texts obtained by the large language model processing the first input text based on its current strategy.

[0043] It can be understood that the embodiments of the present specification do not limit the specific service type provided by the large language model, and the large language model can be a large language model used to provide any text generation type service.

[0044] For example, the current strategy of the large language model can include a current parameter set of the large language model.

[0045] In step S220, the G output texts are combined with the first input text respectively, and the combined results are input into the trained reward model respectively to obtain G reward values corresponding to the G output texts.

[0046] In this step, the electronic device obtains, for any ith output text O i of the G output texts, a concatenation of the ith output text O i and the first input text X to obtain a concatenated text i, and then inputs the concatenated text i into the trained reward model to obtain a reward value r i corresponding to the ith output text O i . In this way, G reward values r corresponding to the G output texts are obtained.

[0047] The reward model has been pre-trained to score a plurality of output texts generated by the large language model to evaluate the matching degree between each output text and its input text. Among them, the greater the reward value r corresponding to the output text represents the more matched the output text and its input text. In some possible examples, the value range of the reward value r can be [0, 1].

[0048] Then, in step S230, the current policy is updated based on the target function, wherein the target function includes a global penalty term, and the global penalty term represents that a penalty value is applied only when the maximum value of the G reward values does not exceed the preset threshold, and the penalty value is determined according to the first output probability of each of the G output texts under the current policy.

[0049] In this step, the target function for updating the large language model includes a global penalty term, and the global penalty term represents that a penalty value is applied only when the maximum value of the G reward values does not exceed the preset threshold. The penalty value is determined according to the first output probability of each of the G output texts under the current policy. The first output probability of each of the G output texts is obtained by the large language model processing each output text and the first input text based on the current policy.

[0050] It can be understood that if the maximum value of the G reward values does not exceed the preset threshold, it can be considered that the G reward values indicate that the G output texts are all not matched with the first input text, and accordingly, a penalty value needs to be applied to expect the large language model to reduce the probability of generating the G output texts for the first input text X.

[0051] Exemplarily, the global penalty term can be represented by the following formula (1):

[0052]

[0053] Wherein, γ represents a hyperparameter for controlling the strength of the global penalty term; τ represents the preset threshold, I(·) represents an indicator function, which takes 1 when the condition in the parentheses is true, and 0 otherwise; π θ (O i represents the first output probability of the i-th output text O i generated by the large language model based on the current policy for the first input text X, and r i represents the i-th reward value of the i-th output text O i .

[0054] Exemplarily, when the value range of the reward value is [0, 1], the minimum value of the preset threshold can be 0, and the maximum value can be set according to the situation, and the maximum value is less than 1.

[0055] In this step, after obtaining the reward value r of each of the G output texts and the first output probability of each of the G output texts, the reward value and the first output probability of each of the G output texts are substituted into the formula of the global penalty term of the objective function to obtain the corresponding penalty value. Further, based on the objective function including at least the global penalty term, the current policy is updated.

[0056] In some possible examples, in order to achieve better optimization of the policy of the large language model, the objective function further includes a first function term, which is determined based on the G advantage values corresponding to the G output texts.

[0057] For example, the first function term J1 can be represented by the following formula (2):

[0058]

[0059] wherein E[·] represents a calculation of an expected value, represents a previous policy of the large language model based on the current policy old The fourth output probability of the i-th output text O i is obtained by the large language model based on the previous policy old processing the i-th output text O i and the first input text X; ∈ represents a clipping parameter, which is used to limit the step length of policy update and ensure the stability of policy update, and is usually set to a small constant, such as 0.2; represents when less than 1-∈, taking 1-∈, when greater than 1+∈, taking 1+∈; A i represents the i-th advantage value corresponding to the i-th output text O i .

[0060] In the case where the objective function further includes the first function term, the method can further include a determination process of the G advantage values corresponding to the G output texts, which includes:

[0061] For any i-th output text in the G output texts, the i-th advantage value corresponding to the i-th output text is determined according to the i-th reward value corresponding to the i-th output text, the average baseline of the reward value and the standard deviation of the reward value.

[0062] In the case where the objective function includes the first function term, the electronic device determines, for any ith output text in the G output texts, an ith advantage value corresponding to the ith output text according to an ith reward value corresponding to the ith output text, the average baseline of the reward values, and the standard deviation of the reward values. Similarly, the respective advantage values corresponding to the respective output texts are obtained, and then the respective advantage values can be substituted into the first function term to jointly calculate the global penalty term, and the objective function is calculated, and then the current policy is updated based on the objective function including the first function term and the global penalty term.

[0063] In some possible examples, the ith advantage value A i may be represented by the following formula (3):

[0064]

[0065] wherein baseline represents the average baseline of the reward values, and std(r) represents the standard deviation of the reward values.

[0066] In yet some possible examples, in order to ensure numerical stability in a low-variance scenario, ensure the stability and convergence speed of policy updating of a large language model, the foregoing process of determining the ith advantage value corresponding to the ith output text can include the following steps 11-12:

[0067] In step 11, if the standard deviation std(r) of the reward values is greater than a preset standard deviation threshold, the ith advantage value is calculated according to the ith reward value, the average baseline, and the standard deviation of the reward values. In step 12, if the standard deviation std(r) of the reward values is not greater than the preset standard deviation threshold, the ith advantage value is determined to be 0.

[0068] In some examples, the preset standard deviation threshold δ can be set to a small value, for example, 1*10 -3 to prevent the standard deviation of the reward values from being too small.

[0069] In the above examples, the ith advantage value A i may be represented by the following formula (4):

[0070]

[0071] From the above formulas (3) and (4), it can be known that the ith advantage value is calculated by first calculating the average baseline and the standard deviation of the reward values. In the embodiments of the present specification, the average baseline baseline of the reward values and the standard deviation std(r) of the reward values can be calculated in various ways. The calculation methods of the average baseline baseline of the reward values and the standard deviation std(r) of the reward values are introduced as follows.

[0072] In some possible examples, the aforementioned average baseline baseline is a mean value of the aforementioned G reward values r. The aforementioned standard deviation std(r) of the reward values is a standard deviation of the aforementioned G reward values r. That is, in this example, a mean value of the G reward values r is calculated to obtain the aforementioned average baseline baseline, and a standard deviation of the G reward values r is calculated to obtain the aforementioned standard deviation std(r) of the reward values, so as to calculate the average baseline of the reward values and the standard deviation of the reward values based on the G output texts corresponding to the first input text.

[0073] Considering that the average baseline of the reward values and the standard deviation of the reward values are calculated based on only the reward values of the G output texts corresponding to the first input text, in the case where the reward values of the G output texts fluctuate greatly, the average baseline of the reward values and the standard deviation of the reward values may fluctuate greatly and be unstable, and accordingly, the advantage values of the output texts fluctuate greatly, which further affects the stability of the policy update. Moreover, based on only the reward values of the G output texts corresponding to the first input text, the average baseline of the reward values and the standard deviation of the reward values are calculated based on a small sample, and the performance of the global policy cannot be well reflected.

[0074] To solve the above problems, the present specification embodiment provides a manner of calculating the average baseline baseline of the reward values and the standard deviation std(r) of the reward values as follows.

[0075] Specifically, in some possible examples, the process of calculating the average baseline baseline of the reward values and the standard deviation std(r) of the reward values can include the following steps 21-24.

[0076] In step 21, based on the G reward values, a first reward mean value and a first reward standard deviation are determined. In this step, a mean value is calculated according to the G reward values of the G output texts to obtain the first reward mean value, and a standard deviation is calculated based on the G reward values of the G output texts to obtain the first reward standard deviation.

[0077] Then, in step 22, a current baseline and a current standard deviation are obtained, which are respectively calculated based on a mean value and a standard deviation of reward values of a plurality of groups of output texts corresponding to a plurality of input texts before the first input text. The plurality of groups of output texts corresponding to the plurality of input texts before the first input text are all generated by the large language model based on the strategy of the large language model processing the corresponding input texts. The plurality of input texts all belong to the aforementioned training set.

[0078] In this step, the electronic device can obtain the current baseline baseline and the current standard deviation std(r) of the reward values from a specified storage area. old and the current standard deviation std(r) old .

[0079] Then, in step 23, the current baseline is updated based on the first reward mean value to obtain an updated baseline as the average baseline. In this step, the updated baseline, which can also be referred to as a moving baseline, is obtained based on a weighted combination of the current baseline and the first reward mean value, and then the updated baseline is taken as the average baseline of the reward value.

[0080] Exemplarily, the updated baseline baseline2 can be represented by the following formula (5):

[0081] baseline2 = a x baseline old + (1-a) x baseline1; (5)

[0082] wherein baseline1 represents the first reward mean value, and a is a preset weight value.

[0083] In step 24, the current standard deviation is updated based on the first reward standard value to obtain an updated standard deviation as the standard deviation of the reward value. In this step, the updated standard deviation, which can also be referred to as a moving standard deviation, is obtained based on a weighted combination of the current standard deviation and the first reward standard deviation, and then the updated standard deviation is taken as the standard deviation of the reward value.

[0084] The updated standard deviation std(r)2 can be represented by the following formula (6):

[0085] std(r)2 = a x std(r) old + (1-a) x std(r)1; (6)

[0086] wherein std(r)1 represents the first reward standard deviation.

[0087] In the above example, after obtaining the updated baseline baseline2 and the updated standard deviation std(r)2, the updated baseline baseline2 can be taken as the average baseline baseline, and the updated standard deviation std(r)2 can be taken as the standard deviation std(r) of the reward value.

[0088] Subsequently, the electronic device will also store the obtained average baseline baseline and the obtained standard deviation std(r) of the reward value to the specified storage area to replace the current baseline baseline old and the current standard deviation std(r) old .

[0089] In the above process, the average baseline of the reward value and the standard deviation of the reward value are jointly determined based on the mean and standard deviation of the reward values of the G output texts based on the first input text, and the current baseline and the current standard deviation calculated based on the mean and standard deviation of the reward values of the several groups of output texts corresponding to the several input texts before the first input text, which can better enhance the stability and accuracy of the baseline estimation, reduce the influence of a single input text on the advantage value calculation, and better ensure the stability of the advantage value calculation and the stability of the policy update of the large language model.

[0090] In some possible examples, the process of calculating the average baseline baseline of the reward value and the standard deviation std(r) of the reward value can include the following steps 31-34:

[0091] In step 31, the reward values of the preceding output texts are obtained, wherein the preceding output texts include the several groups of output texts corresponding to the several input texts before the first input text, and the output texts before the i-th output text in the G output texts.

[0092] In this step, the reward values of the preceding output texts, i.e., the several groups of output texts corresponding to the several input texts before the first input text, and the output texts before the i-th output text in the G output texts, can be stored in a specified storage area of the electronic device, and accordingly, the electronic device can obtain the reward values of the preceding output texts from the specified storage area. The first input text and the several input texts before the first input text belong to the training set for the reinforcement learning training of the large language model.

[0093] It can be understood that after obtaining the G reward values of the G output texts of the first input text, the G reward values of the G output texts of the first input text are also stored in the specified storage area for subsequent use in updating the process of calculating the average baseline baseline and the standard deviation std(r) of the reward value for the input texts after the first output text.

[0094] Then, in step 32, the mean and the standard deviation are calculated based on the i-th reward value and the reward values of the preceding output texts, to obtain the first baseline and the first standard deviation. In this step, the electronic device calculates the mean based on the i-th reward value and the reward values of the preceding output texts to obtain the first baseline baseline3, which can also be referred to as obtaining a global baseline; and calculates the standard deviation based on the i-th reward value and the reward values of the preceding output texts to obtain the first standard deviation std(r)3, which can also be referred to as obtaining a global standard deviation.

[0095] For example, the first baseline baseline3 can be represented by the following formula (7):

[0096]

[0097] wherein, K represents the i-th reward value, the reward values of the output texts ranked before the i-th output text in the G output texts, and the reward values of the aforementioned preceding output texts.

[0098] In this step, based on the obtained multiple reward values, i.e., the i-th reward value, the reward values of the output texts ranked before the i-th output text in the G output texts, and the reward values of the aforementioned preceding output texts, the first baseline and the first standard deviation are jointly determined, which can better enhance the stability and accuracy of baseline estimation, reduce the influence of a single input text on advantage value calculation, better guarantee the stability of advantage value calculation, guarantee the stability of policy update of a large language model, and realize efficient reuse of input texts. It is also applicable to resource-constrained scenarios, i.e., fewer input texts or high annotation cost, and non-Markov decision scenarios.

[0099] It can be understood that the reward model can be obtained by pre-training based on the reward value labels of each output text in the output text group corresponding to the input text and each output text in the output text group, which is obtained by processing the input text by other large language models. The annotation cost mentioned above can refer to the cost of labeling the reward values of each output text in the output text group of each input text when training the reward model.

[0100] Then, in step 33, the average baseline and the standard deviation of the reward value are obtained based on the first baseline and the first standard deviation, respectively. In this step, after obtaining the first baseline and the first standard deviation, the average baseline is obtained based on the first baseline, and the standard deviation of the reward value is obtained based on the first standard deviation.

[0101] In some possible examples, the first baseline baseline3 can be directly determined as the average baseline baseline; and the first standard deviation std(r)3 can be determined as the standard deviation std(r) of the reward value.

[0102] In yet some possible examples, the average baseline can be obtained by jointly using the aforementioned moving baseline and global baseline, and the standard deviation of the reward value can be obtained by jointly using the aforementioned moving standard deviation and global standard deviation. Accordingly, in step 33, steps 331-335 can be included.

[0103] In step 331, the first reward mean and the first reward standard deviation are determined based on the G reward values.

[0104] At step 332, a current baseline and a current standard deviation are obtained based on the mean and the standard deviation of the reward values of the plurality of sets of output texts corresponding to the plurality of input texts before the first input text, respectively;

[0105] At step 333, the current baseline is updated based on the first reward mean, and the current standard deviation is updated based on the first reward standard deviation, to obtain an updated baseline and an updated standard deviation.

[0106] The implementation principles of steps 331-333 are the same as those of steps 21-24, and the implementation process can refer to the implementation process of steps 21-24, which will not be repeated here.

[0107] At step 334, an average baseline is obtained based on a weighted combination of the first baseline and the updated baseline. In this step, the average baseline can be obtained based on the first baseline and the corresponding weight value and the weight value corresponding to the aforementioned updated baseline, i.e., the moving baseline.

[0108] For example, the average baseline baseline can be represented by the following formula (8):

[0109] baseline=a(α×baseline old +(1-α)×baseline1)+b(baseline2); (8)

[0110] Wherein, a and b are the weight values corresponding to the first baseline and the aforementioned updated baseline, respectively, and the sum of a and b is 1.

[0111] At step 335, a standard deviation of the reward value is obtained based on a weighted combination of the first standard deviation and the updated baseline. In this step, the standard deviation of the reward value can be obtained based on the first standard deviation and the corresponding weight value and the weight value corresponding to the aforementioned updated standard deviation, i.e., the moving standard deviation.

[0112] For example, the standard deviation of the reward value std(r) can be represented by the following formula (9):

[0113] std(r)=a×(α×std(r) old +(1-α)×std(r)1)+b×std(r)2,; (9)

[0114] In the above example, the moving baseline and the global baseline are mixed, and the moving standard deviation and the global standard deviation are mixed, to better balance the advantage value and balance the immediate strategy performance and long-term stability of the large language model.

[0115] In some possible examples, the objective function further includes a second function term, which can also be referred to as a second KL divergence term, and the second function term is determined based on G sets of third output probabilities each corresponding to an output text and the first probability, where each third output probability is obtained by the large language model based on processing the corresponding output text and the first input text according to the reference strategy.

[0116] The second function term J2 can be represented by the following formula (10):

[0117] J2 = -β · KL (π θ ||π ref ) ; (10)

[0118] where β represents a hyperparameter for controlling the penalty strength of the KL divergence term, KL (π θ ||π ref ) represents the KL divergence between the current strategy of the large language model and the reference strategy, the reference strategy being the strategy initialized by the large language model, i.e., the strategy that has not been adjusted, π ref represents a set of third output probabilities of the G output texts generated by the large language model based on processing the first input text X according to the reference strategy, which can also be referred to as a probability distribution of the G third output probabilities under the reference strategy, π θ represents a set of first output probabilities of the G output texts generated by the large language model based on processing the first input text X according to the current strategy, which can also be referred to as a probability distribution of the G first output probabilities under the current strategy.

[0119] The second function term can avoid the strategy update step process of the large language model to some extent and improve the stability of the strategy update.

[0120] As can be seen from the above, in some possible examples, the objective function may include a global penalty term, a first function term, and a second function term. Correspondingly, the objective function may be represented by the following formula (11):

[0121]

[0122] In the above examples, the objective function integrates the shear constraint, i.e., the first function term, the KL divergence, i.e., the second function term, and the global penalty term, to form a hierarchical strategy control system, to achieve better optimization and adjustment of the strategy of the large language model, and significantly improve the robustness, stability, and data efficiency of the strategy optimization.

[0123] The process of updating the current strategy based on the objective function will be described below.

[0124] In some possible examples, step S230 can include steps 41-43:

[0125] In step 41, the gradient value of the target function is determined. In this step, the target function is differentiated to obtain the gradient value of the target function. Wherein, the gradient value g can be represented by the following formula (12):

[0126]

[0127] Wherein, represents the derivative symbol.

[0128] Then, in step 42, the updated first strategy is determined based on the gradient value and the current strategy. In this step, the gradient ascent method is used to determine the updated first strategy based on the gradient value and the current strategy.

[0129] Exemplarily, the updated first strategy θ1 can be represented by the following formula (13):

[0130] θ1 = θ + ηg; (13)

[0131] Wherein, θ1 represents the first strategy, i.e. the parameter values of each parameter in it, θ represents the current strategy, i.e. the parameter values of each parameter in it, and η is a preset value.

[0132] Next, in step 43, a new current strategy is determined based on the first strategy.

[0133] In some possible examples, the first strategy θ1 can be directly determined as the new current strategy θ new , and accordingly, the current strategy of the large language model is adjusted to the new current strategy.

[0134] In some possible examples, in order to better ensure the stability of the policy update of the large language model, in step 43, step 431 can be included:

[0135] In step 431, if the first KL divergence is not greater than the specified divergence threshold, the first strategy is taken as the new current strategy, wherein the first KL divergence is determined based on G second output probabilities and G third output probabilities of the G output texts, the G second output probabilities are obtained by the large language model based on the first strategy processing the corresponding output texts and the first input text, and the G third output probabilities are obtained by the large language model based on the reference strategy processing the corresponding output texts and the first input text.

[0136] In this example, after obtaining the first strategy, the KL divergence between the first strategy and the reference strategy is calculated, specifically, the first KL divergence D KL between the first strategy and the reference strategy is determined based on the G second output probabilities and the G third output probabilities of the G output texts.

[0137] The first KL divergence is used to verify whether updating the current policy to the first policy will cause a large policy update step. If the first KL divergence is not greater than the specified divergence threshold, it indicates that the policy update is relatively stable, and accordingly, the first policy is taken as the new current policy, i.e., the new current policy θ new =θ1. If the first KL divergence is greater than the specified divergence threshold, it indicates that the policy update is relatively unstable, and it is necessary to determine a new current policy.

[0138] In some possible examples, at step 43, steps 432-433 can also be included:

[0139] At step 432, if the first KL divergence is greater than the specified divergence threshold, a scaling coefficient is determined based on the first KL divergence and a preset coefficient calculation manner, where the scaling coefficient is used to scale the gradient value.

[0140] In some possible examples, a preset coefficient calculation formula can be pre-stored. If the first KL divergence is greater than the specified divergence threshold, the scaling coefficient s is determined based on the first KL divergence and the specified divergence threshold based on the preset scaling coefficient calculation formula. The preset coefficient calculation formula can be represented by the following formula (14):

[0141] s = sqrt( z / D KL ); (14)

[0142] Wherein, sqrt(.) represents a square root function, and z represents the specified divergence threshold, for example, z can take a value of 0.5.

[0143] In some examples, the aforementioned scaling coefficient s can also be set as: s=z / D KL .

[0144] In the above manner, the scaling coefficient is determined, and then the current policy is updated using the scaled gradient value, which can theoretically make the KL divergence between the updated policy and the reference policy not greater than the specified divergence threshold.

[0145] At step 433, the current policy is adjusted based on the gradient value and the scaling coefficient to determine a new current policy. In this step, the scaling coefficient can be first used to scale the gradient value to obtain a scaled gradient value, and then the gradient ascent method is used to determine the new current policy based on the scaled gradient value and the current policy.

[0146] For example, the new current policy θ new can be represented by the following formula:

[0147] θ new =θ+sηg;

[0148] Wherein, s represents the scaling coefficient.

[0149] In yet some possible examples, the aforementioned calculation of the scaling factor based on the first KL divergence and the preset coefficient can include: employing a dichotomy search method to determine the scaling factor s based on the first KL divergence and a specified divergence threshold.

[0150] For example, let d = ηg; if the first KL divergence is greater than the specified divergence threshold z, let s ∈ [0, 1], and search in this search interval to search for the scaled coefficient s adjusted strategy θmid and the reference strategy ref between the two. as close as possible to the specified divergence threshold z; wherein the process of determining the scaling factor s can be implemented by the following code:

[0151] while True: (# infinite loop until the exit condition is met)

[0152] s mid = (s l + s r ) / 2; (# calculate the midpoint of the current search interval [s l , s r ] as the trial coefficient, where s l represents the left end of the current search interval, and s r represents the right end of the current search interval)

[0153] θ mid = θ + s mid · d; (# update the current strategy θ along the direction d with the current trial coefficient s mid to obtain a new strategy θ mid );

[0154] (# calculate the KL divergence value between the new strategy θ mid and the reference strategy ref )

[0155] (# if the KL divergence value is close to the specified divergence threshold z (e.g. within the allowed range), the exit condition is met and the search is stopped)

[0156] (# if the KL divergence value is greater than the specified divergence threshold z, update the right end s r to the midpoint s mid ; if the KL divergence value is significantly lower than the specified divergence threshold z (the specified divergence threshold z is significantly lower than the KL divergence value​ If the difference exceeds the specified difference, the left endpoint s will be... l Updated to midpoint s mid ).

[0157] The scaling factor s is obtained through the binary search method described above, which makes the strategy π obtained after adjustment by the scaling factor s... θmid With reference strategy π ref KL divergence between Get as close as possible to the specified divergence threshold z. At this point, the scaling factor s is equal to the s when the exit condition is met during the binary search process described above. mid Accordingly, the new current strategy is the s that meets the exit condition during the binary search process described above. mid The corresponding strategy θ mid At this point, the KL divergence between the new current policy and the reference policy is not greater than the specified divergence threshold z.

[0158] This embodiment enables the training of a large language model within a reinforcement learning framework. By setting a global penalty term, a penalty value is applied when the maximum value among the G reward values ​​does not exceed a preset threshold, i.e., when all output texts do not match the first input text. This reduces the probability of the set of output texts being adopted, enhances the global consistency of the strategy, and avoids local optimization deviations.

[0159] Furthermore, it provides multiple methods for calculating the advantage value. Different methods yield advantage values ​​that ensure more stable policy updates for the large language model, guaranteeing stable policy updates and consistently improving the model's text generation capabilities, thus ensuring its reliability under various complex conditions. A preset standard deviation threshold is set so that when the standard deviation of the reward within a group is too small (i.e., the standard deviation of the reward value is not greater than the preset threshold), advantage updates are automatically disabled, setting the advantage value in the first function term to 0. This avoids the propagation of invalid gradient values ​​and improves the stability of the training process.

[0160] Integrating shearing constraints (the first function term), KL divergence (the second function term), and a global penalty term—a three-tiered constraint system—forms a hierarchical policy control system, preventing policy drift and enabling better optimization and adjustment of the policy of large language models. In this process, by introducing techniques such as global penalty terms, conditional thresholds (i.e., setting a preset standard deviation threshold), average baselines, global baselines, or a fusion of both baselines, and three-tiered constraints, the robustness, stability, and data efficiency of large language models in reinforcement learning training can be improved. This further enhances the stability of large language models in handling complex tasks, improves data utilization efficiency to reduce training costs, and supports diverse application scenarios (such as natural language processing and dialogue generation), providing an efficient optimization paradigm for reinforcement learning of large language models.

[0161] The foregoing steps S210-S230 are only described by taking a first input sample as an example to illustrate the reinforcement learning training process of the large language model. In some possible examples, multiple input samples can also be combined to perform reinforcement learning training on the large language model.

[0162] In some examples, the foregoing steps S210-S230 can be iterated multiple times to perform multiple reinforcement learning training on the large language model by using different input texts until a specified convergence condition is met to determine that the training of the large language model is completed. Then, the content generation service is provided by using the trained large language model. In the process of providing the content generation service by using the trained large language model, a to-be-processed text can be obtained, the to-be-processed text is processed based on the policy of the trained large language model to obtain generated text corresponding to the to-be-processed text, and the generated text can be the text with the highest corresponding output probability.

[0163] The above describes specific embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different than the order in the embodiments, and still achieve the desired result. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order shown or sequentially, to achieve the desired results. In certain implementations, multitasking and parallel processing are also possible, or can be advantageous.

[0164] Corresponding to the above method embodiments, the embodiments of the present specification provide a reinforcement learning training device 300 for a large language model, a schematic block diagram of which is shown in Figure 3 The reinforcement learning training device 300 for the large language model comprises:

[0165] A first input module 310 is configured to input a first input text into a large language model, so that the large language model processes the first input text based on a current policy thereof to obtain G output texts.

[0166] A second input module 320 is configured to combine the G output texts with the first input text respectively, and input the combined results into a trained reward model respectively to obtain G reward values corresponding to the G output texts.

[0167] A first updating module 330 is configured to update the current policy based on a target function, wherein the target function comprises a global penalty term, the global penalty term represents that a penalty value is applied only when a maximum value of the G reward values does not exceed a preset threshold, and the penalty value is determined according to first output probabilities of the G output texts respectively under the current policy.

[0168] In some possible examples, the target function further includes a first function term determined based on G advantage values corresponding to the G output texts; and the apparatus further includes:

[0169] a first determining module (not shown in the figure) configured to, for any ith output text in the G output texts, determine an ith advantage value corresponding to the ith output text according to an ith reward value corresponding to the ith output text, an average baseline of reward values, and a standard deviation of reward values.

[0170] In some possible examples, the average baseline is a mean value of the G reward values.

[0171] In some possible examples, the first determining module is specifically configured to, if the standard deviation is greater than a preset standard deviation threshold, calculate the ith advantage value according to the ith reward value, the average baseline, and the standard deviation.

[0172] If the standard deviation is not greater than the preset standard deviation threshold, the ith advantage value is determined to be 0.

[0173] In some possible examples, the apparatus further includes:

[0174] a second determining module (not shown in the figure) configured to determine a first reward mean value and a first reward standard deviation based on the G reward values.

[0175] a first obtaining module (not shown in the figure) configured to obtain a current baseline and a current standard deviation, which are respectively calculated based on a mean value and a standard deviation of reward values of a plurality of sets of output texts corresponding to a plurality of input texts before the first input text.

[0176] a second updating module (not shown in the figure) configured to update the current baseline based on the first reward mean value to obtain an updated baseline as the average baseline.

[0177] a third updating module (not shown in the figure) configured to update the current standard deviation based on the first reward standard value to obtain an updated standard deviation as the standard deviation of reward values.

[0178] In some possible examples, the apparatus further includes a third obtaining module (not shown in the figure) configured to obtain reward values of preceding output texts, wherein the preceding output texts include a plurality of sets of output texts corresponding to a plurality of input texts before the first input text, and output texts in the G output texts that are arranged before the ith output text.

[0179] a first obtaining module (not shown in the figure) configured to, according to the ith reward value and the reward values of the preceding output texts, respectively perform mean value calculation and standard deviation calculation to obtain a first baseline and a first standard deviation.

[0180] The second obtaining module (not shown in the figure) is configured to obtain the average baseline and the standard deviation of the reward value based on the first baseline and the first standard deviation, respectively.

[0181] In some possible examples, the second obtaining module is specifically configured to determine a first reward mean and a first reward standard deviation based on the G reward values.

[0182] The current baseline and the current standard deviation are obtained based on the mean and the standard deviation of the reward values of the plurality of sets of output texts corresponding to the plurality of input texts before the first input text.

[0183] The current baseline is updated based on the first reward mean, and the current standard deviation is updated based on the first reward standard value, to obtain an updated baseline and an updated standard deviation.

[0184] The average baseline is obtained based on a weighted combination of the first baseline and the updated baseline.

[0185] The standard deviation of the reward value is obtained based on a weighted combination of the first standard deviation and the updated baseline.

[0186] In some possible examples, the first updating module 330 includes: a first determining unit (not shown in the figure) configured to determine a gradient value of the target function; a second determining unit (not shown in the figure) configured to determine an updated first policy based on the gradient value and the current policy; and a third determining unit (not shown in the figure) configured to determine a new current policy based on the first policy.

[0187] In some possible examples, the third determining unit is specifically configured to, if a first KL divergence is less than a specified divergence threshold, take the first policy as the new current policy, where the first KL divergence is determined based on G second output probabilities and G third output probabilities of the G output texts, the G second output probabilities are obtained by the large language model based on the first policy processing the corresponding output texts and the first input text, and the G third output probabilities are obtained by the large language model based on its reference policy processing the corresponding output texts and the first input text.

[0188] In some possible examples, the third determining unit is further configured to, if the first KL divergence is not less than the specified divergence threshold, determine a scaling coefficient based on the first KL divergence and a preset coefficient calculation manner, where the scaling coefficient is used to scale the gradient value; and adjust the current policy based on the gradient value and the scaling coefficient to determine the new current policy.

[0189] The device embodiments correspond to the method embodiments, and specific descriptions can be referred to the descriptions of the method embodiments, which will not be repeated here. The device embodiments are based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. Specific descriptions can be referred to the corresponding method embodiments.

[0190] The embodiments of the present specification also provide a computer-readable storage medium, which stores a computer program. When the computer program is executed in a computer, the computer program causes the computer to execute the reinforcement learning training method of the large language model provided by the present specification.

[0191] The embodiments of the present specification also provide a computing device, which includes a memory and a processor. The memory stores executable code, and the processor executes the executable code to implement the reinforcement learning training method of the large language model provided by the present specification.

[0192] Each of the embodiments in the present specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments mainly describes the differences from other embodiments. In particular, the storage medium and the computing device embodiments are basically similar to the method embodiments, and thus are described more simply. The relevant parts can be referred to the descriptions of the method embodiments.

[0193] Those skilled in the art should realize that the functions described in the above one or more examples can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium.

[0194] The above specific embodiments further specifically describe the purposes, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above descriptions are only specific embodiments of the present application, and are not used to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present application shall be included in the protection scope of the present application.

Claims

1. A reinforcement learning training method for a large language model, comprising: The first input text is input into the large language model, so that the large language model processes the first input text based on its current strategy to obtain G output texts; The G output texts are combined with the first input text respectively, and the combined results are input into the trained reward model to obtain the G reward values ​​corresponding to the G output texts. The current policy is updated based on the objective function, wherein the objective function includes a global penalty term, which means that a penalty value is applied only when the maximum value among the G reward values ​​does not exceed a preset threshold, and the penalty value is determined according to the first output probability of each of the G output texts under the current policy.

2. The method as described in claim 1, wherein, The objective function further includes a first function term, which is determined based on G advantage values ​​corresponding to G output texts; the method further includes: For any i-th output text among the G output texts, the i-th advantage value corresponding to the i-th output text is determined based on the i-th reward value corresponding to the i-th output text, the average baseline of the reward value, and the standard deviation of the reward value.

3. The method as described in claim 2, wherein, The average baseline is the mean of the G reward values.

4. The method of claim 2, wherein, Determining the i-th advantage value corresponding to the i-th output text includes: If the standard deviation is greater than a preset standard deviation threshold, the i-th advantage value is calculated based on the i-th reward value, the mean baseline, and the standard deviation. If the standard deviation is not greater than the preset standard deviation threshold, the i-th advantage value is determined to be 0.

5. The method of claim 2, further comprising: Based on the G reward values, determine the mean of the first reward and the standard deviation of the first reward; The current baseline and current standard deviation are obtained, which are calculated based on the mean and standard deviation of the reward values ​​of several sets of output texts corresponding to several input texts preceding the first input text. The current baseline is updated based on the first average reward to obtain the updated baseline, which is used as the average baseline. The current standard deviation is updated based on the first reward standard value to obtain the updated standard deviation, which is then used as the standard deviation of the reward value.

6. The method of claim 2, further comprising: Obtain the reward value of the preceding output text, wherein the preceding output text includes several sets of output texts corresponding to several input texts preceding the first input text, and the output texts in the G output texts that are ordered before the i-th output text; Based on the i-th reward value and the reward value of the preceding output text, the mean and standard deviation are calculated respectively to obtain the first baseline and the first standard deviation; The average baseline and the standard deviation of the reward value are obtained based on the first baseline and the first standard deviation, respectively.

7. The method of claim 6, wherein, The process of obtaining the standard deviation of the mean baseline and the reward value includes: Based on the G reward values, determine the mean of the first reward and the standard deviation of the first reward; The current baseline and current standard deviation are obtained, which are calculated based on the mean and standard deviation of the reward values ​​of several sets of output texts corresponding to several input texts preceding the first input text. The current baseline is updated based on the first reward mean, and the current standard deviation is updated based on the first reward standard value, to obtain the updated baseline and the updated standard deviation; The average baseline is obtained based on a weighted combination of the first baseline and the updated baseline; The standard deviation of the reward value is obtained based on a weighted combination of the first standard deviation and the updated baseline.

8. The method according to any one of claims 1-7, wherein, The step of updating the current policy based on the objective function includes: Determine the gradient value of the objective function; Based on the gradient value and the current policy, determine the updated first policy; Based on the first strategy, a new current strategy is determined.

9. The method of claim 8, wherein, The determination of the new current strategy includes: If the first KL divergence is not greater than a specified divergence threshold, the first strategy is adopted as the new current strategy. The first KL divergence is determined based on the G second output probabilities and G third output probabilities of each of the G output texts. The G second output probabilities are obtained by the large language model processing the corresponding output text and the first input text based on the first strategy. The G third output probabilities are obtained by the large language model processing the corresponding output text and the first input text based on its reference strategy.

10. The method of claim 9, further comprising: If the first KL divergence is greater than a specified divergence threshold, a scaling factor is determined based on the first KL divergence and a preset coefficient calculation method, wherein the scaling factor is used to scale the gradient value; The current policy is adjusted based on the gradient value and the scaling factor to determine a new current policy.

11. A reinforcement learning training device for a large language model, comprising: The first input module is configured to input the first input text into the large language model, so that the large language model processes the first input text based on its current strategy to obtain G output texts. The second input module is configured to combine the G output texts with the first input text respectively, and input the combined results into the trained reward model to obtain the G reward values ​​corresponding to the G output texts; The update module is configured to update the current policy based on an objective function, wherein the objective function includes a global penalty term, which means that a penalty value is applied only when the maximum value among the G reward values ​​does not exceed a preset threshold, and the penalty value is determined according to the first output probability of each of the G output texts under the current policy.

12. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-10.

Images