Large language model reinforcement learning training method, system, terminal and storage medium

Abstract

This application discloses a method, system, terminal, and storage medium for training large language models using reinforcement learning, relating to the field of large language model technology. The method includes: acquiring a training question-and-answer dataset, including training questions, standard answers, and answer formats; obtaining the backbone answer sequence of the training questions using the large language model to be trained and using it as the answer sequence to be processed; performing branch processing on the answer sequence to be processed to obtain branch answer sequences; identifying high-entropy points during branch processing and generating the next layer's answer sequence to be processed; backtracking and concatenating the branch answer sequences to obtain the concatenated answer sequence corresponding to the backbone answer sequence; calculating the normalized reward function value of the concatenated answer sequence; calculating the path weight of the high-entropy points; segmenting the concatenated answer sequence based on the high-entropy points; determining the dominance function value of the sequence segment based on the path weight of the high-entropy points and the normalized reward function value to obtain the dominance function value of the concatenated answer sequence; and updating the gradient of the large language model. This can improve the model training effect.

Description

Technical Field

[0001] This application relates to the field of large language model technology, and in particular to a large language model reinforcement learning training method, system, terminal and storage medium. Background Technology

[0002] With the development of science and technology, the application of large language models is becoming increasingly widespread. Specifically, large language models (LLMs) based on the Transformer architecture and driven by massive amounts of data have made rapid progress in natural language understanding, generation, and cross-modal reasoning, and are widely used in the field of artificial intelligence.

[0003] Large language models typically rely on reinforcement learning training methods, such as Direct Preference Optimization (DPO) or Group Relative Policy Optimization (GRPO). However, these algorithms suffer from insufficient exploration and are prone to getting trapped in local optima. Therefore, the problem with existing technologies is that reinforcement learning training schemes for large language models lack sufficient exploration, which is detrimental to improving model training performance.

[0004] Therefore, the relevant technologies still need to be improved and developed. Summary of the Invention

[0005] The main purpose of this application is to provide a reinforcement learning training method, system, terminal and storage medium for large language models, aiming to solve the technical problem that the reinforcement learning training schemes for large language models are not sufficiently exploratory and are not conducive to improving the training effect of the model.

[0006] To achieve the above objectives, the first aspect of this application provides a reinforcement learning training method for a large language model, wherein the method includes: Obtain the training question-and-answer dataset, wherein each training question-and-answer data in the training question-and-answer dataset includes the training question, the standard answer, and the answer format; Using the large language model to be trained, obtain multiple core answer sequences corresponding to each of the above training questions; The above-mentioned main response sequence is used as the response sequence to be processed. At least one layer of branching is performed on the response sequence to be processed until the branching process ends to obtain the branch response sequence. The branching process includes determining the high-entropy point based on the log probability corresponding to each word in the response sequence to be processed, and generating the next layer of response sequence to be processed based on the high-entropy point. The above branch answer sequences are backtracked and spliced ​​to obtain the spliced ​​answer sequences corresponding to each of the above main answer sequences, and the spliced ​​answer sequences corresponding to the same training question are taken as a sequence data group. For each of the above sequence data sets, the normalized reward function value corresponding to each of the above spliced ​​answer sequences in the above sequence data set is calculated based on all the spliced ​​answer sequences in the above sequence data set, the standard answer and answer format corresponding to the above sequence data set. For each of the aforementioned high-entropy points, the corresponding high-entropy point path weight is calculated based on the distance between the high-entropy point and the end position of the sequence in its respective spliced ​​response sequence. The spliced ​​response sequence is segmented based on the high-entropy points. Based on the high-entropy point path weights and the normalized reward function value, the advantage function value corresponding to each sequence segment is determined to obtain the advantage function value corresponding to each of the aforementioned spliced ​​response sequences. Based on the advantage function value corresponding to the above-mentioned concatenated answer sequence, the gradient of the above-mentioned large language model to be trained is updated to obtain the updated large language model.

[0007] Optionally, obtaining the training question-answering dataset as described above includes: Obtain the initial dataset, wherein one initial data point in the initial dataset includes the training question, the standard answer, and the answer format; For each training question, multiple samples are taken using the large language model to be trained. Based on the answers obtained from each sample, the standard answers, and the answer formats, the correct answer rate for the training question is determined. Based on the correct answer rate corresponding to each of the above training questions, and the correct answer range limited by the preset minimum and maximum correct answer thresholds, the target training question is determined, and the initial data corresponding to the target training question is used as the above training question and answer data to obtain the above training question and answer dataset.

[0008] Optionally, the above-mentioned large language model to be trained obtains multiple backbone answer sequences corresponding to each of the above-mentioned training questions, including: For each of the above training questions, multiple samplings are performed using the large language model to be trained to obtain multiple backbone answer sequences corresponding to each of the above training questions.

[0009] Optionally, the above-mentioned branching process on the above-mentioned answer sequence to be processed, at least one level, until the branching process ends to obtain the branched answer sequence, includes: The negative value of the log probability corresponding to each word in the above-mentioned unprocessed response sequence is used as the entropy value corresponding to the above-mentioned word. Based on the above entropy values ​​and the preset entropy threshold, high entropy points are determined from the above sequence of answers to be processed; For each high-entropy point, based on the training question corresponding to the high-entropy point and the cumulative generated part before the high-entropy point in the unprocessed answer sequence to which the high-entropy point belongs, the model resampled input corresponding to the high-entropy point is constructed. For each model, the input is resampled, and the large language model to be trained is sampled at least once. The sampled answers are used as branch answer sequences. If any preset branch processing termination condition is met, the hierarchical processing of the branch answer sequence ends. Otherwise, the branch answer sequence is used as a new answer sequence to be processed, and the process returns to the step of taking the negative value of the log probability corresponding to each word in the answer sequence to be processed and using it as the entropy value corresponding to the word. The preset branch processing termination condition includes that there are no high entropy points in the sampled answers and that the number of branch processing layers reaches the preset upper limit value.

[0010] Optionally, the above-mentioned backtracking and splicing process on the above-mentioned branch answer sequences to obtain the spliced ​​answer sequences corresponding to each of the above-mentioned main answer sequences includes: For each of the above-mentioned branch answer sequences, the above-mentioned branch answer sequences are concatenated to the cumulative generation before the high-entropy point corresponding to the above-mentioned branch answer sequences to obtain the intermediate sequence; Take the above intermediate sequence as the current sequence, trace back to the high-entropy point corresponding to the previous branch, and then concatenate the above current sequence with the accumulated generation before the above high-entropy point, and update the above intermediate sequence. Return to the above steps, taking the intermediate sequence as the current sequence, tracing upwards to the high-entropy point corresponding to the previous branch, and concatenating the current sequence to the cumulative generation steps before the high-entropy point, until tracing back to the starting point of the main answer sequence, to obtain the complete answer to the above training question, and use it as the concatenated answer sequence.

[0011] Optionally, for each of the aforementioned high-entropy points, the corresponding high-entropy point path weight is calculated based on the distance between the high-entropy point and the end position of the sequence in its respective concatenated response sequence, including: For each high-entropy point, determine the number of subsequent high-entropy points between the high-entropy point and the end position of the sequence in each spliced ​​response sequence to which the high-entropy point belongs, and determine the path weight of the high-entropy point based on the minimum value of the number of subsequent high-entropy points.

[0012] Optionally, after updating the gradient of the large language model to be trained based on the advantage function value corresponding to the concatenated response sequence to obtain the updated large language model, the method further includes: The updated large language model is used as the current large language model to be trained. The process of obtaining multiple backbone answer sequences corresponding to each of the above training questions through the large language model to be trained is repeated until the preset iterative training termination condition is met. The iterative training termination condition includes the number of iterations reaching a preset iteration threshold.

[0013] A second aspect of this application provides a large language model reinforcement learning training system, wherein the system includes: The data acquisition module is used to acquire the training question-and-answer dataset, wherein a training question-and-answer data in the training question-and-answer dataset includes the training question, the standard answer, and the answer format; The data processing module is used to obtain multiple backbone answer sequences corresponding to each of the above training questions through the large language model to be trained. The branching module is used to take the above-mentioned main response sequence as the response sequence to be processed, and perform at least one layer of branching processing on the response sequence to be processed until the branching processing ends to obtain the branch response sequence. The branching processing includes determining the high-entropy point based on the log probability corresponding to each word in the response sequence to be processed, and generating the next layer of response sequence to be processed based on the high-entropy point. The backtracking splicing module is used to perform backtracking splicing processing on the above branch answer sequences to obtain the spliced ​​answer sequences corresponding to each of the above main answer sequences, and to take the spliced ​​answer sequences corresponding to the same training question as a sequence data group. The sequence processing module is used to calculate the normalized reward function value corresponding to each of the spliced ​​answer sequences in the sequence data group based on all the spliced ​​answer sequences in the sequence data group, the standard answer and the answer format corresponding to the sequence data group. The right-of-way calculation module is used to calculate the corresponding right-of-way for each of the above-mentioned high-entropy points based on the distance between the high-entropy point and the end position of the sequence in its respective spliced ​​answer sequence, to segment the spliced ​​answer sequence based on the high-entropy points, and to determine the advantage function value corresponding to each sequence segment based on the high-entropy point right-of-way and the normalized reward function value, so as to obtain the advantage function value corresponding to each of the above-mentioned spliced ​​answer sequences. The gradient update module is used to update the gradient of the large language model to be trained based on the advantage function value corresponding to the above concatenated answer sequence, so as to obtain the updated large language model.

[0014] A third aspect of this application provides a terminal, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements any of the steps of the large language model reinforcement learning training method described above.

[0015] A fourth aspect of this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the steps of the above-described large language model reinforcement learning training method.

[0016] As can be seen from the above, the present application provides a large language model reinforcement learning training method, which includes: acquiring a training question-and-answer dataset, wherein each training question-and-answer data in the training question-and-answer dataset includes a training question, a standard answer, and an answer format; acquiring multiple backbone answer sequences corresponding to each of the above training questions using the large language model to be trained; using the backbone answer sequences as answer sequences to be processed, performing at least one layer of branching processing on the answer sequences to be processed until the branching processing ends to obtain a branch answer sequence, wherein one layer of branching processing includes determining a high-entropy point based on the log probability corresponding to each word in the answer sequence to be processed, and generating the next layer of answer sequences to be processed based on the high-entropy point; performing backtracking concatenation processing on the branch answer sequences to obtain a concatenated answer sequence corresponding to each of the above backbone answer sequences, and concatenating it with the same... The concatenated response sequences corresponding to the training question are treated as a sequence data set. For each of these sequence data sets, the normalized reward function value corresponding to each concatenated response sequence is calculated based on all the concatenated response sequences, the standard response, and the response format corresponding to the sequence data set. For each high-entropy point, the corresponding high-entropy point weight is calculated based on the distance between the high-entropy point and the end position of the sequence in its concatenated response sequence. The concatenated response sequence is segmented based on the high-entropy point weight and the normalized reward function value. The dominance function value corresponding to each sequence segment is then determined to obtain the dominance function value corresponding to each of the concatenated response sequences. Based on the dominance function value corresponding to the concatenated response sequence, the gradient of the large language model to be trained is updated to obtain the updated large language model.

[0017] Thus, during reinforcement learning training of a large language model, multiple backbone response sequences are obtained for each training question. High-entropy points with high uncertainty in these sequences are selected as key points. Iterative expansion is performed based on these high-entropy points, and branch resampling is conducted by filtering high-entropy points using an entropy threshold to obtain branch response sequences. These are then backtracked and concatenated to obtain concatenated response sequences, thereby constructing data sets for each sequence. This allows for the determination of the normalized reward function value corresponding to each concatenated response sequence within the data set. Simultaneously, the path weights of the high-entropy points are calculated to determine the dominance function value of the sequence segments obtained from the high-entropy point partitioning. This determines the dominance function value corresponding to the concatenated response sequence. Based on the dominance function value corresponding to the concatenated response sequence, gradient updates are performed on the large language model to be trained, resulting in an updated large language model. In this way, during reinforcement learning training, exploration based on lexical uncertainty is achieved through high-entropy points, enhancing the sampling exploration and improving model training performance. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a large language model reinforcement learning training method provided in an embodiment of this application; Figure 2 This is a schematic diagram of a branching process provided in an embodiment of this application; Figure 3 This is a schematic diagram of the constituent modules of a large language model reinforcement learning training system provided in an embodiment of this application; Figure 4 This is a block diagram illustrating the internal structure of a terminal provided in an embodiment of this application. Detailed Implementation

[0020] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0021] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0022] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0023] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0024] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to classification to [the described condition or event]."

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0026] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0027] Currently, the application of large language models is becoming increasingly widespread, and their training is becoming increasingly important. In some application scenarios, supervised fine-tuning (SFT) can be used to train large language models. With the support of a large amount of downstream task data, supervised fine-tuning can quickly and directly improve the performance of large models on their corresponding tasks. However, this method gradually reaches its performance ceiling for large models. To seek further improvements, reinforcement learning (RL) can be used for training.

[0028] Large model alignment is a subfield of reinforcement learning, and RL has been proven to significantly improve the performance of large models on tasks such as mathematical reasoning and code generation. Compared to SFT, RL has a more complex structure and is more difficult to train, but it has the potential to break through the current performance limits and further enhance the reasoning ability and intelligence of large models.

[0029] In existing technologies, Proximal Policy Optimization (PPO) is a classic, robust, and widely validated reinforcement learning algorithm. Due to the complex architecture and high deployment costs of large models, reinforcement learning applications for large model alignment have been evolving towards simplified algorithm structures: PPO includes both the Critic and Reward models in addition to the Actor model; Direct Preference Optimization (DPO) simplifies the algorithm by omitting the Reward model by calculating the difference in preferences between positive and negative samples; Group Relative Preference Optimization (GRPO) further simplifies the algorithm by using a normalized inter-group reward assignment advantage function, which eliminates both the Critic and Reward models.

[0030] However, algorithms like DPO and GRPO, while simplifying the structure of traditional reinforcement learning algorithms, sacrifice some exploratory power and feedback accuracy. Furthermore, reinforcement learning applied to large model alignment suffers from problems such as excessively long sampling times, sparse rewards leading to difficulty in convergence, insufficient exploration resulting in getting trapped in local optima, and entropy collapse causing learning to stop. Therefore, how to compensate for the performance losses caused by simplification while maintaining computational feasibility and improving the learning efficiency and stability of algorithms in large model alignment has become a pressing technical problem to be solved in current technologies.

[0031] To address at least one of the aforementioned technical problems, this application proposes a reinforcement learning training method for a large language model. The method includes: acquiring a training question-and-answer dataset, wherein each training question-and-answer data point in the dataset includes a training question, a standard answer, and an answer format; acquiring multiple backbone answer sequences corresponding to each training question using the large language model to be trained; using the backbone answer sequences as unprocessed answer sequences, performing at least one layer of branching processing on the unprocessed answer sequences until the branching processing ends to obtain branch answer sequences, wherein each layer of branching processing includes: determining high-entropy points based on the log probabilities corresponding to each word in the unprocessed answer sequences, and generating the next layer of unprocessed answer sequences based on the high-entropy points; and performing backtracking concatenation processing on the branch answer sequences to obtain concatenated answer sequences corresponding to each of the aforementioned backbone answer sequences. The system is structured as follows: A sequence of concatenated responses corresponding to the same training question is grouped into a single data set. For each data set, a normalized reward function value is calculated based on all concatenated response sequences, the corresponding standard response, and the response format. For each high-entropy point, a corresponding high-entropy point weight is calculated based on the distance between the high-entropy point and the end position of its concatenated response sequence. The concatenated response sequence is then segmented based on these high-entropy points. The dominance function value for each segment is determined based on the high-entropy point weights and the normalized reward function value. Finally, the large language model to be trained is updated using gradients based on the dominance function values ​​of the concatenated response sequences to obtain the updated large language model.

[0032] Thus, during reinforcement learning training of a large language model, multiple backbone response sequences are obtained for each training question. High-entropy points with high uncertainty in these sequences are selected as key points. Iterative expansion is performed based on these high-entropy points, and branch resampling is conducted by filtering high-entropy points using an entropy threshold to obtain branch response sequences. These are then backtracked and concatenated to obtain concatenated response sequences, thereby constructing data sets for each sequence. This allows for the determination of the normalized reward function value corresponding to each concatenated response sequence within the data set. Simultaneously, the path weights of the high-entropy points are calculated to determine the dominance function value of the sequence segments obtained from the high-entropy point partitioning. This determines the dominance function value corresponding to the concatenated response sequence. Based on the dominance function value corresponding to the concatenated response sequence, gradient updates are performed on the large language model to be trained, resulting in an updated large language model. In this way, during reinforcement learning training, exploration based on lexical uncertainty is achieved through high-entropy points, enhancing the sampling exploration and improving model training performance.

[0033] like Figure 1As shown in the embodiments of this application, a method for training reinforcement learning of a large language model is provided. Specifically, the method includes the following steps: Step S100: Obtain the training question-and-answer dataset, wherein each training question-and-answer data in the training question-and-answer dataset includes a training question, a standard answer, and an answer format; Step S200: Using the large language model to be trained, obtain multiple backbone answer sequences corresponding to each of the above training questions; Step S300: Take the above-mentioned main answer sequence as the answer sequence to be processed, and perform at least one layer of branching processing on the answer sequence to be processed until the branching processing ends to obtain a branch answer sequence. The branching processing includes determining the high-entropy point based on the log probability corresponding to each word in the answer sequence to be processed, and generating the next layer of answer sequence to be processed based on the high-entropy point. Step S400: Perform backtracking splicing processing on the above branch answer sequences to obtain spliced ​​answer sequences corresponding to each of the above main answer sequences, and take the spliced ​​answer sequences corresponding to the same training question as a sequence data group; Step S500: For each of the above sequence data groups, calculate the normalized reward function value corresponding to each of the above spliced ​​answer sequences in the above sequence data group based on all the spliced ​​answer sequences in the above sequence data group, the standard answer and answer format corresponding to the above sequence data group. Step S600: For each of the above high-entropy points, calculate the corresponding high-entropy point path weight based on the distance between the high-entropy point and the end position of the sequence in the corresponding spliced ​​answer sequence, segment the spliced ​​answer sequence based on the high-entropy points, and determine the advantage function value corresponding to each sequence segment based on the high-entropy point path weight and the normalized reward function value, so as to obtain the advantage function value corresponding to each of the above spliced ​​answer sequences. Step S700: Based on the advantage function value corresponding to the above-mentioned concatenated answer sequence, perform gradient update on the above-mentioned large language model to be trained to obtain the updated large language model.

[0034] The aforementioned training question-and-answer dataset can be used as the training data for the current batch, for training the large language model to be trained. Specifically, the standard answer serves as the answer label for the corresponding training question, and the answer format indicates the standard paradigm of the answer.

[0035] In this embodiment of the application, obtaining the training question-answering dataset includes: Obtain the initial dataset, wherein one initial data point in the initial dataset includes the training question, the standard answer, and the answer format; For each training question, multiple samples are taken using the large language model to be trained. Based on the answers obtained from each sample, the standard answers, and the answer formats, the correct answer rate for the training question is determined. Based on the correct answer rate corresponding to each of the above training questions, and the correct answer range limited by the preset minimum and maximum correct answer thresholds, the target training question is determined, and the initial data corresponding to the target training question is used as the above training question and answer data to obtain the above training question and answer dataset.

[0036] First, obtain the initial dataset. Each piece of initial data in the above initial dataset contains a training question, a standard answer, and a response format. The initial dataset can be obtained from public datasets (such as the "nvidia / AceReason-Math" dataset on the Huggingface platform), or it can be constructed by yourself according to actual training needs. No specific restrictions are made here.

[0037] To further ensure training effectiveness, the initial dataset needs to be screened: for each training question, multiple samplings (e.g., 10 samplings) are performed using the large language model to be trained. Based on the consistency between the answers obtained from each sampling and the standard answers, as well as the standardization of the sampled answers compared to the answer format, the correct answer rate of the model to be trained when processing the training question is determined, and questions are screened based on the correct answer rate.

[0038] The minimum and maximum accuracy thresholds mentioned above can be set and adjusted according to actual needs. In this embodiment, they are set to 20% and 80% respectively, but are not intended as specific limitations. Target training questions with answer accuracy within the corresponding accuracy range are selected, and the initial data corresponding to these target training questions are used as the training question-and-answer data, ultimately forming the training question-and-answer dataset. The purpose of this selection is to ensure that the rewards, after normalization, can reflect differences in quality to ensure training effectiveness and avoid training ineffectiveness due to excessively high (the model has fully mastered) or excessively low (the model cannot learn) answer accuracy.

[0039] It should be noted that the large language model to be trained can be selected according to actual needs. Specifically, the large language model to be trained can be an existing mature base model. For example, the large language model to be trained can be a mathematical reasoning and question answering large language model. In this embodiment of the application, the large language model to be trained is the Qwen2.5-7B base model, but this is not intended as a specific limitation.

[0040] Furthermore, the above-mentioned large language model to be trained is used to obtain multiple backbone answer sequences corresponding to each of the above training questions, including: For each of the above training questions, multiple samplings are performed using the large language model to be trained to obtain multiple backbone answer sequences corresponding to each of the above training questions.

[0041] It should be noted that a pre-defined question-and-answer template can be embedded in the selected training question-and-answer dataset. This template includes system prompts to avoid duplicate sampling results, such as: "You are a helpful assistant, there may already be an answer prefix, please continue generating." The specific system prompts mentioned above are merely examples; in actual use, they can be adjusted according to specific needs, and no specific limitations are imposed here.

[0042] For each of the training questions in the above training question-answering dataset, after embedding them into the above question-answering template, they are sampled multiple times through the above large language model to be trained (the number of sampling times can be preset according to actual needs, such as N times). Each sampling yields an answer sequence, and finally, multiple backbone answer sequences corresponding to each of the above training questions are obtained.

[0043] Each obtained main response sequence is used as a response sequence to be processed. At least one level of branching is performed on each of these response sequences until the branching termination condition is met, ultimately yielding a branched response sequence, specifically including: The negative value of the log probability corresponding to each word in the above-mentioned unprocessed response sequence is used as the entropy value corresponding to the above-mentioned word. Based on the above entropy values ​​and the preset entropy threshold, high entropy points are determined from the above sequence of answers to be processed; For each high-entropy point, based on the training question corresponding to the high-entropy point and the cumulative generated part before the high-entropy point in the unprocessed answer sequence to which the high-entropy point belongs, the model resampled input corresponding to the high-entropy point is constructed. For each model, the input is resampled, and the large language model to be trained is sampled at least once. The sampled answers are used as branch answer sequences. If any preset branch processing termination condition is met, the hierarchical processing of the branch answer sequence ends. Otherwise, the branch answer sequence is used as a new answer sequence to be processed, and the process returns to the step of taking the negative value of the log probability corresponding to each word in the answer sequence to be processed and using it as the entropy value corresponding to the word. The preset branch processing termination condition includes that there are no high entropy points in the sampled answers and that the number of branch processing layers reaches the preset upper limit value.

[0044] Specifically, for the response sequence to be processed, resampling based on entropy is performed. The negative value of the logprob value corresponding to each token in the response sequence is used as the entropy value for that token. A higher entropy value indicates greater uncertainty for the token and thus greater exploratory value. The logprobability can be obtained directly from sampling; for example, vLLM sampling directly returns the logprobability. During the generation of the large language model, the output of each token is accompanied by a logprobability, which reflects the model's confidence in selecting that token at the current position. Negating the logprobability effectively transforms the confidence into a measure of uncertainty. A higher entropy value indicates greater hesitation in the model's prediction at that position, and more possible choices, which typically corresponds to key decision points or easily confused points in the inference chain.

[0045] Based on a preset entropy threshold (e.g., L=2.0, but not a specific limitation), words with entropy values ​​higher than the threshold are selected as candidate high-entropy points. In some application scenarios, to avoid interference from the initial sequence and adjacent high-entropy points, a protection mechanism for selecting high-entropy points can be further set, and the final high-entropy point can be determined from the candidate high-entropy points based on the above protection mechanism. Specifically, a preset initial protection value (e.g., i1=100, meaning the first 100 words in the generated sequence are not considered high-entropy points) and an interval protection value (e.g., i2=50, meaning the last 50 words of the selected high-entropy points are not considered high-entropy points) can be set, and the k points with the highest entropy values ​​(the value of k can be set and adjusted according to actual needs, e.g., k=6) can be selected as the final high-entropy points.

[0046] Considering that the initial stages of sequence generation typically include fixed formatted outputs or routine introductory phrases, high entropy at these locations often lacks exploratory value. Therefore, an initial protection value is set, meaning the first 100 tokens of the generated sequence are not considered as high-entropy points. Simultaneously, to avoid semantic fragmentation caused by continuous branching in local regions, an interval protection value is set; once a position is selected as a high-entropy point, its subsequent 50 tokens will no longer be included in the candidate set. This mechanism ensures the reasonableness and sparsity of the distribution of branch points across the sequence, allowing the model to focus on nodes that truly present inference challenges.

[0047] Figure 2 This is a schematic diagram of a branching process provided in an embodiment of this application. Figure 2In the process, three high-entropy points are identified in the main response sequence corresponding to the initial path. For each high-entropy point, branch processing is performed separately. For each high-entropy point, the preset prompt, the training question corresponding to the high-entropy point, and the accumulated generated part (i.e., the generated content) before the high-entropy point in the response sequence to be processed are concatenated to form a prefix, which is used as the model resampling input for the high-entropy point.

[0048] For each model, the input is resampled, and the large language model to be trained is sampled at least once to obtain the corresponding branch answer sequence. It is then determined whether any branch processing termination condition is met. The branch processing termination condition includes two criteria: first, there are no high-entropy points in the sampled answers; second, the number of branch processing layers executed reaches a preset upper limit (e.g., 3 layers). These branch processing termination conditions can be set and adjusted according to actual needs, and are not specifically limited here.

[0049] It should be noted that in this embodiment of the application, the main answer sequence is taken as the first layer. After performing one layer of branching, a second layer of branching is obtained. Based on the second layer of branching, another layer of branching is performed to obtain a third layer of branching.

[0050] It should be further explained that in the embodiments of this application, the model resampling input for each high-entropy point is performed only once to generate a branch. In actual use, multiple samplings can be performed, and no specific limitation is made here.

[0051] After the branching process is completed, a backtracking concatenation process is performed on each of the obtained branch answer sequences to finally obtain the concatenated answer sequences corresponding to each of the above-mentioned main answer sequences. The backtracking concatenation process performed on the above-mentioned branch answer sequences to obtain the concatenated answer sequences corresponding to each of the above-mentioned main answer sequences includes: For each of the above-mentioned branch answer sequences, the above-mentioned branch answer sequences are concatenated to the cumulative generation before the high-entropy point corresponding to the above-mentioned branch answer sequences to obtain the intermediate sequence; Take the above intermediate sequence as the current sequence, trace back to the high-entropy point corresponding to the previous branch, and then concatenate the above current sequence with the accumulated generation before the above high-entropy point, and update the above intermediate sequence. Return to the above steps, taking the intermediate sequence as the current sequence, tracing upwards to the high-entropy point corresponding to the previous branch, and concatenating the current sequence to the cumulative generation steps before the high-entropy point, until tracing back to the starting point of the main answer sequence, to obtain the complete answer to the above training question, and use it as the concatenated answer sequence.

[0052] Specifically, from the end ( Figure 2 Starting with the sequence generated on the right side of the middle section, the resampled generation of the high-entropy point is spliced ​​with the accumulated generation of the sequence preceding the high-entropy point. This splicing step is repeated iteratively from back to front until all branches are spliced ​​together to form a complete answer to the input question. Assume that each main branch, after resampling and splicing, yields... A complete answer, plus N main branches generated for each training question, yields the answer for each training question. Each complete answer (i.e., a concatenated answer sequence) serves as a complete answer group (i.e., a sequence data group) for the same training question, preparing for subsequent inter-group normalized rewards. Represents a sequence of data. That is the number of elements (i.e., the group capacity).

[0053] Furthermore, for each sequence of data, the corresponding normalized reward function value is calculated. Specifically, the answers contained in the above-mentioned concatenated answer sequence are first extracted and compared with the above-mentioned standard answer (ground truth) to obtain the correctness score. (1 for correct, -1 for incorrect); Provided that the correctness requirement is met, the above-mentioned concatenated answer sequence is compared with the above-mentioned answer format to obtain a format conformity score. (Standardized values ​​are 1, non-standardized values ​​are -1); Under the premise of correctness, the weighted basic reward value is calculated: obtain the preset weight coefficients, for example, , Weighted calculation of basic reward value : ; It should be noted that if correctness is not satisfied, then standardization is meaningless, and the basic reward value... It is directly recorded as -1.

[0054] It should be further explained that multiple complete sequences belonging to the same problem are connected through high-entropy points. However, considering that segments closer to the starting side are more common to all complete sequences, to differentiate the final performance of each sequence, it is necessary to refer to the feedback of the endpoint reward, so that the advantage function values ​​of the shared starting side segments differ in each sequence. In the embodiments of this application, the normalized reward function value is calculated. As a final reward: ; in, This represents a specific path (i.e., a sequence of concatenated answers). The base reward value, The normalized reward function value representing this path, in the formula above... It represents a sequence of data. and These represent the mean and standard deviation of the base reward values ​​for all sequences in the dataset, respectively. Calculating the normalized reward function value ensures that the reward values ​​are within a reasonable range, facilitating subsequent calculation of the dominance function. Furthermore, it helps establish a connection among multiple complete sequences: a single reward can be compared using inter-group reward normalization.

[0055] Specifically, for each of the aforementioned high-entropy points, the corresponding high-entropy point path weights are calculated based on the distance between the high-entropy point and the end position of the sequence in its respective concatenated response sequence, including: For each high-entropy point, determine the number of subsequent high-entropy points between the high-entropy point and the end position of the sequence in each spliced ​​response sequence to which the high-entropy point belongs, and determine the path weight of the high-entropy point based on the minimum value of the number of subsequent high-entropy points.

[0056] In some application scenarios, the path is recursively moved forward from the end point. For each high-entropy point, the path weights of all its successor high-entropy points are obtained, and the minimum value of the path weights of all the successor high-entropy points is added by 1 as the path weight of the high-entropy point. When there are no successor high-entropy points in the successor branch, the path weight of the end point of the successor branch is counted as 1.

[0057] The aforementioned high-entropy point weights are used to measure the distance of these high-entropy points from the end position of the sequence (reward endpoint). The larger the weight order, the farther the high-entropy point is from the reward endpoint. The weights determine the minimum number of high-entropy points required to reach the reward endpoint, without considering erroneous branches. Specifically, the high-entropy point weights are calculated by backtracking forward using the following formula: ; in, Represents a high entropy point right of way, yes The set of the first high-entropy points after the fork in the road. It is a high-entropy point in the set.

[0058] Based on the high-entropy points mentioned above, the spliced ​​response sequence is segmented. The left and right limits of each sequence segment are high-entropy points, sequence start points, or sequence end points. All words in the same sequence segment have the same dominance function value, thus forming a segment-level segmentation method.

[0059] Specifically, the advantage function value of high-entropy points can be calculated using the following formula: ; in, Represents a high entropy point The dominant function value, It is the exponent of the preset reward decay coefficient. It is a number less than 1 to ensure that the reward decays as the distance increases, thus incentivizing the search for the shortest route. This represents the base reward value at the corresponding point. The specific meaning of the above formula is: the dominance function of a high-entropy point equals the average of the rewards of all branches after the high-entropy point multiplied by different powers of the decay coefficient, with the branch weight as the power of the branch decay coefficient. Therefore, a high-entropy point with a high dominance function value means that most subsequent branches can lead to the rewarded endpoint both correctly and quickly.

[0060] By using the above backtracking calculation steps based on right-of-way to calculate the advantage function, the advantage functions of all high-entropy points can be recursively calculated. The recursion begins by calculating the advantage function of the first high-entropy point. The recursion is triggered when the calculation of the advantage function of each high-entropy point requires the right-of-way of the high-entropy points in subsequent branches. The recursion terminates when the branches following the last high-entropy point have reached the reward endpoint. If a failed branch occurs, the final... If the value is 0, the power factor term of the decay coefficient becomes meaningless because multiplication always results in 0. The power factor series is theoretically infinite, but it doesn't need to be calculated. The zero advantage function of the failed branch will be propagated forward until a high-entropy point has a branch leading to the reward endpoint. At this point, the power factor of the high-entropy point becomes constant again, and the advantage function can be recalculated as a positive number. Thus, the advantage function of each high-entropy point can be recursively calculated.

[0061] Furthermore, the dominance function values ​​for sequence segments are calculated based on the dominance function values ​​at high-entropy points: The dominance function of a segment is equal to the difference between the dominance function value at the high-entropy point at the back end (right limit) and the high-entropy point at the front end (left limit). This restores the dominance function to be the net increase in expected return minus the current value function. The dominance function of a segment is essentially the net approximation value of the distance to the reward endpoint after traversing this segment. Specifically, the dominance function value of a segment can be calculated using the following formula: ; in, Represents the sequence segments obtained from the initial calculation. The dominant function value. For example... Figure 2 As shown, and These represent the front-end high-entropy point and the back-end high-entropy point of the corresponding sequence segment, respectively. and These represent the dominant function values ​​of the front-end high-entropy point and the back-end high-entropy point, respectively.

[0062] It should be further explained that, in order to distinguish the differences in the advantages of the common starting segments in different spliced ​​response sequences, the sequence segment advantage function values ​​obtained in the initial calculation above can be optimized using preset weighting coefficients, as shown in the following formula: ; in, Represents the sequence segments obtained after optimization. The actual advantage function value, This represents the path to which the sequence segment belongs. The normalized reward function value. and These are preset weighting coefficients, and their sum equals 1 to ensure that the optimized actual advantage function value is within a reasonable range. Greater than or equal to This ensures that the feedback from the result is greater than the feedback from the process. In a specific application scenario, it can be set to... and The values ​​are 0.6 and 0.4 respectively, but these are not considered specific limits.

[0063] In this embodiment, the pre-optimized sequence segment dominance function value is used as process feedback, and the weighted optimized sequence segment dominance function value is used as the actual dominance function value for updating. After calculating the actual dominance function value of the sequence segment, the dominance function values ​​of each complete sequence (path) are concatenated. ; in, Representing the The answer sequence should be constructed by piecing together the main body of the answer. (i.e., path) The advantage function value, after concatenation, all complete sequences of the same prompt word (i.e., the same training question) (i.e., the concatenated answer sequence) are regarded as a single sequence data group. . Is this path? The total number of sequence segments. It should be noted that when calculating for a single sequence data set, because all answers to the same training question are counted together as a sequence data set after recursive concatenation. Therefore, the corresponding superscript It can be omitted.

[0064] At this point, the advantage function has been obtained through sampling and backtracking calculation based on the entropy threshold. Next, the gradient will be calculated according to the improved full gradient objective function based on the PPO paradigm, and backpropagation will be used to update the model. In some application scenarios, the gradient update of the large language model to be trained is performed based on the advantage function values ​​corresponding to the concatenated answer sequence, including: Calculate the importance sampling ratio between the current policy and the old policy; where the old policy is the large language model used during sampling, and its parameters are fixed during single-batch training; the current policy is the updated model, which is gradually optimized by starting from the old policy and undergoing multiple gradient updates during single-batch training. When the importance sampling ratio is less than the preset clipping lower limit and the dominance function value is positive, the importance sampling ratio in the objective function is replaced by the product of the clipping lower limit and the original importance sampling ratio divided by the original importance sampling ratio after stopping the gradient. When the importance sampling ratio is greater than the preset pruning upper limit and the dominance function value is positive, the importance sampling ratio in the objective function is replaced by the product of the pruning upper limit and the original importance sampling ratio divided by the original importance sampling ratio after stopping the gradient. The objective function value is calculated based on the importance sampling ratio after replacement, and the model parameters are updated by backpropagation.

[0065] PPO has become the classic paradigm in the field of large model alignment, and most RL algorithm variants are built upon this classic paradigm. GRPO's objective function also adopts the core paradigm of PPO, that is, clipping the importance sampling ratio through a (clip ratio) mechanism. ; in, This represents the value of the objective function; This represents the expected value, i.e., the mean. Group capacity refers to the total number of complete answer sequences corresponding to the same question; Index representing the sequence, The output represents the first The text of a complete sequence of answers; Index representing lexical units; The importance sampling ratio is used to measure the difference in probability between the current strategy and the old strategy on corresponding lexical units. The dominant function, i.e., the first... The first answer In this embodiment of the application, the advantage function value of a word is equal to the advantage function value of the sequence segment to which it belongs; The preset shear constant; This represents a cropping operation, which will... Limited to Within the range.

[0066] The clip ratio mechanism ensures training stability by limiting the magnitude of policy updates. During policy optimization, the importance sampling ratio is first calculated: ; In the above formula, the numerator represents the probability of the current strategy generating the word, and the denominator represents the probability of the old strategy generating the word. This ratio measures the difference between the current strategy and the old strategy in choosing an action. The differences in the above, the pruning mechanism will Limited to Within a certain range, this prevents excessively large policy updates from causing training instability. When When it exceeds the range, additional consideration should be given. >0 or If the value is less than 0, then there are four possible scenarios as shown in Table 1: Table 1

[0067] Table 1. Sampling ratio based on importance With advantage function The different numerical ranges are divided into four cases, as shown in the third column of Table 1. (The first column) represents the network parameters obtained by minimizing the objective function in the first column of data in Table 1 (i.e., the second column in Table 1). gradient, advantage function With constant There are no gradients, but the importance sampling ratio is higher. Includes gradients. Column 5 of Table 1. (The previous column) indicates how to calculate the network parameters from the data in column 4 of Table 1. The gradient.

[0068] In this embodiment, gradient protection is also applied to the objective function term. Specifically, column 4 of Table 1 represents the objective function term used in this application after gradient protection processing. For case 1 in Table 1, when the gradient disappears due to the clipping operation, the objective term can be transformed to restore the gradient. In case 1, the objective function term after gradient protection processing is rewritten as follows: ; in, This represents a stop-gradient operation, where the transformation remains numerically unchanged. During backpropagation, the gradient signal with respect to the policy parameters is recovered, effectively solving the gradient vanishing problem caused by clipping.

[0069] In this application embodiment, an improved full gradient objective function based on the PPO paradigm is proposed for the above four cases: Both scenarios 1 and 2 involve choosing a lower probability event compared to the old strategy, thus making... Less than the lower limit The actual calculation method for the aforementioned high-entropy point is -logprob, i.e. ,in This represents the model strategy; points with higher values ​​are high-entropy points. Points with lower values ​​can be considered relatively high-entropy points compared to older strategies. Although their absolute high-entropy values ​​differ from those of the aforementioned high-entropy points, they represent explorations with lower certainty. These explorations should be encouraged, with greater rewards for success and less punishment for failure.

[0070] The "more" in "greater" or "smaller" both require at least two candidate values. and upper and lower boundaries This perfectly meets the requirement. Whether it is greater than 0 indicates whether it was successful, thus: Scenario 1 is a relatively low-probability exploration that fails, and is therefore given... and smaller value in After multiplying by the dominance function, it becomes .

[0071] Scenario 2 is a relatively low-probability exploration that succeeds, and is rewarded. and Larger value in After multiplying by the dominance function and recovering the gradient, it becomes .

[0072] Cases 3 and 4 both involve choosing the higher-probability event compared to the old strategy, thus... Greater than the upper limit When faced with a strategy of high certainty, success indicates that the strategy's prediction is roughly correct, but it's important to avoid convergence too quickly and getting trapped in local optima. Failure indicates that the strategy's prediction deviates significantly from reality, and it should be severely penalized and corrected promptly. Thus, we have: Scenario 3 is a relatively deterministic prediction, and the result is unsuccessful. and Larger value in After multiplying by the dominance function, it becomes .

[0073] Scenario 4 is a relatively deterministic prediction, and the result is successful, therefore... and smaller value in After multiplying by the dominance function and recovering the gradient, it becomes .

[0074] In summary, the objective function for policy network updates in this embodiment is written in the following closed-form form: ; ; in, This represents the importance sampling ratio of the objective function after gradient protection. This represents the original importance sampling ratio. The group capacity.

[0075] In this embodiment, the improved full gradient objective function based on the PPO paradigm is adopted. Based on this, the loss can be calculated, and then the gradient can be calculated through backpropagation for model updating. After the update, the sampling of the next batch dataset continues, and the above process can be repeated to continuously train until one or more epochs are completed.

[0076] Specifically, after updating the gradient of the large language model to be trained based on the advantage function values ​​corresponding to the concatenated response sequence to obtain the updated large language model, the method further includes: The updated large language model is used as the current large language model to be trained. The process of obtaining multiple backbone answer sequences corresponding to each of the above training questions through the large language model to be trained is repeated until the preset iterative training termination condition is met. The iterative training termination condition includes the number of iterations reaching a preset iteration threshold.

[0077] Specifically, the above iteration thresholds can be set and adjusted according to actual needs, and no specific limitations are made here.

[0078] This application provides a reinforcement learning training method for a large language model. The method includes: acquiring a training question-and-answer dataset, wherein each training question-and-answer data point in the dataset includes a training question, a standard answer, and an answer format; acquiring multiple backbone answer sequences corresponding to each training question using the large language model to be trained; using the backbone answer sequences as answer sequences to be processed, performing at least one layer of branching processing on these answer sequences until the branching processing ends to obtain branch answer sequences, wherein each layer of branching processing includes determining high-entropy points based on the log probabilities of each word in the answer sequences to be processed, and generating the next layer of answer sequences to be processed based on the high-entropy points; performing backtracking concatenation processing on the branch answer sequences to obtain concatenated answer sequences corresponding to each backbone answer sequence, and concatenating these concatenated answer sequences with the same training question... The concatenated answer sequence corresponding to the question is treated as a sequence data set. For each of the above sequence data sets, based on all the concatenated answer sequences, the corresponding standard answer, and the answer format, the normalized reward function value corresponding to each of the concatenated answer sequences in the above sequence data set is calculated. For each of the above high-entropy points, the corresponding high-entropy point weight is calculated based on the distance between the high-entropy point and the end position of the sequence in its concatenated answer sequence. Based on the high-entropy points, the concatenated answer sequence is segmented. Based on the high-entropy point weights and the normalized reward function value, the dominance function value corresponding to each sequence segment is determined to obtain the dominance function value corresponding to each of the above concatenated answer sequences. Based on the dominance function value corresponding to the concatenated answer sequence, the gradient of the large language model to be trained is updated to obtain the updated large language model.

[0079] Thus, during reinforcement learning training of a large language model, multiple backbone response sequences are obtained for each training question. High-entropy points with high uncertainty in these sequences are selected as key points. Iterative expansion is performed based on these high-entropy points, and branch resampling is conducted by filtering high-entropy points using an entropy threshold to obtain branch response sequences. These are then backtracked and concatenated to obtain concatenated response sequences, thereby constructing data sets for each sequence. This allows for the determination of the normalized reward function value corresponding to each concatenated response sequence within the data set. Simultaneously, the path weights of the high-entropy points are calculated to determine the dominance function value of the sequence segments obtained from the high-entropy point partitioning. This determines the dominance function value corresponding to the concatenated response sequence. Based on the dominance function value corresponding to the concatenated response sequence, gradient updates are performed on the large language model to be trained, resulting in an updated large language model. In this way, during reinforcement learning training, exploration based on lexical uncertainty is achieved through high-entropy points, enhancing the sampling exploration and improving model training performance.

[0080] Specifically, this application's scheme explores uncertainties, enhancing the exploratory nature of sampling. It innovatively selects high-entropy tokens as key points along the thought chain, iterates based on these high-entropy points, recursively calculates path weights and advantage functions, and finally iteratively concatenates them into a complete sequence. Only the tree structure is used as an intermediate process, maintaining simplicity. The adoption of an extended sampling method based on high-entropy tokens improves both exploratory nature and sampling efficiency.

[0081] Based on right-of-way theory, a backtracking calculation method for the advantage function is proposed. A segment-level advantage function, a compromise between token-level and sequence-level advantages, is introduced, achieving more accurate feedback without significantly increasing computational cost. Combining process feedback with result feedback improves the fine-grainedness of feedback, effectively alleviating the convergence problem caused by reward sparsity, while avoiding the need for additional models and maintaining deployment costs.

[0082] An improved full-gradient objective function is proposed based on the PPO paradigm, providing tolerance for feedback to relatively low-probability policies and protecting high-entropy exploration to some extent. The improved full-gradient objective function ensures that low-probability policies with relatively high entropy will not have their gradients canceled. Unsynchronized improvements could lead to the pruning of some high-entropy points. The improvement of the objective function, along with sampling high-entropy expansion and dominance function backtracking, complement each other. Gradient protection for low-probability policies with relatively high entropy prevents gradient vanishing, and this, combined with high-entropy point sampling and dominance function backtracking, further improves the stability and convergence speed of model training.

[0083] like Figure 3As shown, corresponding to the above-described large language model reinforcement learning training method, this application embodiment also provides a large language model reinforcement learning training system, which includes: The data acquisition module 310 is used to acquire the training question and answer dataset, wherein a training question and answer data in the training question and answer dataset includes the training question, the standard answer, and the answer format; Data processing module 320 is used to obtain multiple backbone answer sequences corresponding to each of the above training questions through the large language model to be trained; The branch processing module 330 is used to take the above-mentioned main answer sequence as the answer sequence to be processed, and perform at least one layer of branch processing on the answer sequence to be processed until the branch processing ends to obtain the branch answer sequence. The branch processing includes determining the high entropy point according to the log probability corresponding to each word in the answer sequence to be processed, and generating the next layer of answer sequence to be processed based on the high entropy point. The backtracking splicing module 340 is used to perform backtracking splicing processing on the above branch answer sequences to obtain the spliced ​​answer sequences corresponding to each of the above main answer sequences, and to take the spliced ​​answer sequences corresponding to the same training question as a sequence data group. The sequence processing module 350 is used to calculate the normalized reward function value corresponding to each of the spliced ​​answer sequences in the sequence data group based on all the spliced ​​answer sequences in the sequence data group, the standard answer and the answer format corresponding to the sequence data group. The right-of-way calculation module 360 ​​is used to calculate the corresponding right-of-way for each of the above-mentioned high-entropy points based on the distance between the high-entropy point and the end position of the sequence in the corresponding spliced ​​answer sequence, to segment the spliced ​​answer sequence based on the high-entropy points, and to determine the advantage function value corresponding to each sequence segment based on the high-entropy point right-of-way and the normalized reward function value, so as to obtain the advantage function value corresponding to each of the above-mentioned spliced ​​answer sequences. The gradient update module 370 is used to update the gradient of the large language model to be trained based on the advantage function value corresponding to the above-mentioned concatenated answer sequence, so as to obtain the updated large language model.

[0084] Thus, during reinforcement learning training of a large language model, multiple backbone response sequences are obtained for each training question. High-entropy points with high uncertainty in the sequences are selected as key points. Iterative expansion is performed based on these high-entropy points, and branch resampling is conducted by filtering high-entropy points based on entropy thresholds to obtain branch response sequences. These are then backtracked and concatenated to obtain concatenated response sequences, thereby constructing data sets for each sequence to determine the normalized reward function value corresponding to each concatenated response sequence. Simultaneously, the path weights of the high-entropy points are calculated to determine the dominance function value of the sequence segments obtained from the high-entropy point partitioning. This determines the dominance function value corresponding to the concatenated response sequence. Based on the dominance function value corresponding to the concatenated response sequence, the gradient of the large language model to be trained is updated to obtain the updated large language model. In this way, during reinforcement learning training, exploration based on lexical uncertainty is achieved through high-entropy points, enhancing the sampling exploration and improving the model training effect.

[0085] It should be noted that the specific structure and implementation of the above-mentioned large language model reinforcement learning training system and its various modules or units can be referred to the corresponding descriptions in the above method embodiments, and will not be repeated here.

[0086] It should be noted that the division of the modules in the above-mentioned large language model reinforcement learning training system is not unique and is not intended as a specific limitation.

[0087] Based on the above embodiments, this application also provides a terminal, the principle block diagram of which can be as follows: Figure 4 As shown. The terminal includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps of any of the aforementioned large language model reinforcement learning training methods. The display screen can be a liquid crystal display (LCD) or an e-ink display.

[0088] Those skilled in the art will understand that Figure 4 The block diagram shown is only a partial structural diagram related to the solution of this application and does not constitute a limitation on the terminal on which the solution of this application is applied. The specific terminal may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements.

[0089] In one embodiment, a terminal is provided, the terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of any of the large language model reinforcement learning training methods provided in the embodiments of this application.

[0090] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the large language model reinforcement learning training methods provided in this application.

[0091] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0092] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0093] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0094] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0095] In the embodiments provided in this application, it should be understood that the disclosed systems / terminal devices and methods can be implemented in other ways. For example, the system / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units described above is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0096] If the integrated modules / units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, and software distribution media, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0097] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions are not in essence a departure from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A reinforcement learning training method for a large language model, characterized in that, The method includes: Obtain a training question-and-answer dataset, wherein each training question-and-answer data point in the training question-and-answer dataset includes a training question, a standard answer, and an answer format; Using the large language model to be trained, obtain multiple backbone answer sequences corresponding to each training question; The main response sequence is used as the response sequence to be processed. The response sequence to be processed is subjected to at least one layer of branching until the branching process ends to obtain the branch response sequence. The branching process includes determining the high entropy point based on the log probability corresponding to each word in the response sequence to be processed, and generating the next layer of response sequence to be processed based on the high entropy point. The branch answer sequences are backtracked and spliced ​​to obtain spliced ​​answer sequences corresponding to each main answer sequence, and the spliced ​​answer sequences corresponding to the same training question are taken as a sequence data group. For each of the sequence data groups, the normalized reward function value corresponding to each of the spliced ​​answer sequences in the sequence data group is calculated based on all the spliced ​​answer sequences in the sequence data group, the standard answer and the answer format corresponding to the sequence data group; For each high-entropy point, the corresponding high-entropy point path weight is calculated based on the distance between the high-entropy point and the end position of the sequence in its respective spliced ​​response sequence. The spliced ​​response sequence is segmented based on the high-entropy point. Based on the high-entropy point path weight and the normalized reward function value, the advantage function value corresponding to each sequence segment is determined to obtain the advantage function value corresponding to each spliced ​​response sequence. Based on the advantage function value corresponding to the concatenated response sequence, the large language model to be trained is updated by gradient to obtain the updated large language model.

2. The large language model reinforcement learning training method according to claim 1, characterized in that, The acquisition of the training question-answering dataset includes: Obtain an initial dataset, wherein each initial data point in the initial dataset includes a training question, a standard answer, and an answer format; For each training question, multiple samples are taken using the large language model to be trained. Based on the answers obtained from each sample, the standard answer, and the answer format, the answer accuracy corresponding to the training question is determined. Based on the correct answer rate corresponding to each training question and the correct answer range defined by the preset minimum and maximum correct answer thresholds, the target training question is determined, and the initial data corresponding to the target training question is used as the training question and answer data to obtain the training question and answer dataset.

3. The large language model reinforcement learning training method according to claim 1, characterized in that, The process involves obtaining multiple backbone answer sequences corresponding to each training question using a large language model to be trained, including: For each training question, multiple samplings are performed using the large language model to be trained to obtain multiple backbone answer sequences corresponding to each training question.

4. The large language model reinforcement learning training method according to claim 1, characterized in that, The step of performing at least one level of branching on the sequence of answers to be processed until the branching process ends to obtain a branched answer sequence includes: The negative value of the log probability corresponding to each word in the sequence of answers to be processed is used as the entropy value corresponding to the word; Based on the entropy value and the preset entropy threshold, high entropy points are determined from the sequence of answers to be processed; For each high-entropy point, based on the training question corresponding to the high-entropy point and the cumulative generated portion of the unprocessed answer sequence to which the high-entropy point belongs before the high-entropy point, the model resampling input corresponding to the high-entropy point is constructed. For each model, the input is resampled, and the large language model to be trained is sampled at least once. The sampled answers are used as branch answer sequences. If any preset branch processing termination condition is met, the hierarchical processing of the branch answer sequence ends. Otherwise, the branch answer sequence is used as a new answer sequence to be processed, and the process returns to the step of taking the negative value of the log probability corresponding to each word in the answer sequence to be processed and using it as the entropy value corresponding to the word. The preset branch processing termination condition includes that there are no high entropy points in the sampled answers and that the number of layers of branch processing has reached a preset upper limit.

5. The large language model reinforcement learning training method according to claim 1, characterized in that, The backtracking and splicing process for the branch answer sequences to obtain the spliced ​​answer sequences corresponding to each of the main answer sequences includes: For each of the branch response sequences, the branch response sequences are concatenated to the accumulated generation before the high entropy point corresponding to the branch response sequence to obtain the intermediate sequence; Using the intermediate sequence as the current sequence, trace upwards to the high-entropy point corresponding to the previous branch processing, concatenate the current sequence to the cumulative generation before the high-entropy point, and then update the intermediate sequence. Return to the previous step of taking the intermediate sequence as the current sequence, tracing upwards to the high-entropy point corresponding to the previous branch processing, and splicing the current sequence to the cumulative generation before the high-entropy point, until tracing back to the starting point of the main answer sequence, to obtain a complete answer to the training question, and use it as the spliced ​​answer sequence.

6. The large language model reinforcement learning training method according to claim 1, characterized in that, The step of calculating the corresponding high-entropy point path weight for each high-entropy point based on the distance between the high-entropy point and the end position of the sequence in its respective concatenated response sequence includes: For each high-entropy point, determine the number of subsequent high-entropy points between the high-entropy point and the end position of the sequence in each spliced ​​response sequence to which the high-entropy point belongs, and determine the path weight of the high-entropy point based on the minimum value of the number of subsequent high-entropy points.

7. The large language model reinforcement learning training method according to any one of claims 1 to 6, characterized in that, After updating the large language model to be trained by performing gradient updates based on the advantage function values ​​corresponding to the concatenated response sequence to obtain the updated large language model, the method further includes: The updated large language model is used as the current large language model to be trained. The process of obtaining multiple backbone answer sequences corresponding to each training question through the large language model to be trained is repeated until a preset iterative training termination condition is met. The iterative training termination condition includes the number of iterative training iterations reaching a preset iteration threshold.

8. A large language model reinforcement learning training system, characterized in that, The system includes: The data acquisition module is used to acquire the training question-and-answer dataset, wherein a training question-and-answer data in the training question-and-answer dataset includes a training question, a standard answer, and an answer format; The data processing module is used to obtain multiple backbone answer sequences corresponding to each training question through the large language model to be trained; The branching module is used to take the main response sequence as the response sequence to be processed, and perform at least one layer of branching processing on the response sequence to be processed until the branching processing ends to obtain the branched response sequence. The branching processing includes determining the high-entropy point based on the log probability corresponding to each word in the response sequence to be processed, and generating the next layer of response sequence to be processed based on the high-entropy point. The backtracking splicing module is used to perform backtracking splicing processing on the branch answer sequences to obtain the spliced ​​answer sequences corresponding to each of the main answer sequences, and to take the spliced ​​answer sequences corresponding to the same training question as a sequence data group; The sequence processing module is used to calculate, for each of the sequence data groups, the normalized reward function value corresponding to each of the spliced ​​answer sequences in the sequence data group, based on all the spliced ​​answer sequences in the sequence data group, the standard answer and the answer format corresponding to the sequence data group; The right-of-way calculation module is used to calculate the corresponding right-of-way for each high-entropy point based on the distance between the high-entropy point and the end position of the sequence in its respective spliced ​​response sequence, segment the spliced ​​response sequence based on the high-entropy point, and determine the advantage function value corresponding to each sequence segment based on the right-of-way for the high-entropy point and the normalized reward function value, so as to obtain the advantage function value corresponding to each spliced ​​response sequence. The gradient update module is used to update the large language model to be trained based on the advantage function value corresponding to the concatenated response sequence, so as to obtain the updated large language model.

9. A terminal, characterized in that, The terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the steps of the large language model reinforcement learning training method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the large language model reinforcement learning training method as described in any one of claims 1 to 7.

Images