Reinforcement learning training method and system for small-scale large language model

Abstract

The application relates to the technical field of reinforcement learning, in particular to a reinforcement learning training method and system for a small-scale large language model, which comprises the following steps: acquiring a user query and an external tool set and constructing a tool integrated reasoning training task; driving a large language model to generate thinking content and tool calling instructions, and combining external tool execution results to form an interaction track; respectively calculating a format reward, a tool calling correctness reward, a dynamic length reward and a distillation learning reward for the interaction track, and fusing the rewards to obtain a total reward value; performing track reuse and track data augmentation processing on the interaction track to generate an extended training data set; and based on the extended training data set and the total reward value, performing parameter iterative optimization on the small-scale large language model to be trained by using a reinforcement learning strategy. The application can enable the small-scale large language model to stably improve reasoning capability and structured tool calling capability in a complex tool integrated reasoning task.

Description

Technical Field

[0001] This application relates to the field of reinforcement learning technology, and in particular to a reinforcement learning training method and system for small-scale large language models. Background Technology

[0002] Large language models extend text understanding capabilities to the ability to operate external systems by selecting and invoking external tools, thereby accomplishing tasks that are difficult to achieve relying solely on the model's internal parameters. This effectively compensates for the model's shortcomings in real-time information acquisition, accurate calculation, and interaction with specialized systems. In the development of tool-integrated reasoning, the application of models in multi-step information processing, logical execution, and hardware / software system interaction is gradually increasing.

[0003] Currently, the main technical approaches to enabling models to invoke tools include supervised fine-tuning and reinforcement learning. Supervised fine-tuning typically trains the model by building inference trajectory data offline, allowing the model to learn predetermined tool invocation patterns. Reinforcement learning, on the other hand, adjusts the model's strategy by introducing interactive feedback signals to improve its decision-making ability during tool selection and invocation. In resource-constrained vertical business scenarios, due to considerations of computing power cost and deployment efficiency, large language models with relatively small parameter sizes are often used, combined with efficient parameter fine-tuning techniques for task adaptation. However, because offline trajectory data usually has limited coverage, supervised fine-tuning methods have certain limitations in adapting to unseen situations when faced with diverse tool combinations and complex task structures. Reinforcement learning methods in practice often use relatively macroscopic reward evaluation methods, with limited granularity in characterizing tool names, parameter structures, and invocation details, which may affect the accuracy of strategy adjustment in complex tool integration scenarios. Under small-scale model conditions, due to limitations in model expressive power and training data size, these problems are more likely to affect training results.

[0004] Therefore, under the condition of limited computing power and data resources, how to design a training mechanism suitable for small-scale large language models, so that it can stably improve reasoning ability and structured tool invocation ability in complex tool integration reasoning tasks, is an urgent problem to be solved. Summary of the Invention

[0005] This application provides a reinforcement learning training method and system for small-scale large language models, which enables these models to steadily improve their reasoning ability and structured tool invocation capabilities in complex tool integration reasoning tasks. The technical solution provided in this application is as follows:

[0006] Firstly, this application provides a reinforcement learning training method for small-scale large language models, the method comprising:

[0007] Obtain user queries and a set of external tools, and construct a tool integration inference training task based on the user queries and the set of external tools;

[0008] Based on the tool-integrated reasoning training task, the small-scale large language model to be trained generates thinking content and tool call instructions, and combines the execution results of external tools to form an interactive trajectory that includes reasoning process, tool call information and tool observation results;

[0009] For the interaction trajectory, calculate the format reward, tool call correctness reward, dynamic length reward, and distillation learning reward based on the teacher model distribution difference, and fuse the reward items to obtain the total reward value corresponding to the interaction trajectory;

[0010] The interaction trajectory is subjected to trajectory reuse and trajectory data augmentation processing to generate an expanded training data set;

[0011] Based on the expanded training dataset and the total reward value, a reinforcement learning strategy is used to iteratively optimize the parameters of the small-scale large language model to be trained.

[0012] In one specific implementation scheme, the calculation of format reward, tool invocation correctness reward, dynamic length reward, and distillation learning reward based on teacher model distribution differences for the interaction trajectory includes:

[0013] Calculation format reward The format reward is used to evaluate whether the model output meets the preset structured labeling specification;

[0014] Let the current training step number be... The threshold in the early stages of training is When the output format is completely correct and When the format is correct, the reward is a constant of 1; when the output format is completely correct and The format reward decays according to the training progress; when the output format is incorrect, the format reward is 0, and its expression is:

[0015] .

[0016] In one specific implementation scheme, the calculation of format reward, tool invocation correctness reward, dynamic length reward, and distillation learning reward based on teacher model distribution differences for the interaction trajectory further includes:

[0017] Rewards for correct use of computing tools The tool call correctness reward is used to evaluate the degree of matching between the tool name, parameter name, and parameter value;

[0018] Let the set of correctly invoked tool names be... The set of tool names invoked by the model prediction is The tool name matching reward is given in the form of intersection-union ratio, defined as follows:

[0019] ;

[0020] in, Indicates the number of elements in the set;

[0021] Calculate the parameter name matching reward. For each correct tool invocation entry, calculate the degree of parameter name matching. The parameter name matching reward is defined as follows:

[0022] ;

[0023] in, This represents the set of correct tool call entries. Represents a set One of the tool call entries, This represents the parameter key-value mapping structure corresponding to the entry. This represents the parameter key-value mapping structure corresponding to the predicted entry; and These represent the set of correct parameter names and the set of predicted parameter names corresponding to the current entry, respectively; the summation symbol represents the summation of the sets. Each item in the calculation is calculated and accumulated individually;

[0024] Calculate the reward based on the parameter values. The parameter name is The correct parameter values, The parameter name is The predicted parameter values ​​are used to introduce an indicator function. The parameter value represents a value of 1 if the condition is true and 0 if the condition is false. The reward for matching the parameter value is defined as follows:

[0025] ;

[0026] in, Indicates an entry The set of parameter names contained therein;

[0027] Add the three sub-items together to get the matching score:

[0028] ;

[0029] in, Express the maximum possible score, and define the tool call correctness reward as:

[0030] .

[0031] In one specific implementation scheme, the calculation of format reward, tool invocation correctness reward, dynamic length reward, and distillation learning reward based on teacher model distribution differences for the interaction trajectory further includes:

[0032] Calculate dynamic length reward Let the length of the thought content generated by the model be... The target thinking length is Total training steps: Training progress is defined as:

[0033] ;

[0034] The dynamic length reward is defined as follows:

[0035] ;

[0036] in, Used to characterize the progress of training.

[0037] In one specific implementation scheme, the calculation of format reward, tool invocation correctness reward, dynamic length reward, and distillation learning reward based on teacher model distribution differences for the interaction trajectory further includes:

[0038] To calculate the distillation learning reward, the difference in the output distribution of the teacher model is introduced as the distillation learning reward term, and KL divergence is used to measure the difference in the output distribution between the teacher model and the small-scale model under the same input conditions.

[0039] Let the content generated by the teacher model be distributed as follows: The content generated by small-scale models is distributed as follows: The learning reward for distillation is expressed using the KL divergence with clipping, defined as follows:

[0040] ;

[0041] in, The upper bound is used to limit the reward range. This means clipping the KL divergence values ​​to the interval. .

[0042] In a specific feasible implementation, the step of fusing the various reward items to obtain the total reward value corresponding to the interaction trajectory includes:

[0043] The total reward is defined as follows:

[0044] ;

[0045] in, This represents the total reward value corresponding to the interaction trajectory, used to characterize the model's overall performance on that interaction trajectory.

[0046] In one specific implementation, performing trajectory reuse and trajectory data augmentation processing on the interaction trajectory to generate an expanded training dataset includes:

[0047] Perform trajectory reuse operation. For a given interactive trajectory, treat the interactive trajectory as a sequence of multiple consecutive states, and change the position of the trajectory's initial state in the sequence while keeping the order of the steps inside the trajectory unchanged.

[0048] Let the interaction trajectory have a length of The structured trajectory, by selecting the initial state of the trajectory as different intermediate states in the structured trajectory, can be obtained from the length of the trajectory. Structured trajectory derivation is obtained Each training data point inherits the subsequent inference content, tool call information, and observation result information from the corresponding initial state in the original structured trajectory, enabling the same interaction trajectory to be reused for training.

[0049] Based on trajectory reuse, trajectory data augmentation is performed. A small number of high-quality successful trajectories are used as input to automatically process the successful trajectories and generate enhanced trajectory data.

[0050] An extended training data set is constructed by combining the training data obtained from trajectory reuse with the enhanced trajectory data obtained from trajectory data augmentation.

[0051] Secondly, this application provides a reinforcement learning training system for small-scale large language models, employing the following technical solution:

[0052] A reinforcement learning training system for small-scale large language models includes:

[0053] The task construction module is used to obtain user queries and external tool sets, and construct tool integration inference training tasks based on the user queries and external tool sets;

[0054] The trajectory generation module is used to generate thinking content and tool call instructions based on the tool integrated reasoning training task driven by the small-scale large language model to be trained, and to form an interactive trajectory containing reasoning process, tool call information and tool observation results by combining the execution results of external tools.

[0055] The reward calculation module is used to calculate the format reward, tool call correctness reward, dynamic length reward, and distillation learning reward based on the teacher model distribution difference for the interaction trajectory, and to fuse the reward items to obtain the total reward value corresponding to the interaction trajectory.

[0056] The data expansion module is used to perform trajectory reuse and trajectory data augmentation processing on the interaction trajectory to generate an expanded training data set;

[0057] The reinforcement learning module is used to perform iterative parameter optimization on the small-scale large language model to be trained using a reinforcement learning strategy, based on the expanded training data set and the total reward value.

[0058] Thirdly, this application provides an electronic device, the device including a processor and a memory; the memory stores a program, the program being loaded and executed by the processor to implement a reinforcement learning training method for a small-scale large language model as described in the first aspect.

[0059] Fourthly, this application provides a computer-readable storage medium storing a program that, when executed by a processor, is used to implement a reinforcement learning training method for a small-scale large language model as described in the first aspect.

[0060] By constructing a tool integration reasoning training task around user queries and external tool sets, the small-scale large language model to be trained generates thinking content and tool call instructions under the drive of this task. The results of external tool execution are combined to form an interaction trajectory containing reasoning process, tool call information, and tool observation results. On this basis, format rewards, tool call correctness rewards, dynamic length rewards, and distillation learning rewards based on the differences in teacher model distribution are calculated for the interaction trajectory. The rewards are then fused to obtain the total reward value corresponding to the interaction trajectory. Trajectory reuse and trajectory data augmentation processing are then performed on the interaction trajectory to generate an extended training dataset. Finally, reinforcement learning strategies are used to iteratively optimize the parameters of the small-scale large language model based on the extended training dataset and the total reward value. During training, the model's decision-making and updates are no longer limited to fitting fixed offline patterns. Instead, they are continuously constrained and corrected within a closed loop of tool invocation and observation feedback, using interactive trajectories as the carrier. By simultaneously introducing and fusing rewards of different dimensions, such as format, correctness, length, and distribution differences, the reward signals can provide a more granular and targeted evaluation of the structured output and reasoning process of tool invocation. This enables the model to gain a clearer learning direction in complex tool integration reasoning tasks and reduces the instability of the training process. At the same time, trajectory reuse and trajectory data augmentation further expand existing interactive trajectories into a larger-scale training dataset, making the training sample coverage more comprehensive and efficient under limited computing power and data conditions. This allows for the continuous accumulation of effective update signals for tool invocation and reasoning behavior during the iterative optimization of model parameters under low resource constraints. Consequently, the model's reasoning and structured tool invocation capabilities in diverse tool combinations and complex task scenarios exhibit a more stable improvement trend and stronger adaptability.

[0061] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, the preferred embodiments of this application are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating the reinforcement learning training method for small-scale large language models in this application embodiment.

[0063] Figure 2 This is a schematic diagram of the overall process of the reinforcement learning training method for small-scale large language models in the embodiments of this application.

[0064] Figure 3 This is a structural block diagram of a reinforcement learning training system for small-scale large language models in an embodiment of this application.

[0065] Figure 4 This is a block diagram of an electronic device for reinforcement learning training of a small-scale large language model, as described in this application. Detailed Implementation

[0066] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this application, but are not intended to limit the scope of this application.

[0067] Optionally, this application uses the reinforcement learning training method for small-scale large language models provided in various embodiments as an example for application in electronic devices. The electronic device is a terminal or a server. The terminal can be a computer, tablet computer, etc. This embodiment does not limit the type of electronic device.

[0068] Reference Figure 1 This is a flowchart illustrating a reinforcement learning training method for a small-scale large language model according to an embodiment of this application. The method includes at least the following steps:

[0069] Step S101: Obtain user queries and external tool sets, and construct a tool integration inference training task based on user queries and external tool sets.

[0070] In step S101, this step is used to determine the basic elements upon which tool integration reasoning training depends, and to organize user queries and external tool sets in a unified task format. This ensures that the small-scale large language model to be trained has clear input objects and a range of callable tools during the training process, enabling it to perform multi-step reasoning and tool invocation decisions around user queries in subsequent training. The tool integration reasoning training task is used to characterize how the model selects and combines tools during multi-step reasoning, and its training process uses multi-step interaction trajectories as basic sample units.

[0071] Specifically, first obtain user queries User query The task input is in the form of natural language, which triggers the model to generate a multi-step reasoning process and provide a final response. Then, an external toolkit is acquired. ,in, Indicates the number of external tools. Indicates the first External tools are used to provide support during model inference, including information acquisition, computation and logic execution, hardware / software interaction, and content generation. (Based on user queries) With external tool sets The tool integrates inference training tasks and represents the interactive process of the training tasks as a step-by-step progressive sequence of inference states; in the first... The state before each reasoning step It consists of historical interaction records, which contain the first... Step-by-step natural language reasoning content , No. Step-by-step toolkit and the Observation results after step tool execution ,in Observation results This includes execution feedback from the external tool environment and result information returned by the environment. (State-based) The model then outputs natural language reasoning content in the next step. and the set of tools to be invoked This allows the training task to describe the model's step-by-step reasoning and tool-calling behavior using state sequences as a carrier.

[0072] In implementation, by uniformly defining the components of user queries, external tool sets, and interaction state sequences in this step, each interaction trajectory during training can be expressed with a consistent data structure, thereby ensuring that subsequent steps for generating, processing, and training interaction trajectories have clear object boundaries and consistent semantic orientation.

[0073] Step S102: Based on the tool integration reasoning training task, the small-scale large language model to be trained generates thinking content and tool call instructions, and combines the execution results of external tools to form an interaction trajectory that includes reasoning process, tool call information and tool observation results.

[0074] In step S102, this step is used to enable the small-scale large language model to be trained to respond to user queries. With external tool sets Under defined task conditions, multi-step interactive reasoning is performed, outputting thought content and tool invocation instructions at each step, and receiving the execution results of external tools as observation information, thus forming an interactive trajectory that can be uniformly represented. The interactive trajectory records the model's reasoning process, tool invocation process, and corresponding observation results in the form of a progressively advancing state sequence. The tool invocation instructions are encoded using a predefined structured data format, and the external tool environment parses the structured data based on a unified interface protocol and returns standardized observation results, enabling the interactive trajectory to be stored and processed in a parsable data structure.

[0075] Specifically, in the The reasoning state is obtained at the end of the interaction step. Reasoning state Represented as:

[0076] ;

[0077] In state Based on this, a small-scale large language model generates the content for the next step of natural language reasoning. and the set of tools to be invoked To enable the model to autonomously generate inference process records and tool call commands, special markers are used during training to distinguish different types of content. These special markers include... <think>、<tool_call>、 <response>,in <think>Used to identify thoughts,<tool_call> Used to identify tool invocation commands <response>Used to identify the final response content. Tool invocation commands are in...<tool_call> The tags are given in the form of structured parameters, which include at least the tool name and the parameter set.

[0078] Subsequently, the tool invocation command is sent to the external tool environment for execution. The external tool environment performs the corresponding operation based on the tool name and parameter set, and returns the execution result as the observation result. .Will Adding to the state sequence yields a new reasoning state. ,Right now:

[0079] ;

[0080] By repeatedly executing multi-step interactions in the above manner, the model's thought processes, tool usage information, and tool observation results are consistently recorded across multiple inference steps, forming an interaction trajectory. When the model is generated... <response>The interaction terminates when the marked final response is received, or when the preset maximum number of inference steps is reached, and the trajectory space immediately closes.

[0081] Step S103: Calculate the format reward, tool call correctness reward, dynamic length reward, and distillation learning reward based on the teacher model distribution difference for the interaction trajectory, and fuse the reward items to obtain the total reward value corresponding to the interaction trajectory.

[0082] In step S103, this step is used to perform fine-grained reward evaluation on the interaction trajectory formed in step S102. The quality of structured output, the degree of matching of tool calls, the control of inference length, and the consistency of the distribution of unstructured content in the interaction trajectory are quantified into multiple reward items, and these multiple reward items are merged into a single total reward value. By decomposing the training objectives of different dimensions into computable reward items, this step enables reinforcement learning training to simultaneously constrain the format standardization of tool call instructions, the matching accuracy of tool call content, and the progressive control of the length dimension of inference content. Furthermore, a teacher model is introduced to evaluate the distribution difference between the thought content and the final response content, thereby enabling the reward signal to cover both structured behavior and unstructured expression.

[0083] Specifically, the interaction trajectory includes natural language reasoning content, tool invocation information, and tool observation results at each inference step. To ensure consistent evaluation of the trajectory during training, this step defines the reward as the sum of four reward categories, and provides clear calculation rules and variable meanings for each category.

[0084] First, calculate the format reward. The format reward is used to evaluate whether the model output meets the preset structured labeling specification. A decay mechanism related to the number of training steps is introduced during training, giving more significant reward weight to correct formatting in the early stages of training. Let the current training step number be... This refers to the number of steps already executed during the current reinforcement learning training. The threshold in the early stages of training is... When the output format is completely correct and When the format is correct, the reward is a constant of 1; when the output format is completely correct and The format reward decays according to the training progress; when the output format is incorrect, the format reward is 0, and its expression is:

[0085] ;

[0086] According to this definition, format rewards can explicitly guide the model to learn a stable structured output form in the early stages of training, while gradually reducing the influence of format terms on the total reward as training progresses.

[0087] Based on format constraints, further calculate the reward for correct tool call usage. The tool invocation correctness reward is used to evaluate the degree of matching between the tool name, parameter name, and parameter values. Let the set of correctly invoked tool names be... The set of tool names invoked by the model prediction is The tool name matching reward is given in the form of intersection-union ratio, defined as follows:

[0088] ;

[0089] in, The set represents the number of elements in the set. The intersection is used to describe the parts with the same name, and the union is used to describe the range of names covered.

[0090] The parameter name matching reward is then calculated, determining the degree of matching for each correct tool invocation entry. The parameter name matching reward is defined as follows:

[0091] ;

[0092] in, This represents the set of correct tool call entries. Represents a set One of the tool call entries, This represents the parameter key-value mapping structure corresponding to the entry. This represents the parameter key-value mapping structure corresponding to the prediction item. The parameter key-value mapping structure is a structured data object consisting of parameter names and corresponding parameter values. The parameter values ​​are the input data passed to the external tool interface. and These represent the set of correct parameter names and the set of predicted parameter names corresponding to the current entry, respectively; the summation symbol represents the summation of the sets. Each item in the calculation is calculated and accumulated one by one.

[0093] Finally, the reward is calculated based on the parameter values. The parameter name is The correct parameter values, The parameter name is The predicted parameter values ​​are used to introduce an indicator function. The parameter value is set to 1 if the condition is true and 0 if the condition is false. The reward is defined as follows:

[0094] ;

[0095] in, Indicates an entry The set of parameter names contained therein, that is, the set of keys in the parameter key-value mapping structure of this entry.

[0096] Add the three sub-items above to get the matching score:

[0097] ;

[0098] in, Express the maximum possible score, and define the tool call correctness reward as:

[0099] ;

[0100] By breaking down tool call correctness into three levels—name, parameter name, and parameter value—this step enables the reward signal to clearly indicate the level at which the error occurred, thus avoiding the problem of unclear training feedback caused by evaluating tool calls in a coarse-grained manner.

[0101] Beyond ensuring correct tool usage, this step also introduces a dynamic length reward to adaptively constrain the length of the thought content. Let the length of the thought content generated by the model be... The target thinking length is Total training steps: Training progress is defined as:

[0102] ;

[0103] The dynamic length reward is defined as follows:

[0104] ;

[0105] in, To characterize the progress of training, by introducing into the denominator By changing the target constraint with the training progress, the reward for thinking length is dynamically adjusted with the training process, avoiding instability caused by imposing too strong a constraint on thinking length in the early stage of training or by lacking length guidance in the later stage of training.

[0106] Meanwhile, the interaction trajectory also includes unstructured text such as thought content and final response content. To evaluate this part, the difference in the output distribution of the teacher model is introduced as a distillation learning reward, and KL divergence is used to measure the difference in the output distribution between the teacher model and the small-scale model under the same input conditions. Let the distribution of the content generated by the teacher model be... The content generated by small-scale models is distributed as follows: The learning reward for distillation is then expressed using the KL divergence with clipping, defined as follows:

[0107] ;

[0108] in, The upper bound is used to limit the reward range. This means clipping the KL divergence values ​​to the interval. Since a smaller KL divergence indicates that the two distributions are closer, this step negatives the KL divergence and prunes it so that a higher reward is obtained when the distribution difference is small, and the reward is constrained by the pruning mechanism when the distribution difference is large, so that the reward will not fluctuate too much.

[0109] Based on the fact that all the above reward items can be calculated, this step merges the reward items to obtain the total reward value corresponding to the interaction trajectory. The total reward is defined as:

[0110] ;

[0111] in, This represents the total reward value corresponding to the interaction trajectory, used to characterize the model's overall performance on that interaction trajectory.

[0112] In implementation, the above reward design has clear substantive characteristics: First, the reward signal is decomposed into a multi-level evaluation oriented towards structured output and content matching. Tool call correctness is further subdivided into three sub-dimensions: tool name, parameter name, and parameter value. This allows the reward to reflect the specific source of tool call errors, thus avoiding the situation of coarse feedback and unclear learning direction that occurs when only answer matching or a single format item is evaluated. Second, both format rewards and dynamic length rewards introduce dynamic mechanisms related to training progress. Format rewards provide more direct structured output constraints in the early stages of training and decay as training progresses. Dynamic length rewards are determined by training progress parameters. The length constraint strength is adjusted so that the length guidance changes as training progresses, thus making the constraint focus of the reward adjustable for different training stages. Third, the learning reward is distilled to address unstructured parts such as the thinking content and the final response content. Differences in the teacher model distribution are introduced, and the reward magnitude is controlled through a KL divergence pruning mechanism, allowing the evaluation of unstructured content to be incorporated into the total reward in a controllable manner. Through the above combination, this step forms a unified reward system covering both structured tool calls and unstructured expressions, and comprehensively quantifies the interaction trajectory in the form of a total reward value, thus reflecting a design approach that combines fine-grained reward design with teacher model distillation evaluation.

[0113] Step S104: Perform trajectory reuse and trajectory data augmentation processing on the interactive trajectory to generate an expanded training data set.

[0114] In step S104, this step expands the training data scale based on the obtained interaction trajectories, increasing the number of times and coverage of limited training trajectories in reinforcement learning training. An interaction trajectory consists of multiple progressively advancing state records, with each trajectory containing multiple tool calls and corresponding observation results. Since obtaining high-quality interaction trajectories in vertical business scenarios is costly, this step performs post-processing on the interaction trajectories. Multiple training data are generated from a single structured interaction trajectory through trajectory reuse operations, and enhanced trajectory data containing erroneous calls and recovery steps are generated from a small number of high-quality successful trajectories through trajectory data augmentation operations, thus forming an expanded training data set.

[0115] Specifically, the trajectory reuse operation is performed first. For a given interactive trajectory, the trajectory is treated as a sequence of multiple consecutive states, and the position of the initial state of the trajectory within the sequence is changed while maintaining the order of the steps within the trajectory. Let the interactive trajectory be of length *l*. The structured trajectory, by selecting the initial state of the trajectory as different intermediate states within the structured trajectory, can be obtained from the length of the structured trajectory. Structured trajectory derivation is obtained Each training data point inherits the subsequent inference content, tool call information, and observation result information from the corresponding initial state in the original structured trajectory, enabling the same interaction trajectory to be reused for training. For each training data point derived from the original interaction trajectory, the corresponding reward value is recalculated based on the inference steps it contains.

[0116] Building upon trajectory reuse, trajectory data augmentation is further performed. Trajectory data augmentation takes a small number of high-quality successful trajectories as input and automatically processes them to generate enhanced trajectory data. This enhanced trajectory data includes erroneous call trajectories from the tool invocation process and corresponding recovery steps, ensuring that the generated enhanced trajectories cover the process of tool invocation errors and their correction. Erroneous calls are generated by replacing, deleting, or perturbing tool names or parameters in the original successful trajectories. By aggregating the training data obtained from trajectory reuse with the enhanced trajectory data obtained from trajectory data augmentation, an extended training dataset is constructed. This extended training dataset records the reused and augmented interaction trajectory data, maintaining a structured representation consistent with the original interaction trajectories.

[0117] Step S105: Based on the expanded training dataset and total reward value, a reinforcement learning strategy is used to iteratively optimize the parameters of the small-scale large language model to be trained.

[0118] In step S105, this step introduces the total reward value obtained in step S103 and the expanded training data set obtained in step S104 into the reinforcement learning training process. Through a parameter iterative update mechanism, the policy of the small-scale large language model to be trained is optimized, allowing the model's decision-making behavior at the interaction trajectory level to gradually approach the optimal policy. The expanded training data set includes training data derived from trajectory reuse and augmented trajectory data generated through trajectory data augmentation. The total reward value is used to measure the model's overall performance on the corresponding interaction trajectory. This step, under the reinforcement learning policy, utilizes interaction trajectory samples and reward signals to complete the update calculation of model parameters.

[0119] Specifically, interaction trajectory samples are obtained from the expanded training dataset, and each interaction trajectory sample is associated with its corresponding total reward value. The interaction trajectory samples record the natural language reasoning content, tool invocation information, and tool observation results of the model during multi-step inference in the form of a state sequence. The model in the... The state after the interaction is recorded as The model's strategy is denoted as ,Strategy Used in a given state The system outputs the next step of natural language inference and the set of tools to be invoked, i.e.:

[0120] ;

[0121] Under this strategy, the model bases its actions on the current state at each interaction step. It generates decisions about the next toolset and obtains observations through interaction with the external tool environment. And the corresponding rewards. To characterize the theoretical objective of policy optimization, the optimal tool selection can be abstractly represented as maximizing the reward function:

[0122] ;

[0123] in, For external toolkits For calling The subsequent observation results Indicates the state Select tool set And obtain observation results The second one obtained Step reward value. The step reward value is determined using the fusion calculation method in step S103, which involves fusing the output format reward, tool call correctness reward, dynamic length reward, and distillation learning reward corresponding to the step to obtain the fusion reward value for the step. This fusion reward value is then used as the reward evaluation for the step behavior. The same interaction trajectory contains When performing each reasoning step, the steps will be... By accumulating the rewards step by step, the total reward value for the interaction trajectory is obtained. Furthermore, the overall optimization objective of the model is to progressively approximate the optimal strategy that maximizes the expected value of the accumulated reward during the interaction process in the parameter space, and its objective form is expressed as:

[0124] ;

[0125] in, Indicating in strategy The expected outcome is as follows. Based on the above policy definition and objective function, during the parameter iterative optimization process, the model parameters are treated as variables to be optimized, and the parameters are updated in each iteration based on the interaction trajectory samples and corresponding reward evaluations. Parameter updates are based on the policy gradient method. The update process includes: determining the policy based on the current parameters. The output probability distribution under each state is calculated; combining the actual output content recorded in the interaction trajectory samples with the tool call decision, the cumulative reward evaluation of the policy on the trajectory is calculated; then, based on the cumulative reward evaluation, an optimization direction for the policy parameters is constructed, and the model parameters are updated once along this optimization direction. After completing one parameter update, the next iteration begins, and interaction trajectory samples are selected from the expanded training dataset, and the above calculation and update process is repeated, thereby achieving continuous iterative optimization of the model parameters. In implementation, both reused trajectory samples and augmented trajectory samples in the expanded training dataset participate in the reinforcement learning parameter iterative optimization process, and together with the total reward value, constitute the input for policy updates, enabling the training process to continuously perform parameter update calculations based on a limited number of interaction trajectories.

[0126] In summary, combining Figure 2 This application proposes a reinforcement learning training method for small-scale large language models: First, it acquires user queries and external tool sets and constructs a tool-integrated reasoning training task. Under a unified task framework, the model generates thought content and tool invocation instructions through multi-step interactions, and interacts with the external tool environment to obtain tool observation results, thus forming an interaction trajectory containing the reasoning process, tool invocation information, and observation results. Subsequently, a fine-grained reward evaluation mechanism is introduced for the interaction trajectory. The distillation learning constraints corresponding to output format standardization, tool invocation correctness, thought length control, and differences in teacher model distribution are quantified into multiple reward items and fused together to form a comprehensive reward system. The total reward value is used as the basis for trajectory reuse and trajectory data augmentation. Trajectory reuse derives multiple training data from a single structured trajectory by changing the initial state of the trajectory. Trajectory data augmentation automatically generates enhanced trajectory data containing error calls and recovery steps using a small number of high-quality successful trajectories as input, ultimately forming an expanded training data set. Based on the expanded training data set and the total reward value, the parameters of a small-scale large language model are iteratively optimized under the reinforcement learning strategy. This allows the model to continuously adjust its behavior in generating inference content and tool call instructions under different inference states, and gradually converge to a better decision strategy as the reward signal is received.

[0127] Because the training process uses interactive trajectories as a carrier, the model's thought process, tool call instructions, and tool observation results are consistently recorded and incorporated into reinforcement learning optimization. This allows the model to move beyond relying on a single offline trajectory fitting, repeatedly experiencing tool selection, parameter filling, and observation feedback in a real interactive context of multi-step decision-making. Consequently, it is easier to form transferable decision-making patterns rather than fixed patterns when facing diverse tool combinations and complex task structures. Furthermore, the reward signal is decomposed into multi-dimensional constraints oriented towards structured behavior and unstructured expression: the correctness of tool call is subdivided into tool name, parameter name, and parameter value levels; the output format and thought length are dynamically adjusted in relation to training progress; and the distillation learning constraint uses the differences in teacher model distribution to controllably evaluate the thought content and final response content. This enables the model to obtain a clearer learning direction and a more stable gradient source during training, avoiding insufficient feedback and training fluctuations that occur when relying solely on coarse-grained rewards. Meanwhile, trajectory reuse transforms a single structured trajectory into multiple trainable samples, and trajectory data augmentation expands a small number of successful trajectories into enhanced trajectories that cover error invocation and recovery steps. This significantly improves the sample utilization rate under conditions of limited computing power and limited high-quality data, and the training samples are more fully covered in both correct invocation and error correction scenarios. This leads to a more stable improvement in the accuracy and robustness of the model in calling structured tools, and enables the formation of a sustainable iterative reinforcement learning training mechanism under low-resource conditions.

[0128] Figure 3 This is a structural block diagram of a reinforcement learning training system for small-scale large language models provided in one embodiment of this application. The system includes at least the following modules:

[0129] The task construction module is used to obtain user queries and external tool sets, and to construct tool integration inference training tasks based on user queries and external tool sets;

[0130] The trajectory generation module is used to generate thinking content and tool call instructions based on the tool integration reasoning training task driven by the small-scale large language model to be trained, and to form an interactive trajectory containing reasoning process, tool call information and tool observation results by combining the execution results of external tools.

[0131] The reward calculation module is used to calculate the format reward, tool call correctness reward, dynamic length reward, and distillation learning reward based on the teacher model distribution difference for the interaction trajectory, and to fuse the reward items to obtain the total reward value corresponding to the interaction trajectory.

[0132] The data expansion module is used to perform trajectory reuse and trajectory data augmentation processing on the interactive trajectory to generate an expanded training dataset;

[0133] The reinforcement learning module is used to perform iterative parameter optimization on a small-scale large language model to be trained, based on an expanded training dataset and total reward value, using a reinforcement learning strategy.

[0134] For relevant details, please refer to the above method implementation examples.

[0135] Figure 4 This is a block diagram of an electronic device provided in one embodiment of this application. The device includes at least a processor 401 and a memory 402.

[0136] Processor 401 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 401 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 401 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 401 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 401 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0137] Memory 402 may include one or more computer-readable storage media, which may be non-transitory. Memory 402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in memory 402 is used to store at least one instruction, which is executed by processor 401 to implement the reinforcement learning training method for small-scale large language models provided in the method embodiments of this application.

[0138] In some embodiments, the electronic device may also optionally include: a peripheral device interface and at least one peripheral device. The processor 401, memory 402, and peripheral device interface can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface via a bus, signal line, or circuit board. Indicatively, peripheral devices include, but are not limited to: radio frequency circuits, touch displays, audio circuits, and power supplies.

[0139] Of course, electronic devices may also include fewer or more components, and this embodiment does not limit this.

[0140] Optionally, this application also provides a computer-readable storage medium storing a program that is loaded and executed by a processor to implement the reinforcement learning training method for small-scale large language models described in the above method embodiments.

[0141] Optionally, this application also provides a computer product including a computer-readable storage medium storing a program, which is loaded and executed by a processor to implement the reinforcement learning training method for small-scale large language models described in the above method embodiments.

[0142] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.< / response> < / response> < / think> < / response> < / think>

Claims

1. A reinforcement learning training method for small-scale large language models, characterized in that, The method includes: Obtain user queries and a set of external tools, and construct a tool integration inference training task based on the user queries and the set of external tools; the user queries are task inputs in natural language form. Based on the tool-integrated reasoning training task, the small-scale large language model to be trained generates thinking content and tool call instructions, and combines the execution results of external tools to form an interactive trajectory that includes reasoning process, tool call information and tool observation results; For the interaction trajectory, calculate the format reward, tool call correctness reward, dynamic length reward, and distillation learning reward based on the teacher model distribution difference, and fuse the reward items to obtain the total reward value corresponding to the interaction trajectory; Perform trajectory reuse and trajectory data augmentation processing on the interaction trajectory to generate an expanded training data set, including: Perform trajectory reuse operation. For a given interactive trajectory, treat the trajectory as a sequence of multiple consecutive states, and change the position of the initial state of the trajectory within the sequence while maintaining the order of steps within the trajectory. Let the interactive trajectory be of length... The structured trajectory, by selecting the initial state of the trajectory as different intermediate states in the structured trajectory, can be obtained from the length of the structured trajectory. Structured trajectory derivation is obtained Each training data point inherits the subsequent inference content, tool call information, and observation result information from the corresponding initial state in the original structured trajectory, enabling the same interactive trajectory to be reused for training. Based on trajectory reuse, trajectory data augmentation is performed, using a small number of high-quality successful trajectories as input to automatically process the successful trajectories and generate enhanced trajectory data. An extended training data set is constructed by combining the training data obtained from trajectory reuse with the enhanced trajectory data obtained from trajectory data augmentation. Based on the expanded training dataset and the total reward value, a reinforcement learning strategy is used to iteratively optimize the parameters of the small-scale large language model to be trained.

2. The reinforcement learning training method for small-scale large language models according to claim 1, characterized in that, The calculation of format rewards, tool call correctness rewards, dynamic length rewards, and distillation learning rewards based on teacher model distribution differences for the interaction trajectory includes: Calculation format reward The format reward is used to evaluate whether the model output meets the preset structured labeling specification; Let the current training step number be... The threshold in the early stages of training is When the output format is completely correct and When the format is correct, the reward is a constant of 1; when the output format is completely correct and The format reward decays according to the training progress; when the output format is incorrect, the format reward is 0, and its expression is: 。 3. The reinforcement learning training method for small-scale large language models according to claim 2, characterized in that, The calculation of format rewards, tool call correctness rewards, dynamic length rewards, and distillation learning rewards based on teacher model distribution differences for the interaction trajectory also includes: Rewards for correct use of computing tools The tool call correctness reward is used to evaluate the degree of matching between the tool name, parameter name, and parameter value; Let the set of correctly invoked tool names be... The set of tool names invoked by the model prediction is The tool name matching reward is given in the form of intersection-union ratio, defined as follows: ； in, Indicates the number of elements in the set; Calculate the parameter name matching reward. For each correct tool invocation entry, calculate the degree of parameter name matching. The parameter name matching reward is defined as follows: ； in, This represents the set of correct tool call entries. Represents a set One of the tool call entries, This represents the parameter key-value mapping structure corresponding to the entry. This represents the parameter key-value mapping structure corresponding to the predicted entry; and These represent the set of correct parameter names and the set of predicted parameter names corresponding to the current entry, respectively; the summation symbol represents the summation of the sets. Each item in the calculation is calculated and accumulated individually; Calculate the reward based on the parameter values. The parameter name is The correct parameter values, The parameter name is The predicted parameter values ​​are used to introduce an indicator function. The parameter value represents a value of 1 if the condition is true and 0 if the condition is false. The reward for matching the parameter value is defined as follows: ； in, Indicates an entry The set of parameter names contained therein; Add the three sub-items together to get the matching score: ； in, Express the maximum possible score, and define the tool call correctness reward as: 。 4. The reinforcement learning training method for small-scale large language models according to claim 3, characterized in that, The calculation of format rewards, tool call correctness rewards, dynamic length rewards, and distillation learning rewards based on teacher model distribution differences for the interaction trajectory also includes: Calculate dynamic length reward Let the length of the thought content generated by the model be... The target thinking length is Total training steps: Training progress is defined as: ； The dynamic length reward is defined as follows: ； in, Used to characterize the progress of training.

5. The reinforcement learning training method for small-scale large language models according to claim 4, characterized in that, The calculation of format rewards, tool call correctness rewards, dynamic length rewards, and distillation learning rewards based on teacher model distribution differences for the interaction trajectory also includes: To calculate the distillation learning reward, the difference in the output distribution of the teacher model is introduced as the distillation learning reward term, and KL divergence is used to measure the difference in the output distribution between the teacher model and the small-scale model under the same input conditions. Let the content generated by the teacher model be distributed as follows: The content generated by small-scale models is distributed as follows: The learning reward for distillation is expressed using the KL divergence with clipping, defined as follows: ； in, The upper bound is used to limit the reward range. This means clipping the KL divergence values ​​to the interval. .

6. The reinforcement learning training method for small-scale large language models according to claim 5, characterized in that, The process of merging the various reward items to obtain the total reward value corresponding to the interaction trajectory includes: The total reward is defined as follows: ； in, This represents the total reward value corresponding to the interaction trajectory, used to characterize the model's overall performance on that interaction trajectory.

7. A reinforcement learning training system for small-scale large language models, characterized in that, include: The task construction module is used to obtain user queries and external tool sets, and construct tool integration inference training tasks based on the user queries and external tool sets; The user query is a task input in natural language form; The trajectory generation module is used to generate thinking content and tool call instructions based on the tool integrated reasoning training task driven by the small-scale large language model to be trained, and to form an interactive trajectory containing reasoning process, tool call information and tool observation results by combining the execution results of external tools. The reward calculation module is used to calculate the format reward, tool call correctness reward, dynamic length reward, and distillation learning reward based on the teacher model distribution difference for the interaction trajectory, and to fuse the reward items to obtain the total reward value corresponding to the interaction trajectory. The data expansion module is used to perform trajectory reuse and trajectory data augmentation processing on the interactive trajectory to generate an expanded training data set, including: Perform trajectory reuse operation. For a given interactive trajectory, treat the trajectory as a sequence of multiple consecutive states, and change the position of the initial state of the trajectory within the sequence while maintaining the order of steps within the trajectory. Let the interactive trajectory be of length... The structured trajectory, by selecting the initial state of the trajectory as different intermediate states in the structured trajectory, can be obtained from the length of the structured trajectory. Structured trajectory derivation is obtained Each training data point inherits the subsequent inference content, tool call information, and observation result information from the corresponding initial state in the original structured trajectory, enabling the same interactive trajectory to be reused for training. Based on trajectory reuse, trajectory data augmentation is performed, using a small number of high-quality successful trajectories as input to automatically process the successful trajectories and generate enhanced trajectory data. An extended training data set is constructed by combining the training data obtained from trajectory reuse with the enhanced trajectory data obtained from trajectory data augmentation. The reinforcement learning module is used to perform iterative parameter optimization on the small-scale large language model to be trained using a reinforcement learning strategy, based on the expanded training data set and the total reward value.

8. An electronic device, characterized in that, The device includes a processor and a memory; the memory stores a program, which is loaded and executed by the processor to implement a reinforcement learning training method for small-scale large language models as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a program that, when executed by a processor, is used to implement a reinforcement learning training method for small-scale large language models as described in any one of claims 1 to 6.

Images