A language model self-adaption strategy fusion training method and system

Abstract

The application provides a language model adaptive strategy fusion training method and system, which relates to the technical field of language model training and comprises the following steps: tracking the strategy entropy in the reinforcement learning stage and the relative entropy in the supervised fine-tuning stage, and calculating the change ratio of adjacent training steps; evaluating the inference accuracy of offline distillation samples by a verifier and mapping the accuracy to an adaptive coefficient; constructing a mixed offline strategy based on the coefficient, calculating and constraining the ratio weight of the current strategy; combining the change ratio to calculate and clip the gradient contribution weight of the supervised fine-tuning, and modulating the gradient to be optimized cooperatively with the reinforcement learning gradient. The application adjusts the gradient weight by dynamically calculating the entropy change ratio, constructs a mixed strategy based on the accuracy, clips the weight and adjusts the step length, balances the model exploration space, solves the mismatch between offline data and the current strategy distribution, guarantees the training stability, efficiently improves the inference ability of a small model, considers the efficiency and reliability, and is suitable for different samples and scenes.

Description

Technical Field

[0001] This invention relates to the field of language model training technology, and more specifically, to a language model adaptive strategy fusion training method and system. Background Technology

[0002] With the widespread application of artificial intelligence technology in fields such as intelligent customer service, automatic code generation, and scientific computing, lightweight language models have become a core requirement for implementation due to their advantages of low deployment cost and fast response speed. Currently, the training of small models mainly relies on reinforcement learning or supervised fine-tuning of a single policy, and often combines offline distillation data for optimization. However, reinforcement learning is easily limited by the exploration space and finds it difficult to break through the capability boundary, while supervised fine-tuning tends to mechanically imitate large models and lacks generalization. The problem of adapting offline data to the model's current policy has also not been effectively solved, which restricts the improvement of inference performance of small models.

[0003] Traditional language model training methods, especially in pure reinforcement learning training, suffer from rigid model policies that lead to insufficient exploration space and an inability to efficiently expand inference capabilities, resulting in a performance bottleneck in the later stages of training. Pure supervised fine-tuning relies on the fixed trajectory of large model distillation, making it easy for small models to mechanically replicate the output of large models, lacking autonomous inference logic and exhibiting poor generalization to new scenarios. Meanwhile, existing offline strategies are mostly set with fixed values, failing to consider the differences in inference accuracy of offline distillation samples, leading to a mismatch between offline data and the current policy distribution of the model. This can easily cause entropy collapse or a sharp drop in performance, and the lack of dynamic optimization of gradient weights and parameter step sizes makes it difficult to balance training stability and efficiency.

[0004] Therefore, it is necessary to design an adaptive policy fusion training method and system for language models to solve the problems of insufficient exploration in pure reinforcement learning, mechanical imitation in pure supervised fine-tuning, distribution mismatch and entropy collapse caused by fixed offline policies, and lack of dynamic optimization in training, which lead to weak reasoning ability, poor stability and low efficiency of small models. Summary of the Invention

[0005] In view of this, the present invention proposes a language model adaptive policy fusion training method and system, which aims to solve the problems of insufficient exploration of pure reinforcement learning, mechanical imitation of pure supervised fine-tuning, distribution mismatch and entropy collapse caused by fixed offline policies, and lack of dynamic optimization in training, resulting in weak reasoning ability, poor stability and low efficiency of small models.

[0006] In one aspect, this invention proposes a language model adaptive policy fusion training method, comprising: The policy entropy during the tracking reinforcement learning phase and the relative entropy during the supervised fine-tuning phase are compared, and the change ratio of the policy entropy and the relative entropy between adjacent training steps is calculated. The inference accuracy of offline distillation samples is evaluated using a validator, and the inference accuracy is mapped to adaptive coefficients. Based on the adaptive coefficients, a hybrid offline strategy is constructed, the ratio weight of the current strategy relative to the hybrid offline strategy is calculated, and constraints are applied to the ratio weight to obtain the constrained ratio weight. The contribution weights of the supervised fine-tuning gradient are calculated based on the change ratio and the contribution weights are clipped to obtain clipped contribution weights. The supervised fine-tuning gradient is modulated based on the clipped contribution weights and the constraint ratio weights to obtain the modulated supervised fine-tuning gradient. The modulated supervised fine-tuning gradient is then co-optimized with the reinforcement learning gradient. When constructing a hybrid offline strategy, the adaptive coefficient is used as the first weight, and the hybrid offline strategy is calculated using the first weight and the current strategy. When obtaining the pruning contribution weight, the initial contribution weight is obtained based on the relative entropy change ratio and the strategy entropy change ratio, and the initial contribution weight is pruned based on the pruning upper limit and the pruning lower limit to obtain the pruning contribution weight.

[0007] Further, when calculating the change ratio of the policy entropy and the relative entropy between adjacent training steps, the following steps are included: Divide the policy entropy of the next step in adjacent training steps by the policy entropy of the previous step to obtain the policy entropy ratio. The absolute value of the difference between 1 and the policy entropy ratio is the policy entropy change rate. Divide the relative entropy of the next step in adjacent training steps by the relative entropy of the previous step to obtain the relative entropy ratio. The absolute value of the difference between 1 and the relative entropy ratio is the relative entropy change rate.

[0008] Further, when evaluating the inference accuracy of offline distillation samples through a validator and mapping the inference accuracy to adaptive coefficients, the process includes: The reasoning results of the offline distillation samples are input into the verifier, which judges the correctness of the reasoning results and determines the reasoning accuracy based on the ratio of the number of correct results to the total number of samples. Establish the mapping relationship between the inference accuracy and the adaptive coefficient; When the inference accuracy is greater than or equal to the accuracy threshold, the adaptive coefficient is mapped to a value close to 1. When the inference accuracy is less than the accuracy threshold, the adaptive coefficient is mapped to a value close to 0.

[0009] Further, when calculating the hybrid offline policy using the first weight and the current policy, the following steps are included: The difference between 1 and the first weight is used as the second weight; The product of the second weight and the current strategy, plus the adaptive coefficient, yields the hybrid offline strategy.

[0010] Further, calculating the ratio weight of the current policy relative to the hybrid offline policy, and imposing constraints on the ratio weight to obtain the constrained ratio weight includes: Divide the current strategy by the hybrid offline strategy to obtain the ratio weight; The first weight is used as the maximum constraint value of the ratio weight; When the ratio weight is greater than the maximum constraint value, then the constraint ratio weight is equal to the maximum constraint value; When the ratio weight is less than or equal to the maximum constraint value, the constraint ratio weight is equal to the ratio weight.

[0011] Further, the initial contribution weight is obtained based on the relative entropy change ratio and the strategy entropy change ratio, and the initial contribution weight is pruned based on the pruning upper limit and the pruning lower limit to obtain the pruned contribution weight, including: The initial contribution weight is equal to the relative entropy change ratio divided by the strategy entropy change ratio; When the initial contribution weight is less than the clipping lower limit, the initial contribution weight is adjusted to the clipping lower limit to obtain the clipping contribution weight. When the initial contribution weight is greater than the clipping upper limit, the initial contribution weight is adjusted to the clipping upper limit to obtain the clipping contribution weight. When the initial contribution weight is greater than or equal to the lower limit of the clipping and the initial contribution weight is less than or equal to the upper limit of the clipping, then the clipping contribution weight is equal to the initial contribution weight.

[0012] Furthermore, the lower limit of the clipping is kept constant, and the upper limit of the clipping is adjusted based on the number of samples. When the number of samples increases, the upper limit of the cropping value is increased by a preset ratio; When the number of samples decreases, the upper limit of the cropping value is reduced by a preset ratio.

[0013] Further, when modulating the supervised fine-tuning gradient based on the pruning contribution weight and the constraint ratio weight, the process includes: Multiply the clipping contribution weight by the constraint ratio weight to obtain the modulation coefficient; The modulation supervision fine-tuning gradient is obtained by multiplying the modulation coefficient by the supervision fine-tuning gradient.

[0014] Furthermore, when co-optimizing the modulation-supervised fine-tuning gradient and the reinforcement learning gradient, the following steps are included: The modulation-supervised fine-tuning gradient is added to the reinforcement learning gradient to obtain the total gradient, and the training parameters are updated using the total gradient according to the iteration frequency. When the total gradient is greater than the gradient threshold, the step size of the training parameter update is reduced. When the total gradient is less than the gradient threshold, the step size of the training parameter update is increased. When the total gradient equals the gradient threshold, the step size for updating the training parameters remains unchanged.

[0015] Compared with existing technologies, the beneficial effects of this invention are as follows: The language model adaptive policy fusion training method of this invention dynamically calculates the ratio of policy entropy change during the reinforcement learning phase to relative entropy change during the supervised fine-tuning phase, adaptively adjusting the gradient contribution weights of supervised fine-tuning, effectively balancing the model exploration space, and avoiding the shortcomings of insufficient exploration in pure RL or mechanical imitation in pure SFT. Simultaneously, it constructs a hybrid offline policy based on the inference accuracy of offline distilled samples, solving the problem of mismatch between offline data and the current policy distribution, and preventing entropy collapse or performance collapse caused by fixed offline policies. Furthermore, it ensures training stability by pruning weights and dynamically adjusting the parameter update step size, ultimately efficiently improving the inference ability of small models, balancing training efficiency and performance reliability, and adapting to samples of different quality and training scenarios.

[0016] On the other hand, this invention proposes a language model adaptive policy fusion training system, comprising: The entropy change calculation module is used to track the policy entropy in the reinforcement learning phase and the relative entropy in the supervised fine-tuning phase, and to calculate the change ratio of the policy entropy and the relative entropy in adjacent training steps. An adaptive coefficient mapping module is used to evaluate the inference accuracy of offline distillation samples through a validator and map the inference accuracy to adaptive coefficients; The weight calculation module is used to construct a hybrid offline strategy based on the adaptive coefficients, calculate the ratio weight of the current strategy relative to the hybrid offline strategy, and apply constraints to the ratio weight to obtain the constrained ratio weight. The gradient optimization module is used to calculate the contribution weight of the supervised fine-tuning gradient based on the change ratio and to prune the contribution weight to obtain the pruned contribution weight. Based on the pruned contribution weight and the constraint ratio weight, the supervised fine-tuning gradient is modulated to obtain the modulated supervised fine-tuning gradient. The modulated supervised fine-tuning gradient is then co-optimized with the reinforcement learning gradient.

[0017] It is understandable that the above-mentioned language model adaptive strategy fusion training method and system have the same beneficial effects, which will not be elaborated here. Attached Figure Description

[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart of a language model adaptive policy fusion training method provided in an embodiment of the present invention; Figure 2 This is a functional block diagram of a language model adaptive policy fusion training system provided in an embodiment of the present invention. Detailed Implementation

[0019] Exemplary embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art. It should be noted that, without conflict, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Reference Figure 1 As shown in some embodiments of this application, a language model adaptive policy fusion training method includes: The policy entropy during the tracking reinforcement learning phase and the relative entropy during the supervised fine-tuning phase are compared, and the change ratio of the policy entropy and the relative entropy between adjacent training steps is calculated. The inference accuracy of offline distillation samples is evaluated using a validator, and the inference accuracy is mapped to adaptive coefficients. Based on the adaptive coefficients, a hybrid offline strategy is constructed, the ratio weight of the current strategy relative to the hybrid offline strategy is calculated, and constraints are applied to the ratio weight to obtain the constrained ratio weight. The contribution weights of the supervised fine-tuning gradient are calculated based on the change ratio and the contribution weights are clipped to obtain clipped contribution weights. The supervised fine-tuning gradient is modulated based on the clipped contribution weights and the constraint ratio weights to obtain the modulated supervised fine-tuning gradient. The modulated supervised fine-tuning gradient is then co-optimized with the reinforcement learning gradient. When constructing a hybrid offline strategy, the adaptive coefficient is used as the first weight, and the hybrid offline strategy is calculated using the first weight and the current strategy. When obtaining the pruning contribution weight, the initial contribution weight is obtained based on the relative entropy change ratio and the strategy entropy change ratio, and the initial contribution weight is pruned based on the pruning upper limit and the pruning lower limit to obtain the pruning contribution weight.

[0021] Specifically, the reinforcement learning phase refers to the stage in language model training where reinforcement learning techniques are central. The core objective of this phase is to optimize the model's reasoning paths within its existing capabilities. This is achieved through reinforcement learning with verifiable rewards (such as rewards for verifying the correctness of reasoning results). Simultaneously, it shrinks the model's exploration space to reduce redundant generation, serving the function of "refining" the model's existing reasoning patterns, enabling the model to make reasoning decisions more efficiently within its existing capabilities. Policy entropy refers to a quantitative indicator of the uncertainty of the model's policy distribution during the reinforcement learning phase. Its value reflects the activity level of the model's current exploration—a higher policy entropy value indicates a more dispersed policy distribution, more active exploration behavior, and a potential for trying more diverse reasoning paths. The lower the policy entropy value, the more concentrated the model's policy distribution, the less active the exploration behavior, and the tendency to follow fixed inference paths. The supervised fine-tuning stage refers to the stage in language model training where supervised fine-tuning technology is central. This stage uses high-quality inference trajectories generated by the large model (offline distillation data) as supervision signals. The core objective is to transfer the new inference patterns of the large model to the small model, thereby expanding the exploration space of the small model and fulfilling the function of "expanding" the model's capability boundaries, helping the small model break through the existing upper limit of inference ability. Relative entropy refers to the quantitative indicator of the difference between the current policy of the small model and the distillation policy of the large model (i.e., the policy corresponding to the high-quality inference trajectories generated by the large model) during the supervised fine-tuning stage. Its value reflects the difference between the small model's policy and the large model's policy. The alignment degree of quality policies—the lower the relative entropy value, the smaller the difference between the two, and the better the alignment effect between the small model's policy and the large model's quality policy; the higher the relative entropy value, the greater the difference between the two, and the worse the alignment effect; the change ratio refers to the degree of change of entropy value between two adjacent training steps, divided into two categories: policy entropy change ratio and relative entropy change ratio. Its value is used to judge the current exploration activity of the model (a small policy entropy change ratio indicates that reinforcement learning exploration is not active, and a large one indicates that reinforcement learning exploration is active) and the trend of alignment change with the large model's policy (a small relative entropy change ratio indicates that the alignment degree has changed little, and a large one indicates that the alignment degree has changed much). It is the core basis for adjusting the contribution weight of supervised fine-tuning gradient; inference accuracy refers to the accuracy of the model's inference results in offline distillation samples. The correctness ratio, evaluated by a dedicated validator (such as a verification tool to judge the correctness of the solution steps and answers in math problems), is specifically calculated as "the ratio of the number of correct inference results in the offline distillation samples to the total number of samples." It is the direct basis for mapping the adaptive coefficient. A high inference accuracy indicates that the model has strong inference ability on the task corresponding to this type of sample, while a low accuracy indicates that the model needs to rely on the high-quality inference mode of a large model. The adaptive coefficient refers to the parameter mapped from the inference accuracy of the offline distillation samples, with a value range of [0,1]. Its value is adaptively adjusted as the inference accuracy changes. When the inference accuracy is greater than or equal to the preset threshold, the adaptive coefficient approaches 1, indicating that the model has strong ability on this type of task and can rely more on its own strategy.When the inference accuracy is less than a preset threshold, the adaptive coefficient approaches 0, indicating that the model's ability in this type of task is weak and it needs to rely more on the large model strategy. Its core function is to adjust the weights of different strategies in the hybrid offline strategy. The hybrid offline strategy refers to an offline strategy constructed based on the adaptive coefficient, which integrates a preset benchmark value (determined by the common inference strategy of high-accuracy samples in the offline distillation samples) with the model's current strategy. Its core function is to solve the problem of distribution mismatch between offline distillation data and the model's current reinforcement learning strategy, and to avoid entropy collapse or performance collapse that is prone to occur when using a fixed offline strategy. The ratio weight refers to the numerical ratio of the model's current strategy to the hybrid offline strategy, calculated as "the current strategy value divided by the hybrid offline strategy value". The value of the ratio weight reflects the degree of deviation between the current strategy and the hybrid offline strategy—the greater the deviation, the greater the ratio weight. Without constraint, this could lead to gradient explosion or parameter oscillation during training. It is a key indicator for applying constraints to ensure training stability. The constraint ratio weight refers to the result obtained after applying an upper limit constraint to the ratio weight. The upper limit is the ratio of the difference between 1 and (1 minus the adaptive coefficient). The specific processing logic is as follows: when the ratio weight is greater than the upper limit, it is adjusted to the upper limit; when the ratio weight is less than or equal to the upper limit, the original ratio weight is retained. Its core function is to ensure that the gradient is bounded, avoid training instability caused by excessive deviation between the current strategy and the hybrid offline strategy, and ensure the consistency between offline data and the hybrid offline strategy. Efficient fusion of online RL strategies; Supervised fine-tuning gradient refers to the gradient used to update model parameters during the supervised fine-tuning phase. It is calculated from offline distillation data (high-quality inference trajectories of large models). Its role is to transfer the inference patterns of large models to small models, helping small models expand their exploration space and break through the boundaries of their existing inference capabilities. Its contribution in training is adjusted by subsequent contribution weights and pruning contribution weights. Contribution weight refers to the weight used to adjust the contribution of supervised fine-tuning gradient in training. It is calculated as the ratio of the relative entropy change rate to the policy entropy change rate. Its value changes dynamically with the model's exploration state—when reinforcement learning exploration is inactive (the policy entropy change rate is small), the contribution weight is large, enhancing supervised fine-tuning. The gradient guides the exploration space. When reinforcement learning exploration is active (the policy entropy change rate is large), the contribution weight is small to avoid excessive interference of the supervised fine-tuning gradient with the current policy. Pruning the contribution weight refers to the weight obtained after pruning the contribution weight according to a preset range (usually [1, G], where G is the number of samples). The specific processing logic is as follows: when the contribution weight is less than 1, it is adjusted to 1; when the contribution weight is greater than G, it is adjusted to G; when the contribution weight is between [1, G], the original contribution weight is retained. Its core function is to avoid the contribution weight being too small, resulting in insufficient guidance for supervised fine-tuning (insufficient exploration space) or too large, resulting in excessive supervised fine-tuning (excessive exploration space), and to ensure that the contribution of the supervised fine-tuning gradient is within a reasonable range.The reinforcement learning gradient refers to the gradient used to update model parameters during the reinforcement learning phase. It is calculated in reinforcement learning with verifiable rewards (such as rewarding based on the correctness of inference results). Its function is to optimize the model's inference path within its existing capabilities, shrink the model's exploration space to reduce redundant generation, refine existing effective inference patterns, and work in conjunction with the modulated supervised fine-tuning gradient to efficiently enhance the model's inference ability.

[0022] Specifically, during the training process, two key entropy metrics are continuously tracked: one is the policy entropy of the model itself during the reinforcement learning (RL) phase. The first is used to measure the exploration activity of the current strategy; the second is the relative entropy of the model in the supervised fine-tuning (SFT) stage relative to the distillation strategy of the large model. This reflects its alignment with high-quality inference trajectories. Based on the entropy values ​​of consecutive training steps, the rate of change between adjacent steps is calculated: , For each offline distillation data ( The accuracy of its inference results is evaluated using a dedicated validator, and this accuracy is mapped to an adaptive coefficient. If the sample reasoning is correct and the model is able to reproduce it, then If the samples depend on the capabilities of a large model and the model itself is difficult to generate, then 0.

[0023] based on Construct a hybrid offline policy estimate: +(1- )* in This is the current strategy. Importance weights are then calculated. and impose theoretical constraints. This prevents gradient explosion or entropy collapse caused by excessive policy distribution offset, ensuring training stability.

[0024] Set the contribution weight of the SFT gradient to Where (G=8) is the preset upper limit. When RL exploration is insufficient ( When w is relatively small, it increases, enhancing SFT guidance to expand the exploration space; when RL exploration is active ( When the value is large, w decreases to prevent the SFT from excessively interfering with the current policy. Ultimately, the SFT gradient is... Co-modulation and collaborative optimization with RL gradients achieve an adaptive balance between exploration efficiency and external knowledge transfer.

[0025] Understandably, by dynamically calculating the ratio of policy entropy change during the reinforcement learning phase to relative entropy change during the supervised fine-tuning phase, the contribution weights of the supervised fine-tuning gradient are adaptively adjusted, effectively balancing the model's exploration space and avoiding the shortcomings of insufficient exploration in pure RL or mechanical imitation in pure SFT. At the same time, a hybrid offline policy is constructed based on the inference accuracy of offline distilled samples to solve the problem of mismatch between offline data and the current policy distribution, preventing entropy collapse or performance collapse caused by fixed offline policies. Furthermore, training stability is ensured by pruning weights and dynamically adjusting the parameter update step size, ultimately efficiently improving the inference ability of small models, balancing training efficiency and performance reliability, and adapting to samples of different quality and training scenarios.

[0026] In some embodiments of this application, calculating the ratio of change between the policy entropy and the relative entropy in adjacent training steps includes: Divide the policy entropy of the next step in adjacent training steps by the policy entropy of the previous step to obtain the policy entropy ratio. The absolute value of the difference between 1 and the policy entropy ratio is the policy entropy change rate. Divide the relative entropy of the next step in adjacent training steps by the relative entropy of the previous step to obtain the relative entropy ratio. The absolute value of the difference between 1 and the relative entropy ratio is the relative entropy change rate.

[0027] Specifically, the process of obtaining the policy entropy of the previous step in adjacent training steps is as follows: First, determine the previous training step among two adjacent training steps (e.g., the t-1 step in the training iteration process, t≥2). During the reinforcement learning phase of this training step, input the current task sample to be processed (e.g., mathematical reasoning sample) into the language model, so that the model outputs the policy distribution for this sample (i.e., the probability distribution of the model's selection of different inference paths). Then, use the entropy calculation method to numerically calculate this policy distribution, and the final calculation result is the policy entropy of the previous step in adjacent training steps. The process of obtaining the policy entropy of the next step in adjacent training steps is as follows: First, determine the next training step among two adjacent training steps (e.g., the step t-1 in the training iteration process, t≥2). In the reinforcement learning phase of this training step, the task sample type and format input to the model are kept completely consistent with the previous step. The same task sample is input into the language model, and the model outputs the policy distribution for the sample in this training step (the probability distribution of the model's choice of different inference paths in step t). The same entropy calculation method (such as the same Shannon entropy formula) is used to calculate the policy distribution, and the result is the policy entropy of the next step in the adjacent training step. The operation of obtaining the relative entropy of the previous step in the adjacent training steps is as follows: Determine the previous training step (step t-1, t≥2) in the two adjacent training steps. In the supervised fine-tuning phase of this training step, the distillation policy distribution output by the large model for the current offline distillation sample (i.e., the sample corresponding to the high-quality inference trajectory generated by the large model) is first obtained (as the baseline policy distribution). Then, the same offline distillation sample is input into the language model, causing the model to output the policy distribution for that sample in this training step (the probability distribution of the inference path selection for that sample by the small model in step t-1). The relative entropy (KL divergence) calculation method is used to numerically calculate the two policy distributions, and the result is the relative entropy of the previous step in the adjacent training steps. The operation of obtaining the relative entropy of the next step in the adjacent training steps is as follows: determine the next training step in the two adjacent training steps (step t, t... ≥2), during the supervised fine-tuning phase of this training step, the offline distillation sample is kept consistent with the previous step (i.e., high-quality inference trajectory samples generated by the same large model), and the same large model distillation policy distribution as the previous step is obtained (ensuring that the baseline policy distribution remains unchanged). The offline distillation sample is then input into the language model, so that the model outputs the policy distribution for the sample in this training step (the probability distribution of the inference path selection of the small model for the sample in step t). The relative entropy (KL divergence) calculation method, which is exactly the same as the calculation method of the relative entropy in the previous step, is used to numerically calculate the policy distribution of the small model in this step and the distillation policy distribution of the large model. The result obtained is the relative entropy of the next step in the adjacent training step.

[0028] Understandably, by clearly defining the calculation path of "ratio of entropy values ​​between adjacent training steps → absolute difference → change rate," a precise quantitative basis is provided for the degree of change in policy entropy and relative entropy. This calculation method can objectively reflect the current exploration activity of the model (policy entropy change rate) and the alignment trend of the large model's policy (relative entropy change rate), avoiding the subjectivity of entropy change judgment. This provides reliable numerical support for subsequent "adaptive adjustment of supervised fine-tuning of gradient contribution weights," ensuring that the operation of dynamically balancing the exploration space is based on the real training state, rather than empirical judgment.

[0029] In some embodiments of this application, when evaluating the inference accuracy of offline distilled samples through a validator and mapping the inference accuracy to adaptive coefficients, the following steps are included: The reasoning results of the offline distillation samples are input into the verifier, which judges the correctness of the reasoning results and determines the reasoning accuracy based on the ratio of the number of correct results to the total number of samples. Establish the mapping relationship between the inference accuracy and the adaptive coefficient; When the inference accuracy is greater than or equal to the accuracy threshold, the adaptive coefficient is mapped to a value close to 1. When the inference accuracy is less than the accuracy threshold, the adaptive coefficient is mapped to a value close to 0.

[0030] Specifically, the accuracy threshold is a critical value used to distinguish between high and low inference accuracy of offline distillation samples, ranging from [0,1]. Its core function is to serve as a criterion for adaptive coefficient mapping. When the inference accuracy of offline distillation samples reaches or exceeds this threshold, it indicates that the language model has strong autonomous reasoning ability on the task corresponding to this type of sample and can rely more on its own strategy. When the inference accuracy is lower than this threshold, it indicates that the model's reasoning ability on this type of task is weak and it needs to rely more on the high-quality inference patterns of larger models. Its value directly determines the triggering condition for the adaptive coefficients to approach 1 or 0, and is a key parameter for achieving differentiated adaptation of sample quality. The specific operation of obtaining the accuracy threshold is as follows: first, collect all samples to be distilled. For training, offline distillation samples are used. The inference result of each sample is input into a dedicated validator, which judges the correctness of each sample's inference result and records it. The inference accuracy of all offline distillation samples is calculated (the ratio of the number of correct results for each sample to the total number of inference steps or subtasks contained in that sample). Samples with higher inference accuracy values ​​are selected from all samples (such as a range of "high accuracy samples" determined manually or through preliminary model testing, for example, samples with inference accuracy ranking in the top 40%-60%) to form a high accuracy sample group. The inference accuracy values ​​of all samples in this high accuracy sample group are calculated, and the lowest accuracy value is determined. This lowest accuracy value is set as the accuracy threshold.

[0031] Specifically, assuming the accuracy threshold is 0.8 and the adaptive coefficient ranges from [0,1], the offline distillation sample contains 10 inference sub-tasks. The operation process is as follows: input the results of the 10 inference sub-tasks of the offline distillation sample into the validator. The validator judges that 8 results are correct and 2 are incorrect. Based on the ratio of the number of correct results (8) to the total number of samples (10), the inference accuracy is calculated to be 0.8. After establishing the mapping relationship between the inference accuracy and the adaptive coefficient, since the inference accuracy (0.8) is equal to the accuracy threshold (0.8), the adaptive coefficient is mapped to a value close to 1 (such as 0.95). If only 5 results are correct in the 10 inference sub-tasks of another offline distillation sample, the inference accuracy is calculated to be 0.5. Since the accuracy (0.5) is less than the accuracy threshold (0.8), the adaptive coefficient is mapped to a value close to 0 (such as 0.05).

[0032] Understandably, the operation of "validator evaluation of accuracy → differential mapping of adaptive coefficients" establishes a correlation between sample quality and model strategy adaptation. When the sample inference accuracy is high, the coefficient approaches 1, and when it is low, it approaches 0. This differential mapping provides a "quality label" for the subsequent construction of hybrid offline strategies, avoiding the drawbacks of traditional methods that use the same integration strategy for high- and low-quality samples. This allows the model to selectively rely on its own strategy (high-quality samples) or the large model strategy (low-quality samples), improving its adaptability to samples of different quality.

[0033] In some embodiments of this application, when calculating the hybrid offline policy using the first weight and the current policy, the following steps are included: The difference between 1 and the first weight is used as the second weight; The product of the second weight and the current strategy, plus the adaptive coefficient, yields the hybrid offline strategy.

[0034] Specifically, assuming the first weight (i.e., the adaptive coefficient) is 0.95 (because the offline distillation sample inference accuracy ≥ the accuracy threshold, the adaptive coefficient approaches 1) and the quantized value of the current strategy is 0.6, or the first weight (i.e., the adaptive coefficient) is 0.05 (because the offline distillation sample inference accuracy < the accuracy threshold, the adaptive coefficient approaches 0) and the quantized value of the current strategy is 0.6, when the first weight is 0.95, first calculate the difference between 1 and the first weight (0.95) to obtain the second weight of 0.05, and then calculate the second weight (0.05). The product of the first weight and the current strategy (0.6) is 0.03. Finally, the adaptive coefficient (0.95) is added to the product (0.03) to obtain the value of the hybrid offline strategy, which is 0.98. When the first weight is 0.05, the difference between 1 and the first weight (0.05) is calculated to obtain the second weight, which is 0.95. Then, the product of the second weight (0.95) and the current strategy (0.6) is calculated to obtain the value of the hybrid offline strategy, which is 0.57. Finally, the adaptive coefficient (0.05) is added to the product (0.57) to obtain the value of the hybrid offline strategy, which is 0.62.

[0035] Understandably, the implementation of the hybrid offline strategy using the method of "first weight (adaptive coefficient) → second weight → product summation" directly echoes the innovation of "accuracy-aware offline strategy." This calculation method allows the composition of the hybrid strategy to dynamically change with sample quality—high-quality samples correspond to coefficients approaching 1, and the hybrid strategy is closer to the adaptive coefficients (implying high-accuracy sample characteristics); low-quality samples correspond to coefficients approaching 0, and the hybrid strategy is closer to the current strategy. This effectively solves the problem of mismatch between offline data and the current strategy distribution, fundamentally avoiding entropy collapse or performance collapse caused by a fixed offline strategy.

[0036] In some embodiments of this application, calculating the ratio weight of the current policy relative to the hybrid offline policy and imposing constraints on the ratio weight to obtain the constrained ratio weight includes: Divide the current strategy by the hybrid offline strategy to obtain the ratio weight; The first weight is used as the maximum constraint value of the ratio weight; When the ratio weight is greater than the maximum constraint value, then the constraint ratio weight is equal to the maximum constraint value; When the ratio weight is less than or equal to the maximum constraint value, the constraint ratio weight is equal to the ratio weight.

[0037] Specifically, assuming the current strategy's quantization value is 0.6, the hybrid offline strategy's value is 0.98 when the first weight (i.e., the adaptive coefficient, serving as the maximum constraint value) is 0.95, and its value is 0.62 when the first weight (maximum constraint value) is 0.05. When the maximum constraint value is 0.95, dividing the current strategy (0.6) by the hybrid offline strategy (0.98) yields a ratio weight of approximately 0.61. Since this ratio weight (0.61) is less than or equal to the maximum constraint value (0.95), the constraint ratio weight is equal to 0.61. When the maximum constraint value is 0.05, dividing the current strategy (0.6) by the hybrid offline strategy (0.62) yields a ratio weight of approximately 0.97. Since this ratio weight (0.97) is greater than the maximum constraint value (0.05), the constraint ratio weight is equal to 0.05.

[0038] Understandably, the operation of "calculating ratio weights → using the first weight as the maximum constraint value" sets an upper limit on the ratio weights. When the current policy deviates too much from the hybrid offline policy (the ratio weight is greater than the constraint value), it is forcibly adjusted to the constraint value, preventing gradient explosion or parameter oscillation caused by excessive policy deviation. When the deviation is small, the original ratio weights are retained to ensure that offline data can normally assist policy optimization. This step is a key implementation of the innovation of "ensuring training stability," providing a stable weight foundation for subsequent gradient modulation.

[0039] In some embodiments of this application, an initial contribution weight is obtained based on the relative entropy change ratio and the strategy entropy change ratio, and the initial contribution weight is pruned based on the pruning upper limit and the pruning lower limit to obtain the pruned contribution weight, including: The initial contribution weight is equal to the relative entropy change ratio divided by the strategy entropy change ratio; When the initial contribution weight is less than the clipping lower limit, the initial contribution weight is adjusted to the clipping lower limit to obtain the clipping contribution weight. When the initial contribution weight is greater than the clipping upper limit, the initial contribution weight is adjusted to the clipping upper limit to obtain the clipping contribution weight. When the initial contribution weight is greater than or equal to the lower limit of the clipping and the initial contribution weight is less than or equal to the upper limit of the clipping, then the clipping contribution weight is equal to the initial contribution weight.

[0040] Specifically, the initial contribution weight is calculated based on the ratio of relative entropy change during the supervised fine-tuning phase to the ratio of policy entropy change during the reinforcement learning phase. It is used to initially determine the contribution of the supervised fine-tuning gradient during training. Its value directly reflects the proportional relationship between relative entropy change and policy entropy change—the larger the relative entropy change ratio or the smaller the policy entropy change ratio, the larger the initial contribution weight, and the higher the initial contribution of the supervised fine-tuning gradient. Its core function is to provide a basis for subsequent pruning operations. The pruning lower limit is a fixed critical value used to limit the initial contribution weight from being too small. Its value is usually the minimum value (e.g., 1) that ensures the supervised fine-tuning gradient has a basic guiding role. Its core function is to avoid… To prevent insufficient guidance from excessively small initial contribution weights in supervised fine-tuning, thus hindering the effective expansion of the model's exploration space; the pruning upper limit is a dynamic critical value used to limit excessively large initial contribution weights. Its value is related to the number of samples during training (the larger the sample size, the higher the upper limit). Its core function is to prevent excessive interference from supervised fine-tuning in the reinforcement learning phase due to excessively large initial contribution weights, thereby preventing an overabundance of model exploration space; the pruning contribution weight is the final weight obtained by constraining the initial contribution weights within a range of the pruning lower and upper limits. Its value always lies between the pruning lower and upper limits. Its core function is to ensure that the contribution of the supervised fine-tuning gradient is neither excessive nor insufficient, thus providing a foundation for subsequent... The continued modulation of supervised fine-tuning gradients provides stable weight parameters. The initial contribution weights are obtained by first acquiring the ratio of relative entropy change to policy entropy change in adjacent training steps, then dividing the ratio of relative entropy change to policy entropy change by the quotient, which is the initial contribution weight. The lower limit of pruning is obtained by testing the stability of supervised fine-tuning gradients under different training scenarios (e.g., different offline distillation sample sizes, different inference task types), selecting the minimum fixed value that avoids insufficient supervised fine-tuning guidance (i.e., ensures the basic expansion needs of the model's exploration space), and determining this value as the lower limit of pruning. The upper limit of pruning is obtained by first statistically analyzing the sample size in the current training process... The initial quantity (i.e., the batch processing scale of offline distillation samples) is used to establish a correlation between the sample quantity and the upper limit value based on a preset ratio (e.g., the upper limit value increases by 5% for every 10% increase in the sample quantity). The upper limit value for pruning corresponding to the current sample quantity is calculated and determined according to this correlation. The specific operation for obtaining the pruning contribution weight is as follows: the initial contribution weight is compared with the lower limit value and the upper limit value for pruning. If the initial contribution weight is less than the lower limit value for pruning, the lower limit value for pruning is used as the pruning contribution weight. If the initial contribution weight is greater than the upper limit value for pruning, the upper limit value for pruning is used as the pruning contribution weight. If the initial contribution weight is between the lower limit value and the upper limit value for pruning, the initial contribution weight is directly used as the pruning contribution weight.

[0041] Specifically, assuming the relative entropy change rates are 0.2, 0.6, and 1.0, and the strategy entropy change rates are 0.4, 0.3, and 0.1, with the lower limit of pruning set to 1 and the upper limit to 8, when the relative entropy change rate is 0.2 and the strategy entropy change rate is 0.4, the initial contribution weight is equal to 0.2 divided by 0.4, resulting in 0.5. Since 0.5 is less than the lower limit of pruning (1), the initial contribution weight is adjusted to 1, resulting in a pruning contribution weight of 1. When the rate is 0.6 and the strategy entropy change rate is 0.3, the initial contribution weight is equal to 0.6 divided by 0.3, which is 2. Since 2 is greater than or equal to the lower limit of pruning (1) and less than or equal to the upper limit of pruning (8), the pruning contribution weight is equal to the initial contribution weight of 2. When the relative entropy change rate is 1.0 and the strategy entropy change rate is 0.1, the initial contribution weight is equal to 1.0 divided by 0.1, which is 10. Since 10 is greater than the upper limit of pruning (8), the initial contribution weight is adjusted to 8, resulting in a pruning contribution weight of 8.

[0042] Understandably, clearly defining the process of "relative entropy change ratio / policy entropy change ratio → initial contribution weight → pruning" precisely implements the innovation of "dynamically adjusting the contribution weight of supervised fine-tuning gradients." By pruning the lower limit, we avoid excessively small contribution weights (insufficient guidance from supervised fine-tuning, preventing expansion of the exploration space), and by pruning the upper limit, we avoid excessively large contribution weights (over-supervised fine-tuning, interfering with RL policy refinement). This ensures that the contribution level of supervised fine-tuning gradients is always within a reasonable range of "balancing the exploration space," thus leveraging the expansion capabilities of SFT without weakening the effect of RL in refining existing policies.

[0043] In some embodiments of this application, the lower cut limit is kept constant, and the upper cut limit is adjusted based on the number of samples. When the number of samples increases, the upper limit of the cropping value is increased by a preset ratio; When the number of samples decreases, the upper limit of the cropping value is reduced by a preset ratio.

[0044] Specifically, the sample size refers to the actual number of offline distilled samples processed in a single batch during language model training. Its value is directly related to the scale of data processing during training. Its core function is to serve as the basis for adjusting the pruning cap – changes in the sample size lead to changes in the model's need for supervised fine-tuning gradients. Therefore, the pruning cap needs to be dynamically adapted to the sample size to ensure that the contribution weight of supervised fine-tuning gradients is within a reasonable range under different data processing scales, avoiding an overabundance or underabundance of exploration space due to differences in sample size. The preset ratio is a fixed proportional coefficient used to correlate the relationship between the sample size and changes in the pruning cap. Its value is determined based on the model training stability test results under different sample sizes. Its core function is to clarify the corresponding adjustment range of the pruning cap for each certain change in the sample size, making the adjustment of the pruning cap operable and... Consistency is ensured to guarantee the matching degree between the pruning upper limit and the model training requirements under different sample number scenarios. The sample number acquisition operation is as follows: before each offline distillation sample batch training is started, the total number of offline distillation samples to be input into the model in the current training batch is counted. This number is the sample number for this training. If the batch processing scale is adjusted during training, the total number of offline distillation samples in the adjusted batch is recounted and the sample number is updated. The preset ratio acquisition operation is as follows: first, select multiple groups of different sample numbers (such as 50, 100, 150, 200), test and determine the optimal pruning upper limit that can ensure the stability of model training for each group of sample numbers, calculate the ratio of the change range of each group of sample numbers to the corresponding optimal pruning upper limit, and screen out the fixed ratio that can make the training effect meet the standard under different sample numbers. This ratio is determined as the preset ratio.

[0045] Specifically, assuming the lower limit of the clipping is fixed at 1, the upper limit of the clipping is 8 when the initial sample size is 100. The preset ratio is "for every 20% increase in the sample size, the upper limit of the clipping increases by 10%; for every 20% decrease in the sample size, the upper limit of the clipping decreases by 10%". When the sample size increases from 100 to 120 (an increase of 20%), the upper limit of the clipping is adjusted from 8 to 8×(1+10%)=8.8 according to the preset ratio. When the sample size decreases from 100 to 80 (a decrease of 20%), the upper limit of the clipping is adjusted from 8 to 8×(1-10%)=7.2 according to the preset ratio. In both cases, the lower limit of the clipping remains unchanged at 1.

[0046] Understandably, by operating on "sample size → preset ratio → pruning upper limit," the pruning weights can be adapted to training scenarios with different data scales. When the sample size increases, the model's need for supervised gradient fine-tuning is higher (requiring the transfer of more large model strategies), so the upper limit is increased proportionally. When the sample size decreases, the upper limit is lowered to avoid over-fine-tuning. This solves the problem of "poor adaptability" of traditional fixed pruning ranges under different sample sizes, ensuring that the method can function stably in both small-batch and large-batch offline data training, thus improving scenario adaptability.

[0047] In some embodiments of this application, when modulating the supervised fine-tuning gradient based on the pruning contribution weight and the constraint ratio weight to obtain the modulated supervised fine-tuning gradient, the following steps are included: Multiply the clipping contribution weight by the constraint ratio weight to obtain the modulation coefficient; The modulation supervision fine-tuning gradient is obtained by multiplying the modulation coefficient by the supervision fine-tuning gradient.

[0048] Specifically, assuming the clipping contribution weights are 1, 2, and 8, the constraint ratio weights are 0.61 and 0.05, and the quantized value of the supervised fine-tuning gradient is 0.5, when the clipping contribution weight is 1 and the constraint ratio weight is 0.61, multiplying 1 by 0.61 yields a modulation coefficient of 0.61, and then multiplying 0.61 by the supervised fine-tuning gradient of 0.5 yields a modulation supervised fine-tuning gradient of 0.305; when the clipping contribution weight is 2 and the constraint ratio weight is 0.05, multiplying 2 by 0.05 yields a modulation coefficient of 0.1, and then multiplying 0.1 by the supervised fine-tuning gradient of 0.5 yields a modulation supervised fine-tuning gradient of 0.05; when the clipping contribution weight is 8 and the constraint ratio weight is 0.61, multiplying 8 by 0.61 yields a modulation coefficient of 4.88, and then multiplying 4.88 by the supervised fine-tuning gradient of 0.5 yields a modulation supervised fine-tuning gradient of 2.44.

[0049] Understandably, combining the "pruning contribution weights (dynamic adjustment result)" with the "constraint ratio weights (stabilization result)" to calculate the modulation coefficients, and then using them to modulate the supervised fine-tuning gradient, achieves a dual guarantee of "dynamic adjustment" and "training stability." The modulated gradient can carry the high-quality inference pattern of the large model (realizing the "expansion" function of SFT), and because the weights have been pruned and constrained, they will not interfere with the policy optimization in the RL stage (preserving the "refinement" function of RL), further balancing the model's exploration space and policy stability, and ensuring the high efficiency of inference capability improvement.

[0050] In some embodiments of this application, when co-optimizing the modulation-supervised fine-tuning gradient and the reinforcement learning gradient, the following methods are included: The modulation-supervised fine-tuning gradient is added to the reinforcement learning gradient to obtain the total gradient, and the training parameters are updated using the total gradient according to the iteration frequency. When the total gradient is greater than the gradient threshold, the step size of the training parameter update is reduced. When the total gradient is less than the gradient threshold, the step size of the training parameter update is increased. When the total gradient equals the gradient threshold, the step size for updating the training parameters remains unchanged.

[0051] Specifically, the gradient threshold is a critical value used to determine the magnitude of the total gradient and thus adjust the step size for training parameter updates. Its value is related to the stability requirements of parameter updates during model training. Its core function is to serve as a criterion for step size adjustment—when the total gradient exceeds the threshold, it indicates that the current gradient fluctuation is large, and the step size needs to be reduced to avoid parameter update oscillations; when the total gradient is below the threshold, it indicates that the current gradient change is gradual, and the step size can be increased to accelerate convergence; when the total gradient equals the threshold, it indicates that the current gradient state is suitable for the existing step size and no adjustment is needed. It is a key reference quantity for balancing parameter update stability and convergence efficiency. The step size for training parameter updates is the magnitude of each parameter update using the total gradient. Its magnitude directly affects the speed and stability of parameter updates—an excessively large step size can easily cause parameter updates to exceed the optimal range, resulting in training oscillations; an excessively small step size will slow down the parameter update speed and prolong the training cycle. Its core function is to control the magnitude of each parameter update, ensuring that the model is efficient under stable training conditions. Convergence; The specific steps for obtaining the gradient threshold are as follows: First, select multiple different candidate thresholds (e.g., 0.3, 0.5, 0.7). During the model pre-training phase, use each candidate threshold to test parameter updates, record the number of training oscillations and convergence speed in each test, and select the candidate threshold that minimizes the number of training oscillations and optimizes the convergence speed. This candidate threshold is then determined as the final gradient threshold. The specific steps for obtaining the step size for training parameter updates are as follows: First, set an initial step size (e.g., 0.01). During the model pre-training phase, use this initial step size to update parameters and observe the convergence trend of the parameters during training. If the convergence speed is too slow, increase the step size appropriately (e.g., adjust to 0.02) and retest. If significant training oscillations occur, decrease the step size appropriately (e.g., adjust to 0.005) and retest until a step size that balances convergence speed and training stability is found. This step size is then determined as the initial step size for training parameter updates (which can be dynamically adjusted later based on the relationship between the total gradient and the gradient threshold).

[0052] Specifically, assuming a gradient threshold of 0.5 and an initial training parameter update step size of 0.01, the total gradient is 0.7 when the modulation-supervised fine-tuning gradient is 0.3 and the reinforcement learning gradient is 0.4; the total gradient is 0.4 when the modulation-supervised fine-tuning gradient is 0.2 and the reinforcement learning gradient is 0.2; and the total gradient is 0.5 when the modulation-supervised fine-tuning gradient is 0.25 and the reinforcement learning gradient is 0.25. When the total gradient is 0.7 (greater than the gradient threshold of 0.5), the training parameter update step size is reduced from 0.01 to 0.005; when the total gradient is 0.4 (less than the gradient threshold of 0.5), the training parameter update step size is increased from 0.01 to 0.015; and when the total gradient is 0.5 (equal to the gradient threshold of 0.5), the training parameter update step size remains unchanged at 0.01. In all three cases, the training parameters are updated using the total gradient at a preset iteration frequency.

[0053] Understandably, the innovative approach of "ensuring training stability" is achieved through the operation of "calculating the total gradient → dynamically adjusting the step size according to the gradient threshold." When the total gradient is too large, the step size is reduced to prevent parameter updates from exceeding the optimal range and causing oscillations; when the total gradient is too small, the step size is increased to prevent slow convergence and prolonged training cycles; when the total gradient equals the threshold, the step size is maintained to ensure stable convergence. This step ensures that parameter updates always adapt to the current gradient state, guaranteeing both the stability of the training process and improving model convergence efficiency, helping small models reach optimal inference performance faster.

[0054] Reference Figure 2 As shown in some embodiments of this application, a language model adaptive policy fusion training system includes: The entropy change calculation module is used to track the policy entropy in the reinforcement learning phase and the relative entropy in the supervised fine-tuning phase, and to calculate the change ratio of the policy entropy and the relative entropy in adjacent training steps. An adaptive coefficient mapping module is used to evaluate the inference accuracy of offline distillation samples through a validator and map the inference accuracy to adaptive coefficients; The weight calculation module is used to construct a hybrid offline strategy based on the adaptive coefficients, calculate the ratio weight of the current strategy relative to the hybrid offline strategy, and apply constraints to the ratio weight to obtain the constrained ratio weight. The gradient optimization module is used to calculate the contribution weight of the supervised fine-tuning gradient based on the change ratio and to prune the contribution weight to obtain the pruned contribution weight. Based on the pruned contribution weight and the constraint ratio weight, the supervised fine-tuning gradient is modulated to obtain the modulated supervised fine-tuning gradient. The modulated supervised fine-tuning gradient is then co-optimized with the reinforcement learning gradient.

[0055] It is understandable that the above-mentioned language model adaptive strategy fusion training method and system have the same beneficial effects, which will not be elaborated here.

[0056] It should be noted that: Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0057] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of this application and form different embodiments.

[0058] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A language model adaptive strategy fusion training method, characterized in that, include: The policy entropy during the tracking reinforcement learning phase and the relative entropy during the supervised fine-tuning phase are compared, and the change ratio of the policy entropy and the relative entropy between adjacent training steps is calculated. The inference accuracy of offline distillation samples is evaluated using a validator, and the inference accuracy is mapped to adaptive coefficients. Based on the adaptive coefficients, a hybrid offline strategy is constructed, the ratio weight of the current strategy relative to the hybrid offline strategy is calculated, and constraints are applied to the ratio weight to obtain the constrained ratio weight. The contribution weights of the supervised fine-tuning gradient are calculated based on the change ratio and the contribution weights are clipped to obtain clipped contribution weights. The supervised fine-tuning gradient is modulated based on the clipped contribution weights and the constraint ratio weights to obtain the modulated supervised fine-tuning gradient. The modulated supervised fine-tuning gradient is then co-optimized with the reinforcement learning gradient. When constructing a hybrid offline strategy, the adaptive coefficient is used as the first weight, and the hybrid offline strategy is calculated using the first weight and the current strategy. When obtaining the pruning contribution weight, the initial contribution weight is obtained based on the relative entropy change ratio and the strategy entropy change ratio, and the initial contribution weight is pruned based on the pruning upper limit and the pruning lower limit to obtain the pruning contribution weight.

2. The language model adaptive strategy fusion training method according to claim 1, characterized in that, Calculating the ratio of change between adjacent training steps of the policy entropy and the relative entropy includes: Divide the policy entropy of the next step in adjacent training steps by the policy entropy of the previous step to obtain the policy entropy ratio. The absolute value of the difference between 1 and the policy entropy ratio is the policy entropy change rate. Divide the relative entropy of the next step in adjacent training steps by the relative entropy of the previous step to obtain the relative entropy ratio. The absolute value of the difference between 1 and the relative entropy ratio is the relative entropy change rate.

3. The language model adaptive strategy fusion training method according to claim 2, characterized in that, When evaluating the inference accuracy of offline distilled samples using a validator and mapping the inference accuracy to adaptive coefficients, the following steps are included: The reasoning results of the offline distillation samples are input into the verifier, which judges the correctness of the reasoning results and determines the reasoning accuracy based on the ratio of the number of correct results to the total number of samples. Establish the mapping relationship between the inference accuracy and the adaptive coefficient; When the inference accuracy is greater than or equal to the accuracy threshold, the adaptive coefficient is mapped to a value close to 1. When the inference accuracy is less than the accuracy threshold, the adaptive coefficient is mapped to a value close to 0.

4. The language model adaptive strategy fusion training method according to claim 3, characterized in that, When calculating the hybrid offline policy using the first weight and the current policy, the following steps are included: The difference between 1 and the first weight is used as the second weight; The product of the second weight and the current strategy, plus the adaptive coefficient, yields the hybrid offline strategy.

5. The language model adaptive strategy fusion training method according to claim 4, characterized in that, Calculating the ratio weight of the current policy relative to the hybrid offline policy, and imposing constraints on the ratio weight to obtain the constrained ratio weight includes: Divide the current strategy by the hybrid offline strategy to obtain the ratio weight; The first weight is used as the maximum constraint value of the ratio weight; When the ratio weight is greater than the maximum constraint value, then the constraint ratio weight is equal to the maximum constraint value; When the ratio weight is less than or equal to the maximum constraint value, the constraint ratio weight is equal to the ratio weight.

6. The language model adaptive strategy fusion training method according to claim 5, characterized in that, The initial contribution weight is obtained based on the relative entropy change ratio and the strategy entropy change ratio. The initial contribution weight is then pruned based on the pruning upper limit and pruning lower limit to obtain the pruned contribution weight, including: The initial contribution weight is equal to the relative entropy change ratio divided by the strategy entropy change ratio; When the initial contribution weight is less than the clipping lower limit, the initial contribution weight is adjusted to the clipping lower limit to obtain the clipping contribution weight. When the initial contribution weight is greater than the clipping upper limit, the initial contribution weight is adjusted to the clipping upper limit to obtain the clipping contribution weight. When the initial contribution weight is greater than or equal to the lower limit of the clipping and the initial contribution weight is less than or equal to the upper limit of the clipping, then the clipping contribution weight is equal to the initial contribution weight.

7. The language model adaptive strategy fusion training method according to claim 6, characterized in that, Keep the lower limit of the clipping value constant, and adjust the upper limit of the clipping value based on the number of samples; When the number of samples increases, the upper limit of the cropping value is increased by a preset ratio; When the number of samples decreases, the upper limit of the cropping value is reduced by a preset ratio.

8. The language model adaptive strategy fusion training method according to claim 7, characterized in that, Modulating the supervised fine-tuning gradient based on the pruning contribution weight and the constraint ratio weight, to obtain the modulated supervised fine-tuning gradient, includes: Multiply the clipping contribution weight by the constraint ratio weight to obtain the modulation coefficient; The modulation supervision fine-tuning gradient is obtained by multiplying the modulation coefficient by the supervision fine-tuning gradient.

9. The language model adaptive strategy fusion training method according to claim 8, characterized in that, When co-optimizing the modulation-supervised fine-tuning gradient and the reinforcement learning gradient, the following steps are included: The modulation-supervised fine-tuning gradient is added to the reinforcement learning gradient to obtain the total gradient, and the training parameters are updated using the total gradient according to the iteration frequency. When the total gradient is greater than the gradient threshold, the step size of the training parameter update is reduced. When the total gradient is less than the gradient threshold, the step size of the training parameter update is increased. When the total gradient equals the gradient threshold, the step size for updating the training parameters remains unchanged.

10. A language model adaptive strategy fusion training system, characterized in that, The language model adaptive strategy fusion training method according to any one of claims 1-9 includes: The entropy change calculation module is used to track the policy entropy in the reinforcement learning phase and the relative entropy in the supervised fine-tuning phase, and to calculate the change ratio of the policy entropy and the relative entropy in adjacent training steps. An adaptive coefficient mapping module is used to evaluate the inference accuracy of offline distillation samples through a validator and map the inference accuracy to adaptive coefficients; The weight calculation module is used to construct a hybrid offline strategy based on the adaptive coefficients, calculate the ratio weight of the current strategy relative to the hybrid offline strategy, and apply constraints to the ratio weight to obtain the constrained ratio weight. The gradient optimization module is used to calculate the contribution weight of the supervised fine-tuning gradient based on the change ratio and to prune the contribution weight to obtain the pruned contribution weight. Based on the pruned contribution weight and the constraint ratio weight, the supervised fine-tuning gradient is modulated to obtain the modulated supervised fine-tuning gradient. The modulated supervised fine-tuning gradient is then co-optimized with the reinforcement learning gradient.

Images