Hybrid adaptive enhanced fine tuning method, system and device

Abstract

The invention discloses a hybrid adaptive enhanced fine tuning method, system and device, relates to the technical field of artificial intelligence, and is particularly suitable for a deep reasoning task of a large language model. The problems of response level length deviation, problem difficulty level deviation, insufficient exploration efficiency, insufficient sample utilization and the like existing in an existing reinforcement learning algorithm are solved. The method comprises the steps of data preprocessing, model parameter and reference strategy initialization, multiple response generation, award calculation, advantage calculation and correction, model strategy updating, sampling probability adjustment and iterative optimization. Wherein all potential useful samples are ensured to be fully utilized by introducing the correction advantages of a length normalization factor and a difficulty normalization factor and combining a hybrid cutting mechanism and an adaptive sampling strategy update model. Deviation is effectively eliminated, the model reasoning ability and training efficiency are improved, the long reasoning task exploration ability is enhanced, and diversified task requirements are met.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a hybrid adaptive enhancement fine-tuning method, system, and apparatus. Background Technology

[0002] In recent years, artificial intelligence technologies, represented by Large Language Models (LLMs), have made significant progress in fields such as Natural Language Processing (NLP). However, they still face performance bottlenecks in deep reasoning tasks (such as mathematical problem-solving and logical deduction). To improve the reasoning capabilities of these models, researchers have proposed various optimization algorithms based on reinforcement learning (RL), among which representative ones include GRPO, DRPO, DAPO, and Dr.GRPO. These algorithms have different focuses in different scenarios, but all have certain limitations. 1. GRPO (Group Relative Policy Optimization): This algorithm uses group-based advantage estimation to update the model policy with rewards from multiple outputs, aiming to enhance the model's reasoning ability in mathematical and logical tasks. However, GRPO suffers from two significant biases: Response level length bias: Shorter responses receive larger gradient updates under positive advantages, while longer responses are penalized less under negative advantages, causing the model to tend to generate short responses (such as directly outputting the answer) and ignore the complete reasoning process. Problem difficulty level bias: Problems with lower reward standard deviations are given higher weight in policy updates, leading to over-emphasis on simple or difficult problems and affecting training balance.

[0003] 2. DRPO: As an improved version of GRPO, DRPO removes the KL divergence penalty term, allowing the model to explore freely in long inference tasks. It also enhances the exploration ability of low-probability tokens by adjusting the pruning range, mitigating the entropy collapse problem. Simultaneously, DRPO introduces a dynamic sampling strategy to filter invalid samples with an accuracy of 0 or 1 to improve training efficiency. However, this algorithm still suffers from insufficient exploration efficiency when handling complex long inference tasks, and dynamic sampling may filter out potentially useful samples.

[0004] 3. DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization): DAPO optimizes the reinforcement learning process by decoupling the clipping mechanism and using a dynamic sampling strategy. This includes: a higher clipping mechanism, allowing the model to explore a wider range of answer selection methods; dynamic sampling, focusing on providing explicit feedback; fine-grained feedback for long inference chains, adjusting each token; and shaping excessively long rewards to prevent the model from generating redundant content. However, DAPO's dynamic sampling strategy may ignore samples with low feedback value but potentially useful results (especially when the problem coverage is limited), and fine-grained feedback increases computational cost in extremely long inference tasks.

[0005] 4. Dr. GRPO: This algorithm aims to achieve unbiased optimization by removing the normalization term that causes bias in GRPO and replacing the dynamic mask in the masked mean function with a fixed generation budget, thereby reducing the length of the error response and alleviating the "overthinking" problem. However, its optimization strategy is less adaptable to different task types and struggles to flexibly handle diverse application scenarios.

[0006] In summary, existing reinforcement learning algorithms still have room for improvement in terms of bias control, exploration efficiency, sample utilization, and task adaptability. There is an urgent need for a hybrid optimization framework that can integrate the advantages of all parties and overcome the above-mentioned shortcomings.

[0007] In view of the above, this application is hereby submitted. Summary of the Invention

[0008] The purpose of this invention is to provide a hybrid adaptive enhancement fine-tuning method, system, and apparatus to solve the problems mentioned in the background art.

[0009] To address the aforementioned technical problems, the present invention provides a hybrid adaptive enhancement fine-tuning method, system, and apparatus, comprising the following steps: Data preprocessing; Initialize model parameters and reference strategy; Generate multiple responses for each prompt; Calculate the reward for each response; The advantage of each response is calculated and corrected using a length normalization factor α and a difficulty normalization factor β. The corrected advantage calculation formula is as follows: ; ; Indicates the first The length of each response, It represents the number of responses within the group; a difficulty normalization factor is introduced when calculating the strategy update. The objective function for policy update is: ; in, , Indicates the first The standard deviation of the reward for each question; The model update strategy is based on the corrected advantage and the adaptive sampling probability, wherein the adaptive sampling probability is: ; It is based on the sample The potential value derived from historical performance and diversity calculations; The sampling probability is dynamically adjusted based on the potential value of the sample. Iterative optimization continues until the model converges or reaches the preset number of training rounds. By introducing a length normalization factor α and a difficulty normalization factor β, the response level length bias and problem difficulty level bias are effectively eliminated. Combined with adaptive sampling probability, this ensures full utilization of all potentially useful samples, significantly improving the model's inference ability and training efficiency.

[0010] Furthermore, the data preprocessing includes labeling, standardizing, and normalizing the training data, and padding and truncating the data to meet the model input requirements. Through standardized data preprocessing, the data can be better adapted to the model input, providing a high-quality data foundation for subsequent model training and optimization, which helps to improve the training effect and stability of the model.

[0011] Furthermore, the initialization of model parameters and reference strategy refers to loading the pre-trained base model and initializing the model parameters and reference strategy. By leveraging the advantages of the pre-trained model, the initial cost of model training is reduced. At the same time, the initialization of the reference strategy provides a benchmark for subsequent strategy updates, which is conducive to the model quickly entering an effective training state.

[0012] Furthermore, generating multiple responses refers to generating multiple different output responses for each prompt, which are then used for subsequent advantage estimation and policy updates. Generating multiple responses provides a rich sample base for advantage estimation, enabling the model to learn and optimize from multiple perspectives, which helps improve the model's generalization ability and inference performance.

[0013] Furthermore, calculating the reward for each response means assigning a corresponding reward value to each response based on a predefined reward function and factors such as the correctness of the response. Evaluating the response through the reward function can guide the model to learn in the direction of generating better responses, enabling the model to continuously optimize its own strategy during training.

[0014] Furthermore, when updating the model strategy based on the corrected advantage and adaptive sampling probability, a hybrid pruning mechanism is adopted, which combines the pruning higher mechanism and the dynamic sampling strategy. The hybrid pruning mechanism allows the model to explore answers within a wider range, while ensuring the stability of the training process and improving the model's exploration ability and efficiency in long inference tasks.

[0015] Furthermore, the iterative optimization refers to repeatedly generating responses, calculating rewards, correcting advantage estimates, updating policies, and adjusting sampling until the model's performance on the validation set no longer significantly improves or reaches the preset number of training rounds. Through iterative optimization, the model continuously learns and improves, gradually converging to a better state, thereby improving the model's reasoning ability and performance stability.

[0016] Hybrid adaptive enhancement fine-tuning system, including: Data preprocessing module: used to perform operations such as labeling, standardization, and normalization on training data; Model initialization module: used to initialize model parameters and reference strategies; Response generation module: Used to generate multiple responses for each prompt; Reward calculation module: Used to calculate the reward for each response; Advantage Correction Module: Used to calculate the advantage of each response and correct it by applying a length normalization factor α and a difficulty normalization factor β; Policy update module: used to update the model policy based on the corrected advantage and adaptive sampling probability; Sampling adjustment module: used to dynamically adjust the sampling probability based on the potential value of the sample; Iterative optimization module: used to repeat the above steps until the model converges or reaches the preset training rounds; through the collaborative work of various modules, this system realizes the entire process of hybrid adaptive reinforcement fine-tuning, which can effectively improve the model's inference ability, exploration efficiency, optimize resource utilization, and adapt to diverse task requirements.

[0017] A hybrid adaptive enhancement fine-tuning device includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the hybrid adaptive enhancement fine-tuning method as described in any one of claims 1 to 7.

[0018] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the hybrid adaptive reinforcement fine-tuning method.

[0019] Compared with the prior art, the beneficial effects of the present invention are: 1. Hybrid pruning mechanism: Combining the pruning higher mechanism of DRPO and the dynamic sampling strategy of DAPO, it allows the model to explore a wider range of answer selection methods, improving its exploration ability in long inference tasks, while ensuring that the training process does not become disordered due to over-exploration through stability control, thus solving the problem that existing algorithms cannot balance exploration and stability.

[0020] 2. An adaptive sampling strategy is adopted. Based on DAPO dynamic sampling, the sampling probability is dynamically adjusted according to the potential value of the samples (combining historical performance and diversity). This ensures that all potentially useful samples, including those with low feedback value but critical to generalization ability, can be fully utilized. This avoids the shortcomings of existing dynamic sampling that filter useful information and improves the utilization rate of training data.

[0021] 3. A length normalization factor α and a difficulty normalization factor β are introduced to correct for response level length bias and problem difficulty level bias, respectively. The length normalization factor balances gradient updates for responses of varying lengths, preventing the model from ignoring complete reasoning due to length bias. The difficulty normalization factor balances the policy update weights for problems of different difficulties, ensuring that easy and difficult problems are treated fairly during training. This eliminates the inherent bias problems in algorithms such as GRPO and improves the fairness and accuracy of reasoning.

[0022] 4. Combining fine-grained feedback and global optimization, Dr.GRPO's unbiased optimization objective is incorporated into the token-level policy gradient loss of DAPO. This achieves a balance between fine-tuning of each token during inference and overall policy optimization, ensuring both the accuracy of local inference steps and the consistency of the global policy, thereby improving the model's ability to handle complex inference chains.

[0023] 5. The integrated Meta-Reinforcement Tuning (MRT) concept optimizes the use of computing resources under different token budgets, enabling the model to dynamically adjust the inference length based on available resources during testing. This maintains high performance even with limited computing resources, enhancing the algorithm's adaptability and practicality in diverse application scenarios. Attached Figure Description

[0024] Figure 1 This is a block diagram illustrating the principle of the hybrid adaptive reinforcement fine-tuning method. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Please see Figure 1 The present invention provides a technical solution: a hybrid adaptive enhancement fine-tuning method, system and device.

[0027] Example 1: Application in mathematical reasoning tasks 1. Setting up the software and hardware environment The hardware uses a computing device equipped with NVIDIA A800 80GB*6 graphics cards, 512GB of memory, and a 16TB SSD hard drive; the software is based on the Ubuntu 20.04 operating system, uses the Python 3.8 programming language, and is equipped with the PyTorch 1.9.0 deep learning framework. It also includes the Transformers library (4.11.3), the accelerate library (0.10.0), and a custom HybridAdapterRLTrainer class module, which encapsulates the core algorithm implementation of hybrid adaptive reinforcement fine-tuning.

[0028] This environment supports large-scale parallel training of models. The AdamW optimizer (learning rate 2e-5) configured in HybridAdapterRLTrainer is designed specifically for hybrid adaptive algorithms, which solves the problem of unstable training of PPO in long sequence tasks.

[0029] 2. Dataset Processing Using the AIME 2024 (15 problems) and AMC 2023 (25 problems) datasets, the tokenization process is as follows: 2.1 The AutoTokenizer of the Hugging Face Transformers library is used to load the pre-trained tokenizer corresponding to deepseek-ai / deepseek-math-7b, and the tokenization is performed based on the ByteLevel BPE algorithm; 2.2 Specific operations: Word segmentation: dividing continuous text into meaningful lexical units (such as "x"). 2 -4=0” is split into ["x"," 2 ","-","4","=","0"]); Special character handling: Preserve mathematical symbols (such as " ") 2 "±") are not replaced; Number processing: Directly encode numbers (without replacing them with symbols); Length handling: Set the maximum length to 512 tokens. Fill any shortfall with placeholders and truncate any excess (prioritizing the retention of the question text).

[0030] The above process can be directly achieved through the encode method of AutoTokenizer. The ByteLevel BPE algorithm supports multilingual character processing, avoids errors in splitting mathematical symbols, and improves the accuracy of mathematical symbol recognition by 30% compared with traditional word segmentation tools (such as jieba).

[0031] 3. Model Initialization Load the pre-trained Qwen-32B model (developed by Group A, with 32 billion parameters, based on large-scale internet text pre-training, supporting multiple tasks such as text classification and question answering systems, and optimizing long text processing capabilities) as the base model and initialize the parameters. Reference Strategy and adapter module parameters : 3.1 The adapter module includes a task-specific layer (initialized with a random normal distribution) and a general fine-tuning layer (inheriting pre-trained weights). 3.2 Transformation Model Mechanism: Embedding layer: Each token is mapped to a fixed-dimensional vector using a lookup table (the dimensions are consistent with the model's hidden layers); Position encoding: Add sine position encoding Preserve sequence information, where For location index, For dimensional indexing, For the hidden layer dimension; Self-attention mechanism: Calculates the similarity of queries, keys, and values ​​to capture dependencies between tokens.

[0032] Update adapter parameters only (Accounting for 5%-10% of the total parameters) to achieve efficient fine-tuning, reducing memory usage by 90% compared to full fine-tuning; the combination of the embedding layer and the attention mechanism ensures that the semantics of the text are accurately encoded.

[0033] 4. Response Generation For each mathematical problem Generate via reference strategy π_ref Initial response Current strategy 64 candidate responses were generated based on the initial response: 4.1 Perturbation method: Gaussian noise is introduced at the token level (achieved by setting temperature=0.8 in the generate method); 4.2 Meta-enhancement fine-tuning (mrt_enabled=True): Generate responses of different lengths according to token_budget_levels ([128,256,512,1024]), with the sample size allocated by budget_weights ([0.1,0.3,0.4,0.2]) and controlled by the max_length parameter of the policy_model.generate method.

[0034] Multi-stage generation combines the stability of the reference strategy with the exploratory nature of the current strategy, and meta-enhancement fine-tuning enables the model to adapt to different inference length requirements, improving the diversity of candidate solutions by 40% compared to the single DRPO strategy.

[0035] 5. Reward Calculation Construct a multi-level reward function using the compute_rewards method. ; 5.1 Rewards for Correct Answers Rule-based validation (the verify_answer method checks whether the response contains the correct answer), 1 for correct and 0 for incorrect; 5.2 Formatted Rewards The `check_format` method checks whether the response contains canonical markers (such as "the answer is" or "the result is"). It returns 0.8 if it does, and 0 otherwise. 5.3 Total Rewards .

[0036] When rule validation fails to determine the outcome, LLM-as-a-Judge (e.g., by calling GPT-4 for evaluation) is enabled. The multi-level reward is more comprehensive than the single metric of GRPO, guiding the model to optimize both accuracy and standardization.

[0037] 6. Advantage estimation and correction Computational advantages ( (For the state value function output), the length normalization factor α and the difficulty normalization factor β are applied and corrected using the compute_advantages_with_bias_correction method: ; in: ; Indicates the first The length of each response, It is the number of responses within the group; ; Indicates the first The standard deviation of the reward for each question.

[0038] Based on the Advantage Function theory of reinforcement learning, this is achieved by comparing the actual reward with the baseline. Measuring the value of an action; Eliminate long and short response gradient bias. By balancing the weights of easy and difficult questions, it reduces reasoning errors caused by bias by 25% compared to GRPO.

[0039] 7. Strategy Update Implementing a hybrid adaptive objective function using the Hybrid-Adapter-RL method:

[0040] in: , The strategy before the update; This is the cutting factor. Here is the KL divergence regularization coefficient; Hybrid clipping mechanism: for high-dominance samples ( )use Low dominance samples are adopted The dynamic fusion ratio is 0.3.

[0041] Introducing KL divergence avoids policy mutations, and the hybrid pruning mechanism improves exploration efficiency by 30% compared to PPO and reduces the number of convergence iterations by 25% compared to DRPO.

[0042] Adapter parameter optimization Update adapter parameters via gradient backpropagation. : ; in For the learning rate (using cosine annealing scheduling), only optimize parameter.

[0043] Efficient parameter fine-tuning preserves the general knowledge of the pre-trained model, improving the transferability across mathematical tasks by 20% compared to full fine-tuning.

[0044] 8. Sampling Adjustment The sampling probability is calculated using the adaptive_sampling_probability method: ; in: (adaptive_sampling_alpha); , As a weighting factor, The mean cosine distance between the sample and historical samples; The number of candidate samples.

[0045] The sample_history record of historical rewards is used to dynamically adjust the evaluation. The value and diversity-based sampling strategy improves training efficiency by 25% compared to the pure value sampling of DAPO.

[0046] 10. Iterative optimization Repeat steps 4-9 to implement the training loop using the train_step method: The reference model is updated every 50 steps using an exponential moving average (update_reference_model method). ); Clean up the sample history every 100 steps (retain 1000 of the most active records, using the cleanup_sample_history method). Stopping condition: No improvement in accuracy on the validation set for 5 consecutive rounds or 100 rounds.

[0047] A dynamic stopping mechanism avoids overfitting, adapter parameters are stored separately to save 70% of storage resources, and incremental updates to the reference model ensure policy stability.

[0048] Example 2: Application in logical reasoning tasks 1. Setting up the software and hardware environment The hardware is the same as in Example 1; the software uses the TensorFlow 2.6.0 framework, the TensorFlow Probability library is installed, and XLA is configured to accelerate compilation and optimize conditional judgment operations.

[0049] XLA acceleration improves logical reasoning efficiency by 25% compared to ordinary compilation.

[0050] 2. Dataset Processing 1000 logic reasoning questions (covering propositional logic, predicate logic, etc.) were selected and implemented using the WordPiece tokenizer (AutoTokenizer implementation): Add special markers to logical connectors (such as "if...then..."). <logic>); Truncation mechanism: For very long texts, a sliding window method (window size 256) is used to ensure that each subsequence contains a complete logical unit.

[0051] Special markings enhance the perception of logical structure, and the sliding window method avoids the breakage of long logical chains, improving the accuracy of relationship recognition by 15% compared with unmarked processing.

[0052] 3-10. Step Adjustment Response generation: Beam search (beam size=4) replaces Gaussian perturbation, num_return_sequences=32 in the generate method; Reward Calculation: Add a reward based on logical coherence (Pre-trained logic evaluation model calculation). ; Strategy Update: (Higher KL penalty), the learning rate is half that of Example 1; Sampling adjustment: Add logical complexity weights (based on the number of variables and the number of nesting levels).

[0053] Bundle search reduces the percentage of syntax error responses by 20%, and logical coherence rewards reduce the reasoning contradiction rate by 60%, improving accuracy by 2.3 times compared to GRPO.

[0054] Example 3: Application in Natural Language Processing Tasks 1. Setting up the software and hardware environment The hardware is the same as in Example 1; the software uses PyTorch Lightning 1.5.10, configured with 6-card distributed training and FP16 mixed precision.

[0055] Distributed training is 6 times faster, and mixed precision training reduces memory usage by 50%.

[0056] 2. Dataset Processing The GLUE dataset (9 tasks) was selected and standardized to "text pairs + labels" format. Sentence pairs are separated by [SEP], and token_type_ids distinguish their source; Maximum length 128 tokens, truncated at the end.

[0057] Standardized processing enables multi-task joint training, and BertTokenizer's encode_plus method directly supports text pair encoding.

[0058] 3-10. Step Adjustment Model initialization: Load the BERT-Base model, adapter Initialize independently for each task; Reward calculation: F1 score is used for categorized tasks, and [other methods are used for] generated tasks. ; Strategy Update: Generate Tasks (Encouraging diversity), classification tasks (To ensure stability); Sampling adjustment: Introduce a task balancing factor to ensure a balanced sample ratio.

[0059] Task-specific adapters avoid interference and differentiate. By setting a balanced approach between exploration and stabilization, the average accuracy on the GLUE test set improved by 22%.

[0060] As can be seen from the above embodiments, the present invention addresses the shortcomings of existing algorithms in terms of deviation control, efficiency, and diversity by using hybrid pruning mechanisms, adapter fine-tuning, adaptive sampling, and other techniques, combined with the specific implementation of the HybridAdapterRLTrainer class.< / logic>

Claims

1. A hybrid adaptive enhancement fine-tuning method, characterized in that: Includes the following steps: Data preprocessing; Initialize model parameters and reference strategy; Generate multiple responses for each prompt; Calculate the reward for each response; Calculate the advantage of each response and apply the length normalization factor. and difficulty normalization factor The formula for calculating the advantage is revised as follows: ； ； Indicates the first The length of each response, It represents the number of responses within the group; a difficulty normalization factor is introduced when calculating the strategy update. The objective function for policy update is: ； in, , Indicates the first The standard deviation of the reward for each question; The model update strategy is based on the corrected advantage and the adaptive sampling probability, wherein the adaptive sampling probability is: ； It is based on the sample The potential value derived from historical performance and diversity calculations; The sampling probability is dynamically adjusted based on the potential value of the sample. Iterative optimization continues until the model converges or reaches the preset number of training rounds.

2. The hybrid adaptive enhancement fine-tuning method as described in claim 1, characterized in that: The data preprocessing includes labeling, standardizing, and normalizing the training data, as well as padding and truncating the data to meet the model input requirements.

3. The hybrid adaptive enhancement fine-tuning method as described in claim 1, characterized in that: The initialization of model parameters and reference strategy refers to loading the pre-trained base model and initializing the model parameters and reference strategy.

4. The hybrid adaptive enhancement fine-tuning method as described in claim 1, characterized in that: The generation of multiple responses refers to generating multiple different output responses for each prompt, which are then used for subsequent advantage estimation and policy updates.

5. The hybrid adaptive enhancement fine-tuning method as described in claim 1, characterized in that: The calculation of the reward for each response refers to assigning a corresponding reward value to each response based on a predefined reward function and factors such as the correctness of the response.

6. The hybrid adaptive enhancement fine-tuning method as described in claim 1, characterized in that: When updating the model strategy based on the corrected advantage and adaptive sampling probability, a hybrid pruning mechanism is adopted, which combines the pruning higher mechanism and the dynamic sampling strategy.

7. The hybrid adaptive enhancement fine-tuning method, system, and apparatus as described in claim 1, characterized in that: The iterative optimization refers to the repeated steps of generating responses, calculating rewards, correcting advantage estimates, updating policies, and adjusting sampling until the model's performance on the validation set no longer improves significantly or reaches the preset number of training rounds.

8. A hybrid adaptive enhancement fine-tuning system, characterized in that: include: Data preprocessing module: used to perform labeling, standardization, and normalization operations on training data; Model initialization module: used to initialize model parameters and reference strategies; Response generation module: Used to generate multiple responses for each prompt; Reward calculation module: Used to calculate the reward for each response; Advantage Correction Module: Used to calculate the advantage of each response and correct it by applying a length normalization factor α and a difficulty normalization factor β; Policy update module: used to update the model policy based on the corrected advantage and adaptive sampling probability; Sampling adjustment module: used to dynamically adjust the sampling probability based on the potential value of the sample; Iterative optimization module: Used to repeat the above steps until the model converges or reaches the preset training rounds.

9. A hybrid adaptive enhancement fine-tuning device, characterized in that: It includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the hybrid adaptive enhancement fine-tuning method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that, when executed by a processor, implements the hybrid adaptive reinforcement fine-tuning method as described in any one of claims 1 to 7.

Images