Large language model fine tuning method and device based on gradient sensitivity and medium
By using gradient sensitivity to select parameters and dynamically freezing low-sensitivity parameters, the problems of high resource consumption and blind parameter selection in fine-tuning of large language models are solved, achieving efficient and stable model fine-tuning and improving training efficiency and generalization ability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 夏云龙
- Filing Date
- 2025-09-22
- Publication Date
- 2026-05-08
AI Technical Summary
Existing large language models incur high computational and time costs when fine-tuning in specific vertical domains, and existing PEFT techniques suffer from inference latency, blind parameter selection, and overfitting/underfitting issues.
Parameters are dynamically filtered using gradient sensitivity. A small amount of probe dataset is extracted for training, the gradient sensitivity of the parameters is calculated and a threshold is set, low-sensitivity parameters are frozen, and only high-sensitivity parameters are fine-tuned. The parameter freezing is implemented using the PyTorch framework.
It significantly reduces computational resources and time costs, maintains model performance, avoids overfitting and underfitting, and improves training efficiency and generalization ability.
Smart Images
Figure FT_1
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and machine learning, and particularly to transfer learning and fine-tuning techniques for large language models (LMs). Specifically, it relates to a model fine-tuning method, apparatus, and computer-readable storage medium based on gradient sensitivity-based dynamic screening and parameter freezing. Background Technology
[0002] Large Language Models (LLMs) are widely used in various natural language processing tasks due to their powerful versatility. However, when applied to specific vertical domains (such as healthcare, finance, and e-commerce), their performance is often unsatisfactory due to a lack of domain-specific knowledge. Therefore, fine-tuning pre-trained LLMs using domain-specific data has become a standard practice.
[0003] Because LLM parameters are extremely large (typically in the billions or even trillions), full fine-tuning requires enormous computational resources and time, which is often impractical. Therefore, parameter-efficient fine-tuning (PEFT) techniques have emerged, aiming to approximate the performance of full fine-tuning by optimizing a small number of parameters. The principles and limitations of existing major PEFT techniques are as follows: Adapter Tuning
[0004] Principle: Insert small neural network modules (Adapters) between model layers, and train only these modules.
[0005] Advantages: Low memory usage and fast training speed; retains pre-trained knowledge, suitable for multi-task learning.
[0006] Disadvantages: Increased model depth leads to increased inference latency. Adapter architecture design impacts performance (e.g., inter-layer location, dimensionality). LoRA (Low-Rank Adaptation)
[0007] Principle: The update amount of full parameter fine-tuning (ΔW = A·B) is approximated by low-rank matrix decomposition.
[0008] Advantages: No inference latency, extremely low memory usage; can be combined with other methods (such as quantization) (such as QLoRA).
[0009] Disadvantages: The low-rank assumption may limit the model's expressive power. The choice of rank (r) requires parameter tuning and affects the results. Prefix-Tuning
[0010] Principle: Add a learnable continuous vector (Prefix) before the input as a task prompt.
[0011] Advantages: Only a few parameters are optimized, making it suitable for generating tasks; no modification to the model structure is required.
[0012] Disadvantages: Prefix length is sensitive; excessive length affects efficiency; training stability is poor (requiring techniques such as gradient clipping). Prompt Tuning
[0013] Principle: The extended input prompt is a trainable vector that guides the model's output.
[0014] Advantages: Very few parameters (only Prompt-related parameters are optimized); suitable for scenarios with few or zero samples.
[0015] Disadvantages: Sensitive to Prompt initialization, requiring template design; limited effectiveness for complex tasks. P-Tuning Series (P-Tuning v2)
[0016] Principle: Optimize continuous Prompt vectors and extend them to all model layers.
[0017] Advantages: More stable than traditional prompt tuning, suitable for NLU tasks. The number of parameters is still relatively small.
[0018] Disadvantages: The training time is relatively long (requires optimization layer by layer). Freeze-Based Method
[0019] Principle: Freeze most parameters and only fine-tune some layers (such as the top layer or specific modules).
[0020] Advantages: Low memory usage, fast training speed. Simple and easy to implement.
[0021] Disadvantages: Performance depends on the choice of thawing layer and may be underfit. BitFit (Bias-Term Fine-Tuning)
[0022] Principle: Only fine-tune the bias terms in the model.
[0023] Advantages: Very few parameters (usually accounting for 0.1% of the total parameters of the model); extremely low resource consumption.
[0024] Disadvantages: It is only applicable to tasks where the bias has a significant impact, and has poor flexibility.
[0025] Therefore, there is an urgent need in this field for a method that can maintain the integrity of the original model structure, without modifying the input data, and can adaptively and rationally select key parameters for efficient fine-tuning, so as to significantly reduce computational overhead while ensuring performance. Summary of the Invention
[0026] (a) Purpose of the invention The purpose of this invention is to overcome the aforementioned shortcomings of existing PEFT techniques and provide a gradient-sensitive method for fine-tuning large language models. This method aims to dynamically and adaptively identify parameters crucial to the current fine-tuning task using gradient information, and freeze converged or irrelevant parameters. This significantly reduces the computational resources and time required for fine-tuning without altering the model structure or modifying the training data, while simultaneously avoiding overfitting and underfitting, thus ensuring model performance.
[0027] (II) Technical Solution
[0028] To achieve the above-mentioned objectives, the present invention provides the following technical solution: A gradient-sensitive method for fine-tuning large language models includes the following steps: S1: Data Sampling: From the complete fine-tuning training dataset, a small portion (no more than 10%) of the data is extracted as a probe dataset. This dataset is independent and identically distributed from the full dataset and is used to represent the current fine-tuning task.
[0029] S2: Probe Training: Using the probe dataset, train the pre-trained large language model for a small number of epochs. This stage uses all parameters for training, with the aim of obtaining gradient information of the parameters on the current task.
[0030] S3: Gradient Sensitivity Calculation and Threshold Determination: After probe training is completed, calculate the gradient (or gradient norm) of all model parameters, which will be used as an indicator to measure the "sensitivity" or "importance" of the parameter. Based on the preset target training parameter ratio (e.g., planning to train only 1% of the parameters), calculate the corresponding gradient sensitivity threshold (e.g., taking the 99th percentile of the gradient norm).
[0031] S4: Parameter Freeze: Iterate through all model parameters and compare their gradient sensitivity with the threshold. Set parameters with sensitivity below the threshold to an untrainable (frozen) state, and retain only parameters with sensitivity above the threshold for subsequent training.
[0032] S5: Efficient fine-tuning: Using the complete fine-tuning training dataset, train only on the unfrozen parameters selected in step S4 until the model converges or reaches the preset stopping condition.
[0033] S6: Model Saving: Saves the final model after fine-tuning.
[0034] The gradient sensitivity mentioned above is obtained by calculating the L2 norm of the parameter gradient vector.
[0035] The gradient thresholds mentioned above are determined by sorting the gradient sensitivity of all parameters and taking the percentile based on the proportion of parameters to be trained according to the planned training parameters.
[0036] In the PyTorch framework, the aforementioned model freezing is achieved by setting the `requires_grad` attribute of the parameter to `False`.
[0037] The present invention also provides a system for implementing the above method, comprising: 1) Data sampling module, used to generate probe datasets; 2) Probe training module, used to perform initial training and collect gradients; 3) Sensitivity analysis module, used to calculate gradient sensitivity and determine the threshold; 4) Parameter freezing module, used to freeze low-sensitivity parameters; 5) High-efficiency fine-tuning module, used for final fine-tuning of the remaining parameters.
[0038] The present invention also provides an electronic device and a computer-readable storage medium for performing the above-described method.
[0039] (III) Beneficial Effects
[0040] Compared with the prior art, the present invention has the following significant advantages: 1. Lossless structure: No modification is required to the structure of the original pre-trained model, avoiding the additional inference latency and complexity introduced by methods such as Adapter and LoRA, and maintaining the performance of the original model.
[0041] 2. Data lossless: Unlike methods such as Prompt Tuning, it does not require modification of input data, ensuring the stability and simplicity of the training process.
[0042] 3. High efficiency and energy saving: Through precise selection, only 1%-10% of the key parameters are trained, which greatly reduces the memory usage and computation during training and significantly shortens the training time.
[0043] 4. Adaptive Intelligence: It adaptively selects parameters based on gradient signals, overcoming the blindness of manually selecting the frozen layer, effectively avoiding underfitting, and ensuring fine-tuning results.
[0044] 5. Prevent overfitting: Freezing a large number of converged parameters is essentially an efficient regularization method that helps improve the model's generalization ability on the validation set. Attached Figure Description
[0046] Figure 1. Flowchart of the large-scale model fine-tuning optimization of this invention Detailed Implementation
[0047] The present invention will now be described in further detail with reference to the accompanying drawings and examples.
[0048] This invention provides a gradient-sensitive method for fine-tuning large language models, the process of which is shown in Figure 1. The following description uses the BERT model and PyTorch framework as examples, but this method is also applicable to other architectures of LLM (such as GPT, LLaMA, etc.) and other deep learning frameworks.
[0049] S1: Preparing the Model and Data Load the pre-trained model bert-base-uncased and prepare a fine-tuning dataset full_dataset for a specific task (such as medical question answering). Randomly extract 10% of the data from full_dataset to build the probe dataset probe_dataset and create the corresponding data loader probe_dataloader. S2: Model Training
[0050] The model is trained using the probe dataset prepared in step S1.
[0051] S3: Probe Training and Gradient Collection Initialize the optimizer and train it for 1-3 epochs using probe_dataloader. After the backpropagation of each training step, calculate and record the gradient norm of each parameter.
[0052] The Python code example is as follows: import torch from transformers import AutoModel import numpy as np model = AutoModel.from_pretrained("bert-base-uncased") optimizer = torch.optim.AdamW(model.parameters(), lr=1e-5) # Probe training phase model.train() grad_norms = [] # for storing gradient norms for batch in probe_dataloader: optimizer.zero_grad() outputs = model(**batch) loss = outputs. loss.backward() # Collect the gradient norm of all parameters with torch.no_grad(): for p in model.parameters(): if p.grad is not None: grad_norm = p.grad.norm().item() grad_norms.append(grad_norm) else: grad_norms.append(0.0) optimizer.step() S4: Calculate the threshold and freeze the parameters
[0053] Set the target training parameter ratio to 1%, and calculate the 99th percentile of the gradient norm as a threshold. Iterate through all parameters and freeze those whose gradient norm is below this threshold.
[0054] The Python code example is as follows: # Calculate threshold threshold = np.percentile(grad_norms, 99) # Freeze low-sensitivity parameters for name, p in model.named_parameters(): if p.grad is not None: grad_norm = p.grad.norm().item() if grad_norm < threshold: p.requires_grad = False # Freeze this parameter S5: High-efficiency fine-tuning
[0055] Create a full_dataloader to train the model using the full dataset until it converges.
[0056] The Python code example is as follows: # Efficient fine-tuning using the full dataset; this training only trained the parameters that were not frozen in step S4. for epoch in range(num_epochs): for batchinfull_dataloader: # ... Training steps...
[0057] S6: Save and Deploy
[0058] After training is complete, save the model using model.save_pretrained("fine_tuned_model") for subsequent inference deployment.
[0059] The above examples demonstrate that the method of the present invention can significantly reduce the number of trainable parameters and greatly improve training efficiency. At the same time, the performance of the model on specific downstream tasks is comparable to or even better than that of full parameter fine-tuning.
[0060] In summary, this invention effectively overcomes the various shortcomings of the prior art and has high industrial application value.
Claims
1. A method, apparatus, and medium for fine-tuning a large language model based on gradient sensitivity, characterized in that, Includes the following steps: 1) Extract a portion of the data from the fine-tuning training data to create a probe dataset; 2) Use the probe dataset to perform a small amount of training on the pre-trained large language model to obtain gradient information of the model parameters; 3) Calculate the gradient sensitivity of each parameter based on the gradient information, and set the gradient threshold; 4) Set parameters with gradient sensitivity below the threshold to an untrainable state, and keep the remaining parameters in a trainable state; 5) Train the model using all the fine-tuned training data until the model converges; 6) Save the trained model.
2. The method according to claim 1, characterized in that, The size of the probe dataset does not exceed 10% of the total fine-tuning training data.
3. The method according to claim 1, characterized in that, The gradient sensitivity is obtained by calculating the norm of the parameter gradient.
4. The method according to claim 1, characterized in that, The gradient threshold is dynamically determined based on a preset ratio of the parameters to be trained.
5. The method according to claim 4, characterized in that, The gradient threshold is the percentile of the gradient norm, corresponding to the proportion of the parameters to be trained.
6. The method according to claim 1, characterized in that, The untrainable state described in step (d) is achieved by setting the requires_grad attribute of the parameter to False.
7. A high-efficiency fine-tuning system for large language models, used to implement the method of any one of claims 1 to 6, characterized in that, include: 1) Data sampling module, used to sample and generate probe datasets from the full training data; 2) Probe training module, used to perform initial training on the pre-trained model and collect gradients using the probe dataset; 3) Sensitivity analysis module, used to calculate the gradient sensitivity of each parameter and determine the parameters to be frozen accordingly; 4) Parameter freezing module, used to set low-sensitivity parameters to an untrainable state; 5) High-efficiency fine-tuning module, used to fine-tune training only on unfrozen parameters using the full dataset.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.