Large model translation fine-tuning method based on entropy change parameter importance perception
By calculating the entropy change of the Transformer layer, the LoRA rank is dynamically identified and assigned, solving the problem of unreasonable parameter allocation in the LoRA method and improving the performance and efficiency of low-resource language translation models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- KUNMING UNIV OF SCI & TECH
- Filing Date
- 2026-05-11
- Publication Date
- 2026-06-09
AI Technical Summary
Existing LoRA methods, in low-resource language translation tasks, neglect the functional differences between different layers, leading to unreasonable parameter allocation, performance degradation, and an inability to effectively improve model performance.
By calculating the forward entropy changes of the Transformer layers, key layers are dynamically identified and LoRA ranks are assigned differentially to achieve adaptive fine-tuning, avoid gradient noise interference, and focus on the key layers of the task.
Without increasing the overall trainable parameters, it significantly improves the performance of low-resource translation tasks, enhances the model's fine-tuning efficiency and adaptability, and is applicable to a variety of low-resource language translation tasks.
Smart Images
Figure CN122174923A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a large model translation fine-tuning method based on the importance awareness of entropy-varying parameters, belonging to the field of efficient fine-tuning technology for large models. Background Technology
[0002] In recent years, Large Language Models (LLMs) have demonstrated powerful capabilities in tasks such as multilingual translation, but their performance remains significantly limited on low-resource languages. The fundamental reason is the scarcity of parallel corpora, which limits the effectiveness of conventional fine-tuning. Parameter-efficient fine-tuning methods, such as LoRA, have become mainstream solutions by introducing a low-rank update matrix, achieving task adaptation while freezing the backbone parameters.
[0003] However, existing LoRA systems typically employ a uniform rank across all layers, ignoring the functional differences between layers in low-resource tasks. Research indicates that high-resource languages are insensitive to uniform rank, while low-resource languages suffer performance degradation due to inappropriate parameter allocation. Therefore, a method is urgently needed that can dynamically identify critical layers and allocate fine-tuning capacity accordingly.
[0004] Recent work has found that the entropy changes in the output distribution of each layer during model forward propagation can reflect the evolution of its internal representation, providing an unsupervised signal for quantifying the importance of layers. Based on this, this invention proposes a large-scale model translation fine-tuning method based on the importance awareness of entropy-varying parameters. Summary of the Invention
[0005] The purpose of this invention is to provide a large model translation fine-tuning method based on the importance awareness of entropy-varying parameters. This method is an adaptive fine-tuning method that does not rely on gradients and is based solely on forward entropy signals. It solves the problems of low efficiency in LoRA fine-tuning parameter allocation in low-resource tasks and poor model performance caused by data scarcity in low-resource tasks.
[0006] The technical solution of this invention is: a large model translation fine-tuning method based on the importance awareness of entropy-varying parameters, the method comprising the following steps:
[0007] Step 1, Entropy Preheating: Freeze all trainable parameters, feed the input data of the low-resource task into the pre-trained large language model, perform multiple forward propagations, and collect the entropy sequence of the output distribution of each Transformer layer.
[0008] Step 2, Layer Importance Modeling: Based on the entropy value sequence, calculate the importance score of each layer from two dimensions: the intensity of inter-layer information change and the stability of intra-layer response;
[0009] Step 3, Adaptive Rank Allocation: Based on the importance score, dynamically assign differentiated LoRA ranks to each layer, with layers of high importance assigned high rank and layers of low importance assigned low rank.
[0010] Step 4: Fine-tuning training: Enable the LoRA module with the assigned differentiated LoRA rank, and perform standard fine-tuning training on the model to obtain a fine-tuned model adapted to the target low-resource task.
[0011] Furthermore, the specific steps of Step 1 include:
[0012] Step 1.1: Feed the batch input data of the low-resource task into the pre-trained large language model, freeze all backbone parameters and LoRA modules, and only perform forward propagation; given an input batch The model outputs the hidden states of all Transformer layers:
[0013]
[0014] in, Indicates the number of model layers. This represents the output of the l-th layer. Indicates batch size, Indicates the sequence length. The dimension representing the hidden state;
[0015] Step 1.2, for the first The hidden states of the layer first pass through the model's normalization layer and the language model's output head to obtain the corresponding vocabulary probability distribution:
[0016]
[0017] in, Indicates the first The layer in the first The nth sample, the nth The probability distribution of the vocabulary at each time step This represents the output mapping layer of the language model. Presentation layer normalization operation, Indicates the first The layer in the first The nth sample, the nth The output of each time step express function;
[0018] Step 1.3: Based on the vocabulary probability distribution, define the first... The layer in the first The average entropy value under each preheating step is:
[0019]
[0020] in, Indicates the first word in the vocabulary. The predicted probability of each word. This refers to the vocabulary size of the model;
[0021] Step 1.4: Repeat the process described in Step 1.3 during multiple training steps in the warm-up phase to collect a set of entropy values for each layer.
[0022]
[0023] in, Indicates the first The layer in the first The average entropy value calculated under each preheating step This indicates the total number of steps in the preheating phase.
[0024] Furthermore, the specific steps of Step 2 include:
[0025] Step 2.1: First, define inter-layer importance, which is used to measure the intensity of information change between adjacent layers:
[0026]
[0027] in, Indicates the first The average entropy value of the layer during the preheating stage;
[0028] Step 2.2: Next, define intra-layer importance to characterize the response differences of the same layer across different samples.
[0029] ;
[0030] in, Indicates the first The layer in the first The entropy value calculated under each preheating step This indicates the total number of steps in the preheating phase;
[0031] Step 2.3: Use the Min-Max normalization function to normalize and weight the two importance indicators mentioned above to obtain a unified importance score for each layer. :
[0032] ;
[0033] in, Presentation layer normalization operation, , These are the weighting coefficients.
[0034] Furthermore, the specific steps of Step 3 include:
[0035] Step 3.1: Use softmax with temperature parameter to score importance. Normalize:
[0036]
[0037] in, Indicates the first The relative importance weight of each layer in all Transformer layers. The temperature parameter is used to control the degree of difference in rank distribution between different layers, and L represents the number of model layers;
[0038] Step 3.2, given the minimum rank With the maximum rank Under the conditions, the first The LoRA rank of a layer is defined as:
[0039]
[0040] in, Indicates assignment to the first LoRA rank of the layer Functions that represent rounding to the nearest integer. This function truncates a value to a specified interval.
[0041] Furthermore, the specific steps of Step 4 include:
[0042] Step 4.1: After completing the entropy-guided rank allocation, insert the LoRA module into the target linear layer of the Transformer encoder and decoder; the original pre-trained model weight matrix... Completely frozen, training only the two low-rank matrices introduced by LoRA. and For any fitted weight matrix Its effective weights are expressed in the forward propagation as:
[0043]
[0044] in , ,rank For Step 3, the first Differentiated rank of layer allocation, and satisfying , The dimension representing the hidden state is also the dimension of the input to the linear layer. Indicates the dimension of the linear layer output;
[0045] Step 4.2: Subsequently, supervised fine-tuning of A and B is performed using low-resource translation data, with the backbone model parameters kept frozen throughout the training process; finally, the LoRA parameters with the best performance are saved as the fine-tuning result.
[0046] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the large model translation fine-tuning method based on the importance awareness of entropy-varying parameters.
[0047] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the large model translation fine-tuning method based on the importance awareness of entropy-varying parameters.
[0048] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the large model translation fine-tuning method based on the importance awareness of entropy-varying parameters.
[0049] The beneficial effects of this invention are:
[0050] 1. This invention does not require gradient signals, thus avoiding gradient noise interference under low resource conditions;
[0051] 2. The parameter allocation of this invention focuses more on the key layers of the task, improving the efficiency of fine-tuning. Specifically, in the early stage of fine-tuning, the method of this invention records the entropy sequence of the output distribution of each Transformer layer through the "entropy warm-up" stage. It models the importance of each layer to the task from two dimensions: the intensity of inter-layer information change and the stability of intra-layer response. Based on this, it dynamically allocates differentiated LoRA ranks to different layers, and allocates the limited capacity to the key layers of the task first without increasing the overall trainable parameters.
[0052] 3. The total number of trainable parameters in this invention is controllable, which conforms to the principle of efficient fine-tuning;
[0053] 4. The method of this invention is universal and applicable to various low-resource language large-model translation tasks;
[0054] 5. The method of this invention analyzes the entropy changes of the output distribution of each layer of the model in low-resource tasks, dynamically identifies key layers and allocates the rank of LoRA accordingly, thereby achieving a better fine-tuning effect.
[0055] 6. The method of this invention can effectively improve the performance of large models in low-resource translation tasks without increasing the number of trainable parameters. It has good versatility and practicality and is applicable to a variety of low-resource language machine translation tasks. Attached Figure Description
[0056] Figure 1A schematic diagram illustrating the framework structure of the method of the present invention is shown;
[0057] Figure 2 The overall process of the method of the present invention is shown. Detailed Implementation
[0058] Example 1: As Figures 1-2 As shown, a large model translation fine-tuning method based on entropy-varying parameter importance awareness is described, which includes the following steps:
[0059] Step 1, Entropy Preheating: Freeze all trainable parameters, feed the input data of the low-resource task into the pre-trained large language model, perform multiple forward propagations, and collect the entropy sequence of the output distribution of each Transformer layer.
[0060] Furthermore, the specific steps of Step 1 include:
[0061] Step 1.1: Feed the batch input data of the low-resource task into the pre-trained large language model, freeze all backbone parameters and LoRA modules, and only perform forward propagation; given an input batch The model outputs the hidden states of all Transformer layers:
[0062]
[0063] in, Indicates the number of model layers. This represents the output of the l-th layer. Indicates batch size, Indicates the sequence length. The dimension representing the hidden state;
[0064] Step 1.2, for the first The hidden states of the layer first pass through the model's normalization layer and the language model's output head to obtain the corresponding vocabulary probability distribution:
[0065]
[0066] in, Indicates the first The layer in the first The nth sample, the nth The probability distribution of the vocabulary at each time step This represents the output mapping layer of the language model. Presentation layer normalization operation, Indicates the first The layer in the first The nth sample, the nth The output of each time step express function;
[0067] Step 1.3: Based on the vocabulary probability distribution, define the first... The layer in the first The average entropy value under each preheating step is:
[0068]
[0069] in, Indicates the first word in the vocabulary. The predicted probability of each word. This refers to the vocabulary size of the model;
[0070] Step 1.4: Repeat the process described in Step 1.3 during multiple training steps in the warm-up phase to collect a set of entropy values for each layer.
[0071]
[0072] in, Indicates the first The layer in the first The average entropy value calculated under each preheating step This indicates the total number of steps in the preheating phase.
[0073] Step 2, Layer Importance Modeling: Based on the entropy value sequence, calculate the importance score of each layer from two dimensions: the intensity of inter-layer information change and the stability of intra-layer response;
[0074] Furthermore, the specific steps of Step 2 include:
[0075] Step 2.1: First, define Inter-layer Importance, which measures the intensity of information change between adjacent layers:
[0076]
[0077] in, Indicates the first The average entropy value of the layer during the preheating stage;
[0078] Large differences in interlayer entropy values indicate that the information has changed significantly at that layer, suggesting the key role of that layer in the task;
[0079] Step 2.2: Next, define intra-layer importance to characterize the response differences of the same layer on different samples.
[0080] ;
[0081] in, Indicates the first The layer in the first The entropy value calculated under each preheating step This indicates the total number of steps in the warm-up phase; the larger the variance of the entropy value within a layer, the more significant the difference in the processing of different input samples by that layer, and the stronger its task relevance.
[0082] Step 2.3: To eliminate the dimensional differences in importance indicators across different levels, this invention uses a Min-Max normalization function to normalize and weight the two importance indicators, resulting in a unified importance score for each level. :
[0083] ;
[0084] in, Presentation layer normalization operation, , These are the weighting coefficients.
[0085] Step 3, Adaptive Rank Allocation: Based on the importance score, dynamically assign differentiated LoRA ranks to each layer, with layers of high importance assigned high rank and layers of low importance assigned low rank.
[0086] Furthermore, the specific steps of Step 3 include:
[0087] Step 3.1: To map the layer importance score to a specific LoRA rank, this invention uses a softmax algorithm with a temperature parameter to score the importance. Normalize:
[0088]
[0089] in, Indicates the first The relative importance weight of each layer in all Transformer layers. The temperature parameter is used to control the degree of difference in rank distribution between different layers, and L represents the number of model layers;
[0090] Step 3.2, given the minimum rank With the maximum rank Under the conditions, the first The LoRA rank of a layer is defined as:
[0091]
[0092] in, Indicates assignment to the first LoRA rank of the layer Functions that represent rounding to the nearest integer. This function truncates values to a specified interval. This allocation strategy ensures that, with a fixed parameter budget, model capacity is prioritized for key layers highly relevant to the task, while compressing the parameter size of redundant layers.
[0093] Step 4: Fine-tuning training: Enable the LoRA module with the assigned differentiated LoRA rank, and perform standard fine-tuning training on the model to obtain a fine-tuned model adapted to the target low-resource task.
[0094] Furthermore, the specific steps of Step 4 include:
[0095] Step 4.1: After completing the entropy-guided rank allocation, insert the LoRA module into the target linear layer of the Transformer encoder and decoder; the original pre-trained model weight matrix... Completely frozen, training only the two low-rank matrices introduced by LoRA. and For any fitted weight matrix Its effective weights are expressed in the forward propagation as:
[0096]
[0097] in , ,rank For Step 3, the first Differentiated rank of layer allocation, and satisfying , The dimension representing the hidden state is also the dimension of the input to the linear layer. Indicates the dimension of the linear layer output;
[0098] Step 4.2: Subsequently, supervised fine-tuning of A and B is performed using low-resource translation data, with the backbone model parameters kept frozen throughout the training process; finally, the LoRA parameters with the best performance are saved as the fine-tuning result.
[0099] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the large model translation fine-tuning method based on the importance awareness of entropy-varying parameters.
[0100] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the large model translation fine-tuning method based on the importance awareness of entropy-varying parameters.
[0101] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the large model translation fine-tuning method based on the importance awareness of entropy-varying parameters.
[0102] This invention can be applied to scenarios such as low-resource language machine translation, cross-language transfer learning, and efficient model deployment on edge devices. In particular, in multilingual translation tasks where parallel corpora are scarce (such as Chinese to Lao, Chinese to Burmese, etc.), this invention can effectively improve model performance while significantly reducing training and storage overhead.
[0103] To verify the effectiveness of the large model translation fine-tuning method based on the importance awareness of entropy-varying parameters proposed in this invention, a comparative experiment was conducted with the baseline method. The training set and test set sizes used by all methods are shown in the table below.
[0104] Table 1 shows the size of the training and testing sets.
[0105] In Table 1, "ALT-Parallel-Corpus-20191206" is a parallel corpus for Asian languages, a publicly available resource in the field of natural language processing. "FLORES-200-devtest" is the development test set in the FLORES-200 benchmark dataset, primarily used to evaluate the performance of multilingual machine translation models on low-resource languages. The test set uses the industry-recognized FLORES-200 public dataset, and the comparative experimental results are as follows:
[0106] Table 2 shows the comparative experimental results of the method of this invention and different fine-tuning methods on the FLORES-200 dataset.
[0107] In this table, language pairs are represented using the abbreviation form of "source language-target language," and the language codes follow the ISO 639 standard. Specifically, these include: zh-lo (Chinese → Lao), zh-my (Chinese → Burmese), and zh-th (Chinese → Thai). The English and Chinese expressions in Table 2, "spBLEU / XCOMET," refer to two industry-recognized test metrics in the machine translation field. The English and Chinese expressions in Table 2, "Qwen2.5-7B-Istruct" and "Qwen3-4B-Instruct-2507," are the names of open-source large language models. The English and Chinese expressions in Table 2, "LoRA," refer to a publicly known method for efficient fine-tuning of large model parameters.
[0108] Experimental results show that this invention significantly outperforms the original model and the traditional LoRA method in multiple low-resource language translation tasks. As shown in the figure, in three typical low-resource scenarios—zh-lo (Chinese to Lao), zh-my (Chinese to Burmese), and zh-th (Chinese to Thai)—this invention achieves stable improvements in both spBLEU and XCOMET metrics. Specifically, compared to the original model, this invention achieves an average improvement of approximately 4.46 points in spBLEU and approximately 8.77 points in XCOMET; compared to unified-rank LoRA, it achieves an average improvement of approximately 1.52 points in spBLEU and approximately 4.95 points in XCOMET, maintaining superior performance in the vast majority of tasks. Particularly in high-difficulty translation tasks such as Chinese to Lao, this invention demonstrates stronger generalization ability, validating the effectiveness and robustness of this method in low-resource scenarios.
[0109] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.
Claims
1. A large-scale model translation fine-tuning method based on entropy-varying parameter importance awareness, characterized in that, The method includes the following steps: Step 1, Entropy Preheating: Freeze all trainable parameters, feed the input data of the low-resource task into the pre-trained large language model, perform multiple forward propagations, and collect the entropy sequence of the output distribution of each Transformer layer. Step 2, Layer Importance Modeling: Based on the entropy value sequence, calculate the importance score of each layer from two dimensions: the intensity of inter-layer information change and the stability of intra-layer response; Step 3, Adaptive Rank Allocation: Based on the importance score, dynamically assign differentiated LoRA ranks to each layer, with layers of high importance assigned high rank and layers of low importance assigned low rank. Step 4: Fine-tuning training: Enable the LoRA module with the assigned differentiated LoRA rank, and perform standard fine-tuning training on the model to obtain a fine-tuned model adapted to the target low-resource task.
2. The large model translation fine-tuning method based on entropy-varying parameter importance awareness as described in claim 1, characterized in that, The specific steps of Step 1 include: Step 1.1: Feed the batch input data of the low-resource task into the pre-trained large language model, freeze all backbone parameters and LoRA modules, and only perform forward propagation; given an input batch The model outputs the hidden states of all Transformer layers: ； in, Indicates the number of model layers. This represents the output of the l-th layer. Indicates batch size, Indicates the sequence length. The dimension representing the hidden state; Step 1.2, for the first The hidden states of the layer first pass through the model's normalization layer and the language model's output head to obtain the corresponding vocabulary probability distribution: ； in, Indicates the first The layer in the first The nth sample, the nth The probability distribution of the vocabulary at each time step This represents the output mapping layer of the language model. Presentation layer normalization operation, Indicates the first The layer in the first The nth sample, the nth The output of each time step express function; Step 1.3: Based on the vocabulary probability distribution, define the first... The layer in the first The average entropy value under each preheating step is: ； in, Indicates the first word in the vocabulary. The predicted probability of each word. This refers to the vocabulary size of the model; Step 1.4: Repeat the process described in Step 1.3 during multiple training steps in the warm-up phase to collect a set of entropy values for each layer. ； in, Indicates the first The layer in the first The average entropy value calculated under each preheating step This indicates the total number of steps in the preheating phase.
3. The large model translation fine-tuning method based on entropy-varying parameter importance awareness as described in claim 1, characterized in that, The specific steps of Step 2 include: Step 2.1: First, define inter-layer importance, which is used to measure the intensity of information change between adjacent layers: ； in, Indicates the first The average entropy value of the layer during the preheating stage; Step 2.2: Next, define intra-layer importance to characterize the response differences of the same layer across different samples. ； in, Indicates the first The layer in the first The entropy value calculated under each preheating step This indicates the total number of steps in the preheating phase; Step 2.3: Use the Min-Max normalization function to normalize and weight the two importance indicators mentioned above to obtain a unified importance score for each layer. : ； in, Presentation layer normalization operation, , These are the weighting coefficients.
4. The large model translation fine-tuning method based on entropy-varying parameter importance awareness as described in claim 1, characterized in that, The specific steps of Step 3 include: Step 3.1: Use softmax with temperature parameter to score importance. Normalize: ； in, Indicates the first The relative importance weight of each layer in all Transformer layers. The temperature parameter is used to control the degree of difference in rank distribution between different layers, and L represents the number of model layers; Step 3.2, given the minimum rank With the maximum rank Under the conditions, the first The LoRA rank of a layer is defined as: ； in, Indicates assignment to the first LoRA rank of the layer Functions that represent rounding to the nearest integer. This function truncates a value to a specified interval.
5. The large model translation fine-tuning method based on entropy-varying parameter importance awareness as described in claim 1, characterized in that, The specific steps of Step 4 include: Step 4.1: After completing the entropy-guided rank allocation, insert the LoRA module into the target linear layer of the Transformer encoder and decoder; the original pre-trained model weight matrix... Completely frozen, training only the two low-rank matrices introduced by LoRA. and For any fitted weight matrix Its effective weights are expressed in the forward propagation as: ； in , ,rank For Step 3, the first Differentiated rank of layer allocation, and satisfying , The dimension representing the hidden state is also the dimension of the input to the linear layer. Indicates the dimension of the linear layer output; Step 4.2: Subsequently, supervised fine-tuning of A and B is performed using low-resource translation data, with the backbone model parameters kept frozen throughout the training process; finally, the LoRA parameters with the best performance are saved as the fine-tuning result.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the large model translation fine-tuning method based on entropy-varying parameter importance awareness as described in any one of claims 1 to 5.
7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the large model translation fine-tuning method based on entropy-varying parameter importance awareness as described in any one of claims 1 to 5.
8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the large model translation fine-tuning method based on entropy-varying parameter importance awareness as described in any one of claims 1 to 5.