Knowledge migration method from weak model to strong model based on dynamic logits adjustment
By dynamically adjusting the weight distribution of logits in the decoding step, the sub-optimization problem in knowledge transfer from small models to large models is solved, achieving more efficient knowledge adaptability and improving the performance of the model in multi-task scenarios.
Patent Information
- Application Number
- CN202511331181.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-17
- Publication Date
- 2026-02-10
AI Technical Summary
Existing technologies suffer from suboptimal problems in knowledge transfer from small models to large models, especially in multi-task scenarios, where fixed-weight transfer leads to poor knowledge adaptability.
A dynamic logits adjustment method is adopted. By optimizing the model weight allocation in each decoding step, an optimization objective function is constructed. KL divergence is used to measure distribution similarity, and the weights are dynamically adjusted to achieve adaptive knowledge transfer.
It alleviates the underoptimization problem caused by static weight transfer, improves the knowledge adaptability of the model in each decoding step, and enhances the effect of knowledge transfer.
Smart Images
Figure CN121503578A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to knowledge transfer technology, and more particularly to a knowledge transfer method from a weak model to a strong model based on dynamic logits adjustment. Background Technology
[0002] Existing methods can only achieve complex knowledge transfer on small models by calculating the transfer ratio for static knowledge, but this is prone to suboptimal problems. Therefore, we propose a dynamic logits fusion method to achieve knowledge transfer on small models for specific tasks. The main idea is to implement adaptive model weight allocation at each decoding step. Summary of the Invention
[0003] The technical problem to be solved by the present invention is to provide a knowledge transfer method from a weak model to a strong model based on dynamic logits adjustment, which addresses the shortcomings of the existing technology.
[0004] The technical solution adopted by this invention to solve its technical problem is: a knowledge transfer method from a weak model to a strong model based on dynamic logits adjustment, comprising the following steps: 1) Obtain a large model (LLM) and a small model (SLM) with isomorphic architecture and a shared vocabulary. The pre-trained parameter set for the large model (LLM) is... The output logits is The output probability distribution is P; Small model pre-trained SLM: parameter set is , logits is The output probability distribution is Q; 2) After fine-tuning the small model according to the task scenario, knowledge transfer is performed; For a single task, the parameter set is obtained after fine-tuning a small model. , logits is Distribution is For multi-tasking, fine-tune the parameters. , logits is The output probability distribution is ; Among them, multi-task means that the task involves multiple domains, including multiple small SLMs in different domains; Let be the output probability distribution of the t-th domain small model; The knowledge from the fine-tuned small model is transferred to the large model, thus obtaining the output distribution of the large model. .
[0005] According to the above scheme, in step 2), if it is a single task, the specific steps for fine-tuning the small model are as follows: 2.1) Assume that for different model sizes, the distance between the fine-tuned model output and the pre-trained model output is the same in each decoding step, as shown below:
[0006]
[0007] in, Used to measure the distance between two distributions generated by two language models; 2.2) Construct an optimization objective function and optimize the following objective function in each decoding step. Dynamic optimization during decoding. ; ; in, The distribution is represented by the logits arithmetic of the LLM:
[0008] According to the above scheme, in step 2), if it involves multiple tasks, the specific steps for fine-tuning the small model are as follows: Fine-tuning SLM parameters in each domain Thus, multiple task-specific expert models are obtained. ; Make the following assumptions, and the joint distribution Decompose into alignment constraints of multiple edge distributions; for each neighborhood t,
[0009] ; in, Representative from the SLM parameters for each domain The distribution represent The distribution after fusion; Based on the above constraints, an optimization objective function is constructed;
[0010] Among them, the corresponding parameters Let be the hyperparameters of T domains. Dynamic optimization in each decoding step .
[0011] The beneficial effects of this invention are: This invention proposes a knowledge transfer method from a weak model to a strong model based on dynamic logits adjustment. Each small model focuses on a different task. In each decoding step, the problem of poor adaptability of fixed weight α value is solved by KL divergence constraint and dynamic hyperparameter optimization, which alleviates the underoptimization problem caused by static weight transfer. Attached Figure Description The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation
[0012] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0013] First, we can use the Kullback-Leibler (KL) divergence to measure the two distributions generated by the two language models. and The similarity between two language models (they have the same vocabulary) can be viewed as the distance between them.
[0014] (1) If the condition is already implied in the context, we will omit the condition. The conditions, and only use .
[0015] Suppose we have two pre-trained autoregressive models with isomorphic architectures and sharing the same vocabulary: a smaller model with a parameter set of... And a large model with a parameter set of Our goal is to fine-tune the small model to obtain... This fine-tuning knowledge is then transferred to a larger model. Previous work transferred this fine-tuning knowledge to a larger model by designing arithmetic operations between logits, thereby obtaining the output distribution of the larger model. The details are as follows: (2) in, The token representing the current location; This represents the context sequence preceding the current token; , and These represent the logits of the large model, the small model, and the fine-tuned small model, respectively. Their normalized distributions are represented by... , and It means. Here, It is a pre-tuned hyperparameter that controls the degree to which knowledge is transferred from the small model.
[0016] However, using predefined This will lead to the generated LLM distribution Having a fixed trajectory may not be suitable for every step of decoding. Therefore, we need to use dynamic trajectories in the decoding steps. .
[0017] like Figure 1 As shown, a knowledge transfer method from a weak model to a strong model based on dynamic logits adjustment includes the following steps: 1) Obtain a large model (LLM) and a small model (SLM) with isomorphic architecture and a shared vocabulary. The pre-trained parameter set for the large model (LLM) is... The output logits is The normalized probability distribution is P; Small model pre-trained SLM: parameter set is , logits is The probability distribution is Q; 2) Fine-tune the small model. If it is a single task, obtain the parameter set after fine-tuning the small model. , logits is Distribution is For multi-tasking, fine-tune the parameters. , logits is The probability distribution is ; Among them, multi-task means that the task involves multiple domains, including multiple small SLMs in different domains; The fine-tuning knowledge from the small model is transferred to the large model, thereby obtaining the output distribution of the large model. .
[0018] In step 2), if it is a single task, the specific steps for fine-tuning the small model are as follows: 2.1) Assume that for different model sizes, the distance between the fine-tuned model output and the pre-trained model output is the same in each decoding step, as shown below:
[0019]
[0020] in, Used to measure the distance between two distributions generated by two language models; 2.2) Construct an optimization objective function and optimize the following objective function in each decoding step. Dynamic optimization during decoding. ; ; in, The distribution is represented by the logits arithmetic of the LLM: ; By adjusting α, the KL divergence feature of P is made to match Q. ft KL divergence characteristics with Q; In each step of the LLM decoding process (i.e., for each...) (Context), repeat the following steps: Calculate the current context and ; Using the objective function as the criterion, solve for the optimal α; Calculate using the optimal α As the predicted distribution of the current step in the LLM, the next token is generated. .
[0021] When dealing with complex tasks or new domains, a general-purpose LLM may lack the necessary expertise, while a single SLM may not be able to provide sufficient expertise due to capability gaps. Expanding to incorporate multiple smaller, expert SLMs could be considered.
[0022] In step 2), if it involves multiple tasks, the specific steps for fine-tuning the small model are as follows: Fine-tuning SLM parameters in each domain Thus, multiple task-specific expert models are obtained. ; During the decoding process, the knowledge of these domain-specific SLMs is simultaneously transferred to the LLM. The assumptions of equation (3) are modified as follows: the joint distribution... Decompose into alignment constraints of multiple edge distributions;
[0023] ; in, Representative from the SLM parameters for each domain The distribution represent The distribution after fusion; Since it is difficult to obtain the joint distribution of multiple expert models, we decompose the joint distribution constraint problem into a marginal distribution optimization problem with multiple objectives, and construct an optimization objective function based on the above constraints.
[0024] The distribution is represented by the logits arithmetic of LLM as follows: Among them, the hyperparameters corresponding to the knowledge transfer strength parameter of the t-th domain in the multi-task scenario are the hyperparameters of the T domains. t=1,2,3,...T; Here, we use Indicates the first The logits of each expert. Intuitively, this process projects the KL divergence between logits onto Euclidean space and finds a center point with the minimum sum of distances as the optimal KL value. This optimal KL value corresponds to the output distribution of a large model with multi-domain knowledge. This is achieved through optimization. To obtain the optimal KL value. Typically, The value of is between 0 and 2 to avoid over-migrating. In a multi-task setting, we can accelerate the optimization process by optimizing the boundary of only one expert model, for example, by applying only one SLM expert in the current decoding step.
[0025] Dynamic optimization at each decoding step Using a multi-objective function as the criterion, the optimal solution is obtained. Vectors, using optimal calculate Generate the current step token .
[0026] The main contributions of this invention are as follows.
[0027] 1. Existing logit arithmetic methods were re-evaluated, highlighting the significant impact of fusion weights and the limitations of a single small model on test performance.
[0028] 2. A method for autonomous learning and fusion weights through constraint optimization is introduced, which can approximate the large amount of computation required for fine-tuning large-scale basic models.
[0029] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
Claims
1. A knowledge transfer method from a weak model to a strong model based on dynamic logits adjustment, characterized in that, Includes the following steps: 1) Obtain a large model (LLM) and a small model (SLM) with isomorphic architecture and a shared vocabulary. The pre-trained parameter set for the large model (LLM) is... The output logits is The output probability distribution is P; Small model pre-trained SLM: parameter set is , logits is The output probability distribution is Q; 2) After fine-tuning the small model according to the task scenario, knowledge transfer is performed; For a single task, the parameter set is obtained after fine-tuning a small model. , logits is Distribution is For multi-tasking, fine-tune the parameters. , logits is The output probability distribution is ; Among them, multi-task means that the task involves multiple domains, including multiple small SLMs in different domains; Let be the output probability distribution of the t-th domain small model; The knowledge from the fine-tuned small model is transferred to the large model, thus obtaining the output distribution of the large model. .
2. The knowledge transfer method from a weak model to a strong model based on dynamic logits adjustment according to claim 1, characterized in that, In step 2), if it is a single task, the specific steps for fine-tuning the small model are as follows: 2.1) Assume that for different model sizes, the distance between the fine-tuned model output and the pre-trained model output is the same in each decoding step, as shown below: in, Used to measure the distance between two distributions generated by two language models; 2.2) Construct an optimization objective function and optimize the following objective function in each decoding step. Dynamic optimization during decoding. ; ； in, The distribution is represented by the logits arithmetic of the LLM: 。 3. The knowledge transfer method from a weak model to a strong model based on dynamic logits adjustment according to claim 1, characterized in that, In step 2), if it involves multiple tasks, the specific steps for fine-tuning the small model are as follows: Fine-tuning SLM parameters in each domain Thus, multiple task-specific expert models are obtained. ; Make the following assumptions, and the joint distribution Decompose into alignment constraints of multiple edge distributions; for each neighborhood t, ； in, Representative from the SLM parameters for each domain The distribution, represent The distribution after fusion; Based on the above constraints, an optimization objective function is constructed; Among them, the corresponding parameters Let be the hyperparameters of T domains. Dynamic optimization in each decoding step .
4. An electronic device, characterized in that, include: One or more processors; as well as Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 3.
5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 3.