Model fusion method based on dynamic combination modularization expert knowledge
By dynamically combining modular expert knowledge, model knowledge is divided into shared and exclusive parts. Router weights are calculated and exclusive vectors are compressed, which solves the performance gap problem in model fusion and realizes efficient merging and performance improvement of multi-task models.
Patent Information
- Application Number
- CN202511331187.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-17
- Publication Date
- 2026-02-10
AI Technical Summary
Existing technologies ignore the differences between different models and the heterogeneity of data in model fusion, resulting in a significant gap between the performance of the fused model and the fine-tuned model. The lack of flexibility in the general model leads to a performance decline.
We adopt a method based on dynamic combination of modular expert knowledge, which divides knowledge into shared and exclusive parts. We extract shared experts through task arithmetic, calculate router weights and compress exclusive vectors, and dynamically merge shared knowledge and task-specific knowledge to form a customized model.
This method achieves the merging of multiple models into a multi-task model without increasing additional training overhead, thereby improving model performance. In particular, it outperforms other merging methods in discriminative and generative tasks, with an average improvement of 28.34% in standardized scores.
Smart Images
Figure CN121503640A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to large language model technology, and in particular to a model fusion method based on dynamic combination of modular expert knowledge. BACKGROUND
[0002] In the era of large language models, model fusion is a problem that is generally concerned by the industry and academia, which can effectively fuse multiple single models with different specificities to complete complex multi-task requirements without increasing additional training overhead. However, the traditional model fusion method ignores: (1) the difference between different models; (2) data heterogeneity, thereby causing a significant gap between the performance of the fused model and the fine-tuned model, and the performance of the general model is reduced due to the lack of flexibility for diversified data. Research has found that there are two parts of common knowledge and exclusive knowledge in different models, and their differentiated combination plays an important role in the performance of the fused model, while directly combining knowledge will lead to a decline in the overall performance of the combined model. Therefore, a new model fusion method is needed. SUMMARY
[0003] The technical problem to be solved by the present application is to provide a model fusion method based on dynamic combination of modular expert knowledge in view of the defects in the prior art.
[0004] The technical scheme adopted by the present application to solve the technical problem is: a model fusion method based on dynamic combination of modular expert knowledge, comprising the following steps: 1) Knowledge modularization Divide the knowledge into shared and exclusive parts, and separate the common knowledge useful for all tasks; 1.1) Calculate the shared expert, and use the pre-merged model as a natural placeholder to encapsulate the common knowledge important for all tasks ; Use Task Arithmetic technology to extract the shared expert from the initial to-be-merged task model; 1.2) After removing the shared knowledge, retain the exclusive knowledge of each task model; Calculate the exclusive vector: ; After calculation, we can obtain T exclusive vectors, where T is the total number of task models; 2) Dynamic merging, dynamically merging shared knowledge and task-specific knowledge based on input; 2.1) Initialize the model output Y; 2.2) Calculate the router weight according to the semantic features of the input x, obtain the exclusive knowledge useful for the current x and the corresponding task model, and express the usefulness with the weight; Calculate router weight:
[0005] Where R is a router module, and the input is the embedded value of x. The output is the matching score of T tasks, and the router weight is obtained after the score is softmaxed. This represents the last-level word embedding sequence from shared experts. ; The weights are the corresponding weights for the T task models; These are the parameters of the router module, which is a lightweight neural network composed of fully connected layers. It is its weight; 2.3) Combine the shared expert with the exclusive vector of the selected person model into a customized model for the current x;
[0006] 2.4) Substitute the customized model θ into the basic model f(x;θ) to calculate the inference result of x, and add it to the output set Y. ; 2.5) After traversing all inputs x, output the final Y to complete one inference process.
[0007] According to the above scheme, the exclusive vector is a compressed exclusive vector: the exclusive vector is compressed into a compressed exclusive vector for each task using singular value decomposition (SVD).
[0008] According to the above scheme, the singular value decomposition process is as follows: Assumption Rank-m decomposition, Set the compression level to r, and only retain the portions corresponding to the first r largest singular values to achieve a low-rank task space, resulting in... .
[0009] The beneficial effects of this invention are: This invention proposes a method to merge multiple task-specific models into a single multi-task model without additional training. Attached Figure Description
[0010] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation
[0011] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0012] like Figure 1 As shown, a model fusion method based on dynamically combined modular expert knowledge includes the following steps: 1) Knowledge modularization Divide knowledge into shared and exclusive parts, and separate out common knowledge that is useful for all tasks; 1.1) Computational shared experts use pre-merged models as natural placeholders to encapsulate common knowledge important for all tasks. ; The Task Arithmetic technique is used to extract shared experts from the initial task models to be merged. 1.2) After removing shared knowledge, retain the exclusive knowledge of each task model; Calculate the exclusive vector: ; After the calculation is complete, we can obtain T exclusive vectors, where T is the total number of task models; 1.3) Compressed Exclusive Vectors: Singular Value Decomposition (SVD) is used to compress exclusive vectors into compressed exclusive vectors for each task.
[0013] The singular value decomposition process is as follows: Assumption Rank-m decomposition, Set the compression level to r, and only retain the portions corresponding to the first r largest singular values to achieve a low-rank task space, resulting in... .
[0014] 2) Dynamic merging: Based on input, shared knowledge and task-specific knowledge are dynamically merged; 2.1) Initialize the model output Y; 2.2) Based on the semantic features of input x, calculate the router weights to obtain the exclusive knowledge and corresponding task models that are useful to the current x, and use the weights to represent the degree of usefulness; Calculate router weight:
[0015] Where R is a router module, and the input is the embedded value of x. The output is the matching score of T tasks, and the router weight is obtained after the score is softmaxed. This represents the last-level word embedding sequence from shared experts. ; The parameters of the router module are as follows: In this embodiment, the router module is a lightweight neural network composed of fully connected layers. It is its weight; 2.3) Combine the shared expert with the exclusive vector of the selected person model into a customized model for the current x;
[0016] 2.4) Substitute the customized model θ into the basic model f(x;θ) to calculate the inference result of x, and add it to the output set Y. .
[0017] Loop End: After traversing all inputs x, output the final Y, completing one inference cycle.
[0018] The overall steps are as follows: Pre-compute (executed only once): Computing Sharing Expert
[0019] Extract the exclusive knowledge vector for each task-specific weight: right
[0020] Reasoning (main loop): Initialize output
[0021] For input Each of them : Calculate router weight:
[0022] Merge into a single expert
[0023] Perform model inference to produce output: .
[0024] Experimental verification: Discriminative tasks: Using RoBERTa as the backbone network, evaluations were performed on eight discriminative tasks on the GLUE benchmark.
[0025] Generative tasks: Qwen-14B was used as the main model, with task-specific fine-tuning performed using the LoRA method, and the model was evaluated on four generative tasks.
[0026] The effectiveness of the proposed method was validated through extensive experiments on 12 datasets, covering discriminative and generative tasks, various model architectures, and in-domain and out-of-domain settings. Experimental results show that the proposed method consistently outperforms other merging methods on all datasets, and improves the standardized score by an average of 28.34%, even exceeding the upper limit of fine-tuning on generative tasks.
[0027] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
Claims
1. A model fusion method based on dynamically combined modular expert knowledge, characterized in that, Includes the following steps: 1) Modularization of knowledge; Divide knowledge into shared and exclusive parts, and separate out common knowledge that is useful for all tasks; 1.1) Computational shared experts use pre-merged models as natural placeholders to encapsulate common knowledge that is important for all tasks; The task arithmetic technique is used to extract shared experts from the initial task model to be merged. 1.2) After removing shared knowledge, retain the exclusive knowledge of each task model; Calculate the exclusive vector : ; in, Knowledge of task model t; After the calculation is completed, T exclusive vectors are obtained, where T is the total number of task models; 2) Based on the input, dynamically merge shared knowledge and task-specific knowledge to obtain the fused model.
2. The model fusion method based on dynamically combined modular expert knowledge according to claim 1, characterized in that, Step 2) is as follows: 2.1) Initialize the model output Y; 2.2) Based on the semantic features of input x, calculate the router weights to obtain the exclusive knowledge and corresponding task models that are useful to the current x, and use the weights to represent the degree of usefulness; Calculate router weight: Where R is a router module, and the input is the embedded value of x. The output is the matching score of T tasks, and the router weight is obtained after the score is softmaxed. This represents the last-level word embedding sequence from shared experts. ; The weights are the corresponding weights for the T task models; These are the parameters of the router module, which is a lightweight neural network composed of fully connected layers. It is its weight; 2.3) Combine the shared expert with the exclusive vector of the selected person model into a customized model for the current x; 2.4) Substitute the customized model θ into the basic model f(x;θ) to calculate the inference result of x, and add it to the output set Y. ; 2.5) After traversing all inputs x, output the final Y to complete one inference process.
3. The model fusion method based on dynamically combined modular expert knowledge according to claim 1, characterized in that, The exclusive vector is a compressed exclusive vector: the exclusive vector is compressed into a compressed exclusive vector for each task using singular value decomposition (SVD).
4. The model fusion method based on dynamically combined modular expert knowledge according to claim 1, characterized in that, The singular value decomposition process is as follows: Assumption Ranked-m decomposition, Set the compression level to r, and only retain the portions corresponding to the first r largest singular values to achieve a low-rank task space, resulting in... .
5. An electronic device, characterized in that, include: One or more processors; as well as Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 4.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 4.