Multi-task hybrid expert model fine tuning method based on general LoRA and domain-specific LoRA

Abstract

The invention discloses a multi-task hybrid expert model fine tuning method based on general LoRA and domain-specific LoRA, which is characterized in that general experts and domain-specific experts are integrated, and residual connection is introduced, so that the processing capability of the model on general tasks and domain-specific tasks is balanced, and the performance and the stability are improved. According to the method, efficient multi-task learning and new field adaptation of parameters are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a fine-tuning method for a multi-task hybrid expert model based on general LoRA and domain-specific LoRA. Background Art

[0002] In recent years, the field of natural language processing has made significant progress. Open source large language models such as LLaMA, Qwen, and Yi have emerged and achieved certain success in natural language processing tasks. However, the complexity of these models and the rapid growth of parameter size make efficient training under limited computing resources a challenge. To solve this problem, parameter efficient training (PEFT) methods such as LoRA (Low-Rank Adaptation) have been proposed. LoRA improves performance by using low-rank matrices, but has stability issues when processing large, mixed data sets. MoLoRA improves this problem by extending LoRA and integrating the Mixture-of-Expert (MoE) architecture, but there are still some limitations.

[0003] For example: One limitation of MoLoRA is the "lack of LoRA experts in specific fields". In the MoLoRA architecture, all experts perform parameter updates at the same time during training. This consistency may limit its performance ceiling, especially for tasks that differ significantly, such as mathematics and code. On the contrary, adding LoRA experts in specific fields may potentially improve performance. In addition, another problem with this architecture is "limited pluggability": when inserting new domain experts into MoLoRA, all expert parameters need to be retrained with a dataset containing new domain data, which may result in low efficiency and time-consuming.

[0004] However, MoLoRA has the following disadvantages:

[0005] (1) The lack of domain-specific adapters means that the same experts are used for all tasks, which may limit the performance ceiling. Especially for tasks with large differences, such as mathematics and coding tasks, the lack of domain-specific experts may result in performance not being fully utilized.

[0006] (2) Limited scalability: Adding new task capabilities requires retraining all parameters, which is not only inefficient and time-consuming, but also increases training costs, limiting the flexibility and scalability of the model in practical applications.

[0007] The main disadvantages of existing technologies include:

[0008] 1. Dataset fusion problem: It is difficult to fuse multiple datasets to construct a multifunctional dataset. There may be contradictions between different datasets, making it difficult to evaluate data quality.

[0009] 2. Performance degradation: When trained on fused datasets, the performance of large language models may degrade, especially when dealing with domain-specific tasks.

[0010] 3. High consumption of computing resources: Fully training a large language model requires a lot of computing resources and time, which limits the rapid adaptation and deployment of the model.

[0011] 4. Loss of general ability: When performing domain adaptation, the model may sacrifice its general language understanding ability, resulting in decreased performance on tasks in non-target domains.

[0012] 5. Lack of scalability: Existing methods are difficult to flexibly add new tasks or domains, and usually require retraining the entire model.

[0013] 6. Lack of effective domain experts: In the MoLoRA architecture, all experts update parameters simultaneously during training. This consistency may limit its performance ceiling.

[0014] Developing an efficient multi-task learning and domain adaptation method is crucial for the application and development of large language models for the following reasons:

[0015] a) Improve model generalization ability: By training on multiple tasks and domains, the model can learn a wider range of knowledge and skills and improve its performance on unseen tasks.

[0016] b) Resource efficiency: Compared with training a separate model for each task or domain, multi-task learning can utilize computing resources and storage space more efficiently.

[0017] c) Quickly adapt to new domain tasks: A model with good multi-task learning capabilities can adapt to new domains or tasks more quickly, reducing the development time for new application scenarios.

[0018] d) Maintaining general capabilities: When performing domain adaptation, it is important to maintain the general language understanding capabilities of the model so that it can excel in a specific domain without losing the ability to handle general tasks.

[0019] e) Industrial application requirements: In practical applications, the model is often required to be able to handle multiple types of tasks at the same time, such as question and answer, dialogue, summary, etc., which requires the model to have strong multi-task learning capabilities.

[0020] Therefore, developing a method that can effectively perform multi-task learning and domain adaptation while maintaining the general capabilities of the model has important theoretical significance and practical application value. Summary of the invention

[0021] In order to overcome the shortcomings of the prior art, the present invention provides a fine-tuning method based on a general LoRA and domain-specific LoRA multi-task hybrid expert model, which integrates general experts and domain-specific experts and introduces residual connections to balance the model's processing capabilities for general tasks and domain-specific tasks, thereby improving performance and stability. The present invention achieves parameter-efficient multi-task learning and new domain adaptation.

[0022] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0023] Step 1: Build a multi-task hybrid expert model BENLoRA (Blended Enhanced Network with LoRA for Multiple Domain Robust Adaptation) based on general LoRA and domain-specific LoRA. The model architecture includes the following components:

[0024] a) Base language model: a pre-trained large language model is used as the base model;

[0025] b) General LoRA Expert: used to capture common knowledge and capabilities across domains;

[0026] c) Domain-specific LoRA Experts: Specialized adapters for specific domains or tasks;

[0027] d) Residual connection: connects general LoRA experts and domain-specific LoRA experts to maintain the general capabilities of the model;

[0028] e) Routing: used to dynamically assign weights to dedicated LoRA experts in different fields;

[0029] Step 2: Use a three-stage training process;

[0030] a) Phase 1: Only general LoRA experts are trained, while domain-specific LoRA experts and routers are deactivated;

[0031] b) Phase 2: Train each domain-specific LoRA expert separately, and keep the general expert parameters frozen;

[0032] c) Phase 3: The parameters of all LoRA experts are frozen, and only the routing is trained to learn the best combination strategy for different tasks; by training the routing, the large model can dynamically assign weights to different experts according to the needs of different tasks;

[0033] Step 3: Introduce residual connection;

[0034] The input of the domain-specific LoRA expert includes not only the original input but also the output of the general LoRA expert, expressed as:

[0035]

[0036] where h m is the output of the general LoRA expert; the letter r in the upper right corner represents BENLoRA-Res; W0 represents the frozen weight of the base large language model, which is a matrix; x m Represents the hidden state vector of the input BENLoRA-Res module; Represents the weight assigned to each expert by the routing module in the BENLoRA-Res module, B represents the B matrix in each LoRA expert in BENLoRA-Res; A represents the A matrix in each LoRA expert in R in BENLoRA-Res; a LoRA expert contains two matrices A and B, and the vector passes through A first and then B during operation; BENLoRA contains multiple experts, i is the subscript of any expert, and is universal; the summation symbol indicates that the operation results of these experts are uniformly summed without distinction;

[0037] Step 4: Dynamic routing;

[0038] Use a trainable router to dynamically assign weights to different LoRAs. The routing weight is calculated as follows:

[0039]

[0040] in is the weight assigned to the i-th expert by the routing module, Represents the weight matrix of the routing module in the BENLoRA-Res module. The capital R stands for Router. It represents the process of the routing module obtaining the weight of each LoRA expert according to the input latent vector;

[0041] The routing module contains the weight matrix And the softmax function, the hidden vector x m First pass After that, the output of the softmax function is a vector. The i in the lower right corner of the bracket represents the i-th element in the vector, which is the weight of each LoRA expert.

[0042] Preferably, the large language model is LLaMA-2.

[0043] A computer program enables a computer to execute the above-mentioned multi-task hybrid expert model fine-tuning method.

[0044] An electronic device comprises: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the above-mentioned multi-task hybrid expert model fine-tuning method.

[0045] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the multi-task hybrid expert model fine-tuning method.

[0046] A chip includes: a processor, which is used to call and run a computer program from a memory, so that a device equipped with the chip executes the above-mentioned multi-task hybrid expert model fine-tuning method.

[0047] A computer program product, comprising a computer storage medium storing a computer program, wherein the computer program comprises instructions executable by at least one processor, and when the instructions are executed by the at least one processor, the multi-task hybrid expert model fine-tuning method is implemented.

[0048] The beneficial effects of the present invention are as follows:

[0049] 1. The present invention provides a multi-task hybrid expert model based on general LoRA and domain-specific LoRA, which realizes parameter-efficient multi-task learning and new domain adaptation.

[0050] 2. The present invention avoids the problem of data set fusion by separately training general LoRA and domain-specific LoRA, thereby improving the performance of the model in various fields.

[0051] 3. The present invention utilizes LoRA technology to reduce computing resource consumption and achieve efficient model training.

[0052] 4. The present invention designs a residual connection structure to maintain the original general capabilities of the base model while performing domain adaptation.

[0053] 5. The present invention provides a flexible training paradigm that can easily add new tasks or domain experts without retraining the parameters of all experts. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 The structure comparison between MoLoRA and BENLoRA-Flan and BENLoRA-Res of the present invention;

[0055] Figure 2 Detailed schematic diagram of the three-stage fine-tuning paradigm of BENLoRA of the present invention. DETAILED DESCRIPTION

[0056] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0057] In natural language processing, as the scale of large language models (LLMs) continues to expand and the application scenarios become increasingly diverse, how to achieve efficient training with limited computing resources to meet the needs of multi-task learning has become an important challenge currently faced. Traditional training methods often show insufficient stability in multi-task learning and have a large demand for training resources. In addition, some existing methods such as MoLoRA and SiRA lack domain-specific adapters, that is, when dealing with different tasks, especially those with large differences (such as mathematical and code tasks), performance may be limited. At the same time, MoLoRA requires retraining all parameters when adding new task capabilities, which leads to problems of inefficiency and limited scalability.

[0058] The purpose of the present invention is to provide a multi-task hybrid expert model based on general LoRA and domain-specific LoRA to improve the training efficiency and parameter efficiency of large language models in multi-task learning, enhance the versatility and adaptability of the base model, and enable it to better cope with various complex natural language processing tasks.

[0059] The present invention proposes a multi-task hybrid expert model BENLoRA based on general LoRA and domain-specific LoRA, which mainly includes the following steps:

[0060] 1. Model architecture;

[0061] The model architecture proposed in this invention includes the following components:

[0062] a) Base language model: A pre-trained large language model (such as LLaMA-2) is used as the base model.

[0063] b) General LoRA Expert: used to capture common knowledge and capabilities across domains.

[0064] c) Domain-specific LoRA Experts: Specialized adapters for specific domains or tasks.

[0065] d) Residual connection: connects general LoRA and domain-specific LoRA to maintain the general capabilities of the model.

[0066] e) Routing: It is used to dynamically assign weights to dedicated LoRA experts in different fields.

[0067] 2. Training process;

[0068] The present invention adopts a three-stage training process:

[0069] a) Phase 1: Only general experts are trained to quickly adapt to general tasks, while domain-specific experts and routing are disabled. This allows the general expert to learn task-independent representations, providing basic understanding capabilities for subsequent tasks.

[0070] b) Second stage: Each domain-specific expert is trained separately to focus on its corresponding task, and the general expert parameters remain frozen. At this point, the domain-specific experts can focus on learning specific knowledge in their respective fields and improve the performance of the model in specific fields.

[0071] c) The third stage: The parameters of all experts are frozen, and only the routing is trained to learn the best combination strategy for different tasks. By training the routing, the model can dynamically assign the weights of different experts according to the needs of different tasks, thereby achieving the best performance.

[0072] BENLoRA-Flan is based on the BENLoRA paradigm and separates the experts in MoLoRA into general experts and domain-specific experts. In the training process, the general experts are trained on the general dataset in the first stage, the domain-specific experts are trained on their respective domain-specific datasets in the second stage, and all expert parameters are frozen in the third stage, and only routing is trained.

[0073] BENLoRA-Res is a more advanced method that integrates general experts and domain-specific experts and introduces residual connections to balance the model's processing capabilities for general tasks and domain-specific tasks, thereby improving performance and stability. In BENLoRA-Res, the hidden vector is first calculated using a general expert, then refined by a domain-specific expert, and the output of the general expert is directly integrated into the final result through a residual connection, ensuring that key information is retained and enhancing the robustness of the model.

[0074] 3. Residual connection;

[0075] In order to maintain the general capabilities of the model, the present invention introduces residual connections. Specifically, the input of the domain-specific LoRA includes not only the original input, but also the output of the general LoRA, expressed as:

[0076]

[0077] where h m It is the output of universal LoRA, E r (·) indicates the operation of domain-specific LoRA experts.

[0078] 4. Dynamic routing;

[0079] The present invention uses a trainable router to dynamically assign weights to different LoRAs. The calculation of the routing weight is as follows:

[0080]

[0081] where is the weight of the i-th expert.

Claims

1. A fine-tuning method based on general LoRA and domain-specific LoRA multi-task hybrid expert model, characterized in that: The steps include: Step 1: Build a multi-task hybrid expert model BENLoRA based on general LoRA and domain-specific LoRA. The model architecture includes the following components: a) Base language model: a pre-trained large language model is used as the base model; b) General LoRA Expert: used to capture common knowledge and capabilities across domains; c) Domain-specific LoRA Experts: Specialized adapters for specific domains or tasks; d) Residual connection: connects general LoRA experts and domain-specific LoRA experts to maintain the general capabilities of the model; e) Routing: used to dynamically assign weights to dedicated LoRA experts in different fields; Step 2: Use a three-stage training process; a) Phase 1: Only general LoRA experts are trained, while domain-specific LoRA experts and routers are deactivated; b) Phase 2: Train each domain-specific LoRA expert separately, and keep the general expert parameters frozen; c) Phase 3: The parameters of all LoRA experts are frozen, and only the routing is trained to learn the best combination strategy for different tasks; by training the routing, the large model can dynamically assign weights to different experts according to the needs of different tasks; Step 3: Introduce residual connection; The input of the domain-specific LoRA expert includes not only the original input but also the output of the general LoRA expert, expressed as: where h m is the output of the general LoRA expert; the letter r in the upper right corner represents BENLoRA-Res; W0 represents the frozen weight of the base large language model, which is a matrix; x m Represents the hidden state vector of the input BENLoRA-Res module; Represents the weight assigned to each expert by the routing module in the BENLoRA-Res module, B represents the B matrix in each LoRA expert in BENLoRA-Res; A represents the A matrix in each LoRA expert in R in BENLoRA-Res; a LoRA expert contains two matrices A and B, and the vector passes through A first and then B during operation; BENLoRA contains multiple experts, i is the subscript of any expert, and is universal; the summation symbol indicates that the operation results of these experts are uniformly summed without distinction; Step 4: Dynamic routing; Use a trainable router to dynamically assign weights to different LoRAs. The routing weight is calculated as follows: in is the weight assigned to the i-th expert by the routing module, Represents the weight matrix of the routing module in the BENLoRA-Res module. The capital R stands for Router. It represents the process of the routing module obtaining the weight of each LoRA expert according to the input latent vector; The routing module contains the weight matrix And the softmax function, the hidden vector x m First pass After that, the output of the softmax function is a vector. The i in the lower right corner of the bracket represents the i-th element in the vector, which is the weight of each LoRA expert.

2. According to claim 1, a method for fine-tuning a multi-task hybrid expert model based on general LoRA and domain-specific LoRA is characterized in that: The large language model is LLaMA-2.

3. A computer program, characterized in that The computer program enables a computer to execute the method according to any one of claims 1 to 2.

4. An electronic device, characterized in that: include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the method as claimed in any one of claims 1 to 2.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

6. A chip, characterized in that: include: A processor, configured to call and run a computer program from a memory, so that a device equipped with the chip executes a method as claimed in any one of claims 1 to 2.

7. A computer program product, characterized in that The computer program product comprises a computer storage medium storing a computer program, wherein the computer program comprises instructions executable by at least one processor, and when the instructions are executed by the at least one processor, the method according to any one of claims 1 to 2 is implemented.

Images