Large model fine-tuning method based on hybrid experts of Multi-A and Multi-B experts

Abstract

This paper discloses a large-model fine-tuning method based on a hybrid of Multi-A and Multi-B experts. By embedding the LoRa network into a mixture of experts (MoE) architecture and leveraging Multi-A and Multi-B MoEs to capture the differences between different tasks, the fine-tuning efficiency and overall performance of large language models in multi-task scenarios are significantly improved. This method is applicable to various large language models, promoting their application in different fields and providing new ideas for further exploring parameter-efficient fine-tuning methods.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a large model fine-tuning method based on hybrid experts of Multi-A and Multi-B experts. Background Art

[0002] In recent years, the field of natural language processing has made significant progress. Open-source large-scale language models such as LLaMA, Qwen, and Yi have emerged and achieved moderate success in natural language processing tasks. However, the rapid growth in the complexity and parameter size of these models makes efficient training with limited computing resources a challenge. To address this issue, parameter-efficient training (PEFT) methods such as LoRA have been proposed. LoRA improves performance by using low-rank matrices, but suffers from stability issues when processing large, mixed datasets.

[0003] While fine-tuning a single model for a specific task is now possible, the need to fine-tune the LLM in a multi-task environment is becoming increasingly prominent. While it's intuitive to directly fine-tune the basic LLM using a high-rank LoRA adapter, practice has shown that this approach doesn't improve model performance in multi-task scenarios and may even lead to training instability. Another approach is to incorporate a mixture of experts (MoE) architecture into the LoRA adapter, such as the MoLoRA approach. While MoLoRA improves the performance of the basic LLM across multiple tasks to a certain extent, it doesn't fully account for the similarities and differences between tasks, resulting in less than ideal knowledge transfer.

[0004] Existing multi-task fine-tuning techniques, such as directly fusing datasets of multiple tasks for training or using the MoLoRA method, have the following shortcomings:

[0005] (1) Directly fusing datasets from multiple tasks for training may introduce conflicts and noise between tasks, leading to decreased model performance. The quality of datasets from different tasks is difficult to evaluate, and there may be conflicts in semantic space between them.

[0006] (2) Although MoLoRA introduces the MoE architecture into the LoRA-adapter, it does not consider the similarities and differences between tasks. The parameters of all experts are independent, making it difficult to achieve knowledge transfer and sharing between tasks.

[0007] (3) Existing methods lack an adaptive mechanism to dynamically select the most appropriate expert combination and cannot flexibly adjust the model architecture according to the characteristics of the task.

[0008] Developing an efficient multi-task learning and domain adaptation method is crucial for the application and development of large language models for the following reasons:

[0009] a) Improving model generalization capabilities: By training on multiple tasks and domains, the model can learn a wider range of knowledge and skills, improving its performance on unseen tasks.

[0010] b) Resource efficiency: Compared with training a separate model for each task or domain, multi-task learning can more efficiently utilize computing resources and storage space.

[0011] c) Quickly adapt to new domain tasks: A model with good multi-task learning capabilities can adapt to new domains or tasks more quickly, reducing the development time for new application scenarios.

[0012] d) Maintaining general capabilities: When performing domain adaptation, it is important to maintain the model’s general language understanding capabilities so that it can excel in a specific domain without losing the ability to handle general tasks.

[0013] e) Industrial application requirements: In practical applications, models are often required to handle multiple types of tasks simultaneously, such as question answering, dialogue, summarization, etc. This requires the model to have strong multi-task learning capabilities.

[0014] Therefore, developing a method that can effectively perform multi-task learning and domain adaptation while maintaining the general capabilities of the model has important theoretical significance and practical application value. Summary of the Invention

[0015] To overcome the shortcomings of the existing technology, this paper provides a large model fine-tuning method based on a hybrid of Multi-A and Multi-B experts. By embedding the LoRA network into a mixture of experts (MoE) architecture, and using Multi-A and Multi-B MoE to capture the differences between different tasks, the fine-tuning efficiency and overall performance of large language models in multi-task scenarios are significantly improved. This invention is applicable to various large language models, promotes their application in different fields, and provides new ideas for further exploring parameter-efficient fine-tuning methods.

[0016] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0017] Step 1: Build a HybridMoE model;

[0018] The HybridMoE model includes multiple Multi-A MoE and Multi-B MoE modules;

[0019] The Multi-A MoE module consists of N different experts A1 to A n The Multi-B MoE module consists of experts B1 to B n;The bottom layer LLM maintains a set of static weights W0, representing a fixed linear layer;

[0020] The architecture of the HybridMoE model includes the following components:

[0021] a) Base language model: A pre-trained large language model is used as the base model;

[0022] b) Multi-AMoE: used for multiple-choice format tasks to extract task-related features;

[0023] c) Multi-B MoE: generative tasks for generating extended narratives or solutions;

[0024] d) Gating network: used to control the flow of tensors to Multi-A MoE or Multi-B MoE;

[0025] Step 2: Multi-A MoE and Multi-B MoE;

[0026] In the Multi-A and Multi-B MoE modules, LoRA-A and LoRA-B are treated as different experts; the forward process includes utilizing the router and The output score of determines the routing distribution;

[0027] Step 2-1: Multi-AMoE;

[0028] For the Multi-A MoE module, the calculation process of route determination is expressed by formulas (1), (2) and (3);

[0029]

[0030] Among them, x m represents the hidden state vector of the mth token in the input sequence, Indicates that the mth token is assigned to expert A i The routing weight, Indicates a router The weight of B * Represents the LoRA-B layer shared among all LoRA-A experts; Represents the hidden state vector x of Multi-AMoE for the input m The output, E A (.) represents the hidden vector x of all experts in the Multi-AMoE module m The function transformation performed, the superscript A represents the multi-AMoE module;

[0031] Step 2-2: For Multi-B MoE modules, as shown in formulas (4), (5), and (6):

[0032]

[0033] in, Indicates that the mth token is assigned to expert B i The routing weight, Indicates a router The weight of Represents the hidden state vector x of the Multi-B MoE for the input m The output, E B (.) represents the hidden vector x of all experts in the Multi-B MoE module m The function transformation performed, the superscript A represents the multi-B MoE module, A * Represents the A matrix shared by all B experts; in LoRA, it contains two matrices A and B, and the input vector flows through A first and then through B;

[0034] Step 3: Gating the network;

[0035] Using the symbol θ G The gating network, whose weight matrix is ​​W G ;The forward process of HybridMoE is described by equations (7) and (8);

[0036] gate=argmax(θ G (x m ))=argmax(softmax(W G x m )) (7)

[0037]

[0038] During the fine-tuning phase, the gating network θ G Based on the input hidden vector x m The difference between the two modules is used to select the Multi-A MoE or Multi-B MoE module; if the output score of the Multi-B MoE module exceeds the predefined threshold P, the gate parameter gate is set to the binary value 1, indicating that it is activated; otherwise, if the output score does not reach the predefined threshold P, the gate parameter is set to 0, indicating that the Multi-A MoE module is in the activated state.

[0039] Preferably, the large language model is LLaMA-2, Qwen or Yi.

[0040] Preferably, the threshold value P=0.5.

[0041] A computer program enables a computer to execute the large model fine-tuning method.

[0042] An electronic device comprises: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the above-mentioned large model fine-tuning method.

[0043] A computer-readable storage medium stores a computer program, which implements the large model fine-tuning method when executed by a processor.

[0044] A chip includes: a processor for calling and running a computer program from a memory, so that a device equipped with the chip executes the above-mentioned large model fine-tuning method.

[0045] A computer program product includes a computer storage medium storing a computer program, wherein the computer program includes instructions executable by at least one processor, and when the instructions are executed by the at least one processor, the large model fine-tuning method described above is implemented.

[0046] The beneficial effects of the present invention are as follows:

[0047] (1) By introducing the Multi-A and Multi-B MoE architectures, this paper shares part of the low-rank matrix among experts, which can capture the similarity of tasks while retaining the uniqueness of tasks, achieving more effective knowledge transfer and generalization.

[0048] (2) The present invention adopts a gating network to realize the adaptive selection of Multi-A and Multi-B MoE modules, dynamically routing to the most appropriate module according to the task characteristics, and improving resource utilization efficiency and model performance.

[0049] (3) The present invention uses HybridMoE as a general multi-task PEFT method, which is applicable to various large language models, promotes the application of large language models in different fields, and provides new ideas for further exploring efficient parameter fine-tuning methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Schematic diagram for structural comparison of MoLoRA and the Multi-AMoE, Multi-B MoE and HybridMoE of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described below with reference to the accompanying drawings and examples.

[0052] In natural language processing, as large language models (LLMs) continue to expand in size and their application scenarios become increasingly diverse, achieving efficient training with limited computing resources to meet the demands of multi-task learning has become a major challenge. Traditional training methods often suffer from instability in multi-task learning and place high demands on training resources. To address this issue, parameter-efficient training (PEFT) methods such as LoRA have been proposed. LoRA improves performance by using low-rank matrices, but suffers from stability issues when processing large, mixed datasets. MoLoRA improves this issue by extending LoRA and integrating the Mixture-of-Expert (MoE) architecture, but still has some limitations. Existing LoRA MoE-based fine-tuning methods (such as MoLoRA) have improved model performance in multi-task scenarios to some extent, but fail to fully consider the similarities and differences between different tasks (e.g., multiple-choice questions and multi-step reasoning questions), resulting in suboptimal knowledge transfer.

[0053] This paper aims to solve this problem and proposes an innovative PEFT method HybridMoE. By embedding the LoRA network into a mixture of experts (MoE) architecture, Multi-A and Multi-B MoE are used to capture the differences between different tasks, thereby significantly improving the fine-tuning efficiency and overall performance of large language models in multi-task scenarios.

[0054] This paper proposes a hybrid hybrid expert model HybridMoE based on Multi-AMoE and Multi-B MoE, which mainly includes the following technical solutions:

[0055] 1. Model architecture;

[0056] The Hybrid MoE proposed in this paper is a system that integrates multiple Multi-A MoE and Multi-B MoE modules. The Multi-A MoE module consists of N different experts, A1 to An, while the Multi-B MoE module includes experts, B1 to Bn. The underlying LLM maintains a set of static weights, W0, representing the fixed linear layers in the architecture. The core of the Hybrid MoE is to cleverly integrate the expertise of the Multi-A MoE and Multi-B MoE modules, optimizing the system's performance across multiple tasks through multi-task fine-tuning.

[0057] The HybridMoE model architecture includes the following components:

[0058] a) Base language model: A pre-trained large language model (such as LLaMA-2, Qwen, Yi) is used as the base model.

[0059] b) Multi-A MoE: used for multiple-choice format tasks and is good at extracting task-related features.

[0060] c) Multi-B MoE: A generative task for generating extended narratives or solutions.

[0061] d) Gating network: used to control the flow of tensors to Multi-A MoE or Multi-B MoE.

[0062] 2. Multi-A MoE and Multi-B MoE;

[0063] In the Multi-A and Multi-B MoE modules, LoRA-A and LoRA-B are considered as different experts. To ensure the performance of Hybrid MoE, each expert needs to play a role in the forward propagation process of the system. Specifically, the forward process includes utilizing the router and The output score of is used to determine the routing distribution. For the Multi-A MoE module, the routing determination calculation process can be expressed by formulas (1), (2) and (3). m represents the hidden vector of the mth token in the input sequence, Indicates that the mth token is assigned to expert A i The routing weight, represents the weight of the router, B * represents the LoRA-B layer shared among all LoRA-A experts. For the Multi-B MoE module, the situation is similar, as shown in equations (4), (5) and (6).

[0064]

[0065] 3. HybridMoE;

[0066] The HybridMoE architecture seamlessly integrates Multi-A and Multi-B MoE modules, giving the system the ability to handle various downstream tasks. In order to autonomously select between Multi-A and Multi-B MoE modules, the present invention adopts a G The gating network, whose weight matrix is ​​W G The forward process of HybridMoE can be described by equations (7) and (8).

[0067] During the fine-tuning phase, the gating network θ G Will hide the vector x according to the input mThe algorithm accurately selects the Multi-A MoE or Multi-B MoE module based on subtle differences in performance. If the output score of the Multi-B MoE module exceeds a predefined threshold of 0.5, the gate parameter is set to a binary value of 1, indicating its activation. Conversely, if the output score does not reach the required level, the gate parameter is set to 0, indicating that the Multi-A MoE module is active. This dynamic selection process is key to HybridMoE's adaptability to different task requirements.

[0068] gate=argmax(θ G (x m ))=argmax(softmax(W G x m )) (7)

[0069]

Claims

1. A large model fine-tuning method based on hybrid experts of Multi-A and Multi-B experts, characterized by: The steps include: Step 1: Build a HybridMoE model; The HybridMoE model includes multiple Multi-A MoE and Multi-B MoE modules; The Multi-A MoE module consists of N different experts A1 to A n The Multi-B MoE module consists of experts B1 to B n ;The bottom layer LLM maintains a set of static weights W0, representing a fixed linear layer; The architecture of the HybridMoE model includes the following components: a) Base language model: A pre-trained large language model is used as the base model; b) Multi-A MoE: used for multiple-choice format tasks, extracting task-related features; the selection format tasks include multiple-choice questions and multi-step reasoning questions; c) Multi-B MoE: Generative tasks for generating extended narratives or solutions; the generative tasks include question answering, dialogue, and summarization; d) Gating network: used to control the flow of tensors to Multi-A MoE or Multi-B MoE; Step 2: Multi-A MoE and Multi-B MoE; In the Multi-A and Multi-B MoE modules, LoRA-A and LoRA-B are treated as different experts; the forward process includes utilizing the router and The output score of determines the routing distribution; Step 2-1: Multi-AMoE; For the Multi-A MoE module, the calculation process of route determination is expressed by formulas (1), (2) and (3); Among them, x m represents the hidden state vector of the mth token in the input sequence, Indicates that the mth token is assigned to expert A i The routing weight, Indicates a router The weight of B * Represents the LoRA-B layer shared among all LoRA-A experts; Represents the hidden state vector x of the Multi-A MoE for the input m The output, E A (.) represents the hidden vector x of all experts in the Multi-AMoE module m The function transformation performed, the superscript A represents the multi-A MoE module; Step 2-2: For Multi-B MoE modules, as shown in formulas (4), (5), and (6): in, Indicates that the mth token is assigned to expert B i The routing weight, Indicates a router The weight of Represents the hidden state vector x of the Multi-B MoE for the input m The output, E B (.) represents the hidden vector x of all experts in the Multi-B MoE module m The function transformation performed, the superscript A represents the multi-B MoE module, A * Represents the A matrix shared by all B experts; in LoRA, it contains two matrices A and B, and the input vector flows through A first and then through B; Step 3: Gating the network; Using the symbol θ G The gating network, whose weight matrix is ​​W G ;The forward process of HybridMoE is described by equations (7) and (8); gate=argmax(θ G (x m ))=argmax(softmax(W G x m )) (7) During the fine-tuning phase, the gating network θ G Based on the input hidden vector x m The difference between the two modules is used to select the Multi-A MoE or Multi-B MoE module; if the output score of the Multi-B MoE module exceeds the predefined threshold P, the gate parameter gate is set to the binary value 1, indicating that it is activated; otherwise, if the output score does not reach the predefined threshold P, the gate parameter is set to 0, indicating that the Multi-A MoE module is in the activated state.

2. The large model fine-tuning method based on hybrid experts of Multi-A and Multi-B experts according to claim 1, characterized in that: The large language model is LLaMA-2, Qwen or Yi.

3. The large model fine-tuning method based on hybrid experts of Multi-A and Multi-B experts according to claim 1, characterized in that: The threshold value P=0.

5.

4. An electronic device, characterized in that: include: processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 3.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

6. A chip, characterized in that: include: A processor, configured to call and run a computer program from a memory, so that a device equipped with the chip executes the method according to any one of claims 1 to 3.

7. A computer program product, characterized in that The computer program product comprises a computer storage medium storing a computer program, wherein the computer program comprises instructions executable by at least one processor, and when the instructions are executed by the at least one processor, the method according to any one of claims 1 to 3 is implemented.

Images