Large model fine tuning method based on Multi-A and Multi-B expert hybrid hybrid experts

Abstract

The invention discloses a large model fine tuning method based on Multi-A and Multi-B expert hybrid hybrid experts, which is characterized in that a LoRA network is embedded into a hybrid expert (MoE) architecture, and the Multi-A and Multi-B MoE are utilized to capture differences among different tasks, so that the fine tuning efficiency and comprehensive performance of a large language model in a multi-task scene are remarkably improved. The method is suitable for various large language models, the application of the large language models in different fields is promoted, and a new thought is provided for further exploring an efficient parameter fine tuning method.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a large model fine-tuning method based on hybrid experts of Multi-A and Multi-B experts. Background Art

[0002] In recent years, the field of natural language processing has made significant progress. Open source large language models such as LLaMA, Qwen, and Yi have emerged and achieved certain success in natural language processing tasks. However, the complexity of these models and the rapid growth of parameter size make efficient training under limited computing resources a challenge. To solve this problem, parameter efficient training (PEFT) methods such as LoRA have been proposed. LoRA improves performance by using low-rank matrices, but has stability issues when processing large, mixed datasets.

[0003] At present, it is possible to fine-tune a single model for a specific task, but the need to fine-tune the LLM in a multi-task environment is becoming increasingly prominent. Although it is intuitive to directly use a high-rank LoRA-adapter to fine-tune the basic LLM, practice has shown that this method does not improve model performance in multi-task scenarios, but may lead to unstable training. Another idea is to introduce a mixture of experts (MoE) architecture into the LoRA-adapter, such as the MoLoRA method. Although MoLoRA improves the effect of the basic LLM on multiple tasks to a certain extent, it does not fully consider the similarities and differences between tasks, resulting in less than ideal knowledge transfer.

[0004] Existing multi-task fine-tuning techniques, such as directly fusing the datasets of multiple tasks for training or using the MoLoRA method, have the following shortcomings:

[0005] (1) Directly fusing datasets of multiple tasks for training may introduce conflicts and noise between tasks, resulting in decreased model performance. The quality of datasets of different tasks is difficult to evaluate, and there may be conflicts in semantic space between them.

[0006] (2) Although MoLoRA introduces the MoE architecture into the LoRA-adapter, it does not consider the similarities and differences between tasks. The parameters of all experts are independent, making it difficult to achieve knowledge transfer and sharing between tasks.

[0007] (3) Existing methods lack an adaptive mechanism to dynamically select the most appropriate expert combination and cannot flexibly adjust the model architecture according to the characteristics of the task.

[0008] Developing an efficient multi-task learning and domain adaptation method is crucial for the application and development of large language models for the following reasons:

[0009] a) Improve model generalization ability: By training on multiple tasks and domains, the model can learn a wider range of knowledge and skills and improve its performance on unseen tasks.

[0010] b) Resource efficiency: Compared with training a separate model for each task or domain, multi-task learning can utilize computing resources and storage space more efficiently.

[0011] c) Quickly adapt to new domain tasks: A model with good multi-task learning capabilities can adapt to new domains or tasks more quickly, reducing the development time for new application scenarios.

[0012] d) Maintaining general capabilities: When performing domain adaptation, it is important to maintain the general language understanding capabilities of the model so that it can excel in a specific domain without losing the ability to handle general tasks.

[0013] e) Industrial application requirements: In practical applications, the model is often required to be able to handle multiple types of tasks at the same time, such as question and answer, dialogue, summary, etc., which requires the model to have strong multi-task learning capabilities.

[0014] Therefore, developing a method that can effectively perform multi-task learning and domain adaptation while maintaining the general capabilities of the model has important theoretical significance and practical application value. Summary of the invention

[0015] In order to overcome the shortcomings of the prior art, the present invention provides a large model fine-tuning method based on Multi-A and Multi-B expert hybrid experts, by embedding the LoRA network into the mixture of experts (MoE) architecture, using Multi-A and Multi-B MoE to capture the differences between different tasks, thereby significantly improving the fine-tuning efficiency and comprehensive performance of the large language model in multi-task scenarios. The present invention is applicable to various large language models, promotes the application of large language models in different fields, and provides new ideas for further exploring parameter-efficient fine-tuning methods.

[0016] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0017] Step 1: Build the HybridMoE model;

[0018] The HybridMoE model includes a plurality of Multi-A MoE and Multi-B MoE modules;

[0019] The Multi-A MoE module consists of N different experts A1 to A n The Multi-B MoE module consists of experts B1 to B n;The bottom layer LLM maintains a set of static weights W0, representing a fixed linear layer;

[0020] The architecture of the HybridMoE model includes the following components:

[0021] a) Base language model: a pre-trained large language model is used as the base model;

[0022] b) Multi-AMoE: used for multiple-choice format tasks to extract task-related features;

[0023] c)Multi-B MoE: Generative tasks for generating extended narratives or solutions;

[0024] d) Gating network: used to control the flow of tensors to Multi-A MoE or Multi-B MoE;

[0025] Step 2: Multi-A MoE and Multi-B MoE;

[0026] In the Multi-A and Multi-B MoE modules, LoRA-A and LoRA-B are treated as different experts; the forward process includes utilizing the router and The output score of determines the routing distribution;

[0027] Step 2-1: Multi-AMoE;

[0028] For the Multi-A MoE module, the calculation process of route determination is expressed by formulas (1), (2) and (3);

[0029]

[0030] Among them, x m represents the hidden state vector of the mth token in the input sequence, Indicates that the mth token is assigned to expert A i The routing weight, Indicates the router The weight of B * Represents the LoRA-B layer shared among all LoRA-A experts; Represents the hidden state vector x of Multi-AMoE for the input m The output, E A (.) represents all the experts in the Multi-AMoE module for the hidden vector x m The function transformation performed, the superscript A represents the multi-AMoE module;

[0031] Step 2-2: For Multi-B MoE modules, as shown in formulas (4), (5) and (6):

[0032]

[0033] in, Indicates that the mth token is assigned to expert B i The routing weight, Indicates the router The weight of Represents the hidden state vector x of the Multi-B MoE for the input m The output, E B (.) represents all the expert pairs in the Multi-B MoE module. m The function transformation performed, the superscript A represents the multi-B MoE module, A * Represents the A matrix shared by all B experts; in LoRA, it contains two matrices A and B, and the input vector flows through A first and then through B;

[0034] Step 3: Gating the network;

[0035] Using the label θ G The gating network with a weight matrix W G ; The forward process of HybridMoE is described by equations (7) and (8);

[0036] gate=argmax(θ G (x m ))=argmax(softmax(W G x m )) (7)

[0037]

[0038] In the fine-tuning stage, the gating network θ G According to the input hidden vector x m The difference between the two selects the Multi-A MoE or Multi-BMoE module; if the output score of the Multi-B MoE module exceeds the predefined threshold P, the gate parameter gate is set to the binary value 1, indicating that it is activated; conversely, if the output score does not reach the predefined threshold P, the gate parameter is set to 0, indicating that the Multi-A MoE module is in an activated state.

[0039] Preferably, the large language model is LLaMA-2, Qwen or Yi.

[0040] Preferably, the threshold value P=0.5.

[0041] A computer program, wherein the computer program enables a computer to execute the large model fine-tuning method.

[0042] An electronic device comprises: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the above-mentioned large model fine-tuning method.

[0043] A computer-readable storage medium stores a computer program, which implements the large model fine-tuning method when executed by a processor.

[0044] A chip includes: a processor, which is used to call and run a computer program from a memory, so that a device equipped with the chip executes the above-mentioned large model fine-tuning method.

[0045] A computer program product, comprising a computer storage medium storing a computer program, wherein the computer program comprises instructions executable by at least one processor, and when the instructions are executed by the at least one processor, the large model fine-tuning method is implemented.

[0046] The beneficial effects of the present invention are as follows:

[0047] (1) By introducing the Multi-A and Multi-B MoE architectures, this paper shares part of the low-rank matrix among experts, which can capture the similarity of tasks while retaining the uniqueness of tasks, thereby achieving more effective knowledge transfer and generalization.

[0048] (2) The present invention adopts a gating network to realize the adaptive selection of Multi-A and Multi-B MoE modules, dynamically routing to the most suitable module according to the task characteristics, and improving resource utilization efficiency and model performance.

[0049] (3) The present invention uses HybridMoE as a general multi-task PEFT method, which is applicable to various large language models, promotes the application of large language models in different fields, and provides new ideas for further exploring efficient parameter fine-tuning methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Schematic diagram of the structural comparison of MoLoRA and the Multi-AMoE, Multi-B MoE and HybridMoE of the present invention. DETAILED DESCRIPTION

[0051] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0052] In natural language processing, as the scale of large language models (LLMs) continues to expand and the application scenarios become increasingly diverse, how to achieve efficient training with limited computing resources to meet the needs of multi-task learning has become an important challenge currently faced. Traditional training methods often show insufficient stability in multi-task learning and have a large demand for training resources. To solve this problem, parameter efficient training (PEFT) methods such as LoRA have been proposed. LoRA improves performance by using low-rank matrices, but has stability issues when dealing with large, mixed data sets. MoLoRA improves this problem by extending LoRA and integrating the Mixture-of-Expert (MoE) architecture, but there are still some limitations. Although existing fine-tuning methods based on LoRA MoE (such as MoLoRA) have improved the performance of the model in multi-task scenarios to a certain extent, they have failed to fully consider the similarities and differences between different tasks (for example: multiple-choice questions and multi-step reasoning questions), resulting in less than ideal knowledge transfer.

[0053] The present invention aims to solve this problem and proposes an innovative PEFT method, HybridMoE, which embeds the LoRA network into a hybrid of experts (MoE) architecture and uses Multi-A and Multi-B MoE to capture the differences between different tasks, thereby significantly improving the fine-tuning efficiency and overall performance of the large language model in multi-task scenarios.

[0054] The present invention proposes a hybrid hybrid expert model HybridMoE based on Multi-AMoE and Multi-B MoE, which mainly includes the following technical solutions:

[0055] 1. Model architecture;

[0056] The HybridMoE proposed in this invention is a system that integrates multiple Multi-AMoE and Multi-B MoE modules. The Multi-AMoE module consists of N different experts A1 to An, while the Multi-B MoE module contains experts B1 to Bn. The underlying LLM maintains a set of static weights W0, representing the fixed linear layers in the architecture. The core of HybridMoE is to cleverly integrate the expertise of Multi-A MoE and Multi-B MoE modules, and optimize the system's performance on multiple tasks through multi-task fine-tuning.

[0057] The model architecture of HybridMoE includes the following components:

[0058] a) Base language model: A pre-trained large language model (such as LLaMA-2, Qwen, Yi) is used as the base model.

[0059] b) Multi-A MoE: It is used for multiple-choice format tasks and is good at extracting task-related features.

[0060] c)Multi-B MoE: Generative tasks for generating extended narratives or solutions.

[0061] d) Gating network: used to control the flow of tensors to Multi-A MoE or Multi-B MoE.

[0062] 2. Multi-A MoE and Multi-B MoE;

[0063] In the Multi-A and Multi-B MoE modules, LoRA-A and LoRA-B are considered as different experts. To ensure the performance of HybridMoE, each expert needs to play a role in the forward propagation process of the system. Specifically, the forward process includes utilizing the router and The output score of x is used to determine the routing distribution. For the Multi-A MoE module, the calculation process of routing determination can be expressed by formulas (1), (2) and (3). m represents the hidden vector of the mth token in the input sequence, Indicates that the mth token is assigned to expert A i The routing weight, represents the weight of the router, B * represents the LoRA-B layer shared among all LoRA-A experts. For the Multi-B MoE module, the situation is similar, as shown in equations (4), (5) and (6).

[0064]

[0065] 3. HybridMoE;

[0066] The HybridMoE architecture seamlessly integrates Multi-A and Multi-B MoE modules, giving the system the ability to handle a variety of downstream tasks. In order to autonomously select between Multi-A and Multi-B MoE modules, the present invention adopts a G The gating network with a weight matrix W G The forward process of HybridMoE can be described by equations (7) and (8).

[0067] In the fine-tuning stage, the gating network θ G Will hide the vector x according to the input mThe subtle differences in the performance of the proposed method are used to accurately select the Multi-A MoE or Multi-B MoE module. If the output score of the Multi-B MoE module exceeds the predefined threshold of 0.5, the gate parameter gate is set to the binary value 1, indicating its activation. Conversely, if the output score does not reach the required size, the gate parameter is set to 0, indicating that the Multi-A MoE module is activated. This dynamic selection process is the key to HybridMoE's adaptability to different task requirements.

[0068] gate=argmax(θ G (x m ))=argmax(softmax(W G x m )) (7)

[0069]

Claims

1. A large model fine-tuning method based on hybrid experts of Multi-A and Multi-B experts, characterized in that: The steps include: Step 1: Build the HybridMoE model; The HybridMoE model includes a plurality of Multi-A MoE and Multi-B MoE modules; The Multi-A MoE module consists of N different experts A1 to A n The Multi-B MoE module consists of experts B1 to B n ;The bottom layer LLM maintains a set of static weights W0, representing a fixed linear layer; The architecture of the HybridMoE model includes the following components: a) Base language model: a pre-trained large language model is used as the base model; b) Multi-A MoE: used for multiple choice format tasks to extract task-related features; c)Multi-B MoE: Generative tasks for generating extended narratives or solutions; d) Gating network: used to control the flow of tensors to Multi-A MoE or Multi-B MoE; Step 2: Multi-A MoE and Multi-B MoE; In the Multi-A and Multi-B MoE modules, LoRA-A and LoRA-B are treated as different experts; the forward process includes utilizing the router and The output score of determines the routing distribution; Step 2-1: Multi-A MoE; For the Multi-A MoE module, the calculation process of route determination is expressed by formulas (1), (2) and (3); Among them, x m represents the hidden state vector of the mth token in the input sequence, Indicates that the mth token is assigned to expert A i The routing weight, Indicates the router The weight of B * Represents the LoRA-B layer shared among all LoRA-A experts; Represents the hidden state vector x of the Multi-A MoE for the input m The output, E A (.) represents all the experts in the Multi-A MoE module for the hidden vector x m The function transformation performed, the superscript A represents the multi-A MoE module; Step 2-2: For Multi-B MoE modules, as shown in formulas (4), (5) and (6): in, Indicates that the mth token is assigned to expert B i The routing weight, Indicates the router The weight of Represents the hidden state vector x of the Multi-B MoE for the input m The output, E B (.) represents all the experts in the Multi-B MoE module’s hidden vector x m The function transformation performed, the superscript A represents the multi-B MoE module, A * Represents the A matrix shared by all B experts; in LoRA, it contains two matrices A and B, and the input vector flows through A first and then through B; Step 3: Gating the network; Using the label θ G The gating network with a weight matrix W G ; The forward process of HybridMoE is described by equations (7) and (8); gate=argmaxθ G (x m ))=argmax(softmax(W G x m )) (7) In the fine-tuning stage, the gating network θ g According to the input hidden vector x m The difference between the two selects the Multi-AMoE or Multi-B MoE module; if the output score of the Multi-B MoE module exceeds the predefined threshold P, the gate parameter gate is set to the binary value 1, indicating that it is activated; conversely, if the output score does not reach the predefined threshold P, the gate parameter is set to 0, indicating that the Multi-AMoE module is in an activated state.

2. A large model fine-tuning method based on hybrid experts of Multi-A and Multi-B experts according to claim 1, characterized in that: The large language model is LLaMA-2, Qwen or Yi.

3. The large model fine-tuning method based on Multi-A and Multi-B expert hybrid experts according to claim 1, characterized in that: The threshold value P=0.

5.

4. A computer program, characterized in that The computer program enables a computer to execute the method according to any one of claims 1 to 3.

5. An electronic device, characterized in that: include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the method as claimed in any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

7. A chip, characterized in that: include: A processor, configured to call and run a computer program from a memory, so that a device equipped with the chip executes a method as claimed in any one of claims 1 to 3.

8. A computer program product, characterized in that The computer program product comprises a computer storage medium storing a computer program, wherein the computer program comprises instructions executable by at least one processor, and when the instructions are executed by the at least one processor, the method according to any one of claims 1 to 3 is implemented.

Images