Multi-domain question and answer task fine tuning method based on low-rank adaptive matrix and mixed experts

Abstract

The invention belongs to the field of natural language processing and efficient fine tuning of parameters, and particularly relates to a multi-domain question and answer task fine tuning method based on a low-rank adaptive matrix and hybrid experts, and the method comprises the steps: loading and dividing a data set; a sentence vector model is used for extracting data set sentence vector representation, and a K-means clustering algorithm is used for sentence vectors to obtain a category number N; loading the pre-training language model, and freezing original model parameters; constructing N asymmetric low-rank expert modules and routing modules beside a specified structure in the model; sentence vector representation is used as input of a routing module, and the routing module calculates expert weights and performs weighted summation on different experts. The multi-domain question and answer task fine tuning method comprises a low-rank adaptive matrix and a hybrid expert module. According to the method, the number of trainable parameters is remarkably reduced by using the low-rank adaptive matrix, and the training efficiency is improved; a hybrid expert module is used to learn proprietary knowledge of questions in different fields in a complex question and answer task, and the generalization ability of the model is improved.

Description

Technical Field

[0001] The invention belongs to the field of natural language processing and efficient parameter fine-tuning, and specifically relates to a multi-domain question-answering task fine-tuning method based on low-rank adaptive matrices and hybrid experts. Background Art

[0002] Since the introduction of Transformer, pre-trained language models have emerged in an endless stream, and fine-tuning large pre-trained language models on downstream tasks has become an important paradigm for NLP. However, with the expansion of the number of model parameters, it has become increasingly difficult to fine-tune all parameters of the model. Taking GPT-3-175B as an example, if independent instances of the fine-tuned model are deployed, each instance has 175B parameters, and the required computing resources are very expensive, making it difficult to complete full parameter fine-tuning on consumer-grade graphics cards. Therefore, researchers have proposed a variety of efficient parameter fine-tuning methods, among which the method of using low-rank matrix adaptation has become the main choice for most researchers to fine-tune the model. Low-rank matrix adaptation, also known as LoRA, freezes the pre-trained model parameter weights and injects a trainable rank decomposition matrix into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared with full parameter fine-tuning, LoRA can reduce the number of trainable parameters by 10,000 times.

[0003] The hybrid expert model, also known as MoE, was proposed as early as the last century and its predecessor is "ensemble learning". As a deep learning technology controlled by sparse gates composed of expert models and gating models, MoE consists of multiple sub-models (i.e. experts), each of which is a local model that specializes in processing a subset of the input space. Guided by the core idea of ​​"divide and conquer", MoE uses a gating network to determine which model should be used to train each data, thereby reducing the interference between different types of samples. The introduction of the "sparseness" of the gating function allows MoE to activate only a few expert models when processing input data, and most of the expert models are in an inactive state. This "sparse state", as an important advantage of the hybrid expert model, further improves the efficiency of the model training and reasoning process.

[0004] On this basis, some recent studies have combined LoRA with hybrid experts to further enhance the adaptability and performance of the model. The integration of LoRA allows precise modification of weights through low-rank matrices, while MoE utilizes a set of expert networks, each specializing in different tasks or aspects of the data. The synergy between LoRA's target weight adaptation and MoE's expert-driven approach provides a dynamic path for model enhancement.

[0005] However, the traditional MoE framework does not take into account the common knowledge between different tasks or different domains of tasks, and only focuses on proprietary knowledge, which affects the performance of the model. The static top-K expert selection popular in MoE does not fully reflect the differences in mixed experts when dealing with problems in different fields, which prompts the need for a more dynamic selection mechanism that can respond to the complexity and subtlety of problems in different fields in complex question-answering tasks. Summary of the invention

[0006] In order to solve the problems existing in the above prior art, the present invention proposes a multi-domain question-answering task fine-tuning method based on a low-rank adaptive matrix and hybrid experts, which is characterized by comprising: loading and dividing a data set; extracting a sentence vector representation of the data set using a sentence vector model; obtaining the number of categories N using a K-means clustering algorithm for the sentence vector; loading a pre-trained language model and freezing the original model parameters; constructing N asymmetric trainable low-rank expert modules and a trainable routing module next to the structure specified in the model; fine-tuning the model, using the sentence vector representation as the input of the routing module, the routing module calculates the expert weights and sums them weighted, and the original model output is added to the expert module output and fed to the next layer of Transformer blocks. The multi-domain question-answering task fine-tuning method includes a large language model, a low-rank adaptive matrix, and a hybrid expert framework;

[0007] The fine-tuning process for multi-domain question answering tasks using low-rank adaptive matrices and mixture of experts modules includes:

[0008] S1: Load the multi-domain complex question-answering dataset and divide the samples into training set D and test set T;

[0009] S2: Use the SentenceTransformer library to load the sentence vector model and extract the sentence vector representation of the training set D;

[0010] S3: Perform K-means clustering on the sentence vectors of the training set D to obtain the optimal number of clusters N;

[0011] S4: Connect to the Huggingface model library to load the model and freeze the model parameters;

[0012] S5: N asymmetric trainable low-rank expert modules and trainable routing modules are constructed next to the specified structure of the model, usually between the feed-forward layers of the model Transformer block;

[0013] S6: Preprocess the training set D, set the expert weight threshold, input the training set samples into the model to start fine-tuning, and use the sentence vector representation of the training set as the input of the routing module during the training process. The routing module calculates the expert weight, discards the expert modules that do not reach the threshold, and sums the weights of the remaining experts. The output of the hybrid expert module is added to the model output and fed into the next layer of Transformer block;

[0014] S7: After the training is completed, the model effect is verified. The test set T is preprocessed, the sentence vector representation of the test set is extracted, the test samples are input into the model, and the cluster center corresponding to the sentence vector of each sample is used as the input of the routing module. The routing module distributes the input vector to different expert modules. Finally, the accuracy of the answer is used as an indicator after the entire model is calculated.

[0015] Preferably, the process of loading a multi-domain complex question-answering dataset includes: collecting question-answering datasets containing different domains, cleaning the collected data to leave only plain text samples, and dividing these samples into a training set containing 70% of the samples and a test set containing 30%.

[0016] Preferably, use the SentenceTransformers library to load the sentence vector model, and extract the sentence vector representation of the training set D as follows:

[0017] c i =SentenceTransformersModel(x i )

[0018] Preferably, K-means clustering is used for the sentence vectors, and the optimal number of categories N is determined. The core indicator for determining the optimal number of clusters N is the sum of squared errors (SSE), which is calculated as follows:

[0019]

[0020] Preferably, connect to the Huggingface model library to load the model and freeze the model parameters, which is specifically expressed as follows:

[0021] model=Transformers.pretrained(modelpath,requiregrad=False)

[0022] Preferably, the formula of the Adam algorithm is:

[0023]

[0024] Preferably, the method for efficient fine-tuning of parameters of a low-rank adaptive matrix is ​​expressed as:

[0025] W0+ΔW=W0+BA

[0026] Preferably, the low-rank expert module is constructed as,

[0027] E i =B i A

[0028] Preferably, the sample pretreatment is:

[0029] Input(S)=[CLS]Question+Content[SEP]Option 1[SEP]…[SEP]Option i[SEP]

[0030] Preferably, the expert weight is calculated as:

[0031] p i =Softmax(W g c) i

[0032] Preferably, the output of the low-rank mixed expert module is expressed as:

[0033]

[0034] Preferably, the calculation formula for accuracy is:

[0035]

[0036] The present invention utilizes an asymmetric low-rank matrix to solve the problem of learning shared knowledge between different question-answering tasks or different sub-field problems in question-answering tasks in previous low-rank mixed expert methods; the present invention utilizes a mixed expert module to improve the model expression ability of the previous parameter efficient fine-tuning method based on a low-rank adaptive matrix when processing complex tasks, and changes the expert selection method from static top-K experts to dynamic selection of multiple experts based on weight thresholds, thereby solving the differences of mixed experts when facing question-answering tasks in different fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is the overall flow chart of the present invention;

[0038] Figure 2 A diagram for constructing a low-rank adaptive matrix of the present invention;

[0039] Figure 3 Constructing a graph for the model of the present invention;

[0040] Figure 4 This is a diagram of the hybrid expert framework of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] A fine-tuning method for multi-domain question answering tasks based on low-rank adaptive matrices and mixed experts, such as Figure 1 The method includes: loading and dividing the data set; extracting the sentence vector representation of the data set using the sentence vector model; using the K-means clustering algorithm for the sentence vector to obtain the number of categories N; loading the pre-trained language model and freezing the original model parameters; building N asymmetric trainable low-rank expert modules and trainable routing modules next to the structure specified in the model; fine-tuning the model, using the sentence vector representation as the input of the routing module, the routing module calculates the expert weight and sums it weighted, and the original model output is added to the expert module output and fed to the next layer of the network structure. The multi-domain question answering task fine-tuning method includes a large language model, a low-rank adaptive matrix, and a hybrid expert framework;

[0043] like Figure 2 As described above, the efficient fine-tuning method for large model parameters for training low-rank matrix adaptation and multi-layer optimization includes:

[0044] S1: Load a multi-domain complex question-answering dataset and divide the samples into a training set D and a test set T;

[0045] S2: Use the SentenceTransformer library to load the sentence vector model and extract the sentence vector representation of the training set D;

[0046] S3: Perform K-means clustering on the sentence vectors of the training set D to obtain the optimal number of clusters N;

[0047] S4: Connect to the Huggingface model library to load the model and freeze the model parameters;

[0048] S5: N asymmetric trainable low-rank expert modules and trainable routing modules are constructed next to the specified structure of the model, usually between the feed-forward layers of the model Transformer block;

[0049] S6: Preprocess the training set D, set the expert weight threshold, input the training set samples into the model to start fine-tuning, and use the sentence vector representation of the training set as the input of the routing module during the training process. The routing module calculates the expert weight, discards the expert modules that do not reach the threshold, and sums the weights of the remaining experts. The output of the hybrid expert module is added to the model output and fed into the next layer of Transformer block;

[0050] S7: After the training is completed and the model effect is verified, the test set T is preprocessed, the sentence vector representation of the test set is extracted, the test samples are input into the model, and the cluster center corresponding to the sentence vector of each sample is used as the input of the routing module. The routing module distributes the input vector to different expert modules. Finally, after the entire model is calculated, the accuracy of the answer is displayed as an indicator;

[0051] The ScienceQA dataset is used. ScienceQA is a common sense multimodal question-answering dataset collected from elementary and high school science courses. It contains multiple-choice question-answering datasets in different languages ​​for different subjects. It contains 21,208 samples. All the plain text samples are collected, and the training set D and test set T of 6,508 and 2,224 samples are created respectively. ScienceQA has rich diversity from three disciplines: natural sciences, language sciences, and social sciences. In order to answer these questions, the model needs to be consistent with the correct common sense knowledge.

[0052] After obtaining the training set and test set, use the SentenceTransformers library to load the sentence vector model and extract the sentence vector representation of the sample. The simple representation is as follows:

[0053] c i =SentenceTransformersModel(x)

[0054] where c i It represents the extracted sentence vector, and SentenceTransformersModel() represents the loaded sentence vector.

[0055] The main process of obtaining the optimal clustering number N is described as follows.

[0056] Step 1: Run the K-means clustering algorithm for different values ​​of N (usually starting from 1 and gradually increasing).

[0057] Step 2: For each N value, calculate the total internal sum of squares (SSE, Sum of SquaredErrors) of the clustering results, that is, the sum of squared errors within the cluster. The formula is as follows,

[0058]

[0059] Among them, C i is the i-th cluster, p is C i The sample points in m i It is C i The center of mass (C i The mean of all samples in the cluster), SSE is the clustering error of all samples, which represents the quality of clustering effect.

[0060] Step 3: Plot a graph of N value versus SSE.

[0061] Step 4: Observe the trend of SSE as the value of N increases. Generally, as N increases, SSE decreases because the points within the cluster are closer to the cluster center.

[0062] Step 5: Find the "elbow" point where the SSE decreases slowly, that is, the inflection point of the SSE curve. This point is usually considered to be the optimal number of clusters N.

[0063] After obtaining the optimal number of clusters N, clustering is completed for the sentence vectors of all samples in the training set D.

[0064] Connect to the Huggingface model library, load the specified pre-trained model, and freeze the model parameters, specifically expressed as:

[0065] model=Transformers.pretrained(modelpath,requiregrad=False)

[0066] Among them, modelpath is the local model address or model name, requiregrad = False means freezing the model parameters so that they do not participate in the calculation of gradients during back propagation.

[0067] Set the learning rate and optimizer, where the optimizer is Adam. The optimization process can be simply expressed as the following formula:

[0068]

[0069] in, represents the corrected first-order moment estimate, represents the corrected second-order moment estimate, ∈, η are the parameters that need to be adjusted during the training process.

[0070] A low-rank expert module is constructed, where the original parameter efficient fine-tuning method based on a low-rank adaptive matrix can be expressed as:

[0071] W0+ΔW=W0+BA

[0072] Where W0 is the frozen original parameter matrix, ΔW is the update of the parameter matrix, B and A are two low-rank decomposition matrices, BA has the same shape as W0 and is used to fit the parameter update, where r<<min(d,k).

[0073] The hybrid expert framework is used to extend its performance, prompting the model to learn specialized knowledge of problems in different fields. It is constructed in an asymmetric way so that it retains some common knowledge. The specific expert construction is shown as follows:

[0074] E i =B i A

[0075] Among them, A is shared by all experts to learn the common knowledge in different sub-fields, and B i Unique for each expert to learn the specialized knowledge in different sub-fields.

[0076] After the construction is completed, the training set samples are preprocessed, and the content part, question part and option part of each sample are spliced ​​together. Special identifiers are added to the sentences to indicate the beginning, separation and end. The specific expressions are as follows:

[0077] Input(S)=[CLS]Question+Content[SEP]Option 1[SEP]…[SEP]Option i[SEP]

[0078] Among them, S represents the input sentence, [CLS] represents the beginning of the sentence, and [SEP] represents the segmentation and ending mark. The input is converted into word vectors through the word segmenter and input into the model training.

[0079] During the training process, the routing module is trained through the trainable matrix W g The input x of the feed-forward layer (FFN) is assigned to different experts. Specifically, the Softmax function is used for normalization to generate different expert weights. The formula is as follows:

[0080] p i =Softmax(W g c) i

[0081] where p i represents the weight of an expert, W g is a trainable routing module, and c is the cluster center sentence vector corresponding to the sentence vector of the sample.

[0082] Define the weight threshold τ, if p i ≥τ, then select expert i if all weights p i If all are below the threshold, a very high τ may result in no expert being selected. To ensure that at least one expert is selected, we set T = 1 / K as the lower limit of the expert weight, where K is the number of experts. After weight judgment, the output of all experts can be expressed as,

[0083]

[0084] Among them, y represents the output of the expert, p i is the weight of each expert, E i (x) is the result output by the i-th expert, The function indicates that for those that do not satisfy pi Expert weights ≥τ will be discarded.

[0085] The input of each layer is processed by the hybrid expert and then merged with the result of the frozen matrix and output to the next layer. It can be specifically expressed as:

[0086]

[0087] Where W0 is the frozen original parameter matrix, p i is the expert weight, is the threshold control function, B i A is each trainable low-rank expert module.

[0088] After the model training is completed, the test set is used to complete the reasoning and verify the model capability. During the reasoning process, the sentence vector of the test set sample corresponding to the center vector in the clustering model is used as the input of the routing module to improve the generalization ability of the model. After the entire model, the accuracy of the answer is used as the measurement standard of the method. The accuracy is calculated as follows:

[0089]

[0090] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-domain question answering task fine-tuning method based on low-rank adaptive matrix and hybrid experts, characterized in that: include: Load the dataset and divide it; Use the sentence vector model to extract the sentence vector representation of the data set; use the K-means clustering algorithm on the sentence vector to obtain the number of categories N; Load the pre-trained language model and freeze the original model parameters; build N trainable low-rank expert modules and trainable routing modules next to the structure specified in the model; fine-tune the model, use the sentence vector representation as the input of the routing module, the routing module calculates the expert weight and sums it, and the original model output is added to the expert module output and fed to the next layer of Transformer block. The multi-domain question answering task fine-tuning method includes a low-rank adaptive matrix and a hybrid expert module; The fine-tuning process for multi-domain question answering tasks using low-rank adaptive matrices and mixture of experts modules includes: S1: Load the multi-domain question answering dataset and divide the samples into training set D and test set T; S2: Use the SentenceTransformer library to load the sentence vector model and extract the sentence vector representation of the training set D; S3: Perform K-means clustering on the sentence vectors of the training set D to obtain the optimal number of clusters N; S4: Connect to the Huggingface model library to load the model and freeze the model parameters; S5: N trainable low-rank expert modules and trainable routing modules are constructed next to the specified structure of the model, usually between the feed-forward layers of the model Transformer block; S6: Preprocess the training set D, set the expert weight threshold, input the training set samples into the model to start fine-tuning, and use the sentence vector representation of the training set as the input of the routing module during the training process. The routing module calculates the expert weight, discards the expert modules that do not reach the threshold, and sums the weights of the remaining experts. The output of the hybrid expert module is added to the model output and fed into the next layer of Transformer block; S7: After the training is completed and the model effect is verified, the test set T is preprocessed, the sentence vector representation of the test set is extracted, the test samples are input into the model, and the cluster center corresponding to the sentence vector of each sample is used as the input of the routing module. The routing module distributes the model input to different expert modules. Finally, after the entire model is calculated, the accuracy of the answer is used as an indicator.

2. According to claim 1, a multi-domain question answering task fine-tuning method based on low-rank adaptive matrix and hybrid experts is characterized in that: The process of loading a multi-domain question-and-answer dataset includes: collecting a question-and-answer dataset containing different fields, the dataset containing multimodal question-and-answer content. Since the main focus of the present invention is text input, the collected data is cleaned to leave only pure text samples, and these samples are divided into a training set containing 70% of the samples and a test set containing 30%.

3. According to claim 2, a multi-domain question answering task fine-tuning method based on low-rank adaptive matrix and hybrid experts is characterized in that: Use the SentenceTransformers library to load the sentence vector model and extract the sentence vector representation of the training set D: The SentenceTransformers library can load the sentence vector model and extract the sentence vector representation for the sample data. Use the SentenceTransformers library to load the specified sentence vector model and extract the sentence vector representation of all samples in the training set D to prepare for subsequent clustering. The specific formula is as follows. c i =SentenceTransformersModel(x i ) where c i represents the extracted sentence vector, SentenceTransformersModel() represents the loaded sentence vector model, x i Represents each sample in the training set.

4. The multi-domain question answering task fine-tuning method based on low-rank adaptive matrix and hybrid experts according to claim 3, characterized in that: Use N-means clustering on sentence vectors to determine the optimal number of clusters N: The basic idea of ​​the K-means algorithm is to divide the sample data into K categories so that the distance between each data point and the cluster center of its category is minimized, thereby achieving the purpose of clustering. The main process of obtaining the optimal number of clusters N is described as follows. Step 1: Run the K-means clustering algorithm for different values ​​of N (usually starting from 1 and gradually increasing). Step 2: For each N value, calculate the total internal sum of squares (SSE, Sum of Squared Errors) of the clustering results, that is, the sum of squared errors within the cluster. The formula is as follows, Among them, C i is the i-th cluster, p is C i The sample points in m i It is C i The center of mass (C i The mean of all samples in the cluster), SSE is the clustering error of all samples, which represents the quality of clustering effect. Step 3: Plot a graph of N value versus SSE. Step 4: Observe the trend of SSE as the value of N increases. Generally, as N increases, SSE decreases because the points within the cluster are closer to the cluster center. Step 5: Find the "elbow" point where the SSE decreases slowly, that is, the inflection point of the SSE curve. This point is usually considered to be the optimal number of clusters N. After obtaining the optimal number of clusters N, clustering is completed for the sentence vectors of all samples in the training set D.

5. The multi-domain question answering task fine-tuning method based on low-rank adaptive matrix and hybrid experts according to claim 4, characterized in that: Connect to the Huggingface model library to load the model and freeze the model parameters: Connect to the Huggingface model library, load the specified pre-trained model, and freeze the model parameters, specifically expressed as: model=Transformers.pretrained(modelpath,requiregrad=False) Among them, modelpath is the local model address or model name, requiregrad=False means freezing the parameters of the model so that it does not participate in the calculation of gradients during back propagation. Set the learning rate and optimizer, where the optimizer is Adam, and the optimization process is as follows: in, represents the corrected first-order moment estimate, represents the corrected second-order moment estimate, ∈, η are the parameters that need to be adjusted during the training process.

6. The multi-domain question answering task fine-tuning method based on low-rank adaptive matrix and hybrid experts according to claim 5, characterized in that: Construct N trainable low-rank expert modules and trainable routing modules: The parameter-efficient fine-tuning method with low-rank adaptive matrices (LoRA) achieves comparable performance to full parameter fine-tuning on many benchmarks by freezing the pre-trained model weights and inserting the trainable rank factorization matrix into the specified network structure of the pre-trained language model. Specifically, LoRA uses two sequential low-rank matrices A and B to fit the update of the pre-trained language model parameters. The forward calculation process is shown below, W0+ΔW=W0+BA Where W0 is the frozen original parameter matrix, ΔW is the update of the parameter matrix, B and A are two low-rank decomposition matrices, BA has the same shape as W0 and is used to fit the parameter update, where r<<min(d,k). A single rank factorization matrix cannot learn the specialized knowledge of different sub-domains in multi-domain question answering tasks, so a mixture of experts (MoE) framework is used to extend its performance. The MoE framework expands the model complexity and capacity by integrating multiple expert sub-networks, each of which may be specialized for different data segments or tasks. In the MoE framework, the routing module coordinates N independent experts. The routing module is implemented through a trainable matrix W g The input vector x is distributed among these experts. In order to enable the model to learn the proprietary knowledge and common knowledge of different sub-fields in the multi-domain question-answering task, a low-rank expert module is constructed in an asymmetric way. Each expert module can be represented as E i =B i A, where A is shared by all experts to learn common knowledge in different sub-fields, B i Unique for each expert to learn the specialized knowledge in different sub-fields. According to the number of clustering categories N, N trainable low-rank experts and trainable routing modules are constructed between the feed-forward layers (FFN) of each Transformer block.

7. The multi-domain question answering task fine-tuning method based on low-rank adaptive matrix and hybrid experts according to claim 6, characterized in that: Preprocess the training set D, set the weight threshold, input the training set samples into the model to start fine-tuning, and use the sentence vector representation of the training set as the input of the routing module during the training process. The routing module calculates the expert weights, sums the expert weights, and finally adds them to the original model output and feeds them into the next layer of the network structure: Get the training set D, splice the content part, question part, and option part of each sample together, and add special identifiers to the sentence to indicate the beginning, separation, and end. The specific representation is as follows: Input(S)=[CLS]Question+Content[SEP]Option 1[SEP]…[SEP]Option i[SEP] Among them, S represents the input sentence, [CLS] represents the beginning of the sentence, and [SEP] represents the segmentation and ending mark. The input is converted into word vectors through the word segmenter and input into the model training. During the training process, the routing module is trained through the trainable matrix W g The input of the model is assigned to different experts. Specifically, the Softmax function is used for normalization to generate different expert weights. The formula is as follows: p i =Softmax(W g c) i where p i represents the weight of an expert, W g is a trainable routing module, and c is the cluster center sentence vector corresponding to a single sample. Define the weight threshold τ, if p i ≥τ, then select expert i if all weights p i If all are below the threshold, a very high τ may result in no expert being selected. To ensure that at least one expert is selected, we set T = 1 / N as the lower limit of the expert weight, where N is the number of experts. After weight judgment, the output of all experts can be expressed as, Among them, x is the input of a Transformer block in a layer of the pre-trained model, y represents the output of the expert, and p i is the weight of each expert, E i (x) is the result output by the i-th expert, The function indicates that for those that do not satisfy p i Expert weights ≥τ will be discarded. The input x of each feed-forward layer (FFN) is processed by the hybrid expert module and then merged with the calculation result of the frozen parameter matrix and output to the next layer of the network. It can be specifically expressed as: Where W0 is the frozen original pre-trained weight parameter matrix, p i is the expert weight, is the threshold control function, B i A is each trainable low-rank expert module. It is a threshold control function. For a certain expert, if pi ≥ τ, it will be retained, otherwise it will be discarded. The final result can be expressed as, Result = MoELoRAModel(x, c) Where MoELoRAModel() indicates the application of the mixed expert module and low-rank adaptive matrix pre-training model, x represents each sample in the dataset and c is the sentence-oriented clustering center corresponding to each sample.

8. The multi-domain question answering task fine-tuning method based on low-rank adaptive matrix and hybrid experts according to claim 7, characterized in that: Verify the model effect: preprocess the test set T, extract the sentence vector representation of the test set, input the test sample into the model, and use the cluster center corresponding to the sentence vector of each sample as the input of the routing module. The routing module distributes the input vector to different expert modules. Finally, the accuracy value is displayed as the experimental indicator after the entire model calculation: evaluate the model after training, perform the preprocessing method as described in claim 6 on the test set, load the sentence vector model, extract the sentence vector representation of the training set, and input the test sample into the model for reasoning. During the reasoning process, whenever passing through the low-rank mixed expert module, the category center vector closest to the test sample in the clustering model is used as the input of the routing module, and the weights that do not reach the threshold are discarded. Finally, the expert output and the model output are added and fed to the next layer of the network. After the entire model, the accuracy of the answer is used as the performance measure of the method. The accuracy is calculated as follows: 。

Images