Multi-domain question answering task fine-tuning method based on low-rank adaptive matrix and mixed experts

Abstract

The application belongs to the field of natural language processing and parameter efficient fine-tuning, and particularly relates to a multi-domain question answering task fine-tuning method based on a low-rank adaptive matrix and a mixed expert, which comprises the following steps: loading a data set and dividing; extracting a sentence vector representation of the data set by using a sentence vector model, and obtaining a category number N by using a K-means clustering algorithm on the sentence vector; loading a pre-trained language model and freezing original model parameters; constructing N asymmetric low-rank expert modules and a routing module beside a specified structure in the model; inputting the sentence vector representation into the routing module, calculating expert weights by the routing module, and performing weighted summation on different experts; the multi-domain question answering task fine-tuning method comprises a low-rank adaptive matrix and a mixed expert module; the application significantly reduces the number of trainable parameters by using the low-rank adaptive matrix, and improves the training efficiency; the mixed expert module is used to learn the specific knowledge of different domain problems in a complex question answering task, and the generalization ability of the model is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of natural language processing and parameter efficient fine-tuning, and particularly relates to a multi-domain question answering task fine-tuning method based on a low-rank adaptive matrix and a mixed expert. BACKGROUND

[0002] Since the emergence of Transformer, pre-training language models have been emerging in large numbers, and fine-tuning large pre-training language models on downstream tasks has become an important paradigm in NLP. However, with the explosion of model parameters, it is increasingly difficult to fine-tune the model with all parameters. Taking GPT-3-175B as an example, if an independent instance of the fine-tuned model is deployed, each instance has 175B parameters, and the required computing resources are very expensive, making it difficult to complete the fine-tuning of all parameters on a consumer-grade graphics card. Therefore, researchers have proposed various parameter efficient fine-tuning methods, among which the method of adapting to low-rank matrices has become the main choice for most researchers to fine-tune the model. Low-rank matrix adaptation, also known as LoRA, freezes the pre-trained model parameter weights and injects a trainable rank decomposition matrix into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared with full-parameter fine-tuning, LoRA can reduce the number of trainable parameters by 10,000 times.

[0003] The mixed expert model is also known as MoE, which was proposed as early as last century, and its predecessor is "ensemble learning". As a deep learning technology controlled by sparse gates composed of expert models and gating models, MoE is composed of multiple sub-models (i.e. experts), each of which is a local model that specifically handles a subset of the input space. Under the guidance of the core idea of "divide and conquer", MoE uses a gating network to determine which model each data should be trained on, thereby reducing interference between different types of samples. The introduction of the "sparsity" of the gating function allows MoE to activate only a few expert models when processing input data, with most expert models in an inactive state. This "sparse state" as an important advantage of the mixed expert model further improves the efficiency of the model training and inference process.

[0004] On this basis, some recent research combines LoRA with mixed experts to further enhance the adaptability and performance of the model. The integration of LoRA allows precise modification of weights through low-rank matrices, while MoE utilizes a set of expert networks, each of which specifically handles different tasks or aspects of the data. The synergy between the target weight adaptation of LoRA and the expert-driven approach of MoE provides a dynamic approach to model enhancement.

[0005] However, the traditional MoE framework does not consider the common knowledge between different tasks or different fields of the same task, and only focuses on specific knowledge, which affects the performance of the model. The static top-K expert selection popular in MoE does not fully reflect the differences between mixed experts in dealing with problems in different fields, which prompts the need for a more dynamic selection mechanism that can respond to the complexity and subtlety of different field problems in complex question and answer tasks. SUMMARY

[0006] In view of the deficiencies of the prior art, the present application provides a multi-domain question and answer task fine-tuning method based on a low-rank adaptive matrix and a mixed expert, characterized by comprising: loading a data set and dividing; using a sentence vector model to extract a sentence vector representation of the data set; using a K-means clustering algorithm on the sentence vector to obtain a class number N; loading a pre-trained language model and freezing the original model parameters; constructing N asymmetric trainable low-rank expert modules and a trainable routing module beside the specified structure in the model; fine-tuning the model, taking the sentence vector representation as the input of the routing module, the routing module calculating the expert weights and performing weighted summation, and feeding the sum of the original model output and the expert module output into the next layer of Transformer blocks; the multi-domain question and answer task fine-tuning method comprises a large language model, a low-rank adaptive matrix and a mixed expert module;

[0007] The multi-domain question and answer task fine-tuning process using a low-rank adaptive matrix and a mixed expert module comprises:

[0008] S1: Load a multi-domain question and answer data set, and divide the samples into a training set D and a test set T;

[0009] S2: Use the SentenceTransformer library to load a sentence vector model and extract a sentence vector representation of the training set D;

[0010] S3: Perform K-means clustering on the sentence vector of the training set D to obtain the optimal cluster number N;

[0011] S4: Connect the Huggingface model library to load the model and freeze the model parameters;

[0012] S5: Construct N asymmetric trainable low-rank expert modules and a trainable routing module beside the specified structure of the model, usually between the feedforward layers of the model Transformer blocks;

[0013] S6: Preprocess the training set D, set the expert weight threshold, input the training set sample into the model to start fine-tuning, and input the sentence vector representation of the training set as the routing module during the training process. The routing module calculates the expert weight, discards the expert module that does not reach the threshold, and sums the remaining experts, and the mixed expert module output is added to the original model output and fed into the next layer of Transformer block;

[0014] S7: The training is completed, and the model effect is verified. The test set T is preprocessed, the test set sentence vector representation is extracted, the test sample is input into the model, the clustering center corresponding to the sentence vector of each sample is taken as the input of the routing module, the routing module assigns the input vector to different expert modules, and finally the accuracy of the answer is taken as the index after the whole model calculation.

[0015] Preferably, the process of loading multi-domain question and answer data set includes: collecting question and answer data sets containing different domains, the data set containing multi-modal question and answer content, cleaning the collected data, leaving only pure text samples, dividing the samples, the training set containing 70% samples, and the test set containing 30% samples.

[0016] Preferably, the sentence vector model is loaded using the SentenceTransformers library, and the sentence vector representation of the training set D is extracted as:

[0017] c i =SentenceTransformersModel(x i )

[0018] Preferably, the sentence vector is subjected to K-means clustering, and the optimal cluster number N is determined; the core index for determining the optimal cluster number N is the sum of squared errors within clusters (SSE), and the calculation formula is:

[0019]

[0020] Preferably, the model is loaded by connecting the Huggingface model library and the model parameters are frozen, represented as:

[0021] model=Transformers.pretrained(modelpath,requiregrad=False)

[0022] Preferably, the formula of Adam algorithm is:

[0023]

[0024] Preferably, the parameter efficient fine-tuning method of low-rank adaptive matrix is represented as:

[0025] W0+Delta W = W0+BA

[0026] Preferably, the low-rank expert module is constructed as:

[0027] E i = B i A

[0028] Preferably, the sample preprocessing is:

[0029] Input(S) = [CLS] question + content [SEP] option1 [SEP] … [SEP] optioni [SEP]

[0030] Preferably, the expert weight calculation is:

[0031] p i = Softmax(W g c) i

[0032] Preferably, the output of the low-rank mixed expert module is represented as:

[0033]

[0034] Preferably, the accuracy calculation formula is:

[0035]

[0036] The present application uses an asymmetric low-rank matrix to solve the problem of learning common knowledge between different question and answer tasks or different sub-field problems in the previous low-rank mixed expert method; the present application uses a mixed expert module to improve the model expression ability of the previous parameter efficient fine-tuning method based on a low-rank adaptive matrix when processing complex tasks; the expert selection method changes from a static top-K expert to dynamically selecting multiple experts according to a weight threshold, solving the difference problem of the mixed expert when facing different domain question and answer tasks. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The overall flowchart of the present application;

[0038] Figure 2 The construction diagram of the low-rank adaptive matrix of the present application;

[0039] Figure 3 The model construction diagram of the present application;

[0040] Figure 4 The mixed expert framework diagram of the present application. DETAILED DESCRIPTION

[0041] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0042] A multi-domain question answering task fine-tuning method based on low-rank adaptive matrix and mixed experts, as Figure 1 The method comprises the following steps: loading a data set and dividing; extracting a sentence vector representation of the data set using a sentence vector model; obtaining a category number N by using a K-means clustering algorithm on the sentence vector; loading a pre-trained language model and freezing original model parameters; constructing N asymmetric trainable low-rank expert modules and a trainable routing module beside a specified structure in the model; fine-tuning the model, taking the sentence vector representation as the input of the routing module, calculating the expert weight by the routing module and performing weighted summation, and feeding the sum of the original model output and the expert module output into the next layer network structure; the multi-domain question answering task fine-tuning method comprises a large language model, a low-rank adaptive matrix and a mixed expert module.

[0043] As Figure 2 The efficient fine-tuning method of a large model parameter trained by a low-rank matrix adaptation and multi-layer optimization comprises the following steps:

[0044] S1: loading a multi-domain question answering data set and dividing samples into a training set D and a test set T;

[0045] S2: loading a sentence vector model using a SentenceTransformer library and extracting a sentence vector representation of the training set D;

[0046] S3: performing K-means clustering on the sentence vector of the training set D to obtain an optimal clustering number N;

[0047] S4: connecting a Huggingface model library to load a model and freeze model parameters;

[0048] S5: constructing N asymmetric trainable low-rank expert modules and a trainable routing module beside a specified structure of the model, usually between the feedforward layers of the model Transformer block;

[0049] S6: preprocessing the training set D, setting an expert weight threshold, inputting the training set samples into the model to start fine-tuning, taking the sentence vector representation of the training set as the input of the routing module during the training process, calculating the expert weight by the routing module, discarding the expert modules that do not reach the threshold, and performing weighted summation on the remaining experts, feeding the sum of the mixed expert module output and the original model output into the next layer Transformer block;

[0050] S7: Training ends, and the model effect is verified. The test set T is preprocessed, and the sentence vector representation of the test set is extracted. The test samples are input into the model, and the cluster center corresponding to the sentence vector of each sample is used as the input of the routing module. The routing module distributes the input vector to different expert modules. Finally, after the entire model is calculated, the accuracy of the answer is used as the indicator.

[0051] The ScienceQA dataset was used. ScienceQA is a multimodal commonsense question-answering dataset collected from elementary and high school science courses. It contains single-choice question-answering datasets in different subjects and languages, with 21,208 samples. All plain text samples were collected, and training sets D and test sets T, with 6,508 and 2,224 samples respectively, were created. ScienceQA has rich diversity across three disciplines: natural sciences, linguistic sciences, and social sciences. To answer these questions, the model needs to be consistent with correct commonsense knowledge.

[0052] After obtaining the training and test sets, the sentence vector model is loaded using the SentenceTransformers library to extract the sentence vector representation of the samples, as shown in the following simplified representation:

[0053] c i =SentenceTransformersModel(x i )

[0054] Among them, c i The `x` represents the extracted sentence vector, and `SentenceTransformersModel()` represents the loaded sentence vector model. i Let i represent the i-th sample.

[0055] The main process for obtaining the optimal number of clusters N is as follows:

[0056] Step 1: For different values ​​of N, run the K-means clustering algorithm, where N usually starts from 1 and gradually increases;

[0057] Step 2: For each N value, calculate the total internal sum of squared errors (SSE) of the clustering results, i.e., the sum of squared errors within each cluster; the formula is as follows:

[0058]

[0059] Among them, C i It is the i-th cluster, and p is C i The sample points in the middle, m i It is C i The centroid of the cluster is SSE, which is the clustering error of all samples and represents the quality of the clustering.

[0060] Step 3: Draw a chart of N value and SSE;

[0061] Step 4: Observe the trend of SSE changing with the increase of N value; usually, with the increase of N, SSE will decrease, because the points within the cluster are closer to the cluster center;

[0062] Step 5: Find the "elbow" point where the speed of SSE decrease slows down, that is, the inflection point of the SSE curve, which is usually considered as the best cluster number N;

[0063] After obtaining the best cluster number N, the sentence vectors of all samples in the training set D are clustered.

[0064] Connect the Huggingface model library, load the specified pre-trained model, and freeze the model parameters, which can be represented as,

[0065] model = Transformers.pretrained(modelpath, requiregrad=False)

[0066] Where modelpath is the local model address or model name, and requiregrad=False means to freeze the parameters of the model, so that it does not participate in the calculation of the gradient during backpropagation.

[0067] Set the learning rate and optimizer, where the optimizer is Adam, and the optimization process can be simply represented as the following formula,

[0068]

[0069] Where, is the corrected first moment estimate, is the corrected second moment estimate, and ∈ and η are parameters that need to be adjusted during training.

[0070] Construct a low-rank expert module, where the original parameter efficient fine-tuning method based on low-rank adaptive matrix can be represented as,

[0071] W0+ΔW = W0+BA

[0072] Where W0 is the frozen original parameter matrix, ΔW is the update of the parameter matrix, B and A are two low-rank decomposition matrices, BA has the same shape as W0 and is used to fit the update of the parameter, where r << min(d, k).

[0073] The performance of the hybrid expert framework is extended, prompting the model to learn the specialized knowledge of different domain problems, and the asymmetric structure is adopted to retain a part of the common knowledge, and the specific expert structure is represented as follows,

[0074] E i = B i A

[0075] wherein A is shared by all experts to learn the common knowledge of different subfields, and B i is unique to each expert to learn the specialized knowledge of different subfields.

[0076] After the construction is completed, the training set samples are preprocessed, the content part, the question part and the option part of each sample are spliced together, and special identifiers are added in the sentence to represent the beginning, separation and end, and the specific representation is as follows,

[0077] Input(S) = [CLS] question + content [SEP] option1 [SEP] … [SEP] optioni [SEP]

[0078] wherein S represents the input sentence, [CLS] represents the beginning of the sentence, and [SEP] represents the separation and end symbol; the input training sample is converted into a word vector by a word segmenter and input into the model for training.

[0079] In the training process, the routing module assigns the input x of the feedforward layer (FFN) to different experts through a trainable matrix W g , and the specific normalization is performed using the Softmax function to generate different expert weights, and the formula is as follows,

[0080] p i = Softmax(W g c) i

[0081] wherein p i represents the weight of a certain expert, W g is a trainable routing matrix, and c is a cluster center sentence vector corresponding to a single sample.

[0082] Define the weight threshold τ, if p i ≥ τ, select expert i, if all weights p i are lower than the threshold, then a very high τ may result in no expert being selected, in order to ensure that at least one expert is selected, set τ = 1 / N as the lower limit of the expert weight, wherein N is the number of experts; after the weight judgment, the output of all experts can be represented as,

[0083]

[0084] where x is the input of a certain layer of the pre-trained model, y represents the output of the expert, p i is the weight of the i-th expert, E i (x) is the result of the output of the i-th expert, The function indicates that the weight of the expert who does not satisfy p i ≥τ will be discarded.

[0085] The input of each layer is processed by the mixed expert and then combined with the result of the frozen matrix to output to the next layer network, which can be specifically represented as,

[0086]

[0087] where W0 is the frozen original pre-training weight parameter matrix, p i is the weight of the i-th expert, B i A is each trainable asymmetric low-rank expert module, and x represents an input vector. is a threshold control function. If p i ≥τ, the expert is retained, otherwise discarded.

[0088] After the model training is completed, the test set is used to complete the inference to verify the model capability. In the inference process, the center vector corresponding to the sentence vector of the test set sample in the clustering model is used as the input of the routing module to improve the generalization capability of the model. After the whole model, the accuracy of the answer is used as the performance measurement standard of the method, and the calculation method of the accuracy is as follows,

[0089]

[0090] The above examples further illustrate the purpose, technical solutions and advantages of the present application. It should be understood that the above examples are only preferred embodiments of the present application and do not limit the present application. Any modification, equivalent replacement, improvement, etc. made to the present application within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A multi-domain question answering task fine-tuning method based on a low-rank adaptive matrix and a mixed expert, characterized by, The method comprises the following steps: loading a data set and dividing it; extracting a sentence vector representation of the data set using a sentence vector model; obtaining the number of categories N by using a K-means clustering algorithm on the sentence vector; loading a pre-trained language model, freezing the original model parameters; constructing N asymmetric trainable low-rank expert modules and a trainable routing module beside the specified structure in the model; fine-tuning the model, taking the sentence vector representation as the input of the routing module, calculating the expert weight by the routing module and performing weighted summation, and feeding the sum of the original model output and the expert module output into the next layer of the Transformer block; The multi-domain question answering task fine-tuning method comprises a large language model, a low-rank adaptive matrix, and a hybrid expert module; The multi-domain question answering task fine-tuning process using the low-rank adaptive matrix and the hybrid expert module comprises the following steps: S1: loading a multi-domain question answering data set and dividing the samples into a training set D and a test set T; S2: loading a sentence vector model using the SentenceTransformer library and extracting a sentence vector representation of the training set D; S3: performing K-means clustering on the sentence vector of the training set D to obtain the optimal number of clusters N; S4: connecting the Huggingface model library to load the model and freeze the model parameters; S5: constructing N asymmetric trainable low-rank expert modules and a trainable routing module beside the specified structure of the model, usually between the feedforward layers of the model Transformer block; S6: preprocessing the training set D, setting an expert weight threshold, inputting the training set samples into the model to start fine-tuning, taking the sentence vector representation of the training set as the input of the routing module during the training process, calculating the expert weight by the routing module, discarding the expert modules that do not reach the threshold, and performing weighted summation on the remaining experts, feeding the sum of the hybrid expert module output and the original model output into the next layer of the Transformer block, which comprises the following steps: obtaining the training set D, concatenating the content part, the question part, and the option part of each sample together, and adding special identifiers to indicate the beginning, separation, and end in the sentence, which is specifically represented as follows, Input(S)=[CLS]question+content[SEP]option1[SEP]…[SEP]optioni[SEP] wherein S represents the input sentence, [CLS] represents the beginning of the sentence, and [SEP] represents the separation and end markers; the input training sample is converted into a word vector by a word segmenter and input into the model for training; During the training process, the routing module passes through the trainable matrix W g The input of the model is allocated to different experts, and the Softmax function is used for normalization to generate different expert weights, as follows: p i = Softmax(W g c) i where p i represents the weight of a certain expert, W g is a trainable routing matrix, and c is the cluster center sentence vector corresponding to a single sample. Define a weight threshold τ, if p i ≥τ, select expert i, if all weights p i are below the threshold, a very high τ can result in no expert being selected, to guarantee that at least one expert is selected, set τ = 1 / N as a lower bound on the expert weights, where N is the number of experts; after the weight judgment, the output of all experts is represented as, where x is the input of a certain layer of the pre-trained model, y represents the output of the expert, p i is the weight of the i-th expert, E i (x) is the result of the i-th expert output, The function indicates that the expert weight that does not satisfy p i ≥τ will be discarded; the input x of each feedforward layer (FFN) is processed by the hybrid expert module and then combined with the calculation result of the frozen parameter matrix to output to the next layer of network, which is specifically represented as, where W0is the frozen original pre-trained weight parameter matrix, p i is the weight for the i-th expert, B i A is each trainable asymmetric low-rank expert module, x represents the input vector; is a threshold control function, for a certain expert if p i ≥τ, it is retained, otherwise discarded; the final result is represented as, Result=MoELoRAModel(x,c) wherein MoELoRAModel() represents a pre-trained model applying the hybrid expert module and the low-rank adaptive matrix, x represents each sample of the data set, and c is the cluster center sentence vector corresponding to each sample. S7: The training is completed, and the model effect is verified. The test set T is preprocessed, the test set sentence vector representation is extracted, the test sample is input into the model, and the cluster center corresponding to the sentence vector of each sample is taken as the input of the routing module. The routing module distributes the model input to different expert modules. Finally, the accuracy of the answer is taken as the index after the whole model calculation.

2. The multi-domain question answering task fine-tuning method based on a low-rank adaptive matrix and a hybrid expert according to claim 1, characterized in that, The process of loading the multi-domain question and answer dataset includes: collecting question and answer datasets containing different domains, the dataset containing multi-modal question and answer content, cleaning the collected data, and only keeping pure text samples. The samples are divided into training set containing 70% samples and test set containing 30% samples.

3. The multi-domain question answering task fine-tuning method based on low-rank adaptive matrix and mixed experts according to claim 2, characterized in that, Load the sentence vector model using the SentenceTransformers library, and extract the sentence vector representation of the training set D: the SentenceTransformers library loads the sentence vector model, and extracts the sentence vector representation for the sample data; Load the specified sentence vector model using the SentenceTransformers library, and extract the sentence vector representation of all samples in the training set D for subsequent clustering, as shown in the following formula: c i = SentenceTransformersModel(x i ) where c i represents the sentence vector after extraction, SentenceTransformersModel() represents the loaded sentence vector model, x i represents the i-th sample in the training set.

4. The multi-domain question answering task fine-tuning method based on low-rank adaptive matrix and mixed experts according to claim 3, characterized in that, K-means clustering is used for sentence vector to determine the optimal number of clusters N; the basic idea of K-means algorithm is to divide sample data into K categories, so that the distance between each data point and the cluster center of its belonging category is minimized, thereby achieving the purpose of clustering; the main process of obtaining the optimal cluster number N is as follows: Step 1: For different N values, run the K-means clustering algorithm, N is usually increased gradually from 1; Step 2: For each N value, calculate the total sum of squares of the clustering results, i.e. the sum of squares of the within-cluster error; the formula is as follows: where C i is the ith cluster, p is the sample point in C i , m i is the centroid of C i , and SSE is the clustering error of all samples, which represents the quality of clustering effect; Step 3: Draw a chart of N value and SSE; Step 4: Observe the trend of SSE changing with the increase of N value; usually, with the increase of N, SSE will decrease, because the points within the cluster are closer to the cluster center; Step 5: Find the "elbow" point where the SSE decrease slows down, which is the inflection point of the SSE curve, and this point is usually considered as the optimal cluster number N; After obtaining the optimal cluster number N, the sentence vectors of all samples in the training set D are clustered.

5. The multi-domain question answering task fine-tuning method based on low-rank adaptive matrix and mixed experts according to claim 4, characterized in that, Connect the Huggingface model library to load the model and freeze the model parameters: Connect the Huggingface model library to load the specified pre-trained model and freeze the model parameters, which is represented as: model = Transformers.pretrained(modelpath, requiregrad = False) Where modelpath is the local model address or model name, and requiregrad = False means to freeze the model parameters so that they do not participate in the gradient calculation during backpropagation. Set the learning rate and optimizer, where the optimizer is Adam, and the optimization process is as follows: where, denotes the corrected first moment estimate, denotes the corrected second moment estimate, and ∈ and η are parameters to be adjusted during training.

6. The multi-domain question answering task fine-tuning method based on a low-rank adaptive matrix and a hybrid expert according to claim 5, characterized in that, Build N asymmetric trainable low-rank expert modules and a trainable routing module: LoRA (Low-Rank Adaptive Matrix Parameterization) is a parameter-efficient fine-tuning method for pre-trained models that freezes the pre-trained model weights and inserts a trainable rank factorization matrix into the specified network structure of the pre-trained language model, achieving performance comparable to full-parameter fine-tuning in many benchmark tests; LoRA uses two low-rank matrices A and B to fit the updates of the pre-trained language model parameters; the parameter update formula of LoRA is as follows: W0+ΔW=W0+BA where W0is the frozen original parameter matrix, AW is the update of the parameter matrix, B and A are two low-rank decomposition matrices, BA has the same shape as W0and is used to fit the update of the parameter, where r « min(d, k). A single rank decomposition matrix cannot learn the specific knowledge of different sub-fields in a multi-domain question answering task, so a mixed expert (MoE) framework is used to expand its performance; the mixed expert framework expands the model complexity and capacity by integrating multiple expert sub-networks, and each expert sub-network can be dedicated to different data segments or tasks; in the mixed expert framework, N independent experts are coordinated by a routing module The routing module is a trainable matrix W g The input vector x is distributed among these experts; in order to make the model learn the specific knowledge of different sub-fields in a multi-domain question answering task as well as common knowledge, an asymmetric way is used to construct a low-rank expert module, and each expert module can be represented as E i = B i A, where A is shared by all experts to learn common knowledge of different sub-fields, and B i is unique to each expert to learn specific knowledge of different sub-fields; according to the number of clustering categories N, N asymmetric trainable low-rank experts and a trainable routing module are constructed between the feedforward layers (FFN) of each Transformer block.

7. The multi-domain question answering task fine-tuning method based on low-rank adaptive matrix and mixed experts according to claim 6, characterized in that, Verify model effect: preprocess the test set T, extract the test set sentence vector representation, input the test sample into the model, and take the cluster center corresponding to the sentence vector of each sample as the input of the routing module. The routing module assigns the input vector to different expert modules. Finally, the accuracy value is calculated as the experimental index after the whole model calculation: after training, evaluate the model, preprocess the test set, load the sentence vector model, extract the sentence vector representation of the test set, input the test sample into the model for inference. In the inference process, whenever passing through the low-rank mixed expert module, the nearest class center vector in the clustering model to the test sample is taken as the input of the routing module, and the weights that do not reach the threshold are discarded. Finally, the expert output and the model output are added to feed into the next layer network; After passing through the whole model, the accuracy of the answer is used as the performance measurement standard of the method, and the calculation method of the accuracy is as follows:

Images