Dynamic routing parameter efficient fine tuning method and system based on LoRA-MoE

Abstract

The invention discloses a dynamic routing parameter efficient fine tuning method and system based on LoRA-MoE, and relates to the technical field of large model fine tuning. The method comprises the following steps: firstly, constructing a heterogeneous expert architecture-based LoRA module pool based on a multi-field data set; and secondly, coding the hidden layer features of the task through a dynamic gating network, realizing continuous differentiable expert activation, and improving the balance of expert allocation by adopting temperature attenuation and entropy regularization constraint. And then, dynamically selecting and carrying out weighted fusion on a plurality of LoRA parameter increments according to task semantics in a reasoning stage, so as to realize low-cost model adaptive updating. Finally, the module pool is continuously optimized through the confusion degree and manual evaluation feedback, low-efficiency modules are automatically eliminated, and a new LoRA module is generated to maintain task coverage. According to the method, the accuracy and generalization ability of the model in a complex scene can be remarkably improved on the premise of ensuring light weight, and rapid adaptation and dynamic optimization of a large model under a low-resource condition are realized.

Description

Technical Field

[0001] This invention relates to the field of large model fine-tuning technology, and in particular to an efficient method and system for fine-tuning dynamic routing parameters based on LoRA-MoE. Background Technology

[0002] With the advancements in hardware computing power and AI technology in recent years, these models have made significant progress in the field of natural language processing. Large language models have demonstrated strong dominance in general domains, easily handling various language tasks including text generation, understanding, translation, and information extraction. However, their practical deployment in industrial scenarios faces significant challenges. While traditional full-parameter fine-tuning methods can improve model performance in vertical tasks, they require adjusting billions or even hundreds of billions of parameters, resulting in high training costs and difficulty in adapting to scenarios with high real-time requirements. Among parameter-efficient fine-tuning techniques (PEFT), LoRA has become a mainstream solution due to its practicality and compatibility. However, current practices still face the following problems: On the one hand, traditional LoRA modules use fixed-rank matrix factorization, which cannot dynamically adjust low-rank dimensions according to task characteristics, leading to the accumulation of redundant parameters; on the other hand, the expert module structure in existing MoE architectures is simple, making it difficult to capture the multi-dimensional feature differences of tasks. At the same time, most MoE routing mechanisms are based on Top-K fixed routing strategies, lacking fine-grained control, which can easily lead to wasted expert resources or overload.

[0003] Existing dynamic fine-tuning schemes primarily rely on heuristic rules or fixed weights to fuse multiple LoRA modules, lacking adaptive response capabilities to task features. When dealing with complex queries involving cross-domain semantic mixing, such methods are prone to inter-module interference, causing generated content to deviate from expectations. Furthermore, existing dynamic routing mechanisms largely depend on pre-defined expert partitioning strategies, making it difficult to handle novel, unseen task patterns and limiting the model's zero-shot generalization ability. To address these issues, there is an urgent need to construct a dynamic routing framework with online learning capabilities. This framework should deeply integrate the decision-making advantages of hybrid expert (MoE) systems and combine them with real-time task feature analysis to achieve fine-grained activation and collaboration of LoRA modules. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for efficient fine-tuning of dynamic routing parameters based on LoRA-MoE, in order to solve the problems mentioned in the background art, and to solve the problems of parameter redundancy and high computational resource consumption in the fine-tuning process of current large-scale pre-trained models for specific tasks, so as to realize a low-cost, high-performance, and fast-adaptive LoRA-MoE method.

[0005] To achieve the above objectives, the present invention provides the following solution: an efficient method for fine-tuning dynamic routing parameters based on LoRA-MoE, the specific steps of which include the following:

[0006] A LoRA module pool based on a heterogeneous expert architecture is constructed using multi-domain datasets;

[0007] Receive the question-answering task and perform coarse-grained filtering based on the semantic description of the input question, selecting several LoRA modules with the highest fit to the question-answering task as the preliminary filtering results;

[0008] The hidden features of the question-answering task are encoded by a dynamic gating network, and the probabilistic sparse routing mechanism of Gumbel-Softmax is used to realize continuous differentiable expert activation, thereby achieving fine-grained dynamic expert selection.

[0009] Based on the weights output by the dynamic gating network, the parameter increments of each LoRA module are weighted and summed to obtain fusion parameters. These fusion parameters are then dynamically superimposed onto the large language model to achieve dynamic parameter injection and output inference results.

[0010] After generating the inference results, the perplexity and confidence are calculated in real time, and the LoRA module pool is updated and retrained based on the perplexity, confidence and human evaluation feedback.

[0011] Preferably, each LoRA module in the LoRA module pool includes low-rank decomposition parameters, task labels, embedding vectors, performance metrics, and activation frequency information; the expert modules in the LoRA module pool are divided into horizontal experts, vertical experts, and cross-experts according to their functions, to achieve collaborative modeling of feature dimensions, temporal dependencies, and cross-dimensional interactions.

[0012] Preferably, the lateral expert is used to perform directional low-rank decomposition on the input feature matrix to capture the semantic associations of each dimension in the word vectors; the longitudinal expert is used to set up a sliding window on the input sequence to perform time-axis convolution to capture the contextual relationship between adjacent words in the text sequence; and the cross expert is used for cross-dimensional interactive modeling to perform nonlinear interaction between the modeling feature dimension and the temporal dimension.

[0013] Preferably, the specific steps for performing coarse-grained filtering based on the semantic description of the input question are as follows:

[0014] After receiving the question-answering task description, a question representation vector is generated using a sentence vector encoding model;

[0015] The cosine similarity between the problem representation vector and the description vectors of each module in the LoRA module pool is calculated to initially screen the Top-M candidate modules.

[0016] Extract the response features of the Top-M candidate modules in the hidden layers of the basic model, calculate the feature distance between them and the task samples in different layers, and perform a weighted summation of three indicators: cosine similarity, Euclidean distance and Manhattan distance to determine the multiple LoRA modules that are closest to the task distribution.

[0017] Perform perplexity detection, and based on the perplexity detection results, calculate the task fit by weighting the results. Select the LoRA modules with the highest fit as the initial screening results, and use them as the input set for the fine-grained dynamic selection stage.

[0018] Preferably, the specific steps for implementing continuously differentiable expert activation using the probabilistic sparse routing mechanism of Gumbel-Softmax are as follows:

[0019] The input to the dynamic gating network is the dynamic rank allocation vector of the hidden layer features of the question-answering task, with an input dimension of 1024.

[0020] The input features are processed by grouped convolution, compressing the 1024-dimensional hidden layer vector to 256 dimensions, and then generating route logits through low-rank decomposition.

[0021] For each expert, generate Gumbel noise and calculate noisy logits based on the temperature coefficient;

[0022] Based on the noisy logits, continuous distributed weights are generated using the Softmax function, and an entropy maximization regularization term for the expert weight distribution is added to the loss function to balance the expert activation frequency.

[0023] Preferably, the dynamic gating network adopts a single-layer multilayer perceptron structure, with the input being the dynamic rank allocation vector of the hidden layer features of the question-answering task, and the output being the weights of the corresponding expert modules.

[0024] Preferably, when the perplexity fluctuation exceeds a set threshold in several consecutive rounds of reasoning, a route re-exploration mechanism is triggered to redistribute expert weights.

[0025] Preferably, it also includes a parameter reuse mechanism, wherein the parameter passing strategy of the parameter reuse mechanism is as follows:

[0026] Basic adapter sharing: All experts share a low-rank basis matrix, and each expert's private matrix is ​​generated through linear projection;

[0027] Independent updates of incremental parameters: Each expert maintains a dedicated incremental matrix, and parameter reuse is achieved through residual connections;

[0028] Cross-level inheritance mechanism: When the cosine similarity between high-level experts and low-level experts exceeds a set threshold, the parameters of the lower level are directly reused.

[0029] Preferably, the maintenance of the LoRA module pool adopts a contribution metric strategy, which counts the average activation frequency of each LoRA module on the validation set, automatically eliminates low-contribution modules, and triggers an automated fine-tuning process to supplement new modules, ensuring that the coverage of the module pool dynamically expands with the distribution of tasks.

[0030] On the other hand, a LoRA-MoE-based system for efficient fine-tuning of dynamic routing parameters is provided, including a LoRA module pool construction module, a dynamic expert management module, a gated routing module, a parameter fusion module, and a real-time feedback optimization module.

[0031] The LoRA module pool construction module is used to construct a LoRA module pool based on a heterogeneous expert architecture based on multi-domain datasets.

[0032] The dynamic expert management module is used to receive question-and-answer tasks and perform coarse-grained filtering based on the semantic description of the input questions, selecting several LoRA modules with the highest fit with the question-and-answer tasks as the initial filtering results.

[0033] The gated routing module is used to encode the hidden features of the question-answering task through a dynamic gated network, and to realize continuous differentiable expert activation by using the probabilistic sparse routing mechanism of Gumbel-Softmax, thereby achieving fine-grained dynamic expert selection.

[0034] The parameter fusion module is used to perform weighted summation of the parameter increments of each LoRA module based on the weights output by the dynamic gating network to obtain fusion parameters, and dynamically superimpose the fusion parameters onto the large language model to realize dynamic parameter injection and output inference results.

[0035] The real-time feedback optimization module is used to calculate the perplexity and confidence in real time after generating the inference result, and update and retrain the LoRA module pool based on the perplexity, the confidence and the feedback results of human evaluation.

[0036] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0037] (1) By innovatively developing the MoE network structure, we can capture multi-dimensional features of the dataset. At the same time, we can change the traditional Top-K expert selection structure to a more flexible fine-grained parameter activation mechanism and give the conditions for parameter reuse, thus realizing a faster and more efficient LoRA-MoE structure.

[0038] (2) In traditional MoE architectures, expert modules typically employ a homogeneous structure, making it difficult to capture the multi-dimensional feature differences of a task. This invention innovatively proposes a heterogeneous low-rank expert decomposition mechanism, dividing the expert module into three types of functionally complementary sub-modules: horizontal experts focus on global correlation analysis between feature dimensions, achieving semantic association modeling of word vectors through directional low-rank decomposition of the input feature matrix; vertical experts employ sliding window temporal convolution to capture local contextual dependencies; and cross-experts introduce a dual-gating mechanism to model the nonlinear interaction between features and temporal dimensions. This ternary heterogeneous design achieves, for the first time, collaborative modeling of feature correlation, temporality, and cross-dimensional interaction within the MoE framework.

[0039] (3) A parameter reuse enhancement strategy is proposed to address the redundancy problem: First, by sharing the basic adapter, all experts share the low-rank basis matrix, generating only a lightweight private projection matrix; second, incremental parameter updates are adopted, with each expert maintaining their own incremental matrix and achieving task-specific adjustments through residual connections; finally, a cross-layer inheritance mechanism is designed, where parameters are directly reused when the cosine similarity between higher and lower level experts is >0.7. This mechanism maintains the differentiation of expert functions while significantly compressing the number of parameters, thus significantly reducing computational overhead.

[0040] (4) Breakthrough improvements in differentiable dynamic routing and sparsity control mechanisms: Addressing the rigid selection defects of traditional Top-K routing, this invention proposes a probabilistic sparse routing (PSR) mechanism, achieving fine-grained expert control through three technological innovations: Continuously differentiable expert activation: Employing Gumbel-Softmax sampling technology, discrete expert selection is transformed into a continuous weight distribution, supporting soft combination activation of 0-3 experts. Combined with temperature coefficient adaptive scheduling, exploratory routing is encouraged in the early stages of training, gradually converging to deterministic selection in the later stages. Simultaneously, an entropy maximization regularization term is introduced into the loss function to force the gating network to evenly utilize expert resources, avoiding overload or idleness of some experts. This mechanism achieves optimal matching between expert combination and task characteristics under dynamic sparsity constraints. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart of the method of the present invention;

[0043] Figure 2 This is a structural diagram of the LoRA-MoE system of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] The purpose of this invention is to provide an efficient method for fine-tuning dynamic routing parameters based on LoRA-MoE, such as... Figure 1 As shown, the specific steps include the following:

[0046] S1. Construct a LoRA module pool based on a heterogeneous expert architecture using multi-domain datasets;

[0047] S2. Receive the question-answering task and perform coarse-grained filtering based on the semantic description of the input question, selecting several LoRA modules with the highest fit with the question-answering task as the preliminary filtering results;

[0048] S3. The hidden features of the question-answering task are encoded by a dynamic gating network, and the continuous differentiable expert activation is realized by using the probabilistic sparse routing mechanism of Gumbel-Softmax, so as to achieve fine-grained dynamic selection of experts.

[0049] S4. Based on the weights output by the dynamic gating network, the parameter increments of each LoRA module are weighted and summed to obtain the fusion parameters. The fusion parameters are then dynamically superimposed onto the large language model to achieve dynamic parameter injection and output the inference results.

[0050] S5. After generating the inference results, calculate the perplexity and confidence in real time, and update and retrain the LoRA module pool based on the perplexity, confidence and human evaluation feedback.

[0051] Furthermore, each LoRA module in the LoRA module pool contains low-rank decomposition parameters, task labels, embedding vectors, performance metrics, and activation frequency information. By designing heterogeneous low-rank expert modules, traditional MoE experts are split into horizontal experts to handle feature dimension correlations; vertical experts to capture temporal dependencies; and cross-experts to model cross-dimensional interactions. Nonlinear mappings are introduced to achieve collaborative modeling of feature dimensions, temporal dependencies, and cross-dimensional interactions. The specific process is as follows:

[0052] Lateral expert: The core of directional low-rank decomposition of the input feature matrix is ​​to capture the global correlation between feature dimensions through linear combinations of feature dimension columns, i.e., the semantic associations between different dimensions in the word vectors. The parameter passing formula is:

[0053] ;

[0054] in, , , Low-rank dynamic allocation, input features The output feature is .

[0055] Longitudinal expert: A sliding window is used to perform time-axis convolution on the input sequence to capture the contextual relationships between adjacent words in the text sequence. The output is then projected into a low-rank format, i.e.:

[0056] ;

[0057] in, , , , where k is the size of the sliding window, dynamically allocated in low rank. The mathematical model for Conv1D is:

[0058] ;

[0059] in, The output feature value represents the value at time position t and channel c in the output feature map; t represents the time index of the output feature map, c represents the channel index, k represents the size of the convolution kernel, and i is the index within the convolution kernel. The weight parameters of the convolution kernel represent the weight values ​​at position i and channel c of the convolution kernel. This represents the value of the input feature map, i.e., the time position of the input sequence. The value at channel c; p represents the fill size. This indicates the bias term.

[0060] Cross-expertise: Cross-dimensional interaction modeling, modeling the nonlinear interaction between the feature dimension and the temporal dimension. A dual-gating mechanism is introduced, first generating cross-dimensional interaction weights through nonlinear mapping, i.e.:

[0061] ;

[0062] in This is used to control the intensity of the interaction.

[0063] Furthermore, the parameter passing strategy in S1 is as follows:

[0064] Strategy 1: Shared basic adapter; all experts share the basic low-rank matrix. Generate each expert's private matrix through linear projection:

[0065] ,in ;

[0066] Strategy 2: Incremental parameters are updated independently, and each expert maintains their own incremental matrix. Parameter reuse is achieved through residual joins:

[0067] ;

[0068] Strategy 3: Cross-level inheritance mechanism: High-level experts use similarity detection:

[0069] ;

[0070] If the cosine similarity is greater than 0.7, inherit the underlying parameters:

[0071] ;

[0072] Each expert shares the basic adapter matrix and uses incremental adapters to achieve task-specific adjustments. During the module pool construction phase, the backbone network is frozen and only routing gating is trained. The expert capability differentiation loss function is used to promote the separation of expert functions.

[0073] Specifically, such as Figure 2 The diagram shown illustrates the system architecture of this invention, demonstrating the interactive process of dynamic routing, module pool optimization, and real-time feedback. In this embodiment, the LoRA-MoE-based efficient fine-tuning method for dynamic routing parameters proposed in this invention is applied to a medical question-answering scenario. Its core objective is to significantly improve the accuracy and robustness of a large language model in medical knowledge question-answering tasks under the constraint of limited parameter update volume, and to enhance the model's ability to understand complex semantics and cross-domain medical questions. The specific implementation steps are as follows:

[0074] S1. Construct a LoRA module pool for the medical field.

[0075] Based on the LLaMA-8B model, we performed domain-specific fine-tuning on datasets covering different medical subfields. The main fine-tuning parameters are as follows: Rank is set to 8; coefficient is 32; Dropout is 0.05; AdamW optimizer is used; learning rate is 2e-5; each module is trained for 2 rounds; approximately 10k samples are used per round.

[0076] The datasets include MedQA (medical question answering), PubMedQA (scientific abstract question answering), BioASQ (biomedical fact-based question answering), and Clinical-Trial-QA (clinical research text question answering), covering four major areas: disease diagnosis, pharmacological mechanisms, clinical case analysis, and medical reasoning. To ensure data consistency, all data were uniformly converted to Alpaca format: instruction is the user's question, input is supplementary background, and output is the standard answer. Duplicate samples were removed, and the label structure was normalized. During fine-tuning, the rank allocation strategy was adaptively adjusted based on gradient variance, assigning higher ranks to layers with faster feature changes and lower ranks to stable layers. After training, all LoRA modules were integrated into a LoRA module pool. Each module includes its task label, embedding vector, historical activation frequency, and performance metrics (PPL and EM scores) for subsequent dynamic selection.

[0077] S2. Upon receiving a new medical question-answering task, perform coarse-grained filtering based on the semantic description of the input question. The specific process is as follows:

[0078] S2.1, Construction of medical problem task vectors.

[0079] Suppose a user inputs the question: "I've been having persistent headaches lately, what could be the cause? Which department should I see?" The system uses a sentence vector encoding model to generate a question representation vector, while extracting keywords: "symptom description", "headache", "diagnostic suggestions", and "department guidance".

[0080] S2.2 Semantic matching and initial screening.

[0081] The cosine similarity between the input vector and the vector features generated from the training corpus of each LoRA module in the module pool is calculated to obtain the Top-3 similar modules.

[0082] S2.3 and PPL performance preliminary test.

[0083] Each selected LoRA module is dynamically loaded into the LLaMA-8B model, and inference is run on the medical question-answering validation set. The average perplexity (PPL) and accuracy (EM) of each combined model are recorded. The two modules with the best performance are selected for use in the fine-grained stage.

[0084] The input question is then converted into a 1024-dimensional embedding vector using a medical-specific Bio-Encoder model. Cosine similarity is calculated between this vector and the semantic description vectors of each module in the LoRA module pool to select Top-M (M=10) candidate modules. Next, the response features of these candidate modules in the hidden layers of the base model are extracted, and the feature distances between them and the task samples at different layers are calculated. A weighted sum of three metrics—cosine similarity, Euclidean distance, and Manhattan distance—is used to determine the five LoRA modules closest to the task distribution as coarse-grained candidates. Then, perplexity level (PPL) testing is performed on each candidate module combined with the base model to evaluate the generation quality on a small sample of medical question answering, and the modules are reordered in ascending order of PPL. Finally, the five modules with the highest combined similarity and PPL scores are selected as the input set for the fine-grained dynamic selection stage.

[0085] S3. In the fine-grained adaptation stage, a probabilistic sparse routing (PSR) mechanism based on Gumbel-Softmax is introduced to achieve continuously differentiable expert activation.

[0086] The gating network input is the hidden dynamic rank allocation vector (DRA) of the downstream task, with a dimension of 1024. This is compressed to 256 dimensions through grouped convolution (Group=4), and then low-rank decomposition (r=8) is used to generate the routing logits corresponding to each expert. Each expert represents a LoRA module, and the network output weights use a temperature coefficient. Initialize, and decrease by 5% every 1000 steps using an exponential decay strategy until convergence. By introducing Gumbel noise Calculate noisy Continuously distributed weights are generated using the Softmax function. It supports 0–3 experts to be activated as needed. The parameters of the gating network are initialized with a normal distribution and a bias of zero. An entropy maximization regularization term is added to the loss function to balance the frequency of expert calls.

[0087] During the inference phase, the weights calculated by the gating network Used to increment the parameters output by each LoRA module. Perform weighted fusion: The fusion results are dynamically superimposed onto the corresponding layers of the LLaMA-8B backbone network, achieving efficient adaptation specific to medical tasks. To prevent over-reliance on a few experts, the system monitors the average activation frequency of each module. When the activation rate of any expert falls below 5% in five consecutive rounds of inference, a route re-exploration process is automatically triggered to recalculate the hidden layer similarity and adjust the expert allocation.

[0088] S4. During model operation, establish a real-time feedback and module pool update mechanism.

[0089] After each inference, the perplexity score of the generated answer and the score assessed by medical professionals (composed of samples scored by medical language experts) are calculated. If the perplexity score (PPL) fluctuates by more than ±3% for three consecutive times, adaptive weight reallocation is triggered. The module pool is statistically analyzed based on the contribution of each LoRA module on the validation set (i.e., average activation frequency × answer accuracy). Modules with consistently low contributions are either left unused or removed, and lightweight LoRA retraining is automatically triggered to supplement new modules. Validated on the MedicalQA public test set, the model significantly improves answer accuracy, enhancing cross-domain question-answering generalization ability while maintaining significantly low computational overhead.

[0090] On the other hand, a LoRA-MoE-based system for efficient fine-tuning of dynamic routing parameters is provided, including a LoRA module pool construction module, a dynamic expert management module, a gated routing module, a parameter fusion module, and a real-time feedback optimization module.

[0091] The LoRA module pool building module is used to build a LoRA module pool based on a heterogeneous expert architecture based on multi-domain datasets.

[0092] The dynamic expert management module is used to receive question-answering tasks and perform coarse-grained filtering based on the semantic description of the input questions, selecting several LoRA modules with the highest fit to the question-answering task as the initial filtering results.

[0093] The gated routing module is used to encode the hidden features of the question-answering task through a dynamic gated network, and to realize continuous differentiable expert activation by using the probabilistic sparse routing mechanism of Gumbel-Softmax, thereby achieving fine-grained dynamic expert selection.

[0094] The parameter fusion module is used to perform weighted summation of the parameter increments of each LoRA module based on the weights output by the dynamic gating network to obtain fused parameters. The fused parameters are then dynamically superimposed onto the large language model to achieve dynamic parameter injection and output inference results.

[0095] The real-time feedback optimization module is used to calculate the perplexity and confidence in real time after generating inference results, and update and retrain the LoRA module pool based on the perplexity, confidence and human evaluation feedback results.

[0096] The overall architecture includes a LoRA module pool, routers, a base model, and a performance monitoring unit. The LoRA module pool stores LoRA modules trained in different medical fields and their metadata; the routers are responsible for receiving hidden layer features of the task and calculating expert weights to achieve fine-grained dynamic expert selection; the base model is an LLaMA-8B network with a frozen backbone, responsible for text generation and output; and the performance monitoring unit performs perplexity analysis and collects human feedback on the inference results.

[0097] During operation, the dynamic expert management module retrieves similar modules from the module pool based on the input task description vector and provides a candidate set to the router. The router employs a probabilistic sparse routing mechanism to achieve differentiable activation, calculates the weight distribution of each expert through a gating network, and selects Top-K (K=3) activation modules to participate in incremental fusion. The parameter fusion module then performs the parameter fusion based on the weights... The corresponding LoRA parameters are incrementally added to the backbone model to achieve dynamic parameter injection. After the output is generated, the real-time feedback optimization module calculates the PPL index and confidence score. When the performance is unstable or falls below the set threshold, the module pool update and retraining process is triggered to maintain the model's rapid response capability to emerging medical tasks.

[0098] Through the above system design, this invention achieves adaptive expert allocation and efficient parameter updating for different task characteristics in medical question-answering scenarios, significantly improving inference quality and domain generalization ability while ensuring model lightweightness.

[0099] In summary, this invention constructs a continuously optimized LoRA-MoE efficient fine-tuning framework by integrating dynamic routing, heterogeneous expert structure, and real-time feedback mechanism. This framework enables low-rank adaptation and expert collaboration in complex medical question-answering scenarios, thus providing a reliable solution for the rapid deployment of large models in vertical fields.

[0100] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A LoRA-MoE-based dynamic routing parameter efficient fine-tuning method, characterized in that, The specific steps include the following: A LoRA module pool based on a heterogeneous expert architecture is constructed based on a multi-field data set; An answer question task is received, and coarse-grained screening is performed according to a semantic description of an input question, and a plurality of LoRA modules with the highest adaptation degree to the answer question task are selected as a preliminary screening result; An answer question task hidden layer feature is encoded through a dynamic gating network, and continuous differentiable expert activation is realized by using a Gumbel-Softmax probability sparse routing mechanism to realize fine-grained dynamic selection of experts; Based on the weight output by the dynamic gating network, the parameter increments of each LoRA module are weighted and summed to obtain fused parameters, and the fused parameters are dynamically superimposed on a large language model to realize dynamic parameter injection and output a reasoning result; After generating the reasoning result, the perplexity and confidence are calculated in real time, and the LoRA module pool is updated and retrained according to the perplexity, the confidence and the artificial evaluation feedback result.

2. The LoRA-MoE-based dynamic routing parameter fine-tuning method according to claim 1, characterized in that, Each LoRA module in the LoRA module pool includes low-rank decomposition parameters, a task label, an embedding vector, a performance index and activation frequency information; the expert modules in the LoRA module pool are divided into horizontal experts, vertical experts and cross experts according to functions to realize collaborative modeling of feature dimensions, time sequence dependence and cross-dimension interaction.

3. The LoRA-MoE-based dynamic routing parameter fine-tuning method according to claim 2, characterized in that, The horizontal experts are used for directional low-rank decomposition of an input feature matrix to capture the semantic association of each dimension in a word vector; the vertical experts are used for time axis convolution of a sliding window set for an input sequence to capture the context relationship between adjacent words in a text sequence; and the cross experts are used for cross-dimension interaction modeling to realize nonlinear interaction between modeling feature dimensions and time sequence dimensions.

4. The LoRA-MoE-based dynamic routing parameter fine-tuning method according to claim 1, wherein, The specific steps of coarse-grained screening according to the semantic description of the input question are as follows: After receiving a question description, a sentence vector encoding model is used to generate a question representation vector; The question representation vector and the module description vectors in the LoRA module pool are calculated for cosine similarity to preliminarily select Top-M candidate modules; The response features of the Top-M candidate modules in the hidden layer of the base model are extracted, the feature distances between the Top-M candidate modules and the task samples at different layers are calculated, and the cosine similarity, Euclidean distance and Manhattan distance are weighted and summed to determine a plurality of LoRA modules closest to the task distribution; Perplexity detection is performed, the task adaptation degree is calculated based on the perplexity detection result, a plurality of LoRA modules with the highest adaptation degree are selected as the preliminary screening result, and the input set for the fine-grained dynamic selection stage is obtained.

5. The LoRA-MoE-based dynamic routing parameter fine-tuning method according to claim 1, characterized in that, The specific steps of realizing continuous differentiable expert activation by using the Gumbel-Softmax probability sparse routing mechanism are as follows: The input of the dynamic gating network is a dynamic rank allocation vector of a question hidden layer feature, and the input dimension is 1024; Group convolution is used to process the input feature, and the 1024-dimensional hidden layer vector is compressed to 256-dimensional, and then low-rank decomposition is performed to generate routing logits; Gumbel noise is generated for each expert, and noisy logits are calculated according to a temperature coefficient; A continuous distribution weight is generated by a Softmax function based on the noisy logits, and an entropy maximization regular term of the expert weight distribution is added to the loss function to balance the expert activation frequency.

6. The LoRA-MoE-based dynamic routing parameter fine-tuning method according to claim 5, characterized in that, The dynamic gating network adopts a single-layer multi-layer perception structure, the input is a dynamic rank allocation vector of the question and answer task hidden layer feature, and the output corresponds to the expert module weight.

7. The LoRA-MoE-based dynamic routing parameter fine-tuning method according to claim 1, wherein, When the fluctuation amplitude of the perplexity in a plurality of consecutive reasoning rounds exceeds a set threshold, a routing re-exploration mechanism is triggered to re-distribute the expert weight.

8. The LoRA-MoE-based dynamic routing parameter fine-tuning method according to claim 1, characterized in that, The parameter reuse mechanism also includes a parameter delivery strategy: Basic adapter sharing: all experts share a low-rank basis matrix, and generate private matrices for each expert through linear projection; Incremental parameter independent update: each expert maintains a dedicated incremental matrix, and realizes parameter reuse through residual connection; Cross-layer inheritance mechanism: when the cosine similarity between high-level experts and low-level experts exceeds a set threshold, the bottom layer parameters are directly reused.

9. The LoRA-MoE-based dynamic routing parameter fine-tuning method according to claim 1, characterized in that, The LoRA module pool maintenance adopts a contribution quantification strategy, calculates the average activation frequency of each LoRA module on the validation set, automatically eliminates low-contribution modules, triggers an automatic fine-tuning process to supplement new modules, and ensures that the coverage range of the module pool dynamically expands with the task distribution.

10. A LoRA-MoE based dynamic routing parameter efficient fine-tuning system, characterized in that, The LoRA module pool construction module, the dynamic expert management module, the gating routing module, the parameter fusion module, and the real-time feedback optimization module are included. The LoRA module pool construction module is configured to construct a LoRA module pool based on a heterogeneous expert architecture based on multi-domain datasets. The dynamic expert management module is configured to receive a question and answer task, and perform coarse-grained screening according to the semantic description of an input question, and select a plurality of LoRA modules with the highest adaptation degree to the question and answer task as a preliminary screening result. The gating routing module is configured to encode question and answer task hidden layer features through a dynamic gating network, realize continuous differentiable expert activation by using a Gumbel-Softmax probability sparse routing mechanism, and realize fine-grained dynamic selection of experts. The parameter fusion module is configured to perform weighted summation on the parameter increments of each LoRA module based on the weight output by the dynamic gating network, obtain fusion parameters, dynamically superimpose the fusion parameters on a large language model, realize dynamic parameter injection, and output a reasoning result. The real-time feedback optimization module is configured to calculate perplexity and confidence in real time after generating a reasoning result, and update and retrain the LoRA module pool according to the perplexity, the confidence, and the artificial evaluation feedback result.

Images