Multi-lora adaptive fusion inference method and system, electronic device, and storage medium

Abstract

The application relates to a multi-LoRA adaptive fusion reasoning method and system, electronic equipment and a storage medium. A plurality of groups of LoRA expert adapters are mounted on an attention projection layer and / or a feedforward projection layer of a pre-trained main model to form an LoRA expert library. Meanwhile, a gating routing module is configured on a target linear projection layer of the main model, and the gating routing module is used for dynamically evaluating and selecting experts according to a current layer token hidden state in a reasoning stage. When reasoning, an input text is encoded to obtain a token sequence and a corresponding hidden state, the gating routing module calculates expert scores for each token at each layer, performs top-k expert selection and obtains fusion weights; low-rank incremental outputs of selected LoRA experts are fused by weighting according to the weights, and then are superimposed with outputs of a main linear layer in a residual manner to obtain fusion layer outputs. The application can improve task adaptability and reasoning quality, reduce adapter switching and deployment and maintenance costs, and improve service throughput and stability.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and data processing technology, and in particular to a multi-LoRA adaptive fusion inference method, system, electronic device and storage medium based on gated routing. Background Technology

[0002] With the widespread application of large-scale pre-trained models in scenarios such as intelligent question answering, content generation, and multi-task reasoning, the industry is placing higher demands on the models' adaptability, inference quality, and service throughput under multi-task, multi-domain, and complex instruction inputs. To reduce the training and deployment costs of full-scale fine-tuning, parameter-efficient fine-tuning methods such as LoRA, which introduce low-rank incremental parameters into the linear layers of the backbone model to achieve task adaptation, have become common solutions.

[0003] However, existing LoRA adaptations mostly adopt a static model of "manually specifying adapters" or "single-task single-adapter". In scenarios with mixed multi-tasks, unclear instruction boundaries, or dynamically changing contexts, static selection often fails to match real-world needs, easily leading to inaccurate adaptation and fluctuations in inference quality. Simultaneously, frequent switching between multiple adapters increases deployment and maintenance complexity, impacting online service stability and throughput. Using fixed rule overlays or fixed weighted fusion of multiple LoRA experts makes it difficult to characterize the differentiated adaptation needs of different tokens and layers, hindering the full utilization of the complementary advantages of multiple experts.

[0004] Therefore, there is an urgent need for a technical solution that can perform gated routing based on the hidden states of each token at each layer during inference, realize top-k sparse selection and weight normalization fusion, and superimpose the fused low-rank increment with the backbone output residual; at the same time, a training method is needed that updates only the gated routing module while freezing the backbone and LoRA expert parameters, and combines multi-task balanced sampling and joint loss optimization to enable it to learn to adaptively select experts and fusion weights under different task and context conditions, so as to improve the adaptability, inference quality and service efficiency in multi-task and composite instruction scenarios. Summary of the Invention

[0005] In view of this, the purpose of this application is to propose a multi-LoRA adaptive fusion inference method, system, electronic device and storage medium based on gated routing to solve the above problems, so as to realize dynamic expert selection and fusion without manual adapter specification, improve task adaptability and inference quality in multi-task and compound instruction scenarios, reduce adapter switching and deployment and maintenance costs, and improve online service throughput and stability.

[0006] The technical solution of this invention to solve the above-mentioned technical problems is as follows: A multi-LoRA adaptive fusion inference method, comprising the following steps:

[0007] Step S101, obtain the pre-trained backbone model and LoRA expert library: by obtaining the pre-trained backbone model and the LoRA expert library consisting of n low-rank LoRA adaptors, the model foundation and adaptor expert resources are obtained.

[0008] Step S102, Mount LoRA experts and configure gated routing modules: Based on the backbone model and LoRA expert library obtained in step S101, mount multiple LoRA experts on the target linear projection layer of the backbone model and configure gated routing modules on the same layer to complete the model structure construction.

[0009] Step S103, Input Text Preprocessing and Hidden State Acquisition: Based on the model built in step S102, the input text to be processed is obtained. Token representations are obtained through word segmentation and encoding preprocessing, and then the hidden state vectors of each token in each layer are obtained through forward computation of the backbone. ;

[0010] Step S104, Gated route scoring and top-k expert selection: based on the hidden state vector obtained in step S103. Input it into the gated routing module, and output the routing score vector. Perform top-k selection to obtain the set of activated experts. ;

[0011] Step S105: Calculate the fusion weights and fuse the LoRA incremental output: based on the activation expert set obtained in step S104. The fusion weights are calculated by normalizing the route score s. Then, the low-rank incremental outputs of the activated experts are weighted and fused together. The total increment is obtained. ;

[0012] Step S106: Residual superposition yields the fusion layer output: based on the weighted fusion total increment obtained in step S105. The residual is superimposed with the backbone output of the target linear projection layer to obtain the fused layer output;

[0013] Step S107: Complete the forward computation and generate the final result: Based on the fusion layer output obtained in step 6, complete the subsequent forward computation of the backbone model, input the final output into the decoder / task head, and generate the target text or task result.

[0014] Step S108, Gated Routing Module Training: Based on the multi-task dataset, freeze the backbone and LoRA parameters, update only the gated routing module, and obtain the trained gated module through joint loss optimization.

[0015] In some possible implementations, step S101 specifically involves: directly loading the pre-trained backbone model; loading n sets of LoRA expert adapters trained independently for different tasks, combining them to form a LoRA expert library; each LoRA expert i contains a low-rank matrix pair. , ,satisfy , ,and ;

[0016] Step S102 specifically involves: attaching multiple experts from the LoRA expert database to at least one target linear projection layer in the attention module of the backbone model, including the Query projection layer, Key projection layer, Value projection layer, Output projection layer, and / or feedforward projection layer; setting up a gated routing module that corresponds one-to-one with the target linear projection layer, forming a layer-by-layer independent gated network; the input of the gated routing module may include the hidden state of the current layer token, the shared feature pooling vector, and the task meta-feature concatenation vector.

[0017] Step S101 reuses the pre-trained model and trained LoRA experts, eliminating the need for full retraining and significantly reducing training costs and time; it also builds a standardized expert library to provide a unified resource pool for subsequent multi-task adaptation. Step S102 implants adaptation capabilities into the key linear layers of the model to ensure adaptation accuracy; it independently configures gating layer by layer to achieve hierarchical differentiated routing; and it introduces task meta-features to improve the stability of adaptation in composite instruction scenarios.

[0018] In some possible implementations, step S103 specifically involves: performing word segmentation, embedding, and encoding on the input text to be processed, converting it into a token sequence that the model can accept; inputting the token sequence into the backbone model to perform forward computation, and outputting the hidden state vector corresponding to each token layer by layer. ;

[0019] Step S104 specifically involves: extracting the hidden state vector of the current layer and the current token. Input the gated routing module, which outputs routing scores s for n LoRA experts; perform top-k sparse selection based on the scores to select the set of LoRA experts to be activated. Each layer and each token executes the selection independently, and different tokens / layers can activate different experts to achieve dynamic switching and fusion of token level and layer.

[0020] Step S103 converts natural language into a vector representation that the model can compute; it obtains fine-grained semantic features at the token and hierarchical levels to provide accurate decision-making basis for gating routing. Step S104 dynamically selects experts at the token and hierarchical levels to adapt to fine-grained differences in context; top-k sparse selection reduces computation and improves inference throughput; it avoids full expert activation and reduces resource consumption.

[0021] In some possible implementations, step S105 specifically includes:

[0022] For sets The routing score within the area is normalized to obtain the fusion weight. ,satisfy:

[0023]

[0024] The i-th LoRA expert inputs the hidden state vector to the target linear projection layer. The low-rank increment output satisfies:

[0025]

[0026] in, This represents the input hidden state vector for the corresponding linear projection layer. , ,and , This is the scaling factor;

[0027] Weighted distribution across all activated experts We obtain the total increment by weighted summation. .

[0028] Step S105 weight normalization ensures stable fusion values; weighted fusion by score allows high-quality experts to contribute more; LoRA low-rank characteristics are preserved to maintain efficient adaptation.

[0029] In some possible implementations, step S106 specifically involves performing residual superposition calculations, as shown in the following formula:

[0030]

[0031] in, The backbone weight matrix of the target linear projection layer is... Input the hidden state vector to this layer. For the activated LoRA expert group;

[0032] Step S107 specifically involves: using the output of the fusion layer as the input of the next layer to complete the forward computation of all layers; and sending the final output of the model into the decoder or task output head to obtain the inference result.

[0033] Step S106: The residual structure retains the capabilities of the backbone model, avoiding adaptation that could damage basic performance; incremental fusion achieves smooth adaptation and improves output quality. Step S107: The backbone model's inference chain is fully preserved; the output after dynamic fusion better meets the needs of multi-task / composite instructions, improving inference accuracy.

[0034] In some possible implementations, step S108 specifically includes:

[0035] Obtain multi-task training data containing input text and task labels / adapter identifiers;

[0036] Freeze the backbone model and LoRA expert parameters, and only allow the gated routing module parameters to be updated;

[0037] The model output is obtained through forward computation, and the total loss is calculated. : ,in, Generate loss for the language model. For gated routing loss, The loss weighting coefficient;

[0038] Backpropagation only updates the gated routing parameters;

[0039] Multi-task balanced sampling is adopted to reduce the bias of high-frequency tasks.

[0040] Step S108 has extremely low training cost and does not disrupt existing models and experts; joint loss ensures routing accuracy and generation quality; balanced sampling improves the stability of composite instruction scenarios.

[0041] Based on the same inventive concept, a second aspect of the present invention provides a multi-LoRA adaptive fusion inference system for applying the above-described multi-LoRA adaptive fusion inference method, comprising:

[0042] The model and expert database acquisition module provides the backbone large model foundation and LoRA-adapted expert resources required for inference and training;

[0043] The mounting and configuration module completes the mounting of LoRA experts on the backbone model and the layer-by-layer deployment of the gated routing module;

[0044] The encoding and representation acquisition module converts the input text into token representations and extracts the hidden states of each token at each layer;

[0045] The gated routing module calculates expert scores based on hidden states to complete top-k selection and fusion weight calculation;

[0046] The incremental fusion module weights and fuses the LoRA increments and superimposes them with the backbone output residuals to obtain the fusion layer output.

[0047] The output generation module performs inference based on the fused backbone output and generates the final task result.

[0048] The gating routing module training module trains the gating routing module to learn expert selection and weight allocation strategies.

[0049] In some possible implementations, the gated routing module training module includes:

[0050] The data acquisition module collects multi-task training data, labels task tags / adapter identifiers, and performs balanced sampling;

[0051] The training model building module constructs a training model containing a backbone, a LoRA expert library, and gated routes, and freezes the backbone and LoRA parameters.

[0052] The joint optimization module calculates the training loss by forward computation to obtain the output; backpropagation only updates the gated routing parameters; iterative training continues until convergence, and the optimal gated parameters are saved.

[0053] Based on the same inventive concept, a third aspect of this application provides an electronic device, the electronic device comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device enables the above-described adaptive fusion inference method.

[0054] Based on the same inventive concept, the fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer's processor, causes the computer to perform the aforementioned adaptive fusion inference method.

[0055] This application provides a multi-LoRA adaptive fusion inference method, system, electronic device, and storage medium based on gated routing. First, a pre-trained backbone model and multiple groups of LoRA experts are loaded to form an expert library. LoRA experts are then attached to the target linear projection layer of the attention projection layer and / or feedforward projection layer, and a gated routing module is configured. Next, the input text is segmented and encoded, and the hidden state vectors of each token in each layer are obtained through forward computation. The gated routing module outputs expert routing scores based on these scores and performs top-k sparse selection to obtain the set of activated experts and fusion weights. Simultaneously, the low-rank incremental outputs of the activated LoRA experts are weighted and fused according to the fusion weights, and the residuals are superimposed onto the backbone linear projection layer output to obtain the fused layer output. Finally, the target output text or task result is generated based on the fused backbone network output.

[0056] The beneficial effects of this invention are: it can automatically match more suitable expert combinations in multi-task and composite instruction scenarios, improving task adaptability and inference quality; and during the training phase, freezing the backbone and LoRA experts and only updating the parameters of the gated routing module, combined with joint loss and balanced sampling strategies, can reduce training and maintenance costs, reduce deployment complexity caused by adapter switching, thereby improving online service throughput and stability. Attached Figure Description

[0057] Figure 1 A flowchart of a multi-LoRA adaptive fusion inference method based on gated routing provided in this application embodiment;

[0058] Figure 2 A schematic diagram of gated routing and multi-LoRA weighted fusion calculation for a single-layer linear projection layer provided in an embodiment of this application;

[0059] Figure 3 A schematic diagram of the gated routing module structure and the top-k sparse selection and weight normalization process provided in the embodiments of this application;

[0060] Figure 4 is a schematic diagram of the training process of the gated routing module provided in an embodiment of this application;

[0061] Figure 5 This is a schematic diagram of the mounting structure of the multi-LoRA expert library in the main large model provided in the embodiments of this application. Detailed Implementation

[0062] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0063] As described in the background section, LoRA adaptation in related technologies typically employs static selection or manually designated adapters. In scenarios with mixed inputs from multiple tasks, complex instructions, and unclear task boundaries, static selection is prone to inaccurate adaptation and fluctuations in inference quality. Simultaneously, switching between multiple adapters increases deployment and maintenance complexity and impacts online service stability and throughput. Therefore, this application proposes a technical solution that performs gated routing based on the hidden state vectors of each token at each layer during the inference phase, executes top-k expert selection, and adaptively fuses the low-rank incremental outputs of multiple LoRA experts. Furthermore, it provides a training method for the gated routing module, enabling it to learn stable routes and fusion weight allocations while freezing the backbone and LoRA experts, thereby reducing training and maintenance costs and improving routing stability.

[0064] Example 1

[0065] Reference Figure 1 The multi-LoRA adaptive fusion inference method based on gated routing provided in this application includes the following steps:

[0066] Step S101: Input Acquisition and Encoding Preprocessing

[0067] The input text to be processed is obtained, and the input text is preprocessed by word segmentation, embedding and encoding to obtain the token representation of the input sequence; during the forward propagation of the main model, the hidden state vector corresponding to each token in each layer is obtained.

[0068] Step S102: Obtaining the backbone model and LoRA expert database

[0069] Obtain the pre-trained backbone model and a LoRA expert library consisting of n low-rank LoRA fitting experts trained for different tasks. Each LoRA expert corresponds to a low-rank matrix pair. ,in , ,and .

[0070] Step S103: O+FFN Expert Mounting and Layer-by-Layer Gating Configuration

[0071] Multiple LoRA experts are mounted on the target linear projection layer of the backbone model, and a gated routing module is configured. Preferably, the target linear projection layer includes: the output projection layer O of the self-attention module and / or the linear projection layer of the feedforward network FFN.

[0072] The gated routing module is preferably configured layer by layer, associated with the target linear projection layer of the corresponding layer, and used to dynamically evaluate and select routes for the hidden state vector of the current token in the current layer. The LoRA expert library can be a shared expert library, with gated routing modules at different layers selecting the set of activated experts from the same expert library as needed.

[0073] In some embodiments, the shared expert database reuses the same set of LoRA expert parameters across different layers, but the gated routing modules at each layer output (I, w) independently, thereby achieving hierarchical adaptive fusion of "parameter sharing and hierarchical decision-making".

[0074] In some embodiments, the gated routing module can reuse the same set of routing results (I, w) for the O projection layer and the FFN projection layer within the same layer to reduce routing calculation overhead and improve online inference throughput; or it can set up independent gated routing modules for the O projection layer and the FFN projection layer respectively to improve fine-grained adaptation capabilities.

[0075] Step S104: Layer-by-layer, token-by-token routing scoring and top-k expert selection

[0076] During inference, for each token hidden state vector of the target linear projection layer... ,Will Input the gated routing module and output the routing score vectors for n LoRA experts. Based on the scores, a top-k sparse selection process is performed to obtain the set of activated experts. ,in .

[0077] In some embodiments, the gated routing module can calculate the routing confidence (e.g., based on the score entropy or maximum value interval) after outputting the routing score, and adaptively adjust the size of k or trigger a backoff strategy accordingly to reduce the number of activated experts with simple inputs and improve fusion capabilities with complex inputs.

[0078] In some embodiments, the top-k selection may incorporate a temperature coefficient or threshold sparsity strategy to make the weight distribution smoother or sparser, thereby improving routing stability.

[0079] Step S105: Weight normalization and LoRA incremental weighted fusion and residual superposition

[0080] For sets The internal expert scores were normalized to obtain the fusion weight. ,satisfy:

[0081]

[0082] The activated first The low-rank incremental output generated by each LoRA expert satisfies:

[0083]

[0084] The low-rank incremental outputs of the activated experts are weighted and summed according to the fusion weights, and then residually superimposed with the output of the backbone linear projection layer to obtain the fusion layer output:

[0085]

[0086] in The weight matrix of the main linear projection layer. This is the scaling factor.

[0087] In some embodiments, the LoRA increment obtained by weighted fusion is... Apply amplitude constraints or clipping to suppress incremental explosions caused by anomalous routing and improve inference stability.

[0088] In some embodiments, a learnable or configurable residual scaling factor is introduced. To ensure that the fused output meets the requirements ,in It can vary with the layer to adapt to the representation scale of different layers.

[0089] Step S106: Output Generation

[0090] The subsequent forward computation of the backbone model is completed based on the fused layer output, and the final output is input into the decoder or output head to generate the target output text or task result, thereby realizing multi-LoRA adaptive fusion and dynamic switching by token and by layer.

[0091] Reference Figure 2 , Figure 2 This illustrates the computational process at a single linear projection layer: Input hidden state vector Simultaneously, it enters the main linear transformation branch (and obtains) The gated routing branch outputs a score and performs top-k and weight normalization; then, the low-rank incremental outputs of the selected experts' LoRA are weighted and fused to obtain the final score. The fusion result is obtained by superimposing the residual of the main output. .

[0092] Reference Figure 3 The internal process of the gated routing module can be summarized as follows: the input is the current layer token and the hidden state vector. The gated routing module outputs expert routing scores. Perform top-k expert selection on the ratings to obtain the set of activated experts. ; For sets The internal scores are normalized to obtain the fusion weights. (Only for) (Non-zero, otherwise set to 0); Output routing results. This will allow for subsequent access to the expert database and completion of incremental weighted fusion.

[0093] In some embodiments, the input of the gated routing module may optionally be concatenated with at least one of the following: task identifier, instruction type, language identifier, domain identifier, or context length statistical features, to enhance routing stability in complex instruction scenarios.

[0094] In some embodiments, the gated routing module is a lightweight gated network that includes linear mapping and nonlinear activation to map the input hidden state to expert scores; optionally, the gated routing module simultaneously accesses sequence-level pooled vectors and task meta-features to achieve joint routing of token-level information and global semantic information.

[0095] Reference Figure 4 To enable the gated routing module to select appropriate LoRA experts and their fusion weights under different tasks and contexts, this application provides a training process for the gated routing module. The training focuses on teaching the gate to output reasonable... During the training phase, the backbone and LoRA expert parameters are kept frozen. This process includes:

[0096] Obtain training samples, which include training input text and task labels or adapter identifiers corresponding to the training input text;

[0097] With the main model and LoRA expert parameters frozen, forward computation is performed to obtain the hidden state vectors of each token in each layer. ;

[0098] right Expert scores are output by the gated routing module. And perform top-k to obtain the active set. For sets Internal score normalization yields fusion weights ;

[0099] Construct a route supervision signal based on the task label or adapter identifier and calculate the gated route loss. ; the language model generation loss With gated routing loss Form a joint optimization objective:

[0100]

[0101] Backpropagation is performed on the joint loss, and only the parameters of the gated routing module are updated, so that the gated routing module learns the expert selection strategy and fusion weight allocation under different tasks and different context conditions.

[0102] After training is complete, the parameters of the trained gating routing module are output and used for layer-by-layer, token-by-token routing during the inference phase.

[0103] In some embodiments, a balanced sampling strategy may be optionally applied to multi-task samples, or a routing stability / load balancing regularization term may be added to reduce the bias towards high-frequency tasks and suppress expert selection collapse.

[0104] Reference Figure 5The input text is segmented and embedded to obtain a sequence representation, which is then fed into the frozen backbone model (containing multiple Transformer Blocks). Within each Block, LoRA expert incremental computation capabilities are preferably mounted at the output projection layer O of the self-attention module and the linear projection layer of the feedforward network FFN. The gating routing module is configured layer by layer, receiving the hidden state vector of the current token in the current layer and outputting... It selects activated experts from the shared LoRA expert database and calculates the weighted LoRA increment. Finally, the weighted increment is superimposed on the corresponding O or FFN backbone output in the form of residuals, thereby realizing dynamic expert selection and fusion by layer and by token across the entire network hierarchy.

[0105] As can be seen, this embodiment mounts a shared LoRA expert library on the O projection layer and FFN projection layer of the backbone model, and configures a gated routing module in each layer to score the hidden state of each token, perform top-k selection and normalized fusion. This allows the model to dynamically call appropriate expert combinations and complete incremental fusion during the inference phase without manually specifying adapters. At the same time, the backbone and LoRA experts are frozen during the training phase, and only the parameters of the gated routing module are updated. The stability of routing can be improved by combining joint loss and sampling / regularization strategies. This improves task adaptability and inference quality in multi-task and compound instruction scenarios, reduces adapter switching and deployment and maintenance costs, and improves online service throughput and stability.

[0106] Example 2

[0107] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a multi-LoRA adaptive fusion inference system based on gated routing, comprising:

[0108] The model and expert database acquisition module is used to acquire the pre-trained backbone model and the LoRA expert database consisting of n LoRA low-rank adapting experts.

[0109] The mounting and configuration module is used to mount multiple LoRA experts at at least one target linear projection layer in the attention projection layer and / or feedforward projection layer of the backbone model, and to configure a gated routing module at the target linear projection layer.

[0110] The encoding and representation acquisition module is used to acquire the input text and perform encoding preprocessing to obtain the token representation, and obtain the hidden state vector of each token in each layer in the forward computation;

[0111] The gated routing module is used to output a routing score based on the hidden state vector of the current layer token and perform top-k expert selection to obtain the set of activated experts and the fusion weights;

[0112] The incremental fusion module is used to weight and fuse the low-rank incremental outputs of activated LoRA experts according to the fusion weights and then perform residual superposition with the outputs of the backbone linear projection layer to obtain the fused layer outputs.

[0113] The output generation module is used to generate target output text or task results based on the fused backbone network output.

[0114] The gating routing module training module trains the gating routing module to learn expert selection and weight allocation strategies.

[0115] The gated routing module training module includes:

[0116] The data acquisition module is used to acquire a multi-task training dataset containing training input text and task labels or adapter identifiers;

[0117] The training model building module is used to attach n LoRA low-rank adaptation experts and configure the gated routing module at at least one target linear projection layer in the attention projection layer and / or feedforward projection layer of the pre-trained backbone model.

[0118] The joint optimization module is used to perform forward computation based on the training input text and calculate the training loss, and to update the parameters of the gated routing module through backpropagation based on the training loss.

[0119] Example 3

[0120] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides an electronic device, the electronic device comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device enables the electronic device to implement any of the above embodiments.

[0121] Example 4

[0122] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, cause the computer to execute the multi-LoRA adaptive fusion inference method based on gated routing and the inference process of the gated routing module as described in any of the above embodiments.

[0123] The computer-readable storage medium of this embodiment includes permanent and non-permanent, removable and non-removable media, which can be used to store information by any method or technology. The information may be computer-readable instructions, data structures, program modules, or other data. Examples of the computer-readable storage medium include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other storage technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transfer medium for storing information accessible by a computing device.

[0124] In some embodiments, the computer instructions are used to cause the computer to perform the following operations: acquire input text and perform word segmentation and encoding preprocessing to obtain token representations and corresponding hidden state vectors; load a pre-trained backbone model and a shared expert library consisting of n LoRA low-rank adaptation experts; attach LoRA experts to the output projection layer O of the self-attention module and / or the linear projection layer of the feedforward network FFN and configure gated routing modules layer by layer; during the inference phase, the gated routing module outputs a routing score based on the current layer's token hidden state vector and performs top-k expert selection and weight normalization; weightedly fuse the low-rank incremental outputs of the activated LoRA experts and superimpose them with the backbone output residuals to complete subsequent forward computation and generate output text or task results.

[0125] In other embodiments, the computer instructions are also used to cause the computer to perform a gated routing module training process, including: acquiring training samples containing training input text and task labels or adapter identifiers; freezing the backbone model and LoRA expert parameters; outputting routing scores from the gated routing module based on the hidden state vectors of each token in each layer and performing top-k and weight normalization; calculating the gated routing loss and optionally combining it with the language model generation loss to form a joint optimization objective; updating only the gated routing module parameters through backpropagation; and saving the trained gated routing module parameters for layer-by-layer routing and fusion during the inference phase.

[0126] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the gated routing multi-LoRA adaptive fusion method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments. For example, without updating the backbone large model and LoRA expert parameters, dynamic expert selection and fusion of multi-task and multi-scenario inputs are realized through layer-by-layer gating, which improves the model's adaptability in multi-task transfer and generalization scenarios, while reducing training and deployment costs and improving the controllability and stability of the inference process. Further details will not be elaborated here.

[0127] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application, which are not provided in detail for the sake of brevity.

[0128] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the provided drawings may or may not show known power / ground connections to integrated circuit chips and other components. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be within the understanding of those skilled in the art).

[0129] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, different gated network structures can be used to implement route scoring; gated routing modules can be arranged at different levels (by block, by sub-layer, or by linear projection layer); the top-k selection strategy or normalization method can be changed; different forms of routing regularization can be optionally introduced to improve stability; or LoRA experts can be mounted at different projection positions of attention and feedforward, etc.

[0130] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-LoRA adaptive fusion inference method, characterized in that, The steps include the following: Step S101, obtain the pre-trained backbone model and LoRA expert library: by obtaining the pre-trained backbone model and the LoRA expert library consisting of n low-rank LoRA adaptors, the model foundation and adaptor expert resources are obtained. Step S102, Mount LoRA experts and configure gated routing modules: Based on the backbone model and LoRA expert library obtained in step S101, mount multiple LoRA experts on the target linear projection layer of the backbone model and configure gated routing modules on the same layer to complete the model structure construction. Step S103, Input Text Preprocessing and Hidden State Acquisition: Based on the model built in step S102, the input text to be processed is obtained. Token representations are obtained through word segmentation and encoding preprocessing, and then the hidden state vectors of each token in each layer are obtained through forward computation of the backbone. ; Step S104, Gated route scoring and top-k expert selection: based on the hidden state vector obtained in step S103. Input it into the gated routing module, and output the routing score vector. Perform top-k selection to obtain the set of activated experts. ; Step S105: Calculate the fusion weights and fuse the LoRA incremental output: based on the activation expert set obtained in step S104. The fusion weights are calculated by normalizing the route score s. Then, the low-rank incremental outputs of the activated experts are weighted and fused together. The total increment is obtained. ; Step S106: Residual superposition yields the fusion layer output: based on the weighted fusion total increment obtained in step S105. The residual is superimposed with the backbone output of the target linear projection layer to obtain the fused layer output; Step S107: Complete the forward computation and generate the final result: Based on the fusion layer output obtained in step 6, complete the subsequent forward computation of the backbone model, input the final output into the decoder / task head, and generate the target text or task result. Step S108, Gated Routing Module Training: Based on the multi-task dataset, freeze the backbone and LoRA parameters, update only the gated routing module, and obtain the trained gated module through joint loss optimization.

2. The multi-LoRA adaptive fusion inference method according to claim 1, characterized in that, Step S101 specifically involves: directly loading the pre-trained backbone model; loading n sets of LoRA expert adapters trained independently for different tasks, combining them to form a LoRA expert library; each LoRA expert i contains a low-rank matrix pair. , ,satisfy , ,and ; Step S102 specifically involves attaching multiple experts from the LoRA expert library to at least one target linear projection layer of the attention module of the backbone large model, which includes the Query projection layer, Key projection layer, Value projection layer, Output projection layer, and / or feedforward projection layer. The gated routing module is set up in a one-to-one correspondence with the target linear projection layer, forming an independent gated network layer by layer; the input of the gated routing module can include the hidden state of the current layer token, the shared feature pooling vector, and the task meta-feature concatenation vector.

3. The multi-LoRA adaptive fusion inference method according to claim 1, characterized in that, Step S103 specifically involves: performing word segmentation, embedding, and encoding on the input text to be processed, converting it into a token sequence that the model can accept; inputting the token sequence into the main backbone model to perform forward computation, and outputting the hidden state vector corresponding to each token layer by layer. ; Step S104 specifically involves: extracting the hidden state vector of the current layer and the current token. Input the gated routing module, and the module outputs a routing score s for n LoRA experts; Perform top-k sparse selection based on scores to select the set of LoRA experts that need to be activated. Each layer and each token executes the selection independently, and different tokens / layers can activate different experts to achieve dynamic switching and fusion of token level and layer.

4. The multi-LoRA adaptive fusion inference method according to claim 1, characterized in that, Step S105 is as follows: For sets The routing score within the area is normalized to obtain the fusion weight. ,satisfy: The i-th LoRA expert inputs the hidden state vector to the target linear projection layer. The low-rank increment output satisfies: in, This represents the input hidden state vector for the corresponding linear projection layer. , ,and , This is the scaling factor; Weighted distribution across all activated experts We obtain the total increment by weighted summation. .

5. The multi-LoRA adaptive fusion inference method according to claim 1, characterized in that, Step S106 specifically involves performing residual superposition calculation, using the following formula: in, The backbone weight matrix of the target linear projection layer is... Input the hidden state vector to this layer. For the activated LoRA expert group; Step S107 specifically involves: using the output of the fusion layer as the input of the next layer to complete the forward computation of all layers; and sending the final output of the model into the decoder or task output head to obtain the inference result.

6. The multi-LoRA adaptive fusion inference method according to claim 1, characterized in that, Step S108 is as follows: Obtain multi-task training data containing input text and task labels / adapter identifiers; Freeze the backbone model and LoRA expert parameters, and only allow the gated routing module parameters to be updated; The model output is obtained through forward computation, and the total loss is calculated. : ,in, Generate loss for the language model. For gated routing loss, The loss weighting coefficient; Backpropagation only updates the gated routing parameters; Multi-task balanced sampling is adopted to reduce the bias of high-frequency tasks.

7. A multi-LoRA adaptive fusion inference system, used to apply the multi-LoRA adaptive fusion inference method as described in any one of claims 1-6, characterized in that, include: The model and expert database acquisition module provides the backbone large model foundation and LoRA-adapted expert resources required for inference and training; The mounting and configuration module completes the mounting of LoRA experts on the backbone model and the layer-by-layer deployment of the gated routing module; The encoding and representation acquisition module converts the input text into token representations and extracts the hidden states of each token at each layer; The gated routing module calculates expert scores based on hidden states to complete top-k selection and fusion weight calculation; The incremental fusion module weights and fuses the LoRA increments and superimposes them with the backbone output residuals to obtain the fusion layer output. The output generation module performs inference based on the fused backbone output and generates the final task result. The gating routing module training module trains the gating routing module to learn expert selection and weight allocation strategies.

8. The multi-LoRA adaptive fusion inference system according to claim 7, characterized in that, The gated routing module training module includes: The data acquisition module collects multi-task training data, labels task tags / adapter identifiers, and performs balanced sampling. The training model building module constructs a training model containing a backbone, a LoRA expert library, and gated routes, and freezes the backbone and LoRA parameters. The joint optimization module calculates the training loss by forward computation to obtain the output; backpropagation only updates the gated routing parameters; iterative training continues until convergence, and the optimal gated parameters are saved.

9. An electronic device, characterized in that, The electronic device includes: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the multi-LoRA adaptive fusion inference method as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the multi-LoRA adaptive fusion inference method as described in any one of claims 1-6.

Images