Hybrid expert model training method, system and device based on LORA technology and medium

Abstract

The invention discloses a mixed expert model training method, system and device based on an LORA technology and a medium, mainly relates to the technical field of expert model training, and aims to solve the problems that in a multi-task learning scene, a single LoRA adjustment method may not fully cope with diversity and subtle differences between different tasks, needs of all tasks are difficult to consider, and the training efficiency is high. Therefore, the overall performance is reduced. Comprising the following steps: dynamically selecting a LoRA layer from a plurality of LoRA layers for parameter adjustment through a gating network according to training data corresponding to a current task in a process of training by utilizing a preset fine tuning data set, and adjusting the output of the LoRA layer through the gating network to obtain the output of a pre-training network; and after the training of the multi-task pre-training model and the LoRA layer is completed, freezing the parameters of the multi-task pre-training model and the LoRA layer.

Description

Technical Field

[0001] The present application relates to the technical field of hybrid expert models, and in particular to a hybrid expert model training method, system, device and medium based on LORA technology. Background Art

[0002] In the field of deep learning, especially natural language processing, pre-trained models have demonstrated their excellent ability to handle complex language tasks by virtue of their pre-training on large-scale text data. However, when faced with specific tasks or data sets, it is often difficult to achieve the best results by directly applying these pre-trained models, so they need to be fine-tuned to adapt to new requirements.

[0003] The existing full parameter fine-tuning method mainly uses LoRA (Long Range Radio based on spread spectrum technology) technology to insert a trainable low-rank matrix in a specific layer of the pre-trained model to update the parameters. However, in the multi-task learning (MTL) scenario, a single LoRA adjustment method may not be able to fully cope with the diversity and subtle differences between different tasks. Since different tasks may require the model to focus on different features or knowledge representations, a single LoRA adjustment often fails to take into account the needs of all tasks, resulting in a decrease in overall performance.

[0004] Therefore, in order to further improve the processing capabilities of the model in multi-task learning scenarios, a hybrid expert model training method, system, equipment and medium based on LoRa technology are urgently needed to solve the problem that in multi-task learning scenarios, a single LoRA adjustment method may not be able to fully cope with the diversity and subtle differences between different tasks, and it is difficult to take into account the needs of all tasks, thus resulting in a decline in overall performance. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present application provides a hybrid expert model training method, system, device and medium based on LoRa technology to solve the problem that in the existing multi-task learning scenario, a single LoRA adjustment method may not be able to fully cope with the diversity and subtle differences between different tasks, and it is difficult to take into account the needs of all tasks, thus resulting in a decline in overall performance.

[0006] In a first aspect, the present application provides a hybrid expert model training method based on LORA technology, the method comprising: According to a preset performance selection procedure, a multi-task pre-trained model that meets the preset performance requirements is determined from several multi-task pre-trained models; a preset fine-tuning data set of the Chinese environment corresponding to the tasks in the multi-task pre-trained model is created; using the preset fine-tuning data set, several LoRA layers corresponding to the Transformer block in the multi-task pre-trained model are trained; wherein, in the process of training using the preset fine-tuning data set, a LoRA layer is dynamically selected from several LoRA layers according to the training data corresponding to the current task through a gating network for parameter adjustment, and the output of the LoRA layer is adjusted through the gating network to obtain the output of the pre-trained network; after completing the training of the multi-task pre-trained model and the LoRA layer, the parameters of the multi-task pre-trained model and the LoRA layer are frozen; the maximum number of tokens processed by each LoRA layer in a single batch and the preset multi-objective optimization strategy are obtained, and load balancing optimization of the operation of the multi-task pre-trained model is performed.

[0007] The hybrid expert model training method provided in the embodiment of the present application solves the problem that in the existing multi-task learning scenario, a single LoRA adjustment method may not be able to fully cope with the diversity and subtle differences between different tasks, and it is difficult to take into account the needs of all tasks, resulting in a decrease in overall performance, by adding data of several LoRA layers to the Transformer block in the training multi-task pre-training model. In addition, the LoRA layer is dynamically selected from several LoRA layers according to the training data corresponding to the current task through the gating network for parameter adjustment, and the output of the LoRA layer is adjusted through the gating network to obtain the output of the pre-training network; the above enables the present application to intelligently select and combine different LoRA layers for fine-tuning according to the characteristics of different tasks, so as to achieve accurate adaptation and efficient training between tasks. At the same time, on the basis of retaining the stable inheritance of LoRA expert characteristics, the LoRA combination can be optimized to improve the overall performance, and the contributions between different LoRA experts can be balanced to avoid the risk of overfitting.

[0008] In one implementation of the present application, the method also includes: in the model inference stage of the trained multi-task pre-trained model, following the parameter freezing principle, freezing the parameters of the multi-task pre-trained model and the LoRA layer except for the gated network.

[0009] In one implementation of the present application, a LoRA layer is dynamically selected from several LoRA layers for parameter adjustment according to the training data corresponding to the current task through a gating network, specifically including: By formula: ; ; = ; Calculate the activation weight of the i-th LoRA layer corresponding to task t ;in, represents the embedding representation of task t; , , and is the preset value; N represents the total number of LoRA layers; By formula: , in order to adapt to task t, update the parameters of the Lth layer of the multi-task pre-trained model ; in, Indicates the parameters of the Lth layer before updating, Represents the low-rank update matrix provided by the i-th LoRA layer to the L-th layer.

[0010] In one implementation of the present application, the output of the LoRA layer is adjusted by a gating network to obtain the output of the pre-trained network, specifically including: Get the input data x of the LoRA layer; By formula: , concatenate and normalize the outputs of all LoRA layers to obtain data ; in, represents the output of the i-th LoRA layer, and ,in, represents the feedforward neural network layer, LN represents layer normalization, represents the parameters learned by the i-th LoRA layer of the current task during the training process, and N represents the number of all LoRA layers; By formula: , calculate the gated input of each LoRA layer; By formula: , calculate the weight of the i-th LoRA layer; where τ represents the preset temperature parameter; By formula: , calculate the weighted sum of all LoRA layers output by the gating network; By formula: , the output of the gating network Added to the output of multi-task pre-trained models In the output of the pre-trained network, .

[0011] In one implementation of the present application, the method further includes: By formula: ; ; Calculate the total loss function of the multi-task pre-trained model ; in, represents the loss function corresponding to the task, represents a preset constant, and ∈(0,1), represents the actual label of the configuration, Represents the preset balancing hyperparameters.

[0012] In one implementation of the present application, the maximum number of tokens processed by each LoRA layer in a single batch and the preset multi-objective optimization strategy are obtained to perform load balancing optimization of the multi-task pre-training model operation, specifically including: By setting the expert capacity threshold formula: , calculate the maximum number of tokens processed by the i-th LoRA layer in a single batch ; in, represents the minimum value of the preset capacity threshold of the i-th LoRA layer, represents the maximum value of the preset capacity threshold of the i-th LoRA layer, represents the average of the first k-1 batch requests, Indicates the preset adjustment step size, Indicates the maximum number of tokens processed by the i-th LoRA layer in a single batch before optimization.

[0013] In one implementation of the present application, the maximum number of tokens processed by each LoRA layer in a single batch and the preset multi-objective optimization strategy are obtained to perform load balancing optimization of the multi-task pre-training model operation, specifically including: By solving the preset multi-objective optimization strategy formula: ; Calculate and obtain a solution set X that reaches the maximum number of iterations or the improvement of the solution set is less than a preset threshold; in, Represents the variance of the LoRA layer load corresponding to the current solution set X, represents the average system response time corresponding to the current solution set X, represents the resource utilization corresponding to the current solution set X, , , Indicates the preset weight coefficient.

[0014] In the second aspect, the present application provides a hybrid expert model training system based on LORA technology, the system comprising: The pre-trained model preparation module is used to select a program according to preset performance and determine a multi-task pre-trained model that meets the preset performance requirements from several multi-task pre-trained models; the data set construction module is used to create a preset fine-tuning data set for the Chinese environment corresponding to the tasks in the multi-task pre-trained model; the network fine-tuning module is used to use the preset fine-tuning data set to train several LoRA layers corresponding to the Transformer block in the multi-task pre-trained model; wherein, in the process of training using the preset fine-tuning data set, the LoRA layer is dynamically selected from several LoRA layers according to the training data corresponding to the current task through the gating network for parameter adjustment, and the output of the LoRA layer is adjusted through the gating network to obtain the output of the pre-trained network; the model processing module is used to freeze the parameters of the multi-task pre-trained model and the LoRA layer after completing the training of the multi-task pre-trained model and the LoRA layer; the maximum number of tokens processed by each LoRA layer in a single batch and the preset multi-objective optimization strategy are obtained to perform load balancing optimization for the operation of the multi-task pre-trained model.

[0015] In a third aspect, the present application provides a hybrid expert model training device based on LORA technology, the device comprising: processor; And a memory having executable code stored thereon, which, when the executable code is executed, enables the processor to execute a hybrid expert model training method based on LORA technology as any one of the above items.

[0016] In a fourth aspect, the present application provides a non-volatile computer storage medium on which computer instructions are stored. When the computer instructions are executed, a hybrid expert model training method based on LORA technology as described above is implemented.

[0017] Those skilled in the art can understand that the present application has at least the following beneficial effects: The present application discloses a hybrid expert model training method, system and medium based on LORA technology. By configuring several LoRA layers in the preset layers or modules of the training multi-task pre-training model, introducing multi-task LoRA fine-tuning based on the gating mechanism, hierarchical LoRA combination optimization, parameter freezing strategy, intelligent load balancing and other technical means, accurate adaptation and efficient training of different tasks are achieved, and a new solution is provided for multi-task learning in the field of deep learning. It solves the problem that a single LoRA adjustment method may not be able to fully cope with the diversity and subtle differences between different tasks in the existing multi-task learning scenario, and it is difficult to take into account the needs of all tasks, resulting in a decrease in overall performance. In addition, the LoRA layer is dynamically selected from several LoRA layers according to the training data corresponding to the current task through the gating network to adjust the parameters, and the output of the LoRA layer is adjusted through the gating network to obtain the output of the pre-training network; the above makes the present application intelligently select and combine different LoRA layers for fine-tuning according to the characteristics of different tasks, so as to achieve accurate adaptation and efficient training between tasks. At the same time, it can optimize the LoRA combination and improve the overall performance on the basis of retaining the stable inheritance of the LoRA expert characteristics, and balance the contributions between different LoRA experts to avoid the risk of overfitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Some embodiments of the present disclosure are described below with reference to the accompanying drawings, in which: Figure 1 This is a flow chart of a hybrid expert model training method based on LORA technology provided in an embodiment of the present application.

[0019] Figure 2 It is a schematic diagram of the internal structure of a hybrid expert model training system based on LORA technology provided in an embodiment of the present application.

[0020] Figure 3 This is a schematic diagram of the internal structure of a hybrid expert model training device based on LORA technology provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] It should be understood by those skilled in the art that the embodiments described below are only preferred embodiments of the present disclosure, and do not mean that the present disclosure can only be implemented through the preferred embodiments. The preferred embodiments are only used to explain the technical principles of the present disclosure, and are not used to limit the protection scope of the present disclosure. Based on the preferred embodiments provided by the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work should still fall within the protection scope of the present disclosure.

[0022] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0023] The technical solution proposed in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0024] The embodiment provides a hybrid expert model training method based on LORA technology, such as Figure 1 As shown, the method provided in the embodiment of the present application mainly includes the following steps: Step 110: According to a preset performance selection procedure, a multi-task pre-trained model that meets the preset performance requirements is determined from a plurality of multi-task pre-trained models.

[0025] It should be noted that the preset performance selection program is used to select multi-task pre-trained models with excellent performance (for example, Qwen2), and then the hardware resources will be planned and configured. While ensuring that the model runs efficiently and stably, considering the parameter efficiency of LoRA, the selected model should be flexible enough to adapt to different fine-tuning tasks.

[0026] Step 120: Create a preset fine-tuning dataset for the Chinese environment corresponding to the tasks in the multi-task pre-training model.

[0027] It should be noted that the fine-tuning dataset (preset fine-tuning dataset) for the Chinese environment covers scenarios such as summary generation, intelligent question answering, and text classification. The dataset should be diverse to fully optimize the multi-task processing capabilities of the model. During the fine-tuning process of the model, the following formula can be used to update the model parameters: ; in, represents the fine-tuned model parameters, represents the pre-trained model parameters, denotes the parameters learned by fine-tuning.

[0028] For multi-task learning scenarios, this application can define a composite loss function, which is the weighted sum of the loss functions of each task: ; in, is the total loss function, N is the number of tasks, is the loss function of the i-th task (such as cross entropy loss, mean square error loss, etc.), is the preset weight of the i-th task, which is used to balance the impact of different tasks on model training.

[0029] Step 130, using a preset fine-tuning data set to train several LoRA layers corresponding to the Transformer block in the multi-task pre-training model; wherein, in the process of training using the preset fine-tuning data set, a LoRA layer is dynamically selected from several LoRA layers according to the training data corresponding to the current task through a gating network for parameter adjustment, and the output of the LoRA layer is adjusted through the gating network to obtain the output of the pre-training network.

[0030] It should be noted that in multi-task learning scenarios, different tasks may require the model to focus on different features or knowledge representations. As a parameter-efficient fine-tuning method, LoRA adapts to new tasks by adding low-rank matrices without adjusting all parameters of the pre-trained model. In order to further improve the performance of LoRA in multi-task scenarios, this application can regard each LoRA layer as an "expert" and design a gating network to intelligently select the most appropriate LoRA expert (LoRA layer) for fine-tuning.

[0031] In addition, the goal of the gating network is to assign a weight to each LoRA layer (i.e., expert) based on the characteristics or context information of the current task, thereby determining which LoRA layers (experts) should be activated for fine-tuning. For example, the gating network dynamically selects a LoRA layer from several LoRA layers for parameter adjustment based on the training data corresponding to the current task, which can be: By formula: ; ; = ; Calculate the activation weight of the i-th LoRA layer corresponding to task t ;in, represents the embedding representation of task t; , , and is the preset value; N represents the total number of LoRA layers; By formula: , in order to adapt to task t, update the parameters of the Lth layer of the multi-task pre-trained model ; in, Indicates the parameters of the Lth layer before updating, Represents the low-rank update matrix provided by the i-th LoRA layer to the L-th layer.

[0032] By way of further example, the algorithm pseudo code of this step may be: Algorithm: Multi-task LoRA fine-tuning based on gating mechanism; Input: pre-trained model, LoRA layer set { }, task set {T}, task embedding { }; Output: fine-tuned model for each task; 1. Initialize the gating network parameters; 2. for each task t in T do; 3. et = get the embedding representation of task t; 4. gt = softmax(MLP(et)) # Calculate the gating vector; 5. for each layer of L in pre-trained model do; 6. # Update the parameters of this layer; 7. Use of end for Fine-tune the model to suit task t; 8. end for.

[0033] In addition, this application introduces a learnable gating function to achieve fine-grained control of weight distribution. Assume that the pre-trained model is defined by parameter θ, which contains multiple Transformer blocks. For each Transformer block, this application trains multiple LoRA parameter sets { }, where i = 0, 1, ..., N, N is the number of LoRA layers. Given an input x ∈ R^{L×d}, where L and d represent the sequence length and input dimension respectively, the output of the LoRA layer is adjusted through the gating network to obtain the output of the pre-trained network, specifically: Get the input data x of the LoRA layer; By formula: , concatenate and normalize the outputs of all LoRA layers to obtain data ; in, represents the output of the i-th LoRA layer, and ,in, represents the feedforward neural network layer, LN represents layer normalization, represents the parameters learned by the i-th LoRA layer of the current task during the training process, and N represents the number of all LoRA layers; By formula: , calculate the gated input of each LoRA layer; By formula: , calculate the weight of the i-th LoRA layer; where τ represents the preset temperature parameter, represents the gating function; By formula: , calculate the weighted sum of all LoRA layers output by the gating network; By formula: , the output of the gating network Added to the output of multi-task pre-trained models In the output of the pre-trained network, .

[0034] In addition, in the hierarchical LoRA combination optimization, this application not only focuses on the fine-grained control of weights through gating functions, but also hopes to optimize this process so that the combined LoRA can achieve better performance on specific tasks. To this end, this application introduces Gating BaLancing Loss to balance the contributions of different LoRA experts and promote the overall performance of the model. This application uses entropy regularization to encourage uniform distribution of weights: ; where δ is a small positive number, and ∈(0,1), which is used to avoid the computational problems of the logarithmic function.

[0035] Combining standard task losses (such as cross entropy loss for classification tasks) and Gating BaLancing Loss, the overall training objective can be expressed as: ; Among them, y is the true label, is the task-dependent loss function, and λ is a hyperparameter that balances the two loss terms.

[0036] In this way, this application not only optimizes the LoRA combination to improve task performance, but also balances or prefers the contributions of different LoRA experts through GatingBaLancing Loss, thereby achieving more refined regulation.

[0037] Step 140: After completing the training of the multi-task pre-trained model and the LoRA layer, freeze the parameters of the multi-task pre-trained model and the LoRA layer.

[0038] In some embodiments, except for the gating function parameters, all trained LoRA parameters and pre-trained model basic parameters are frozen. This strategy helps retain expert knowledge and avoid overfitting. The formula is as follows: ; Among them, θfrozen represents the frozen model parameters.

[0039] Step 150: Obtain the maximum number of tokens processed by each LoRA layer in a single batch and the preset multi-objective optimization strategy to perform load balancing optimization for the operation of the multi-task pre-training model.

[0040] It should be noted that this application can achieve dynamic allocation and efficient use of resources by setting a reasonable expert capacity threshold to limit the maximum number of tokens processed by each LoRA expert in a single batch. In addition to setting an expert capacity threshold to limit the maximum token processing volume of each LoRA expert in a single batch, a more refined algorithm can be introduced to dynamically adjust this threshold. The queue length prediction algorithm (ARIMA model based on time series) is used to estimate the number of requests in the next few batches, and the expert capacity threshold is adjusted in advance accordingly.

[0041] As an example, the maximum number of tokens processed by each LoRA layer in a single batch can be obtained as follows: By setting the expert capacity threshold formula: , calculate the maximum number of tokens processed by the i-th LoRA layer in a single batch ; in, represents the minimum value of the preset capacity threshold of the i-th LoRA layer, represents the maximum value of the preset capacity threshold of the i-th LoRA layer, represents the average of the first k-1 batch requests, Indicates the preset adjustment step size, Indicates the maximum number of tokens processed by the i-th LoRA layer in a single batch before optimization.

[0042] In addition, the present application can also design a load balancing strategy based on multi-objective optimization, where the objective function can include not only the load balancing degree, but also the system response time, resource utilization, etc. as optimization objectives. The specific process can be: By solving the preset multi-objective optimization strategy formula: ; Calculate and obtain a solution set X that reaches the maximum number of iterations or the improvement of the solution set is less than a preset threshold; in, Represents the variance of the LoRA layer load corresponding to the current solution set X, represents the average system response time corresponding to the current solution set X, represents the resource utilization corresponding to the current solution set X, , , Indicates the preset weight coefficient.

[0043] In order to solve the above multi-objective optimization problem, the basic steps of the algorithm used in this application can be: 1. Initialize a set of random solutions (called the population).

[0044] 2. Evaluate the objective function value for each solution.

[0045] 3. Generate a new solution set based on the objective function value selection, crossover and mutation operations.

[0046] 4. Repeat steps 2 and 3 until the stopping condition is met (such as reaching the maximum number of iterations or the improvement of the solution is less than a preset threshold).

[0047] The method also includes: In the model inference phase, the parameter freezing principle is followed to ensure that all LoRA parameters and pre-trained model parameters remain unchanged except for the gating function. This helps to improve the generalization ability and stability of the model under limited training data conditions. The relevant formula is as follows: ; Among them, θeffective is the set of parameters actually used for inference, which is determined by: ; here, e is the part of the LoRA increment that is activated according to the gating function g(x) (or the empty set if all LoRA parameters are frozen). g(x) is a binary or continuous valued function that decides whether to apply a LoRA increment (i.e. g(x)∈{0,1} or g(x)∈[0,1]) based on the input x and possible additional conditions.

[0048] In addition, this application Figure 2 A hybrid expert model training system based on LORA technology is provided in the embodiment of the present application. Figure 2 As shown, the system provided in the embodiment of the present application mainly includes: The pre-trained model preparation module 210 is used to select a multi-task pre-trained model that meets the preset performance requirements from a plurality of multi-task pre-trained models according to a preset performance selection procedure; The data set construction module 220 is used to create a preset fine-tuning data set for the Chinese environment corresponding to the tasks in the multi-task pre-training model.

[0049] The network fine-tuning module 230 is used to train several LoRA layers corresponding to the Transformer block in the multi-task pre-training model using a preset fine-tuning data set; wherein, in the process of training using the preset fine-tuning data set, a LoRA layer is dynamically selected from several LoRA layers according to the training data corresponding to the current task through a gating network for parameter adjustment, and the output of the LoRA layer is adjusted through the gating network to obtain the output of the pre-trained network.

[0050] The model processing module 240 is used to freeze the parameters of the multi-task pre-trained model and the LoRA layer after completing the training of the multi-task pre-trained model and the LoRA layer; obtain the maximum number of tokens processed by each LoRA layer in a single batch and the preset multi-objective optimization strategy, and perform load balancing optimization for the operation of the multi-task pre-trained model.

[0051] The above is a method embodiment of the present application. Based on the same inventive concept, the present application embodiment also provides a hybrid expert model training device based on LORA technology. Figure 3 As shown, the device includes: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes a hybrid expert model training method based on LORA technology as in the above embodiment.

[0052] Specifically, the server side determines a multi-task pre-trained model that meets the preset performance requirements from several multi-task pre-trained models according to a preset performance selection program; creates a preset fine-tuning data set for the Chinese environment corresponding to the tasks in the multi-task pre-trained model; uses the preset fine-tuning data set to train several LoRA layers corresponding to the Transformer block in the multi-task pre-trained model; wherein, in the process of training using the preset fine-tuning data set, a LoRA layer is dynamically selected from several LoRA layers according to the training data corresponding to the current task through a gating network for parameter adjustment, and the output of the LoRA layer is adjusted through the gating network to obtain the output of the pre-trained network; after completing the training of the multi-task pre-trained model and the LoRA layer, the parameters of the multi-task pre-trained model and the LoRA layer are frozen; the maximum number of tokens processed by each LoRA layer in a single batch and the preset multi-objective optimization strategy are obtained, and load balancing optimization of the operation of the multi-task pre-trained model is performed.

[0053] In addition, an embodiment of the present application also provides a non-volatile computer storage medium on which executable instructions are stored. When the executable instructions are executed, a hybrid expert model training method based on LORA technology as described above is implemented.

[0054] So far, the technical solutions of the present disclosure have been described in combination with the above multiple embodiments, but it is easy for those skilled in the art to understand that the protection scope of the present disclosure is not limited to these specific embodiments. Without departing from the technical principles of the present disclosure, those skilled in the art can split and combine the technical solutions in the above-mentioned various embodiments, and can also make equivalent changes or replacements to the relevant technical features. Any changes, equivalent replacements, improvements, etc. made within the technical concept and / or technical principle of the present disclosure will fall within the protection scope of the present disclosure.

Claims

1. A hybrid expert model training method based on LORA technology, characterized in that: The method comprises: Determine, according to a preset performance selection procedure, a multi-task pre-trained model that meets preset performance requirements from a plurality of multi-task pre-trained models; Create a preset fine-tuning dataset for the Chinese environment corresponding to the tasks in the multi-task pre-training model; Using a preset fine-tuning data set, train several LoRA layers corresponding to the Transformer block in the multi-task pre-training model; wherein, in the process of training using the preset fine-tuning data set, dynamically select a LoRA layer from several LoRA layers for parameter adjustment according to the training data corresponding to the current task through a gating network, adjust the output of the LoRA layer through the gating network, and obtain the output of the pre-training network; After completing the training of the multi-task pre-trained model and the LoRA layer, freeze the parameters of the multi-task pre-trained model and the LoRA layer; Get the maximum number of tokens processed by each LoRA layer in a single batch and the preset multi-objective optimization strategy to perform load balancing optimization for multi-task pre-training model operation.

2. The hybrid expert model training method based on LORA technology according to claim 1 is characterized in that: The method further comprises: In the model inference stage of the trained multi-task pre-trained model, the parameter freezing principle is followed to freeze the parameters of the multi-task pre-trained model and the LoRA layer except for the gating network.

3. The hybrid expert model training method based on LORA technology according to claim 1 is characterized in that: Through the gating network, the LoRA layer is dynamically selected from several LoRA layers according to the training data corresponding to the current task for parameter adjustment, including: By formula: ； ； = ； Calculate the activation weight of the i-th LoRA layer corresponding to task t ;in, represents the embedding representation of task t; , , and is the preset value; N represents the total number of LoRA layers; By formula: , in order to adapt to task t, update the parameters of the Lth layer of the multi-task pre-trained model ; in, Indicates the parameters of the Lth layer before updating, Represents the low-rank update matrix provided by the i-th LoRA layer to the L-th layer.

4. The hybrid expert model training method based on LORA technology according to claim 1 is characterized in that: The output of the LoRA layer is adjusted through the gating network to obtain the output of the pre-trained network, including: Get the input data x of the LoRA layer; By formula: , concatenate and normalize the outputs of all LoRA layers to obtain data ; in, represents the output of the i-th LoRA layer, and ,in, represents the feedforward neural network layer, LN represents layer normalization, represents the parameters learned by the i-th LoRA layer of the current task during the training process, and N represents the number of all LoRA layers; By formula: , calculate the gated input of each LoRA layer; By formula: , calculate the weight of the i-th LoRA layer; where τ represents the preset temperature parameter, represents the gating function; By formula: , calculate the weighted sum of all LoRA layers output by the gating network; By formula: , the output of the gating network Added to the output of multi-task pre-trained models In the output of the pre-trained network, .

5. The hybrid expert model training method based on LORA technology according to claim 4 is characterized in that: The method further comprises: By formula: ； ； Calculate the total loss function of the multi-task pre-trained model ; in, represents the loss function corresponding to the task, represents a preset constant, and ∈(0,1), represents the actual label of the configuration, Represents the preset balancing hyperparameters.

6. The hybrid expert model training method based on LORA technology according to claim 1 is characterized in that: Obtain the maximum number of tokens processed by each LoRA layer in a single batch and the preset multi-objective optimization strategy to perform load balancing optimization for multi-task pre-training model operation, including: By setting the expert capacity threshold formula: , calculate the maximum number of tokens processed by the i-th LoRA layer in a single batch ; in, represents the minimum value of the preset capacity threshold of the i-th LoRA layer, represents the maximum value of the preset capacity threshold of the i-th LoRA layer, represents the average of the first k-1 batch requests, Indicates the preset adjustment step size, Indicates the maximum number of tokens processed by the i-th LoRA layer in a single batch before optimization.

7. The hybrid expert model training method based on LORA technology according to claim 1 is characterized in that: Obtain the maximum number of tokens processed by each LoRA layer in a single batch and the preset multi-objective optimization strategy to perform load balancing optimization for multi-task pre-training model operation, including: By solving the preset multi-objective optimization strategy formula: ； Calculate and obtain a solution set X that reaches the maximum number of iterations or the improvement of the solution set is less than a preset threshold; in, Represents the variance of the LoRA layer load corresponding to the current solution set X, represents the average system response time corresponding to the current solution set X, represents the resource utilization corresponding to the current solution set X, , , Indicates the preset weight coefficient.

8. A hybrid expert model training system based on LORA technology, characterized in that: The system comprises: A pre-trained model preparation module is used to select a program according to a preset performance and determine a multi-task pre-trained model that meets the preset performance requirements from a number of multi-task pre-trained models; The dataset construction module is used to create a preset fine-tuning dataset for the Chinese environment corresponding to the tasks in the multi-task pre-training model; A network fine-tuning module is used to train several LoRA layers corresponding to the Transformer block in the multi-task pre-training model using a preset fine-tuning data set; wherein, in the process of training using the preset fine-tuning data set, a LoRA layer is dynamically selected from several LoRA layers according to the training data corresponding to the current task through a gating network for parameter adjustment, and the output of the LoRA layer is adjusted through the gating network to obtain the output of the pre-training network; The model processing module is used to freeze the parameters of the multi-task pre-trained model and the LoRA layer after completing the training of the multi-task pre-trained model and the LoRA layer; obtain the maximum number of tokens processed by each LoRA layer in a single batch and the preset multi-objective optimization strategy, and perform load balancing optimization for the operation of the multi-task pre-trained model.

9. A hybrid expert model training device based on LORA technology, characterized in that: The device comprises: processor; And a memory having executable code stored thereon, which, when the executable code is executed, enables the processor to execute a hybrid expert model training method based on LORA technology as described in any one of claims 1-7.

10. A non-volatile computer storage medium, characterized in that: Computer instructions are stored thereon, and when the computer instructions are executed, a hybrid expert model training method based on LORA technology as described in any one of claims 1-7 is implemented.

Images