Model fine tuning method and device, electronic equipment and nonvolatile storage medium

Abstract

The invention discloses a model fine tuning method and device, electronic equipment and a nonvolatile storage medium. The method comprises the steps that prompt information is acquired, and the prompt information is used for representing the type of a task needing to be executed by a model; the prompt information is input into a backbone network of the model, a low-rank adapter corresponding to the prompt information is selected before the prompt information passes through the backbone network and the model for the first time to generate a first lexical element, and the low-rank adapter is used for conducting fine tuning training on the model according to the task type corresponding to the prompt information to obtain a first lexical element; different low-rank adapters correspond to different task types; and in the process of generating subsequent lexical elements by the model, performing fine tuning training on the model by reusing the selected low-rank adapter. According to the method and the device, the technical problem of relatively high storage and calculation overhead of a fine adjustment method in a multi-tenant environment in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a model fine-tuning method, device, electronic device, and non-volatile storage medium. Background Art

[0002] In recent years, large-scale pre-trained language models (LLMs) have made significant progress in a wide range of natural language processing (NLP) tasks. However, in practical applications, fine-tuning is still required to adapt LLMs to specific tasks. Although full-parameter fine-tuning can achieve good performance, due to the huge scale of LLMs (typically reaching billions or even hundreds of billions of parameters), the computational cost and storage requirements of full-parameter fine-tuning are extremely high. Therefore, parameter-efficient fine-tuning (PEFT) methods have gradually become mainstream. They adapt to specific tasks by adjusting only a small number of parameters, thereby significantly reducing computing resource consumption.

[0003] Currently, Low-Rank Adaptation (LoRA), as one of the most successful PEFT methods, is widely used in the fine-tuning task of LLM. LoRA learns task-specific adjustments by adding a low-rank trainable matrix to the frozen pre-trained model weights without modifying the original parameters. This method significantly reduces storage overhead, allowing the same LLM to efficiently adapt to multiple tasks through different LoRA adaptation layers. Although LoRA performs well in many tasks, it still faces some challenges in a multi-tenant environment. In this environment, multiple LoRA adaptation layers need to be stored at the same time and switched according to different tasks, resulting in a linear increase in video memory overhead with the number of tasks and causing increased inference latency.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] The embodiments of the present application provide a model fine-tuning method, device, electronic device and non-volatile storage medium to at least solve the technical problem of high storage and computing overhead of the fine-tuning method in the related art in a multi-tenant environment.

[0006] According to one aspect of an embodiment of the present application, a model fine-tuning method is provided, including: obtaining prompt information, wherein the prompt information is used to characterize the type of task that the model needs to perform; inputting the prompt information into the backbone network of the model, and before the prompt information passes through the backbone network for the first time and the model generates the first word element, selecting a low-rank adapter corresponding to the prompt information, wherein the low-rank adapter is used to fine-tune the model for the task type corresponding to the prompt information, and different low-rank adapters correspond to different task types; in the process of subsequent word elements generated by the model, reusing the selected low-rank adapter to fine-tune the model.

[0007] Optionally, selecting a low-rank adapter corresponding to the prompt information includes: performing embedding conversion on the prompt information to obtain an embedding representation corresponding to the prompt information; inputting the embedding representation corresponding to the prompt information into the backbone network of the model to obtain hidden state information corresponding to the prompt information; using a pooling layer to perform a pooling operation on the hidden state information to obtain a target vector corresponding to the prompt information, wherein the pooling operation includes at least one of the following: pooling based on the last word, mean pooling, maximum pooling, and attention-based pooling; using a low-rank adaptation router to determine the low-rank adapter corresponding to the prompt information based on the target vector.

[0008] Optionally, the low-rank adaptation router includes: a linear layer, a normalization function, and a selection function; using the low-rank adaptation router to determine the low-rank adapter corresponding to the prompt information based on the target vector includes: using a linear layer to calculate the original probability of the target vector being assigned to each low-rank adapter; using the normalization function to normalize the original probability to obtain the probability distribution corresponding to the low-rank adapter, wherein the probability distribution is used to characterize the target probability of each low-rank adapter being selected, and the target probability is used to characterize the degree of match between the low-rank adapter and the task type represented by the prompt information; using the selection function, based on the probability distribution, select a preset number of low-rank adapters with the highest target probability from all low-rank adapters, and activate the selected low-rank adapter.

[0009] Optionally, the backbone network of the model includes: a Transformer backbone network; reusing the selected low-rank adapter to fine-tune the model includes: in the process of the model generating each subsequent word except the first word, using a low-rank adaptation router to determine the activated low-rank adapter at the Transformer level, and using the activated low-rank adapter to fine-tune the model when each word is generated.

[0010] Optionally, the method also includes: during the training process of the model, determining the prompt ratio corresponding to each low-rank adapter in each training batch, wherein the training data set of each training batch contains multiple training samples, and each training sample contains: prompt information, an output label corresponding to the prompt information, and the prompt ratio is used to characterize the proportion of the number of prompt information allocated to the low-rank adapter in the current training batch; determining the average probability quality index corresponding to each low-rank adapter in each training batch, wherein the average probability quality index is the average value of the probability of the low-rank adapter being selected for each training sample in the current training batch; determining the load balancing loss function value based on the prompt ratio and the average probability quality index, wherein the load balancing loss function value is used to characterize the degree of balance of the use of each low-rank adapter during the training process; adjusting the model parameters of the model based on the load balancing loss function value, wherein the model parameters include: parameters of the low-rank adapter and the low-rank adaptation router.

[0011] Optionally, the method also includes: during the model training process, obtaining the word sequence output by the model based on the prompt information; determining the model output loss function value based on the word sequence and the output label corresponding to the prompt information in the training data set, wherein the model output loss function value is used to characterize the degree of error between the output value of the model and the actual output label in the training data set.

[0012] Optionally, the method further includes: adjusting the model parameters of the model according to the loss function value output by the model, and adjusting the learnable parameters in the activation function corresponding to the low-rank adaptive router.

[0013] According to another aspect of an embodiment of the present application, a model fine-tuning device is also provided, including: an information acquisition module for acquiring prompt information, wherein the prompt information is used to characterize the type of task that the model needs to perform; an expert selection module for inputting the prompt information into the backbone network of the model, and selecting a low-rank adapter corresponding to the prompt information before the prompt information passes through the backbone network for the first time and the model generates the first word element, wherein the low-rank adapter is used to fine-tune the model for the task type corresponding to the prompt information, and different low-rank adapters correspond to different task types; a fine-tuning training module for reusing the selected low-rank adapter to fine-tune the model during the subsequent word element generation process of the model.

[0014] According to another aspect of the embodiments of the present application, an electronic device is provided, including: a memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the model fine-tuning method is executed when the program is run.

[0015] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, wherein the non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the model fine-tuning method by running the computer program.

[0016] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, which implements the steps of the model fine-tuning method when the computer program is executed by a processor.

[0017] In an embodiment of the present application, prompt information is obtained, wherein the prompt information is used to characterize the type of task that the model needs to perform; the prompt information is input into the backbone network of the model, and before the prompt information passes through the backbone network for the first time and the model generates the first word, a low-rank adapter corresponding to the prompt information is selected, wherein the low-rank adapter is used to fine-tune the model for the task type corresponding to the prompt information, and different low-rank adapters correspond to different task types; in the process of subsequent word elements generated by the model, the selected low-rank adapter is reused to fine-tune the model, and a routing mechanism based on prompt information is adopted. By introducing prompt information (Prompt), the most suitable low-rank adapter (LoRA adaptation layer) is pre-determined during inference, thereby avoiding recalculation of the selection strategy when each word element (token) is generated, thereby achieving the purpose of improving the inference speed and reducing the computational overhead during inference, thereby solving the technical problem of high storage and computational overhead of the fine-tuning method in the related art in a multi-tenant environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 This is a hardware structure block diagram of a computer terminal (or electronic device) for implementing a method for model fine-tuning provided in an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of a model fine-tuning method process provided according to an embodiment of the present application;

[0021] Figure 3 This is a schematic diagram of an algorithm architecture for LoRA routing fine-tuned for a large model provided in accordance with an embodiment of the present application;

[0022] Figure 4 It is a structural schematic diagram of a model fine-tuning device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] To facilitate those skilled in the art to better understand the embodiments of the present application, some technical terms or nouns involved in the embodiments of the present application are explained as follows:

[0026] Large language models are AI models designed to understand and generate human language. They are trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more.

[0027] MOE (Mixture of Experts): A deep learning architecture whose core concept is to use multiple "expert" sub-models to process different types of data and select the most appropriate expert through a gating network to calculate the output. This improves the model's computational efficiency and performance and is commonly used in large-scale natural language processing (NLP), computer vision, and recommender systems. MOE primarily consists of an expert network, a gating network, and a weighted summation algorithm. During execution, only some experts are activated, reducing computational costs.

[0028] LoRA (Low-Rank Adaptation) fine-tuning is a parameter-efficient fine-tuning (PEFT) method designed for large-scale pre-trained models. It learns task-specific adjustments by adding a low-rank matrix to the pre-trained weights without changing the original model parameters, significantly reducing the number of training parameters and memory usage. Suitable for tasks such as NLP, LoRA improves fine-tuning efficiency while maintaining model performance, making it particularly suitable for large model deployments and multi-task adaptation.

[0029] PEFT (Parameter-Efficient Fine-Tuning): PEFT is a method for optimizing large-scale pre-trained models. It aims to reduce the number of parameters required for fine-tuning, improve computational efficiency, and maintain model performance. Common techniques include LoRA (Low-Rank Adaptation), Adapter, Prefix-Tuning, and Prompt-Tuning. These methods achieve efficient fine-tuning by adjusting a small number of parameters or additional layers rather than modifying the entire model weights. They are widely used in tasks such as Natural Language Processing (NLP) and are particularly well-suited for large-scale model deployment and multi-task adaptation.

[0030] Related art approaches such as MOE LoRA (Mixture of Experts LoRA) and DoRA (Dynamic Routing LoRA) employ a multi-expert system-style LoRA selection mechanism, training multiple LoRA modules and using a gating network to select the appropriate LoRA adaptation layer for the current input during inference. While this approach can reduce the storage overhead of single-task LoRA, it still suffers from a key issue: during inference, each token generated requires recalculating the LoRA selection strategy, resulting in significant computational overhead and slowing down inference speed.

[0031] To address the above issues, the present application provides a solution and proposes a more efficient LoRA variant, MiLoRA (Mixture of Low-Rank Adaptation). MiLoRA mainly optimizes the inference computation overhead of MOE LoRA and proposes a prompt-aware routing mechanism, which is described in detail below.

[0032] According to an embodiment of the present application, a method embodiment of model fine-tuning is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal (or electronic device) for implementing a model fine-tuning method. Figure 1As shown, the computer terminal 10 (or electronic device) may include one or more (illustrated as 102a, 102b, ..., 102n in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0034] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or electronic device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0035] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the model fine-tuning method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned model fine-tuning method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0036] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0037] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or electronic device).

[0038] In the above operating environment, the embodiment of the present application provides a model fine-tuning method. Figure 2 is a schematic diagram of a model fine-tuning method process provided in an embodiment of the present application, such as Figure 2 As shown, the method includes the following steps:

[0039] Step S202: Obtain prompt information, wherein the prompt information is used to characterize the type of task that the model needs to perform;

[0040] Step S204: Input the prompt information into the backbone network of the model, and before the prompt information passes through the backbone network for the first time and the model generates the first word, select a low-rank adapter corresponding to the prompt information. The low-rank adapter is used to fine-tune the model for the task type corresponding to the prompt information. Different low-rank adapters correspond to different task types.

[0041] Step S206: During the subsequent word-gram generation process of the model, the selected low-rank adapter is reused to fine-tune the model.

[0042] Through the above steps, a routing mechanism based on prompt information is adopted. By introducing prompt information (Prompt), the most suitable low-rank adapter (LoRA adaptation layer) is pre-determined during inference, avoiding recalculation of the selection strategy when each token is generated, thereby achieving the purpose of improving inference speed and reducing computational overhead during inference, thereby solving the technical problem of high storage and computational overhead of fine-tuning methods in related technologies in multi-tenant environments.

[0043] Figure 3 This is a schematic diagram of an algorithm architecture for LoRA routing for large model fine-tuning according to an embodiment of the present application. Figure 3 The model fine-tuning method in steps S202 to S206 of the embodiment of the present application is further introduced.

[0044] In an embodiment of the present application, an expert route can be selected based on a prompt input by the user. Specifically, in this embodiment, a low-rank adapter (LoRA) can be selected as an expert, and an appropriate LoRA module is selected according to the routing mechanism. The overall process includes the following steps: inputting a prompt, calculating the hidden state h, selecting an appropriate expert module through the prompt routing mechanism, obtaining the topK experts, applying the LoRA adapter (reusing expert selection), generating a token, and continuing decoding until the sequence ends. The process is described in detail below.

[0045] First, receive the prompt input by the user and calculate the hidden state based on the prompt. The specific steps are as follows.

[0046] In some embodiments of the present application, selecting a low-rank adapter corresponding to the prompt information includes the following steps: performing embedding conversion on the prompt information to obtain an embedding representation corresponding to the prompt information; inputting the embedding representation corresponding to the prompt information into the backbone network of the model to obtain hidden state information corresponding to the prompt information; using a pooling layer to perform a pooling operation on the hidden state information to obtain a target vector corresponding to the prompt information, wherein the pooling operation includes at least one of the following: pooling based on the last word, mean pooling, maximum pooling, and attention-based pooling; using a low-rank adaptation router to determine the low-rank adapter corresponding to the prompt information based on the target vector.

[0047] Specifically, the input prompt is converted into an embedding representation that the model can understand, and then the embedded prompt is input into the backbone network of the model (taking the Transformer model as an example) to calculate the hidden state.

[0048] In this embodiment, the hidden state of prompt It refers to the hidden representation obtained before the Lth layer of the Transformer model. Specifically, when the input prompt is hidden through the backbone network of the Transformer model, the hidden state of the Lth layer is where n p is the length of prompt, d is the hidden state of each layer. Here, for different models, the calculation details are different. Taking the embedding of the BERT model as an example, the embedding calculation of each token is as follows, including token embedding, position embedding and paragraph embedding. Corresponding to the above

[0049]

[0050] Therefore, after embedding and before input to the first transformer layer, the dimension of prompt is

[0051]

[0052] After obtaining the hidden state information, the expert selection strategy can be determined. Specifically, the pooling layer can be used to aggregate the hidden state information and convert H l Transformed to h l (i.e. the target vector mentioned above) That is h l =Pooler(H l ).

[0053] In this embodiment, the Pooler operation may include but is not limited to: 1) Pooling based on the last word: using the representation of the last token in the prompt information as the pool. l ; 2) Mean pooling 3) Maximum pooling 4) Attention-based pooling.

[0054] Then, the low-rank adaptive router (LoRA router R l ) for the target vector h l Processing is performed to determine the low-rank adapter corresponding to the prompt information, that is, the currently input prompt information is assigned to the most appropriate LoRA expert (low-rank adapter). The specific steps are as follows.

[0055] In some embodiments of the present application, a low-rank adaptation router includes: a linear layer, a normalization function, and a selection function; using the low-rank adaptation router, based on the target vector, determining the low-rank adapter corresponding to the prompt information includes the following steps: using a linear layer to calculate the original probability of the target vector being assigned to each low-rank adapter; using the normalization function, normalizing the original probability to obtain the probability distribution corresponding to the low-rank adapter, wherein the probability distribution is used to characterize the target probability of each low-rank adapter being selected, and the target probability is used to characterize the degree of match between the low-rank adapter and the task type represented by the prompt information; using the selection function, based on the probability distribution, selecting a preset number of low-rank adapters with the highest target probability from all low-rank adapters, and activating the selected low-rank adapter.

[0056] Specifically, the target vector h l By activation function g and LoRA router R l Processing, R lResponsible for assigning the current input prompt to the most appropriate LoRA expert. The router contains: linear layer, normalization function (Softmax function), selection function (Top-k selection function), among which the linear layer is used to calculate h l The original probability is assigned to each LoRA expert (low-rank adapter), and then the Softmax function is used to convert the output of the linear layer into a probability distribution, that is, to construct the probability distribution of the LoRA expert. It ensures that the sum of the probabilities of all LoRA experts is 1, thus forming a valid probability distribution. The output of the Softmax function represents the probability of each LoRA expert being selected. These probabilities reflect the adaptability of each expert to the current input prompt. Finally, the Top-k selection function is used to select the k experts with the highest probability quality from the probability distribution. The formal expression is shown below:

[0057]

[0058] in, is the weight matrix of the router. The LoRA router dynamically selects the most suitable k experts during the inference process and is called only once before generating a new token. The activated LoRA experts will be reused throughout the generation process.

[0059] In an embodiment of the present application, the activation decision for the LoRA router is calculated only when the input prompt information passes through the Transformer backbone network for the first time, and the activated expert is determined before the first new token is generated. Subsequently, this decision is reused throughout the entire generation process. During reuse, the activated LoRA expert is determined at the Transformer layer level, that is, the corresponding LoRA module is selected to modify the Transformer module. The expert reuse strategy in this application is introduced below, as follows.

[0060] In some embodiments of the present application, the backbone network of the model includes: a Transformer backbone network; reusing the selected low-rank adapter to fine-tune the model includes the following steps: in the process of the model generating each subsequent word except the first word, using a low-rank adaptation router to determine the activated low-rank adapter at the Transformer level, and when each word is generated, using the activated low-rank adapter to fine-tune the model.

[0061] Specifically, the embodiments of the present application only calculate the LoRA router activation decision once, when the input prompt first passes through the Transformer backbone network and before the first new token is generated. These decisions are reused in subsequent generation steps. In other words, the MiLoRA method only calculates which LoRA experts should be activated once, when the input prompt first passes through the Transformer network and before the first new token is generated. These decisions are reused in subsequent generation steps, without needing to be recalculated at each generation step. Therefore, during each subsequent token generation process, the LoRA router determines the activated LoRA experts at the Transformer level and selects which Transformer module is modified by its corresponding LoRA module. The LoRA router dynamically selects the most appropriate k experts for inference for each input prompt, rather than calling multiple routers at each generation step. Therefore, during training, the generation of each subsequent token activates the LoRA experts in the corresponding Transformer layer for training, while inactivated LoRA modules do not participate in model training.

[0062] In addition, in this application example, a load balancing loss term can be added to the training loss to improve the generalization ability and adaptability of the model, as follows.

[0063] In some embodiments of the present application, the method also includes the following steps: during the training of the model, determining the prompt ratio corresponding to each low-rank adapter in each training batch, wherein the training data set of each training batch contains multiple training samples, and each training sample contains: prompt information, output label corresponding to the prompt information, and the prompt ratio is used to characterize the proportion of the number of prompt information allocated to the low-rank adapter in the current training batch; determining the average probability quality index corresponding to each low-rank adapter in each training batch, wherein the average probability quality index is the average value of the probability of the low-rank adapter being selected for each training sample in the current training batch; determining the load balancing loss function value based on the prompt ratio and the average probability quality index, wherein the load balancing loss function value is used to characterize the degree of balance of the use of each low-rank adapter during the training process; adjusting the model parameters of the model based on the load balancing loss function value, wherein the model parameters include: parameters of the low-rank adapter and the low-rank adaptation router.

[0064] In some embodiments of the present application, the method also includes the following steps: during the model training process, obtaining the word sequence output by the model based on the prompt information; determining the model output loss function value based on the word sequence and the output label corresponding to the prompt information in the training data set, wherein the model output loss function value is used to characterize the degree of error between the output value of the model and the actual output label in the training data set.

[0065] Specifically, the model training process includes the following steps: first, preparing a training dataset, in which the training samples include input prompt information and corresponding output labels; dividing the dataset into multiple batches, each batch containing a certain number of samples; in the forward propagation phase of training, the prompt input information in the training batch is fed into the backbone network of the model, the hidden state of the prompt information is calculated, the most appropriate LoRA expert is selected based on the hidden state using a LoRA router, and the output sequence is generated using the selected LoRA expert.

[0066] When calculating loss, we first calculate the task loss (i.e., the model output loss function value) based on the output sequence generated by the model and the actual output labels in the training dataset. We also calculate the load balancing loss function value. Finally, we add the task loss and load balancing loss to obtain the total loss. We then backpropagate this total loss and calculate the gradients of the model parameters. The optimizer then updates the model parameters based on the gradients.

[0067] The specific method for calculating the load balancing loss function value is as follows.

[0068] Assume that the training batch B contains N B samples, define represents the proportion of tips assigned to the i-th LoRA expert at level l:

[0069]

[0070] in, is the probability of the jth expert output by the router layer l. Let is the average probability quality indicator received by the i-th expert:

[0071]

[0072] Then the load balancing loss function value is:

[0073]

[0074] By introducing load balancing loss, the model training process will encourage a more balanced use of all LoRA experts, thereby improving the generalization ability and adaptability of the model. This loss function is particularly suitable for multi-task learning scenarios, where the model needs to switch and adapt between different tasks.

[0075] On the other hand, in the embodiments of the present application, the model can also automatically learn the optimal activation function configuration during the training process, thereby improving the performance and adaptability of the model. The details are as follows.

[0076] In some embodiments of the present application, the method further includes the following steps: adjusting the model parameters of the model according to the loss function value output by the model, and adjusting the learnable parameters in the activation function corresponding to the low-rank adaptive router.

[0077] Specifically, in related art, the activation function in the PEFT module is usually set to the ReLU function, and the PEFT modules of different Transformer layers usually use the same activation function. However, in the embodiment of the present application, the activation function is made learnable during the training process, as shown in the following formula:

[0078]

[0079] Among them, a j and b i is a learnable parameter. The Padé function approximates common activation functions and adapts to them during training. This design avoids the hassle of manually tuning hyperparameters and automatically learns the optimal activation function for different tasks.

[0080] When fine-tuning downstream tasks, a rational activation function can be used to learn the activation function of the LoRA router. Let the set of parameters for the learnable activation function be θ, and the set of other parameters for the LoRA router and LoRA expert be Ω. In this example, θ is treated as an architectural parameter and optimized together with Ω through bi-level optimization.

[0081] Adjusting model parameters by comprehensively considering both the model output loss and the load balancing loss is a key step in achieving efficient fine-tuning in this technical solution. By simultaneously optimizing these two loss functions, the model not only improves its output quality for specific tasks but also ensures balanced utilization of all low-rank adapters, avoiding resource waste and performance bottlenecks.

[0082] The expert selection mechanism of the present application only selects experts when calculating the prompt (that is, the LoRA router is only called before the prompt is input and the first token is generated), and the generation of subsequent tokens will reuse the same expert selection, reducing the computational overhead during inference; compared to the routing test class that calculates each token, the computational complexity is reduced from O(N) to O(k). In addition, although there will still be multiple experts in each layer of the present application, only a small number of experts will be fixed to participate in the calculation for each generated sequence, thanks to the top-k expert selection strategy. Therefore, the number of experts selected in each layer of the present application will also be smaller than the traditional solutions in the related technology. In addition, the present application uses a learnable activation function to replace the original ReLU activation function, and further improves the optimization of the activation function and other parameters of the LoRA module through double-layer optimization.

[0083] According to an embodiment of the present application, an embodiment of a model fine-tuning device is also provided. Figure 4 Schematic diagram of a model fine-tuning device according to an embodiment of the present application. Figure 4 As shown, the device includes:

[0084] An information acquisition module 40 is used to acquire prompt information, wherein the prompt information is used to characterize the type of task that the model needs to perform;

[0085] Expert selection module 42, configured to input the prompt information into the backbone network of the model and select a low-rank adapter corresponding to the prompt information before the prompt information passes through the backbone network for the first time and the model generates the first word. The low-rank adapter is used to fine-tune the model for the task type corresponding to the prompt information. Different low-rank adapters correspond to different task types.

[0086] The fine-tuning training module 44 is used to reuse the selected low-rank adapter to fine-tune the model during the subsequent word-unit generation process of the model.

[0087] Optionally, selecting a low-rank adapter corresponding to the prompt information includes: performing embedding conversion on the prompt information to obtain an embedding representation corresponding to the prompt information; inputting the embedding representation corresponding to the prompt information into the backbone network of the model to obtain hidden state information corresponding to the prompt information; using a pooling layer to perform a pooling operation on the hidden state information to obtain a target vector corresponding to the prompt information, wherein the pooling operation includes at least one of the following: pooling based on the last word, mean pooling, maximum pooling, and attention-based pooling; using a low-rank adaptation router to determine the low-rank adapter corresponding to the prompt information based on the target vector.

[0088] Optionally, the low-rank adaptation router includes: a linear layer, a normalization function, and a selection function; using the low-rank adaptation router to determine the low-rank adapter corresponding to the prompt information based on the target vector includes: using a linear layer to calculate the original probability of the target vector being assigned to each low-rank adapter; using the normalization function to normalize the original probability to obtain the probability distribution corresponding to the low-rank adapter, wherein the probability distribution is used to characterize the target probability of each low-rank adapter being selected, and the target probability is used to characterize the degree of match between the low-rank adapter and the task type represented by the prompt information; using the selection function, based on the probability distribution, select a preset number of low-rank adapters with the highest target probability from all low-rank adapters, and activate the selected low-rank adapter.

[0089] Optionally, the backbone network of the model includes: a Transformer backbone network; reusing the selected low-rank adapter to fine-tune the model includes: in the process of the model generating each subsequent word except the first word, using a low-rank adaptation router to determine the activated low-rank adapter at the Transformer level, and using the activated low-rank adapter to fine-tune the model when each word is generated.

[0090] Optionally, the model fine-tuning device is also used to: determine the prompt ratio corresponding to each low-rank adapter in each training batch during the training process of the model, wherein the training data set of each training batch contains multiple training samples, and each training sample contains: prompt information, output label corresponding to the prompt information, and the prompt ratio is used to characterize the proportion of the number of prompt information allocated to the low-rank adapter in the current training batch; determine the average probability quality index corresponding to each low-rank adapter in each training batch, wherein the average probability quality index is the average value of the probability of the low-rank adapter being selected for each training sample in the current training batch; determine the load balancing loss function value based on the prompt ratio and the average probability quality index, wherein the load balancing loss function value is used to characterize the degree of balance of the use of each low-rank adapter during the training process; adjust the model parameters of the model based on the load balancing loss function value, wherein the model parameters include: parameters of the low-rank adapter and the low-rank adaptation router.

[0091] Optionally, the model fine-tuning device is also used to: obtain the word sequence output by the model based on the prompt information during the model training process; determine the model output loss function value based on the word sequence and the output label corresponding to the prompt information in the training data set, wherein the model output loss function value is used to characterize the degree of error between the output value of the model and the actual output label in the training data set.

[0092] Optionally, the model fine-tuning device is further used to: adjust the model parameters of the model according to the model output loss function value, and adjust the learnable parameters in the activation function corresponding to the low-rank adaptive router.

[0093] It should be noted that the various modules in the above-mentioned model fine-tuning device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0094] It should be noted that the model fine-tuning device provided in this embodiment can be used to perform Figure 2 The model fine-tuning method shown, therefore, the relevant explanations and descriptions of the above-mentioned model fine-tuning method are also applicable to the embodiments of this application and will not be repeated here.

[0095] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the following model fine-tuning method by running the computer program: obtaining prompt information, wherein the prompt information is used to characterize the type of task that the model needs to perform; inputting the prompt information into the backbone network of the model, and before the prompt information passes through the backbone network for the first time and the model generates the first word element, selecting a low-rank adapter corresponding to the prompt information, wherein the low-rank adapter is used to fine-tune the model for the task type corresponding to the prompt information, and different low-rank adapters correspond to different task types; in the process of subsequent word elements generated by the model, reusing the selected low-rank adapter to fine-tune the model.

[0096] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the model fine-tuning method described in each embodiment of the present application: obtaining prompt information, wherein the prompt information is used to characterize the type of task that the model needs to perform; inputting the prompt information into the backbone network of the model, and before the prompt information passes through the backbone network for the first time and the model generates the first word element, selecting a low-rank adapter corresponding to the prompt information, wherein the low-rank adapter is used to fine-tune the model for the task type corresponding to the prompt information, and different low-rank adapters correspond to different task types; in the process of subsequent word elements generated by the model, reusing the selected low-rank adapter to fine-tune the model.

[0097] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0098] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0100] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0101] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0102] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0103] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A model fine-tuning method, characterized in that: include: Obtaining prompt information, wherein the prompt information is used to characterize the type of task that the model needs to perform; Inputting the prompt information into the backbone network of the model, and before the prompt information passes through the backbone network for the first time and the model generates the first word, selecting a low-rank adapter corresponding to the prompt information, wherein the low-rank adapter is used to fine-tune the model for the task type corresponding to the prompt information, and different low-rank adapters correspond to different task types; In the process of subsequent word units generated by the model, the selected low-rank adapter is reused to fine-tune the model.

2. The model fine-tuning method according to claim 1, characterized in that: The low-rank adapter corresponding to the prompt information is selected as follows: Performing embedding conversion on the prompt information to obtain an embedding representation corresponding to the prompt information; Inputting the embedding representation corresponding to the prompt information into the backbone network of the model to obtain hidden state information corresponding to the prompt information; Using a pooling layer, a pooling operation is performed on the hidden state information to obtain a target vector corresponding to the prompt information, wherein the pooling operation includes at least one of the following: last-word-based pooling, mean pooling, maximum pooling, and attention-based pooling; A low-rank adaptive router is used to determine the low-rank adapter corresponding to the prompt information according to the target vector.

3. The model fine-tuning method according to claim 2, characterized in that: The low-rank adaptive router includes: a linear layer, a normalization function, and a selection function; Using a low-rank adaptive router, determining the low-rank adapter corresponding to the prompt information according to the target vector includes: Using the linear layer, calculating the original probability of the target vector being assigned to each low-rank adapter; Using the normalization function, the original probability is normalized to obtain the probability distribution corresponding to the low-rank adapter, wherein the probability distribution is used to characterize the target probability of each low-rank adapter being selected, and the target probability is used to characterize the degree of match between the low-rank adapter and the task type represented by the prompt information; Using the selection function, according to the probability distribution, a preset number of low-rank adapters with the highest target probability are selected from all low-rank adapters, and the selected low-rank adapters are activated.

4. The model fine-tuning method according to claim 3, characterized in that: The backbone network of the model includes: Transformer backbone network; reusing the selected low-rank adapter to fine-tune the model includes: In the process of the model generating each subsequent word-gram except the first word-gram, the low-rank adaptation router is used to determine the activated low-rank adapter at the Transformer level, and when each word-gram is generated, the model is fine-tuned using the activated low-rank adapter.

5. The model fine-tuning method according to claim 1, characterized in that: The method further comprises: During the training process of the model, determining the prompt ratio corresponding to each low-rank adapter in each training batch, wherein the training data set of each training batch includes multiple training samples, and each training sample includes: the prompt information and an output label corresponding to the prompt information, and the prompt ratio is used to represent the proportion of the number of the prompt information assigned to the low-rank adapter in the current training batch; Determine an average probability quality index corresponding to each of the low-rank adapters in each of the training batches, wherein the average probability quality index is an average value of the probability of the low-rank adapter being selected for each of the training samples in the current training batch; Determining a load balancing loss function value according to the prompt proportion and the average probability quality indicator, wherein the load balancing loss function value is used to characterize the degree of balance of use of each of the low-rank adapters during the training process; According to the load balancing loss function value, the model parameters of the model are adjusted, wherein the model parameters include: parameters of the high-rank adapter and the low-rank adaptation router.

6. The model fine-tuning method according to claim 5, characterized in that: The method further comprises: During the model training process, obtaining a word sequence output by the model according to the prompt information; Based on the word sequence and the output label corresponding to the prompt information in the training data set, the model output loss function value is determined, wherein the model output loss function value is used to characterize the degree of error between the output value of the model and the actual output label in the training data set.

7. The model fine-tuning method according to claim 6, characterized in that: The method further comprises: According to the loss function value output by the model, the model parameters of the model are adjusted, and the learnable parameters in the activation function corresponding to the low-rank adaptive router are adjusted.

8. A model fine-tuning device, characterized in that: include: An information acquisition module, configured to acquire prompt information, wherein the prompt information is used to characterize the type of task that the model needs to perform; an expert selection module, configured to input the prompt information into the backbone network of the model and, before the prompt information passes through the backbone network for the first time and the model generates the first word, select a low-rank adapter corresponding to the prompt information, wherein the low-rank adapter is used to fine-tune the model for the task type corresponding to the prompt information, and different low-rank adapters correspond to different task types; A fine-tuning training module is used to reuse the selected low-rank adapter to fine-tune the model during the subsequent word-unit generation process of the model.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the model fine-tuning method according to any one of claims 1 to 7 is executed when the program is run.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the model fine-tuning method according to any one of claims 1 to 7 by running the computer program.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the model fine-tuning method according to any one of claims 1 to 7 are implemented.

Images