Intelligent low-rank adaptive fine-tuning model method

Abstract

The invention provides an intelligent low-rank adaptive fine-tuning model method and system, and belongs to the technical field of model fine tuning. The method comprises the steps of obtaining an output result after a language sample is input into a pre-training language model, and randomly selecting at least one layer from the current pre-training language model as a presetting layer; obtaining a current state s based on an output result of the pre-training language model and feature statistics of the presetting layer; constructing an action space, and traversing the action space to obtain a next state s reached after the pre-training language model selects to execute a certain action a in the current state; calculating a reward value based on the current state s, the next state s and the corresponding output result; attenuation of the action step length is carried out according to the result of each reward value, and then parameter fine adjustment is carried out on the pre-training language model. According to the method, the optimal insertion position is intelligently selected from different layers of the pre-trained language model, so that the fine tuning efficiency is improved, the over-fitting risk is reduced, and the generalization ability of the model is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of model fine-tuning, and in particular, relates to a method and system for intelligent low-rank adaptive fine-tuning of a model. Background Art

[0002] Large pre-trained language models (LLMs) such as BERT and PaLM have achieved remarkable success in the field of natural language processing (NLP) and are widely used in tasks such as text classification, question answering systems, and machine translation. These models learn rich language representations by pre-training on large-scale datasets, and then adapt to specific tasks through fine-tuning. The fine-tuning process involves adjusting the weights of the pre-trained model to improve the performance of new tasks. However, due to the large number of model parameters, fine-tuning is expensive, and researchers are committed to developing efficient fine-tuning methods, such as low-rank adaptation, to reduce computing resource consumption while maintaining model performance. This is especially important in resource-constrained application scenarios.

[0003] In the field of fine-tuning large pre-trained language models (LLMs), traditional fine-tuning methods usually involve updating all parameters of the model, which is not only computationally expensive but also may lead to overfitting. Low-Rank Adaptation techniques such as LoRA provide an alternative with higher parameter and computational efficiency by introducing low-rank matrices in specific layers of the model to achieve parameter updates.

[0004] However, there are some deficiencies in the existing technologies when applying LoRA to attention-based sequence models (Transformer models). First, these methods may not fully utilize the characteristics of the Transformer architecture, such as the self-attention mechanism, resulting in poor fine-tuning effects. Second, the choice of location for inserting the low-rank matrix is ​​often fixed and not optimized according to the specific task or dataset, which may limit the improvement of model performance. In addition, existing low-rank adaptation methods may lack consideration of the interactions between different layers of the model, which are crucial to the final performance of the model. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a technical solution of an intelligent low-rank adaptive fine-tuning model.

[0006] The first aspect of the present invention discloses a method for intelligent low-rank adaptive fine-tuning model; the method comprises:

[0007] Step S1, obtaining the output result after the language sample is input into the pre-trained language model, and randomly selecting at least one layer from the current pre-trained language model as a pre-adjusted layer; the pre-trained language model includes a feedforward network layer, a multi-head attention layer and other network layers;

[0008] Step S2, obtaining the current state s based on the output result of the pre-trained language model and the feature statistics of the pre-adjusted layer;

[0009] Step S3, constructing an action space and traversing the action space to obtain a subsequent state s' reached after the pre-trained language model selects to perform a certain action a in the current state;

[0010] Step S4, calculating the reward value based on the output results corresponding to the current state s and the next state s';

[0011] Step S5: Attenuate the action step length according to the result of each reward value, and then fine-tune the parameters of the pre-trained language model.

[0012] In step S1, the output results of the pre-trained language model include model accuracy, computing resource consumption of the model, inference time of the model, and model parameter efficiency.

[0013] In step S3, the actions in the action space include:

[0014] Insert QLoRA module in one or several specific layers;

[0015] Do not insert a QLoRA module in the current layer; and

[0016] Adjust the parameters of the inserted QLoRA module.

[0017] In step S4, the calculation formula of the reward value is:

[0018] R=w 1 ·A+w 2 ·(1-R resource )+w 3 ·(1-T inference )+w 4 ·E

[0019] Among them, R is the total reward value; w 1 , w 2 , w 3 , w 4 is the weight coefficient; A is the model accuracy; R resource is the computational resource consumption of the model; T inference is the inference time of the model; E is the model parameter efficiency.

[0020] In step S5, the action corresponding to the maximum reward value is selected, and the parameters of the pre-trained language model are fine-tuned.

[0021] The second aspect of the present invention discloses a system for intelligent low-rank adaptive fine-tuning model; the system comprises:

[0022] The first processing module is configured to obtain an output result after a language sample is input into a pre-trained language model, and randomly select at least one layer from the current pre-trained language model as a pre-adjustment layer; the pre-trained language model includes a feedforward network layer, a multi-head attention layer, and other network layers;

[0023] The second processing module is configured to obtain a current state s based on an output result of the pre-trained language model and feature statistics of the pre-adjusted layer;

[0024] The third processing module is configured to construct an action space and traverse the action space to obtain a subsequent state s' reached after the pre-trained language model selects to perform a certain action a in the current state;

[0025] A fourth processing module is configured to calculate a reward value based on output results corresponding to a current state s and a subsequent state s';

[0026] The fifth processing module is configured to attenuate the action step length according to the results of each reward value, and then fine-tune the parameters of the pre-trained language model.

[0027] According to the system of the second aspect of the present invention, the output results of the pre-trained language model include model accuracy, computing resource consumption of the model, inference time of the model, and model parameter efficiency.

[0028] According to the system of the second aspect of the present invention, the actions in the action space include:

[0029] Insert QLoRA module in one or several specific layers;

[0030] Do not insert a QLoRA module in the current layer; and

[0031] Adjust the parameters of the inserted QLoRA module.

[0032] The third aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps in any one of the methods of intelligent low-rank adaptive fine-tuning models in the first aspect of the present disclosure are implemented.

[0033] The fourth aspect of the present invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any one of the methods of intelligent low-rank adaptive fine-tuning models in the first aspect of the present disclosure are implemented.

[0034] In summary, the solution proposed in the present invention has the following technical effects: by intelligently inserting QLoRA (a low-intelligence adaptation based on quantization method) modules in different layers of pre-trained language models (such as Transformer models), and using the Strategy Location Optimization Module (SLOM) to dynamically select the best insertion position, the shortcomings of the existing technology are solved. This method not only improves the efficiency and effect of fine-tuning, but also reduces the risk of overfitting, so that the model can better adapt to specific tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 It is a flowchart of a method for intelligent low-rank adaptive fine-tuning model according to an embodiment of the present invention;

[0037] Figure 2 A network architecture design diagram of a strategic location optimization module according to an embodiment of the present invention;

[0038] Figure 3 The figure is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0040] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, a first image may be referred to as a second image, and similarly, a second image may be referred to as a first image without departing from the scope of this application. Both the first image and the second image are images, but they are not the same image.

[0041] The selection of the location for inserting fine-tunable parameters in fine-tuning of large language models mainly relies on empirical rules, uniform distribution, fixed patterns or gradient-based methods. These traditional methods have problems such as lack of adaptability, low efficiency, limited generalization ability and risk of overfitting. They are usually unable to dynamically adjust the insertion position to adapt to specific tasks, resulting in waste of computing resources and lack of theoretical support. The present invention introduces a strategic location optimization module (SLOM) and a reinforcement learning algorithm to intelligently select the optimal insertion position in different layers of a pre-trained language model (such as a Transformer model), thereby improving fine-tuning efficiency, reducing the risk of overfitting, and enhancing the generalization ability of the model.

[0042] Specifically, according to an embodiment of the present invention, in a first aspect, a method for intelligent low-rank adaptive fine-tuning model is provided; see Figure 1 , the method comprising:

[0043] Step S1, obtaining the output result after the language sample is input into the pre-trained language model, and randomly selecting at least one layer from the current pre-trained language model as a pre-adjusted layer; the pre-trained language model includes a feedforward network layer, a multi-head attention layer and other network layers;

[0044] The pre-trained language model can be a Transformer model. The overall architecture of Transformer consists of four parts: (1) Input part

[0045] 1) Source text embedding layer and its position encoder

[0046] 2) Target text embedding layer and its position encoder

[0047] (2) Output part

[0048] 1) Linear layer: obtains the output of the specified dimension by linearly changing the previous step, that is, the role of converting dimensions;

[0049] 2) Softmax processor: scales the numbers in the last dimension of the vector to the probability range of 0-1 and satisfies that their sum is 1.

[0050] (3) Encoder part

[0051] 1) It is composed of N encoder layers stacked together;

[0052] 2) Each encoder layer consists of two sub-layer connected structures;

[0053] 3) The first sub-layer connection structure includes a multi-head self-attention sub-layer and a normalization layer as well as a residual connection;

[0054] 4) The second sub-layer connection structure includes a feed-forward fully connected sub-layer and a normalization layer as well as a residual connection.

[0055] (4) Decoder

[0056] 1) It is composed of N decoder layers stacked together; each decoder layer consists of three sub-layer connection structures;

[0057] 2) The first sub-layer connection structure includes a multi-head self-attention sub-layer and a normalization layer as well as a residual connection;

[0058] 3) The second sub-layer connection structure includes a multi-head attention sub-layer and a normalization layer as well as a residual connection;

[0059] 4) The third sub-layer connection structure includes a feed-forward fully connected sub-layer and a normalization layer as well as a residual connection.

[0060] In step S1, the output results of the pre-trained language model include model accuracy, computing resource consumption of the model, inference time of the model, and model parameter efficiency.

[0061] Step S2, obtaining the current state s based on the output result of the pre-trained language model and the feature statistics of the pre-adjusted layer;

[0062] In model fine-tuning, the states include:

[0063] Feature statistics of the current layer (such as the mean, variance, gradient, etc. of the weights).

[0064] The performance indicators of the model before the current layer (such as accuracy, loss function value).

[0065] The computational resource consumption and inference time of the model.

[0066] Step S3, constructing an action space and traversing the action space to obtain a subsequent state s' reached after the pre-trained language model selects to perform a certain action a in the current state;

[0067] In step S3, the actions in the action space include:

[0068] Insert QLoRA module in one or several specific layers;

[0069] Do not insert a QLoRA module in the current layer; and

[0070] Adjust the parameters of the inserted QLoRA module.

[0071] Step S4, calculating the reward value based on the output results corresponding to the current state s and the next state s';

[0072] In step S4, the calculation formula of the reward value is:

[0073] R=w1 ·A+w 2 ·(1-R resource )+w 3 ·(1-T inference )+w 4 ·E

[0074] Among them, R is the total reward value; w 1 , w 2 , w 3 , w 4 is the weight coefficient; A is the model accuracy; R resource is the computational resource consumption of the model; T inference is the inference time of the model; E is the model parameter efficiency.

[0075] Step S5: Attenuate the action step length according to the result of each reward value, and then fine-tune the parameters of the pre-trained language model.

[0076] In step S5, the action corresponding to the maximum reward value is selected, and the parameters of the pre-trained language model are fine-tuned.

[0077] Optionally, the method is implemented by a strategy position optimization module based on a reinforcement learning algorithm. The architecture design of the strategy position optimization module based on the reinforcement learning algorithm is as follows: Figure 2 As shown, including:

[0078] 1. Action Network: The Actor network generates the actions that should be taken in a given state. In model fine-tuning, generally, model fine-tuning relies on empirical rules, uniform distribution, fixed patterns, or gradient-based methods. The action network provides an intelligent basis for model fine-tuning, which determines in which layers of the Transformer model to insert fine-tunable parameters to optimize the performance of specific tasks.

[0079] Define a state space that contains enough information to help the agent make decisions. In model fine-tuning, the state includes:

[0080] Feature statistics of the current layer (such as the mean, variance, gradient, etc. of the weights).

[0081] The performance indicators of the model before the current layer (such as accuracy, loss function value).

[0082] The computational resource consumption and inference time of the model.

[0083] Each action in the action space should correspond to a specific decision. In this scenario, the actions are:

[0084] Insert QLoRA modules in certain specific layers.

[0085] Do not insert the QLoRA module in the current layer.

[0086] Adjust the parameters of the inserted QLoRA module

[0087] 2. Evaluation Network (Critic Network): The Critic Network evaluates the potential value of the action selected by the Actor. It provides feedback by estimating the expected cumulative reward after taking a specific action, helping the Actor learn how to choose the best action in different states. The reward value function is designed as follows:

[0088] R=w 1 ·A+w 2 ·(1-R resource )+w 3 ·(1-T inference )+w 4 ·E

[0089] Among them, R is the total reward value; w 1 , w 2 , w 3 , w 4 is the weight coefficient; A is the model accuracy; R resource is the computational resource consumption of the model; T inference is the inference time of the model; E is the model parameter efficiency, which can be the inverse of the number of parameters or other efficiency indicators.

[0090] 3. Update Parameters: Initial parameter settings provide a starting point for the fine-tuning process. Based on the feedback from the Critic network, the model parameters are updated to optimize performance. This involves adjusting the weights of the fine-tunable parameter layer to suit the specific downstream task.

[0091] 4. Q Gradient: Q gradient is the gradient information about the action output by the Critic network, indicating how to adjust the action to increase the expected cumulative reward. During fine-tuning, Q gradient guides the optimization of fine-tunable parameters.

[0092] 5. Policy Network Gradient: Policy Network Gradient is the gradient information about the state output by the Actor network, indicating how to choose the best action in different states. This gradient information is used to update the Actor network to improve the selection of actions.

[0093] The gradients are used to update the network weights to guide the agent to learn in a direction that improves reward.

[0094] 6. Policy State: The policy state is the output of the Actor network in a given state, which determines the action that the agent should take in that state.

[0095] 7. State: A description of the environment, based on which the agent chooses actions. In model fine-tuning, the state refers to the current configuration of the model.

[0096] 8. Select Action: Based on the output of the Actor network and the Critic network, the agent selects an action in the current state. In model fine-tuning, this involves deciding at which layers of the model to apply fine-tunable parameters.

[0097] 9. Tunable Parameters Layer: refers to the adjustable parameter layers in the model, which will be updated during the fine-tuning process to adapt to specific tasks.

[0098] The second aspect of the present invention discloses a system for intelligent low-rank adaptive fine-tuning model; the system comprises:

[0099] The first processing module is configured to obtain an output result after a language sample is input into a pre-trained language model, and randomly select at least one layer from the current pre-trained language model as a pre-adjustment layer; the pre-trained language model includes a feedforward network layer, a multi-head attention layer, and other network layers;

[0100] The second processing module is configured to obtain a current state s based on an output result of the pre-trained language model and feature statistics of the pre-adjusted layer;

[0101] The third processing module is configured to construct an action space and traverse the action space to obtain a subsequent state s' reached after the pre-trained language model selects to perform a certain action a in the current state;

[0102] A fourth processing module is configured to calculate a reward value based on output results corresponding to a current state s and a subsequent state s';

[0103] The fifth processing module is configured to attenuate the action step length according to the results of each reward value, and then fine-tune the parameters of the pre-trained language model.

[0104] According to the system of the second aspect of the present invention, the output results of the pre-trained language model include model accuracy, computing resource consumption of the model, inference time of the model, and model parameter efficiency.

[0105] According to the system of the second aspect of the present invention, the actions in the action space include:

[0106] Insert QLoRA module in one or several specific layers;

[0107] Do not insert a QLoRA module in the current layer; and

[0108] Adjust the parameters of the inserted QLoRA module.

[0109] The third aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps in any one of the methods of intelligent low-rank adaptive fine-tuning models in the first aspect of the present disclosure are implemented.

[0110] Figure 3 is a structural diagram of an electronic device according to an embodiment of the present invention, such as Figure 3 As shown, the electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, near field communication (NFC) or other technologies. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a button, a trackball or a touch pad set on the housing of the electronic device, or an external keyboard, touch pad or mouse, etc.

[0111] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a structural diagram of the part related to the technical solution of the present disclosure, and does not constitute a limitation on the electronic device to which the technical solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0112] The fourth aspect of the present invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method of intelligent low-rank adaptive fine-tuning model in any one of the first aspects of the present disclosure are implemented.

[0113] In summary, the technical solution proposed by the present invention has the following technical effects:

[0114] (1) This application introduces a strategic location optimization module (SLOM) and uses an optimized reward value function to dynamically adjust the insertion position of the QLoRA module in the Transformer architecture. Since SLOM can automatically find the optimal parameter adaptation position based on the performance of the model on a specific task, it solves the performance limitation caused by the fixed insertion position in the traditional fine-tuning method and achieves a more efficient and accurate model fine-tuning effect.

[0115] (2) This application implements an intelligent allocation strategy for fine-tunable parameters by adopting the Actor-Critic reinforcement learning framework. The Actor network is responsible for generating fine-tuning actions, while the Critic network evaluates the potential value of these actions, providing an adaptive policy update mechanism. This approach solves the problem of the lack of flexibility and efficiency of traditional fine-tuning strategies, realizes adaptive fine-tuning on different tasks and datasets, and improves the generalization ability and cross-task performance of the model.

[0116] (3) This application designs a reward function based on multiple objectives such as model accuracy, resource consumption, and inference time to guide the agent to learn to choose the best action in different states. This multi-objective optimization mechanism solves the suboptimal strategy problem that may be caused by single-objective optimization, reduces computing resource consumption while maintaining model performance, and improves the feasibility of the model in practical applications.

[0117] (4) This application reduces the number of model parameters and computational complexity by introducing low-rank adaptation technology. Since low-rank adaptation technology only updates parameters related to the intrinsic rank, this method solves the high computational cost problem caused by traditional full parameter updates and achieves efficient model deployment in resource-constrained environments.

[0118] (5) This application provides flexibility and reusability of model parameters through a pluggable LoRA module design. This design allows the LoRA parameters to be separated from the model after training, solving the problem of model rigidity caused by fixed parameters in traditional fine-tuning methods, enabling the sharing and cross-task reuse of model parameters, and enhancing the adaptability and flexibility of the model.

[0119] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the technical solutions described in the above embodiments can still be modified, or some or all of the technical features therein can be replaced by equivalents, and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent low-rank adaptation fine-tuning model, characterized in that: The method comprises: Step S1, obtaining the output result after the language sample is input into the pre-trained language model, and randomly selecting at least one layer from the current pre-trained language model as a pre-adjusted layer; the pre-trained language model includes a feedforward network layer, a multi-head attention layer and other network layers; Step S2, obtaining the current state s based on the output result of the pre-trained language model and the feature statistics of the pre-adjusted layer; Step S3, construct an action space, and traverse the action space to obtain the next state s reached after the pre-trained language model selects to perform a certain action a in the current state; Step S4, based on the current state s and the next state s, the corresponding output results calculate the reward value; Step S5: Attenuate the action step length according to the result of each reward value, and then fine-tune the parameters of the pre-trained language model.

2. The method according to claim 1, characterized in that: In step S1, the output results of the pre-trained language model include model accuracy, computing resource consumption of the model, inference time of the model, and model parameter efficiency.

3. The method according to claim 2, characterized in that In step S3, the actions in the action space include: Insert QLoRA module in one or several specific layers; Do not insert a QLoRA module in the current layer; and Adjust the parameters of the inserted QLoRA module.

4. The method according to claim 3, characterized in that In step S4, the calculation formula of the reward value is: R=w1·A+w2·(1-R resource )+w3·(1-T inference )+w4·E Among them, R is the total reward value; w1, w2, w3, w4 are weight coefficients; A is the model accuracy; R resource is the computational resource consumption of the model; T inference is the inference time of the model; E is the model parameter efficiency.

5. The method according to claim 4, characterized in that In step S5, the action corresponding to the maximum reward value is selected, and the parameters of the pre-trained language model are fine-tuned.

6. A model fine-tuning system based on artificial intelligence, characterized in that: The system comprises: The first processing module is configured to obtain an output result after a language sample is input into a pre-trained language model, and randomly select at least one layer from the current pre-trained language model as a pre-adjustment layer; the pre-trained language model includes a feedforward network layer, a multi-head attention layer, and other network layers; The second processing module is configured to obtain a current state s based on an output result of the pre-trained language model and feature statistics of the pre-adjusted layer; The third processing module is configured to construct an action space and traverse the action space to obtain a subsequent state s reached by the pre-trained language model after selecting to perform a certain action a in the current state; A fourth processing module is configured to calculate a reward value based on the output results corresponding to the current state s and the next state s; The fifth processing module is configured to attenuate the action step length according to the results of each reward value, and then fine-tune the parameters of the pre-trained language model.

7. The system according to claim 6, characterized in that The output results of the pre-trained language model include model accuracy, model computing resource consumption, model inference time, and model parameter efficiency.

8. The system according to claim 7, characterized in that The actions in the action space include: Insert QLoRA module in one or several specific layers; Do not insert a QLoRA module in the current layer; and Adjust the parameters of the inserted QLoRA module.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps in the method for intelligent low-rank adaptive fine-tuning model described in any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the method for intelligent low-rank adaptive fine-tuning model described in any one of claims 1 to 5 are implemented.

Images