Model fine tuning method and device, computer equipment and storage medium

Abstract

The invention provides a model fine tuning method and device, computer equipment and a storage medium, and the method comprises the steps: loading a pre-training model and a model fine tuning module in a client, so as to enable the model fine tuning module to generate a first matrix and a second matrix of the pre-training model; elements of the first matrix are extracted from random Gaussian distribution; the elements of the second matrix are all zero; performing model training on the pre-training model through the local data sets to obtain a first model, wherein the local data sets of the plurality of clients are different from each other; at least one third matrix is received, each third matrix is sent by an adjacent client, and each third matrix is obtained by updating the second matrix through a preset optimization algorithm by the adjacent client; a second model is generated based on the first model and the at least one third matrix. According to the embodiment of the invention, the consensus error can be reduced, so that the model performance is greatly improved, and the communication traffic required in the model updating process can be remarkably reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of model training technology, and in particular to a model fine-tuning method, device, computer equipment and storage medium. Background Art

[0002] LoRA (Low Rank Adaptation) is an efficient fine-tuning technology for pre-trained models. It is designed to solve the fine-tuning of large-scale pre-trained models, especially for scenarios with limited computing resources and storage resources. LoRA fine-tuning technology can significantly reduce the number of trainable parameters, thereby reducing the fine-tuning cost and making the model more adaptable to specific tasks.

[0003] The existing technology focuses on applying LoRA fine-tuning technology in centralized federated learning scenarios, using LoRA technology for fine-tuning on the local side to reduce storage resource requirements, and reducing communication overhead by only transmitting low-rank matrices when communicating with the central server. However, there is a lack of exploration of applying LoRA fine-tuning in decentralized federated learning scenarios.

[0004] Then, in the process of applying LoRA fine-tuning technology to decentralized federated learning scenarios, since there is no central server in decentralized federated learning scenarios, each node can only exchange model parameter information with neighboring nodes and aggregate. After each round of aggregation, the models of each node are not the same, but through multiple rounds of aggregation, global information can be obtained through information transmission of multi-hop nodes in the communication system. However, there is still an error between the model of each node and the global model. We call this error consensus error. Consensus error will cause inconsistent model parameter updates between nodes, thereby affecting the performance of the global model. Due to the different data distribution between nodes, consensus error will aggravate the deviation between the global model and the local model, which may lead to slower convergence or even no convergence during training. In addition, consensus error will reduce the robustness of the system to malicious nodes, affecting the overall security and reliability of the model. Summary of the invention

[0005] In view of this, the present disclosure proposes a model fine-tuning method, apparatus, computer device and storage medium to solve the problem in the related art that consensus error may lead to inconsistent model parameter updates between nodes, thereby affecting the performance of the global model.

[0006] The first aspect of the present disclosure provides a model fine-tuning method, the method comprising:

[0007] The method is applied to a model system to be fine-tuned, wherein the model system to be fine-tuned includes multiple clients; for any one of the multiple clients, the method includes:

[0008] Loading the pre-trained model and the model fine-tuning module in the client, so that the model fine-tuning module generates a first matrix and a second matrix of the pre-trained model; the elements of the first matrix are extracted from a random Gaussian distribution; the elements of the second matrix are all zero;

[0009] Performing model training on the pre-trained model through a local data set to obtain a first model, wherein the local data sets of the multiple clients are different;

[0010] Receive at least one third matrix, wherein each third matrix is ​​sent by a neighboring client, and each third matrix is ​​obtained by updating the second matrix by a neighboring client through a preset optimization algorithm;

[0011] A second model is generated based on the first model and the at least one third matrix.

[0012] The disclosed embodiment selectively freezes the first matrix, focuses on training the second matrix, and performs model fine-tuning, which can effectively reduce the consensus error and thus greatly improve the model performance; and can significantly reduce the communication overhead required in the model update process, thereby improving communication efficiency.

[0013] In an embodiment of the present disclosure, the method further includes:

[0014] In the process of training the pre-trained model through a local data set to obtain a first model, the second matrix is ​​updated through a preset optimization algorithm to obtain a third matrix, and the third matrix is ​​sent to at least one adjacent client that has a communication connection with the client.

[0015] In an embodiment of the present disclosure, generating a second model based on the first model and the at least one third matrix includes:

[0016] For any one of the at least one third matrix, calculating a model update parameter based on the third matrix, the first matrix, and a first weight coefficient corresponding to the third matrix;

[0017] Calculating model parameters based on the first model and a second weight coefficient corresponding to the first model;

[0018] The second model is generated based on the model parameters and a plurality of model update parameters corresponding one-to-one to the at least one third matrix.

[0019] In the embodiment of the present disclosure, the third matrix is ​​obtained by updating the second matrix through a preset optimization algorithm, including:

[0020] Calculate the loss function based on the output result of the first model and the true label;

[0021] Calculating the gradient of the loss function with respect to the second matrix by a back propagation algorithm;

[0022] The third matrix is ​​calculated based on the second matrix and the gradient of the loss function with respect to the second matrix.

[0023] In an embodiment of the present disclosure, the preset optimization algorithm includes at least one of the following: a distributed parallel gradient descent algorithm, a distributed parallel stochastic gradient descent algorithm, and an adaptive moment estimation algorithm.

[0024] In the embodiment of the present disclosure, communication between any two clients among the multiple clients is mutually reachable.

[0025] In the embodiment of the present disclosure, the pre-trained model and the model fine-tuning module are loaded in the multiple clients respectively, including:

[0026] Loading a model fine-tuning module in a target client so that the model fine-tuning module generates a first matrix and a second matrix of the pre-trained model; the target client is any one of the multiple clients;

[0027] The first matrix is ​​sent to other clients through multi-hop communication broadcast; the other clients are clients other than the target client among the multiple clients.

[0028] The embodiment of the second aspect of the present disclosure provides a model fine-tuning device, comprising:

[0029] A loading module, used to load the pre-trained model and the model fine-tuning module in the client, so that the model fine-tuning module generates a first matrix and a second matrix of the pre-trained model; the elements of the first matrix are extracted from a random Gaussian distribution; the elements of the second matrix are all zero;

[0030] A model training module, used to perform model training on the pre-trained model through a local data set to obtain a first model, wherein the local data sets of the multiple clients are different;

[0031] A matrix receiving module, used to receive at least one third matrix, wherein each third matrix is ​​sent by an adjacent client, and each third matrix is ​​obtained by an adjacent client updating the second matrix through a preset optimization algorithm;

[0032] An aggregation module is used to generate a second model based on the first model and the at least one third matrix.

[0033] An embodiment of the third aspect of the present disclosure provides an electronic device, which includes a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the model fine-tuning method described in the first aspect by executing the computer instructions.

[0034] An embodiment of the fourth aspect of the present disclosure provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to enable a computer to execute the model fine-tuning method described in the first aspect.

[0035] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will become apparent from the following description or be learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] By reading the detailed description of the preferred embodiment below, various other advantages and benefits will become clear to those of ordinary skill in the art. The accompanying drawings are only used for the purpose of illustrating the preferred embodiment and are not considered to be limitations of the present disclosure. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings.

[0037] In the attached picture:

[0038] Figure 1 A schematic diagram of an existing LoRA fine-tuning module provided by an embodiment of the present disclosure is shown;

[0039] Figure 2 A schematic diagram of a DLoRA fine-tuning module provided in an embodiment of the present disclosure is shown;

[0040] Figure 3 A schematic diagram of a centralized federated learning scenario provided by an embodiment of the present disclosure is shown;

[0041] Figure 4 A schematic diagram of a decentralized federated learning scenario provided by an embodiment of the present disclosure is shown;

[0042] Figure 5 A schematic diagram of a flow chart of a model fine-tuning method provided by an embodiment of the present disclosure is shown;

[0043] Figure 6 A schematic diagram showing a flow chart of another model fine-tuning method provided by an embodiment of the present disclosure;

[0044] Figure 7 A schematic diagram showing a comparison of model performance under different data sets provided by an embodiment of the present disclosure;

[0045] Figure 8A schematic diagram showing a comparison of the results of fine-tuning the Bert model on the SST2 dataset provided by an embodiment of the present disclosure;

[0046] Fig. 9 A schematic diagram of the structure of a model fine-tuning device provided by an embodiment of the present disclosure is shown;

[0047] Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown;

[0048] Fig.11 A schematic diagram of a storage medium provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0049] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0050] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present disclosure should have the common meanings understood by those skilled in the art to which the present disclosure belongs.

[0051] The following describes the technical names and technical scenarios involved in the embodiments of the present disclosure.

[0052] Large base models, including large language models, visual conversion models, diffusion models, and LLM-based multimodal models, affect the entire machine learning lifecycle from training to deployment. These models can effectively capture complex patterns and features through pre-training on large-scale datasets, thus demonstrating excellent capabilities in a variety of downstream applications. However, in order to enable these base models to exhibit higher accuracy and adaptability in specific application scenarios, parameter-efficient fine-tuning techniques (such as low-rank adaptation, LoRA) have become key. LoRA significantly reduces computational and storage requirements while maintaining model performance through low-rank adaptive adjustment of model parameters. This makes LoRA show great potential in personalized model tuning.

[0053] In traditional federated learning (FL), LoRA has been proven to effectively reduce communication overhead and improve model adaptability. The centralized architecture of FL aggregates model updates from various clients through a central server, thereby avoiding direct data transmission, reducing the risk of data leakage and alleviating compliance challenges associated with data regulations such as GDPR. Although centralized federated learning can improve aggregation efficiency, it also introduces the risk of single point failure and potential privacy leakage issues. Therefore, decentralized federated learning (DFL) as an alternative learning paradigm is receiving increasing attention. DFL eliminates the central server and allows each node to communicate directly and aggregate model updates, thereby enhancing the robustness and privacy protection of the system. However, when applying LoRA technology to federated learning scenarios (including centralized federated learning and decentralized federated learning), there are several technical challenges:

[0054] High communication overhead: The aggregation nature of federated learning requires nodes to frequently exchange model parameters to ensure the convergence and consistency of the global model. Although LoRA has the advantage of reducing the number of parameters for a single model update, frequent inter-node communication still leads to high communication overhead. As the network scales, this communication demand may cause bandwidth bottlenecks, increase network latency, and significantly increase the energy consumption of the system, thus limiting the application of LoRA fine-tuning in large-scale federated learning scenarios;

[0055] Update inconsistency: In the LoRA fine-tuning structure, low-rank matrix decomposition is used to adaptively obtain fine-tuning information, thereby significantly reducing the number of parameters and memory requirements. In the inference and prediction phase of the model, the weight matrix that actually acts on the input tensor is the product of two low-rank matrices, but in the process of federated aggregation, the aggregation of the two low-rank matrices is independent, which causes the update inconsistency problem. Specifically, in the FL and DFL scenarios, the update inconsistency problem can be expressed as follows:

[0056]

[0057]

[0058] Amplification effect of consensus errors: In particular, in a DFL environment, each node only exchanges model parameters with its neighboring nodes through limited communication, lacking central coordination. This decentralized communication mode is prone to consensus errors because model updates of different nodes are difficult to fully synchronize. The semi-quadratic parameter structure of LoRA is particularly sensitive in this environment. Even small consensus errors may be amplified in the model update, resulting in a significant decrease in model performance or even difficulty in convergence. This problem is particularly prominent when the network topology is sparse and the data heterogeneity is high.

[0059] In view of the above problems existing in applying LoRA fine-tuning in FL and DFL, the present invention proposes a generative artificial intelligence distributed model fine-tuning method DLoRA, such as Figure 1 , 2 shown.

[0060] like Figure 1 As shown: The existing LoRA basic module adapts the model to new tasks by adjusting the A and B matrices while keeping the pre-trained model weights W unchanged. This approach is particularly suitable for fine-tuning large models because it allows the model to improve the performance of specific tasks by adjusting a small number of parameters while keeping most of the pre-trained weights unchanged.

[0061] Compared with the LoRA basic module, the embodiment of the present disclosure adds a freezing process (such as Figure 2 DLoRA effectively controls the amplification effect of update inconsistency and consensus errors in a decentralized environment by selectively freezing non-zero initialized matrices and focusing on training zero-initialized low-rank matrices for model fine-tuning. This strategy can maintain the stability of model parameter updates under limited communication conditions, thereby improving the accuracy of the global model. At the same time, freezing non-zero initialized matrices effectively reduces the number of parameters that need to be transmitted between different clients during the model update process, thereby significantly reducing the communication overhead and energy consumption of the system.

[0062] Centralized federated learning, such as Figure 3As shown: It is a centralized machine learning setting that allows multiple clients (such as mobile devices or entire organizations) to collaboratively train models under a central server (also known as a central server), while ensuring that the training data is not uploaded to the cloud, thereby protecting the privacy and security of the data. In centralized federated learning, there is generally only one central server. The central server needs to process the user's parameters (generally speaking, the processing process includes three parts: collection of model parameters, weighted averaging, and distribution to each user). In centralized federated learning, data aggregation and model management are coordinated by a single central node, which aggregates periodically shared model parameters and averages the model parameters uploaded by each node. A typical algorithm for centralized federated learning is FedAvg, which is an iterative process in which the process of each round is roughly as follows:

[0063] The server initializes a global model parameter w 0 , and distribute it to participating clients

[0064] Each client k uses its own dataset F locally k and the learning rate η on w 0 Perform E rounds of training to obtain updated model parameters

[0065] Client k updates the model parameters Sent to the server, the server Average to get the new global model parameter w t+1 , the formula is as follows:

[0066]

[0067] Decentralized federated learning has similar goals to federated learning. It also allows multiple users (such as mobile devices or entire organizations) to collaboratively train machine learning models without sharing local data. In a decentralized federated learning setting, there is no central server. Instead, communication between adjacent users is used to exchange model parameters, so as to achieve the purpose of sharing the model across the entire network (that is, making the model parameters held by each user approach the average of all user model parameters, that is, the consensus average of the model). Since it does not rely on a central server, the decentralized federated learning system has stronger robustness, security and higher communication efficiency.

[0068] like Figure 4As shown in the figure: A decentralized federated learning network consisting of a user group and the communication links between them. Each user has his own local machine learning model (specifically, some parameters corresponding to this model) and local data set. Only users at both ends of the communication link will conduct two-way communication when updating consensus, and exchange model parameters through two-way communication. Two-way communication on the communication link is the main source of communication overhead. Decentralized federated learning mainly includes the following two steps:

[0069] Step 1: Local Update

[0070] At each iteration t, user i will optimize the machine learning model x held by him through the optimization algorithm i (t) is updated. Currently, the optimization algorithms used in decentralized federated learning scenarios mainly include DPGD, DPSGD, Adam, DAdam, etc. The process of local update is as follows:

[0071]

[0072] Among them, h i (t) is the local update of user i at iteration t.

[0073] Step 2: Consensus Update

[0074] In order to ensure the consensus average of the parameter model, each user will share the model parameters with the neighboring users with whom they have communication links. The consensus update is still an update of the local machine learning model parameters. The result of the consensus update is the weighted average of the user's result (i.e. model parameters) in step 1 (i.e. local update) and the neighbor user's result in step 1. The weighted average process is as follows:

[0075]

[0076] The above formula expresses the operation of weighted averaging the model parameters of the user and the model parameters of its neighbors received from the neighbors.

[0077] Where N iis the set of all neighbor users of user i, and W is called the consensus matrix (its elements are the weights used in weighted averaging). The mathematical properties of the consensus matrix W are crucial to ensuring the convergence of the consensus algorithm. Specifically, if W can satisfy row randomness and column randomness, and its second largest eigenvalue β<1, then through consensus update (weighted averaging with the neighbor's model), the parameters of the models held by all users in the network will be able to reach a consensus average (that is, the same and tend to the average value); in contrast, in federated learning, the consensus average of model parameters is achieved by directly collecting the model parameters of each user through the central server, and then averaging them and sending them to each user.

[0078] Comparing the architecture and aggregation methods of centralized federated learning and decentralized federated learning, it can be seen that centralized federated learning generates a global model after each round of aggregation and sends it to each client. Therefore, before the next round of local training begins, the model parameters of all client nodes are completely consistent. In the decentralized federated learning scenario, due to communication restrictions, each node can only exchange model parameter information with neighboring nodes and aggregate. After each round of aggregation, the models of each node are not the same, but through multiple rounds of aggregation, it is possible to achieve multi-hop information transmission in the communication system to obtain global information, but the models of each node are still not completely consistent with the global model. We call this error consensus error. Consensus error will cause inconsistent model parameter updates between nodes, thereby affecting the performance of the global model. Due to the different data distribution between nodes, consensus error will aggravate the deviation between the global model and the local model, which may lead to slower or even non-convergence during the training process. In addition, consensus error will reduce the robustness of the system to malicious nodes, affecting the overall security and reliability of the model. Therefore, in the decentralized federated learning scenario, reducing consensus error is a key measure to ensure model performance.

[0079] LoRA fine-tuning, LoRA (Low Rank Adaptation) is a parameter-efficient fine-tuning technique designed to address the challenges of fine-tuning large-scale pre-trained models, especially for scenarios with limited computing and storage resources. LoRA significantly reduces the number of trainable parameters by selectively injecting trainable low-rank decomposition matrices into specific layers of the pre-trained model, thereby reducing the cost of fine-tuning and making the model more adaptable to specific tasks. The core idea of ​​LoRA is to approximate the changes required for fine-tuning through low-rank matrices that can capture the necessary task-specific adjustments without directly modifying the weights of the original large model. Specifically, for the weight matrix W in the pre-trained model 0 ∈R d×d , LoRA introduces a low-rank decomposition:

[0080] W=W 0+BA 0

[0081] Among them, B∈R d×r and A∈R r×d is a low-rank matrix whose dimensions are carefully chosen to reduce computational overhead while maintaining the expressiveness of the model. During the forward propagation, ΔW = BA is scaled by a factor Scaling, where α is a hyperparameter and r is the rank of the low-rank matrix. This scaling ensures that the impact of fine-tuning remains balanced relative to the pre-trained matrix. Matrix A is initialized with a random Gaussian, while matrix B is initialized to zero, which ensures that the model remains close to its pre-trained state at the beginning and then gradually updates during fine-tuning.

[0082] Current application status of LoRA in federated learning

[0083] At present, the application of LoRA technology in federated learning focuses on centralized federated learning scenarios, mainly using LoRA technology for fine-tuning on the local side to reduce storage resource requirements, and reducing communication overhead by only transmitting low-rank matrices when communicating with the central server. In addition, a few studies have explored the personalized federated learning problems and privacy protection issues caused by LoRA in centralized federated learning scenarios. The combination of LoRA and FL has been successful, but there is still a lack of exploration of applying LoRA fine-tuning in decentralized federated learning scenarios.

[0084] According to an embodiment of the present disclosure, an embodiment of a model fine-tuning method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0085] In this embodiment, a model fine-tuning method is provided. The method is applied to a model system to be fine-tuned. The model system to be fine-tuned includes multiple clients.

[0086] In the embodiment of the present disclosure, the model system to be fine-tuned can be understood as the above-mentioned Figure 4 Decentralized federated learning in .

[0087] Figure 5 is a flowchart of a model fine-tuning method according to an embodiment of the present disclosure, and the execution subject of the flowchart is any one of the above-mentioned multiple clients, such as Figure 5 As shown, the process includes the following steps:

[0088] Step S101, loading the pre-trained model and the model fine-tuning module in the client, so that the model fine-tuning module generates a first matrix and a second matrix of the pre-trained model.

[0089] In the embodiment of the present disclosure, when the model system to be fine-tuned is started, each client in the system loads the same pre-trained model and model fine-tuning module (i.e., DLoRA module), and initializes the low-rank matrix; wherein the low-rank matrix includes a first matrix (e.g. Figure 2 A matrix in) and the second matrix (i.e. Figure 2 The first matrix is ​​initialized with random Gaussian, which means that the elements of the first matrix are drawn from random Gaussian distribution; the second matrix is ​​initialized with zero, which means that the elements of the second matrix are all zero.

[0090] In some specific embodiments, the above step S101 includes steps S1011 and S1012:

[0091] Step S1011, loading a model fine-tuning module in a target client so that the model fine-tuning module generates a first matrix and a second matrix of the pre-trained model; the target client is any one of the multiple clients;

[0092] Step S1012: sending the first matrix to other clients through multi-hop communication broadcast; the other clients are clients other than the target client among the multiple clients.

[0093] In the above steps S1011-S1012, in order to ensure that all clients in the model system to be fine-tuned have the same first matrix, any one client can be selected from all clients as the target client; then, the model fine-tuning module is initialized on the target client to obtain the first matrix and the second matrix of the pre-trained model; then, the first matrix is ​​forwarded to all clients by multi-hop communication broadcast. The maximum communication round is the same as the maximum path length of the network topology of the model system to be fine-tuned.

[0094] Step S102: training the pre-trained model using a local data set to obtain a first model.

[0095] In the embodiment of the present disclosure, the local data sets of multiple clients in the model system to be fine-tuned are different. Therefore, the first models obtained by each client through model training of the pre-trained model using its own local data set are also different.

[0096] In some specific embodiments, in the process of training the pre-trained model through a local data set to obtain a first model, the second matrix is ​​updated through a preset optimization algorithm to obtain a third matrix, and the third matrix is ​​sent to at least one adjacent client that has a communication connection with the client.

[0097] In the disclosed embodiment, communication between any two clients among the multiple clients of the model system to be fine-tuned is mutually reachable, for example: for any two clients, client A and client B, client A can send information to client B through a communication link; client B can also send information to client A through the same communication link.

[0098] In some specific embodiments, the specific implementation steps of obtaining the first model by training the model through the local data set can be performed in the following form:

[0099] W=W 0 +BA 0

[0100] Where W represents the model parameters of the first model, W 0 represents the model parameters of the pre-trained model, B represents the second matrix, A 0 Represents the first matrix.

[0101] During the model training process, first, ensure that the model parameters W of the pre-trained model 0 and the first matrix A 0 remains unchanged; secondly, the initialized second matrix B is updated by using the optimization algorithm to obtain an updated second matrix, that is, the third matrix; finally, the third matrix is ​​multiplied by the first matrix and merged into the model parameters of the pre-trained model to obtain the model parameters of the first model.

[0102] In some specific embodiments, the above-mentioned updating of the second matrix by a preset optimization algorithm to obtain the third matrix includes steps a1 to a3:

[0103] Step a1, calculating the loss function according to the output result of the first model and the true label;

[0104] Specifically, the output result can be obtained by processing the input data through the weights of the pre-trained model and the model fine-tuning module (i.e., the LoRA module), as follows:

[0105] h=W 0 x+BAx

[0106] Among them, h represents the output result, W 0 represents the weight of the pre-trained model, x represents the input data, A represents the first matrix, and B represents the second matrix.

[0107] Step a2, calculating the gradient of the loss function with respect to the second matrix by a back propagation algorithm;

[0108] Step a3: Calculate the third matrix based on the second matrix and the gradient of the loss function with respect to the second matrix.

[0109] Specifically, the third matrix can be calculated as follows:

[0110]

[0111] Among them, B new represents the third matrix, B old represents the second matrix, represents the gradient of the loss function with respect to the second matrix, and η represents the learning rate.

[0112] In the embodiment of the present disclosure, each client will send the updated second matrix, that is, the third matrix, to the adjacent client after training the model through the local data set to obtain the first model. Compared with LoRA transmitting the first matrix A and the second matrix B at the same time, the improved solution DLoRA provided by the embodiment of the present disclosure only transmits the updated second matrix, thereby reducing the communication overhead by half.

[0113] In some specific embodiments, the above-mentioned preset optimization algorithm includes at least one of the following: distributed parallel gradient descent algorithm (DPGD, Distributed Parallel Gradient Descent), distributed parallel stochastic gradient descent algorithm (DPSGD, Distributed Parallel Stochastic Gradient Descent), adaptive moment estimation algorithm (Adam, Adaptive Moment Estimation) and DAdam (which can be understood as a distributed improved version of Adam that does not converge in some cases).

[0114] Step S103: receiving at least one third matrix.

[0115] In the embodiment of the present disclosure, each third matrix received is sent by an adjacent client, and each third matrix is ​​obtained by an adjacent client updating the second matrix through a preset optimization algorithm. The specific implementation steps of the update can refer to the above steps a1 to a3.

[0116] Step S104: generating a second model based on the first model and the at least one third matrix.

[0117] In some specific embodiments, the above step S104 includes steps S1041 to S1043:

[0118] Step S1041: For any one of the at least one third matrix, a model update parameter is calculated based on the third matrix, the first matrix, and a first weight coefficient corresponding to the third matrix.

[0119] In the embodiment of the present disclosure, during the process of calculating the model update parameters,

[0120] First, for each neighboring client, the third matrix sent by the neighboring client is calculated using the following formula according to the third matrix, the first matrix and the pre-trained model to obtain the first model of the neighboring client;

[0121] X j =W 0 +B 1 A

[0122] Among them, X j represents the first model of the jth neighboring client, W 0 represents the pre-trained model, B 1 represents the third matrix, and A represents the first matrix.

[0123] Then, the product of the first weight coefficient corresponding to the neighboring client and the first model is used as the model update parameter of the neighboring client, such as the following formula:

[0124] Y i,j =W i,j X j

[0125] Among them, Y i,j represents the model update parameter of the jth neighboring client of client i, W i,j represents the first weight coefficient of the jth neighboring client of client i; X j represents the first model of the jth neighboring client.

[0126] Step S1041: Calculate model parameters based on the first model and a second weight coefficient corresponding to the first model.

[0127] In the embodiment of the present disclosure, the model parameters of the i-th client can be calculated by the following formula:

[0128] Y i =W i X i

[0129] Among them, Y i represents the model parameters of the i-th client, W i represents the second weight coefficient of the i-th client, X i Represents the first model of the i-th client.

[0130] Step S1041: Generate the second model based on the model parameters and a plurality of model update parameters corresponding one-to-one to the at least one third matrix.

[0131] In the embodiment of the present disclosure, the second model can be calculated by the following formula:

[0132]

[0133] Among them, N i represents the set of all neighboring clients of the i-th client; Z i represents the second model of the i-th client; Y i represents the model parameters of the i-th client; W represents the sum of the model update parameters of all neighboring clients of the i-th client. i and W i,j Represents the weight coefficient of the corresponding client, which can be dynamically adjusted according to the trust or connection strength between clients to ensure more efficient model parameter fusion.

[0134] The disclosed embodiment performs model fine-tuning by adding a selective freezing of non-zero initialization matrix process outside the model fine-tuning module (i.e., the LoRA basic module), thereby reducing consensus error and improving model accuracy; and reducing communication overhead and improving communication efficiency.

[0135] Corresponding to the implementation of the above model fine-tuning method, the present disclosure also provides a specific implementation method, such as Figure 6 As shown:

[0136] Step 1: Model initialization

[0137] When the system starts, each node loads the same pre-trained model and DLoRA module, and initializes the low-rank matrix. The A matrix is ​​initialized with random Gaussian, and the B matrix is ​​initialized with zero. To ensure that all participating nodes have the same initialization matrix A, a single node can be used to initialize the DLoRA module and broadcast it to all participating nodes through multi-hop communication. The maximum communication round is the same as the maximum path length of the network topology.

[0138] Step 2: Local training

[0139] Each node performs model training on the local data set. The local training update formula is shown in the above formula (1). The pre-trained model parameter W 0 and A 0 The matrix remains frozen, and the optimization algorithm is used to update the zero-initialized B matrix. That is, the model parameters are organized in the following form:

[0140] W=W 0 +BA 0

[0141] Step 3: Parameter passing

[0142] After the local training is completed, the node sends the updated B matrix to the neighboring nodes through the communication module. Compared with LoRA, which transmits both matrices B and A at the same time, our improved solution DLoRA only transmits matrix B, thereby reducing the communication overhead by half.

[0143] Step 4: Consensus Update

[0144] The node receives the B matrix parameter updates from neighboring nodes, performs weighted averaging on these updates through the parameter aggregation module, and updates the local B matrix. The update formula is shown in the above formula (2). During the weighted averaging process, the weight coefficient can be dynamically adjusted according to the trust or connection strength between nodes to ensure more efficient model parameter fusion.

[0145] The system model of the fine-tuning solution meets the following basic constraints:

[0146] 1) The network composed of client nodes is T-strongly connected, the weight matrix is ​​non-negative and satisfies double randomness (i.e. row randomness and column randomness, row randomness means that the sum of the elements in any row of the matrix is ​​1, column randomness means that the sum of the elements in any column of the matrix is ​​1), and the second largest eigenvalue of the weight matrix β < 1. The second largest eigenvalue of the weight matrix is ​​related to the network connection density. The larger the network connection density, the smaller the second largest eigenvalue of the weight matrix.

[0147] The network has synchronized clocks, and the update rule (3) is applied to all nodes at a uniform time interval; the local update is bounded and satisfies the following formula:

[0148] max(||h B (k)||,||h A (k)||)≤D.

[0149] Among them, h B (k) is the vector composed of the update amount of the local B matrix of all client nodes in the kth round, h A (k) is the vector composed of the update amounts of the local A matrix of all client nodes in the kth round.

[0150] The above constraints are common basic constraints in decentralized federated learning scenarios. When the above constraints are met, the upper bound of the consensus error of our proposed method satisfies

[0151]

[0152] Under the same conditions, the upper bound of the consensus error of directly integrating LoRA fine-tuning into the DFL architecture satisfies:

[0153]

[0154] The embodiments of the present disclosure have the following technical effects:

[0155] 1. Freezing the A matrix for fine-tuning has lower consensus error than directly integrating decentralized federated learning and LoRA fine-tuning;

[0156] 2. Compared with direct LoRA fine-tuning, it is less sensitive to network topology and data distribution heterogeneity, and the model performance has better robustness;

[0157] 3. Compared with directly using LoRA fine-tuning, the communication overhead in the aggregation process is reduced by half, saving the computational overhead of calculating the gradient of the A matrix during back-propagation.

[0158] The disclosed embodiment also provides an analysis of experimental results, including the results of consensus error, model accuracy under different network topologies and data heterogeneity. The details are as follows:

[0159] like Figure 7 As shown: Comparison of model performance (accuracy) under different data sets, showing the comparison of model performance of the disclosed solution and the benchmark solution LoRA under different data sets and different data distribution heterogeneity. The experiment uses the Bert model, and the data sets used all belong to the GLUE data set, where sst2 and qnli are two-classification problems, and mnli_matched and mnli_mismatched are three-classification problems. It can be seen that the disclosed solution shows better model performance on different data sets and different data distributions, especially in the three-classification problem. Directly applying LoRA to the DFL scenario basically cannot achieve the training effect (the random classification accuracy is 33.33%).

[0160] like Figure 8 As shown: the results of fine-tuning the Bert model on the sst2 dataset are compared. The sst2 dataset uses Dirichlet partitioning to achieve Non-iid distribution of the data, and the Dirichlet coefficient α = 0.25. At the same time, it can be seen from the figure that the LoRA scheme performs better than DLoRA in some rounds, but the model performance stability is poor, it is difficult to achieve stable convergence, and the confidence interval is large. In contrast, the disclosed scheme has higher stability and robustness, can provide stable performance and a smaller confidence interval.

[0161] As shown in Table 1: The solution proposed in the embodiment of the present disclosure reduces the number of parameters by half by selectively freezing the non-zero initialization matrix. Therefore, in terms of communication overhead, the communication overhead of the solution of the present disclosure is reduced by more than half compared with LoRA. The present disclosure adds low-rank modules of different sizes to the Q matrix, V matrix and the last two linear layers of the Bert model, and measures the communication overhead of LoRA and DLoRA. The specific results are shown in Table 1 (the model parameters use Float32 precision, and the communication volume is measured by the size of the model parameters sent by node i to node k in a single time).

[0162] Table 1

[0163]

[0164] Corresponding to the implementation of the above model fine-tuning method, the embodiment of the present disclosure also provides a model fine-tuning device for performing the above Figures 1 to 8 The model fine-tuning method described in any of the embodiments shown in the figure. Fig. 9 As shown, the model fine-tuning device includes:

[0165] A loading module, used to load the pre-trained model and the model fine-tuning module in the client, so that the model fine-tuning module generates a first matrix and a second matrix of the pre-trained model; the elements of the first matrix are extracted from a random Gaussian distribution; the elements of the second matrix are all zero;

[0166] A model training module, used to perform model training on the pre-trained model through a local data set to obtain a first model, wherein the local data sets of the multiple clients are different;

[0167] A matrix receiving module, used to receive at least one third matrix, wherein each third matrix is ​​sent by an adjacent client, and each third matrix is ​​obtained by an adjacent client updating the second matrix through a preset optimization algorithm;

[0168] An aggregation module is used to generate a second model based on the first model and the at least one third matrix.

[0169] Optionally, the device also includes: a matrix update module, which is used to update the second matrix through a preset optimization algorithm to obtain a third matrix during the process of training the pre-trained model through a local data set to obtain a first model, and send the third matrix to at least one adjacent client that has a communication connection with the client.

[0170] Optionally, the model fine-tuning module is also used to: for any one of the at least one third matrices, calculate the model update parameters based on the third matrix, the first matrix and a first weight coefficient corresponding to the third matrix; calculate the model parameters based on the first model and a second weight coefficient corresponding to the first model; generate the second model based on the model parameters and a plurality of model update parameters corresponding one-to-one to the at least one third matrix.

[0171] Optionally, the matrix update module is also used to: calculate the loss function based on the output results of the first model and the true label; calculate the gradient of the loss function with respect to the second matrix through the back propagation algorithm; and calculate the third matrix based on the second matrix and the gradient of the loss function with respect to the second matrix.

[0172] Optionally, the preset optimization algorithm includes at least one of the following: a distributed parallel gradient descent algorithm, a distributed parallel stochastic gradient descent algorithm, and an adaptive moment estimation algorithm.

[0173] Optionally, the loading module is also used to: load a model fine-tuning module in a target client so that the model fine-tuning module generates a first matrix and a second matrix of the pre-trained model; the target client is any one of the multiple clients; the first matrix is ​​sent to other clients via multi-hop communication broadcast; the other clients are clients other than the target client among the multiple clients.

[0174] The model fine-tuning device provided in the above-mentioned embodiment of the present disclosure and the model fine-tuning method provided in the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0175] The present disclosure also provides an electronic device to perform the above-mentioned model fine-tuning method. Fig.10 , which shows a schematic diagram of an electronic device provided by some embodiments of the present disclosure. Fig.10 As shown, the electronic device 10 includes: a processor 1000, a memory 1001, a bus 1002 and a communication interface 1003, wherein the processor 1000, the communication interface 1003 and the memory 1001 are connected via the bus 1002; the memory 1001 stores a computer program that can be run on the processor 1000, and the processor 1000 executes the aforementioned Figures 1 to 8 The model fine-tuning method provided by any of the illustrated embodiments.

[0176] The memory 1001 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 1003 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0177] The bus 1002 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 1001 is used to store programs. After receiving the execution instruction, the processor 1000 executes the program. Figures 1 to 8 The model fine-tuning method disclosed in any of the illustrated embodiments may be applied to the processor 1000 or implemented by the processor 1000 .

[0178] The processor 1000 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1000. The above processor 1000 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1001, and the processor 1000 reads the information in the memory 1001 and completes the steps of the above method in combination with its hardware.

[0179] The electronic device provided by the embodiment of the present disclosure and the model fine-tuning method provided by the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by them.

[0180] The present disclosure also provides a computer-readable storage medium corresponding to the model fine-tuning method provided in the above embodiment. Fig.11 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the model fine-tuning method provided by any of the aforementioned embodiments will be executed.

[0181] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0182] The computer-readable storage medium provided by the above-mentioned embodiments of the present disclosure and the model fine-tuning method provided by the embodiments of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0183] It should be noted that:

[0184] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present disclosure can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0185] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present disclosure, the various features of the present disclosure are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed disclosure requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, the inventive aspects lie in less than all the features of the single embodiment disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present disclosure.

[0186] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of this disclosure and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0187] The above is only a preferred specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A model fine-tuning method, characterized in that: The method is applied to a model system to be fine-tuned, wherein the model system to be fine-tuned includes a plurality of clients; For any one of the multiple clients, the method includes: Loading the pre-trained model and the model fine-tuning module in the client, so that the model fine-tuning module generates a first matrix and a second matrix of the pre-trained model; the elements of the first matrix are extracted from a random Gaussian distribution; the elements of the second matrix are all zero; Performing model training on the pre-trained model through a local data set to obtain a first model, wherein the local data sets of the multiple clients are different; Receive at least one third matrix, wherein each third matrix is ​​sent by a neighboring client, and each third matrix is ​​obtained by updating the second matrix by a neighboring client through a preset optimization algorithm; A second model is generated based on the first model and the at least one third matrix.

2. The method according to claim 1, characterized in that: The method further comprises: In the process of training the pre-trained model through a local data set to obtain a first model, the second matrix is ​​updated through a preset optimization algorithm to obtain a third matrix, and the third matrix is ​​sent to at least one adjacent client that has a communication connection with the client.

3. The method according to claim 1 or 2, characterized in that: Generating a second model based on the first model and the at least one third matrix includes: For any one of the at least one third matrix, calculating a model update parameter based on the third matrix, the first matrix, and a first weight coefficient corresponding to the third matrix; Calculating model parameters based on the first model and a second weight coefficient corresponding to the first model; The second model is generated based on the model parameters and a plurality of model update parameters corresponding one-to-one to the at least one third matrix.

4. The method according to claim 2, characterized in that: The third matrix is ​​obtained by updating the second matrix by a preset optimization algorithm, including: Calculate the loss function based on the output result of the first model and the true label; Calculating the gradient of the loss function with respect to the second matrix by a back propagation algorithm; The third matrix is ​​calculated based on the second matrix and the gradient of the loss function with respect to the second matrix.

5. The method according to claim 1 or 2, characterized in that: The preset optimization algorithm includes at least one of the following: a distributed parallel gradient descent algorithm, a distributed parallel stochastic gradient descent algorithm and an adaptive moment estimation algorithm.

6. The method according to claim 1 or 2, characterized in that: The communication between any two clients among the multiple clients is mutually reachable.

7. The method according to claim 6, characterized in that The pre-trained model and the model fine-tuning module are loaded in the multiple clients respectively, including: Loading a model fine-tuning module in a target client so that the model fine-tuning module generates a first matrix and a second matrix of the pre-trained model; the target client is any one of the multiple clients; The first matrix is ​​sent to other clients through multi-hop communication broadcast; the other clients are clients other than the target client among the multiple clients.

8. A model fine-tuning device, characterized in that: The device comprises: A loading module, used to load the pre-trained model and the model fine-tuning module in the client, so that the model fine-tuning module generates a first matrix and a second matrix of the pre-trained model; the elements of the first matrix are extracted from a random Gaussian distribution; the elements of the second matrix are all zero; A model training module, used to perform model training on the pre-trained model through a local data set to obtain a first model, wherein the local data sets of the multiple clients are different; A matrix receiving module, used to receive at least one third matrix, wherein each third matrix is ​​sent by an adjacent client, and each third matrix is ​​obtained by an adjacent client updating the second matrix through a preset optimization algorithm; An aggregation module is used to generate a second model based on the first model and the at least one third matrix.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the model fine-tuning method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the model fine-tuning method according to any one of claims 1 to 7.

Images