Federal learning multi-granularity grouping fine tuning method with high communication efficiency

Abstract

The invention discloses a communication-efficient federated learning multi-granularity grouping fine tuning method, relates to the technical field of information security, and solves the technical problems that the communication overhead of the current federated learning is large and the model performance is weak in a non-independent identically distributed data (Non-IID) scene. The method comprises the following steps: constructing a pre-training model of a client through a LoRA low-rank parameter matrix; the client performs asymmetric fine adjustment on the parameter matrix B and uploads the matrix B to the server; the server performs multi-granularity grouping and calculates an intra-group weighted average low-rank matrix; and the server performs weighted averaging on the expert gating parameters corresponding to the clients to obtain an average expert gating parameter, and the average expert gating parameter is used for reasoning of a global model. According to the invention, the training and communication overhead of the LoRA is reduced by using the asymmetric training model of the low-rank adaptation matrix, and the precision of the model in a Non-IID scene is improved through a training mode of combining the LoRA with the mixed experts.

Description

Technical Field

[0001] The present invention belongs to the field of information security technology, and specifically is a communication efficient federated learning multi-granularity grouping fine-tuning method. Background Art

[0002] In the field of information security, federated learning is an emerging machine learning paradigm that can complete some model training tasks that require joint execution by users or institutions without leaving the local data, thereby effectively protecting the privacy and security of users and institutions.

[0003] With the development of deep learning, the performance of models is getting stronger and stronger, especially large language models and multimodal models developed based on pre-trained models. However, the size of these models is very large, and direct training of these large models by different users or institutions through federated learning will incur huge communication overhead.

[0004] One-Shot federated learning is a communication-efficient federated learning paradigm. Different from the multi-round communication federated learning method, One-Shot federated learning expects to complete the training process of federated learning through one round of communication, thereby greatly reducing the traffic overhead caused by multiple rounds of communication. Existing One-Shot federated learning methods mainly include: ensemble learning-based methods, knowledge distillation methods, and model fusion methods.

[0005] Methods based on ensemble learning usually require the client to transfer the trained complete model to the server, where it is integrated into a large model for inference. The drawback of this approach is that the size of the integrated model is related to the number of clients. When the number of clients is large, the size of the integrated model will be very large, and the total communication overhead will also increase. Methods based on knowledge distillation usually require synthetic or public datasets on the server for model training on the server; for knowledge distillation methods that do not require additional datasets, additional computational overhead is usually required. Methods based on model fusion usually require fine-tuning of all parameters and then transmitting the entire model, and even require transmitting additional auxiliary parameter matrices for model fusion on the server. When using large models for federated learning training, the communication overhead is still very large.

[0006] The present invention provides a communication efficient federated learning multi-granularity grouping fine-tuning method to solve the above technical problems. Summary of the invention

[0007] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a communication-efficient federated learning multi-granularity grouping fine-tuning method to solve the technical problems of high communication overhead of current federated learning and weak model performance in non-independent and identically distributed (Non-IID) data scenarios.

[0008] To achieve the above object, a first aspect of the present invention provides a communication efficient federated learning multi-granularity grouping fine-tuning method, comprising:

[0009] S1: Build a pre-trained model of the client through the LoRA low-rank parameter matrix; wherein the LoRA low-rank parameter matrix includes parameter matrix A and parameter matrix B;

[0010] S2: The client performs asymmetric fine-tuning on the parameter matrix B and uploads the parameter matrix B to the server;

[0011] S3: Calculate the similarity vector between each client;

[0012] S4: The server performs multi-granularity grouping and calculates the weighted average low-rank matrix within the group;

[0013] S5: The client performs low-rank hybrid expert setting and loads several sets of low-rank matrices to obtain expert LoRA;

[0014] S6: The client fine-tunes the expert gating parameters through low-rank hybrid experts and uploads them to the server;

[0015] S7: The server performs weighted averaging on the expert gating parameters corresponding to each client to obtain an average expert gating parameter, and uses it for reasoning of the global model.

[0016] Preferably, the pre-training model of the client is constructed by using the LoRA low-rank parameter matrix, including:

[0017] All clients use the same pre-trained model as the base model for subsequent fine-tuning and freeze the pre-trained model parameter matrix. And add the LoRA low-rank parameter matrix; among them, the LoRA low-rank parameter matrix includes the parameter matrix and the parameter matrix R is a real number, r is a low-rank dimension, d in is the input dimension of the pre-trained model parameter matrix, d out is the output dimension of the pre-trained model parameter matrix.

[0018] Preferably, the client performs asymmetric fine-tuning on the parameter matrix B and uploads the parameter matrix B to the server, including:

[0019] X1: All clients use the same LoRA initialization method. After initialization, the LoRA parameters of all clients are the same.

[0020] X2: Each client uses a local data set to train a pre-trained model. During training, parameter matrix A needs to be frozen and only parameter matrix B needs to be trained.

[0021] X3: The client uploads the trained parameter matrix B to the server, and parameter matrix A does not need to be uploaded.

[0022] The present invention utilizes the asymmetry of LoRA to fine-tune the parameter matrix B. Since the parameter matrix A extracts features from the input and the parameter matrix B outputs data based on the features during model fine-tuning using LoRA, fine-tuning only the parameter matrix B is as effective as fine-tuning both the parameter matrix A and the parameter matrix B at the same time, and the amount of parameters that need to be learned and transmitted is only half of the latter. In the present invention, the client needs to freeze the parameter matrix A when training the pre-trained model and only train the parameter matrix B. The client uploads the trained parameter matrix B to the server, and the parameter matrix A does not need to be uploaded, which is beneficial to reducing the communication overhead of federated learning.

[0023] Preferably, the calculating of the similarity vectors between the clients includes:

[0024] Set the number of clients to N, and the pre-trained model of each client has H layers, B n,h Denotes the parameter matrix B of the h-th layer of the n-th client, then the similarity score calculation formula of the parameter matrix B corresponding to the h-th layer of the n+1-th client and the h-th layer of the n-th client is:

[0025]

[0026] After calculating the layer_score (n+1, n, h) of a total of h layers, the calculation formula for the similarity client_score between the n+1th client and the nth client is as follows:

[0027]

[0028] The similarity vector α of the nth client n for:

[0029] α n =[client_score(n,1),client_score(n,2),…,client_score(n,n)];

[0030] Where n = 1, 2, ..., N, N is the total number of clients; h = 1, 2, ..., H, H is the total number of layers of the pre-trained model in the client.

[0031] The present invention calculates the similarity vectors between the various clients. The similarity vectors can represent the similarity between data sets in different clients, making it convenient to subsequently group the clients according to the similarity vectors.

[0032] Preferably, the server performs multi-granularity grouping and calculates the weighted average low-rank matrix within the group, including:

[0033] Use the similarity vector and K-Means algorithm to group the parameter matrix B into multiple granularities to obtain M groups, which are marked as i. Perform weighted averaging on all parameter matrices in group i according to the amount of data corresponding to the client to obtain a weighted average low-rank matrix within the group. And sent to the client; where i = 1, 2, ..., M, M is the total number of groups; the multi-granularity grouping methods include:

[0034] Client granularity grouping: Calculate the average vector of the similarity vectors of all layers of the pre-trained model in the client to obtain the client granularity similarity vector, and then use the K-Means algorithm to divide the client granularity similarity vector into M groups;

[0035] Model block granularity grouping: calculate the average vector of the similarity vectors of all layers in the same block of the same client to obtain the similarity vector of the model block granularity, and then use the K-Means algorithm to divide the similarity vector of the model block granularity into M groups;

[0036] Model layer granularity grouping: directly use the K-Means algorithm to divide the similarity vectors of the model layer granularity into M groups.

[0037] The present invention uses the K-Means algorithm to perform multi-granularity grouping according to the similarity vector, and groups clients with similarity into the same group, which is beneficial to improving the performance of federated learning in Non-IID scenarios and effectively reduces the amount of parameters transmitted using the hybrid expert method when the number of clients is large.

[0038] Preferably, the client performs low-rank mixed expert setting and loads several groups of low-rank matrices to obtain expert LoRA, including:

[0039] The client downloads M in-group weighted average low-rank matrices The local pre-trained model of the client remains unchanged, the local parameter matrix A of the client is copied M times, and the M downloaded group weighted average low-rank matrices are used Replace the local parameter matrix B to obtain M The matrix, in turn The matrix is ​​labeled as the i-th expert LoRA, and M expert LoRAs are obtained.

[0040] Preferably, the client fine-tunes the expert gating parameters by using a low-rank hybrid expert and uploads the parameters to the server, including:

[0041] The client freezes the pre-trained model and expert LoRA. Freezing expert LoRA includes freezing parameter matrix A and parameter matrix Fine-tune the expert gating parameter G of client n, and upload the trained expert gating parameter G to the server after fine-tuning. For the expert gating of the hybrid expert model, the calculation formula for the allocation probability of the i-th expert LoRA is:

[0042]

[0043] Among them, x is the input of expert gating, G i is the expert gating parameter corresponding to the i-th expert LoRA, e is a natural constant, and the softmax() function is used to calculate the allocation probability of the expert gating unit;

[0044] By formula Calculate the total output of the model O(x).

[0045] It should be noted that the client only needs to fine-tune the expert gating parameters for a small number of rounds, otherwise the client model is prone to overfitting.

[0046] Preferably, the server performs weighted averaging on the expert gating parameters corresponding to each client to obtain an average expert gating parameter, including:

[0047] The server performs a weighted average of the expert gating parameters corresponding to the client according to the amount of data received from the client to obtain the average expert gating parameter The average expert gating parameter and expert LoRA are loaded into the global model of the server, and the global model is based on the average expert gating parameters Reasoning with expert LoRA.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] 1. The present invention utilizes the asymmetry of LoRA to fine-tune the parameter matrix B. Since the parameter matrix A extracts features from the input and the parameter matrix B outputs data based on the features during model fine-tuning using LoRA, fine-tuning only the parameter matrix B is as effective as fine-tuning both the parameter matrix A and the parameter matrix B at the same time, and the amount of parameters that need to be learned and transmitted is only half of the latter. In the present invention, the client needs to freeze the parameter matrix A when training the pre-trained model and only train the parameter matrix B. The client uploads the trained parameter matrix B to the server, and the parameter matrix A does not need to be uploaded, which is beneficial to reducing the communication overhead of federated learning. In addition, the accuracy of the model in the Non-IID scenario is improved by combining LoRA with the training method of hybrid experts.

[0050] 2. The present invention calculates the similarity vectors between each client and uses the K-Means algorithm to perform multi-granularity grouping based on the similarity vectors. The similarity vectors can represent the similarity between data sets in different clients, and clients with similarities are grouped into the same group, which is beneficial to improving the performance of federated learning in Non-IID scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 A flow chart of the communication efficient federated learning multi-granularity grouping fine-tuning method of the present invention;

[0053] Figure 2 A schematic diagram of the communication-efficient federated learning multi-granularity grouping fine-tuning method of the present invention. DETAILED DESCRIPTION

[0054] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] See also Figure 1-Figure 2 The first aspect of the present invention provides a method for fine-tuning multi-granularity grouping of federated learning with high communication efficiency, including:

[0056] S1: Build a pre-trained model of the client through the LoRA low-rank parameter matrix; wherein the LoRA low-rank parameter matrix includes parameter matrix A and parameter matrix B;

[0057] S2: The client performs asymmetric fine-tuning on the parameter matrix B and uploads the parameter matrix B to the server;

[0058] S3: Calculate the similarity vector between each client;

[0059] S4: The server performs multi-granularity grouping and calculates the weighted average low-rank matrix within the group;

[0060] S5: The client performs low-rank hybrid expert setting and loads several sets of low-rank matrices to obtain expert LoRA;

[0061] S6: The client fine-tunes the expert gating parameters through low-rank hybrid experts and uploads them to the server;

[0062] S7: The server performs weighted averaging on the expert gating parameters corresponding to each client to obtain an average expert gating parameter, and uses it for reasoning of the global model.

[0063] In this embodiment, the pre-trained model of the client is constructed by using the LoRA low-rank parameter matrix, including:

[0064] All clients use the same pre-trained model as the base model for subsequent fine-tuning and freeze the pre-trained model parameter matrix. And add the LoRA low-rank parameter matrix; among them, the LoRA low-rank parameter matrix includes the parameter matrix and the parameter matrix R is a real number, r is a low-rank dimension, d in is the input dimension of the pre-trained model parameter matrix, d out is the output dimension of the pre-trained model parameter matrix.

[0065] It should be noted that LoRA is a commonly used model fine-tuning method. It freezes the parameter W0 of the pre-trained model and only trains two low-rank parameter matrices A and B. The BA matrix is ​​used to approximate the change in the entire model parameters during the fine-tuning process. Since the gradient of the entire model does not need to be maintained during back propagation, the fine-tuning overhead of the model is reduced.

[0066] In this embodiment, the client performs asymmetric fine-tuning on the parameter matrix B and uploads the parameter matrix B to the server, including:

[0067] X1: All clients use the same LoRA initialization method. After initialization, the LoRA parameters of all clients are the same.

[0068] X2: Each client uses a local data set to train a pre-trained model. During training, parameter matrix A needs to be frozen and only parameter matrix B needs to be trained.

[0069] X3: The client uploads the trained parameter matrix B to the server, and parameter matrix A does not need to be uploaded.

[0070] Exemplarily, the parameter matrix A in all clients is the same and is initialized using Gaussian randomization, and the parameter matrix B in all clients is initialized to zero.

[0071] The present invention utilizes the asymmetry of LoRA to fine-tune the parameter matrix B. Since the parameter matrix A extracts features from the input and the parameter matrix B outputs data based on the features during model fine-tuning using LoRA, fine-tuning only the parameter matrix B is as effective as fine-tuning both the parameter matrix A and the parameter matrix B at the same time, and the amount of parameters that need to be learned and transmitted is only half of the latter. In the present invention, the client needs to freeze the parameter matrix A when training the pre-trained model and only train the parameter matrix B. The client uploads the trained parameter matrix B to the server, and the parameter matrix A does not need to be uploaded, which is beneficial to reducing the communication overhead of federated learning.

[0072] In this embodiment, calculating the similarity vector between the clients includes:

[0073] Set the number of clients to N, and the pre-trained model of each client has H layers, B n,h Denotes the parameter matrix B of the h-th layer of the n-th client, then the similarity score calculation formula of the parameter matrix B corresponding to the h-th layer of the n+1-th client and the h-th layer of the n-th client is:

[0074]

[0075] After calculating the layer_score (n+1, n, h) of a total of h layers, the calculation formula for the similarity client_score between the n+1th client and the nth client is as follows:

[0076]

[0077] The similarity vector α of the nth client n for:

[0078] α n =[client_score(n,1),client_score(n,2),…,client_score(n,n)];

[0079] Where n = 1, 2, ..., N, N is the total number of clients; h = 1, 2, ..., H, H is the total number of layers of the pre-trained model in the client.

[0080] The present invention calculates the similarity vectors between the various clients. The similarity vectors can represent the similarity between data sets in different clients, making it convenient to subsequently group the clients according to the similarity vectors.

[0081] In this embodiment, the server performs multi-granularity grouping and calculates the weighted average low-rank matrix within the group, including:

[0082] Use the similarity vector and K-Means algorithm to group the parameter matrix B into multiple granularities to obtain M groups, which are marked as i. Perform weighted averaging on all parameter matrices in group i according to the amount of data corresponding to the client to obtain a weighted average low-rank matrix within the group. And sent to the client; where i = 1, 2, ..., M, M is the total number of groups; the multi-granularity grouping methods include:

[0083] Client granularity grouping: Calculate the average vector of the similarity vectors of all layers of the pre-trained model in the client to obtain the client granularity similarity vector, and then use the K-Means algorithm to divide the client granularity similarity vector into M groups;

[0084] Model block granularity grouping: calculate the average vector of the similarity vectors of all layers in the same block of the same client to obtain the similarity vector of the model block granularity, and then use the K-Means algorithm to divide the similarity vector of the model block granularity into M groups;

[0085] Model layer granularity grouping: directly use the K-Means algorithm to divide the similarity vectors of the model layer granularity into M groups.

[0086] For example, assume that each group has K clients, and mark a single client in each group as k. The data set owned by client k is D k , then the total data set The proportion of client k’s data set to the total data set is Wherein, k=1, 2, ..., K, K is the total number of clients in a single group;

[0087] The multi-granularity grouping methods include:

[0088] Client granularity grouping: Since the training data of different clients in federated learning are different, the client granularity grouping reflects the similarity of client data. Clients with more similar data are often grouped into the same group, and all layers of the client are grouped according to the same client similarity vector. The parameter matrix corresponding to the K clients in group i is Then through the formula The parameter matrix B within the group k Perform weighted averaging to obtain the weighted average low-rank matrix within the group And sent to the client; among them, B k Represents the parameter matrix corresponding to the kth client in the group;

[0089] Model block granularity grouping: Since different blocks of pre-trained models tend to capture different types of knowledge, model block granularity grouping more reflects the similarity of block-level features; for all layers in the same block, they are grouped according to the same model block similarity vector; assuming that each client has J model blocks, the parameter matrix of the jth block in the kth client is marked as B k,j , then by the formula The parameter matrix B within the group k,j Perform weighted averaging to obtain the weighted average low-rank matrix within the group And send it to the client; perform intra-group weighted averaging on J model blocks; where j = 1, 2, ..., J, J is the total number of pre-trained model blocks in a single client;

[0090] Model layer granularity grouping: The same block of the pre-trained model often contains many layers. The model layer granularity grouping is a finer-grained parameter grouping, which reflects the similarity of hierarchical features; each layer of the pre-trained model has its corresponding similarity vector, and all layers are grouped according to different similarity vectors. Assume that the pre-trained model in the k-th client has a total of Q layers, and the parameter matrix of the q-th layer in the k-th client is marked as B k,q , then by the formula The parameter matrix B within the group k,q Perform weighted averaging to obtain the weighted average low-rank matrix within the group And send it to the client; perform intra-group weighted averaging on the Q model layers; where q = 1, 2, ..., Q, Q is the total number of layers of the pre-trained model in a single client.

[0091] The present invention uses the K-Means algorithm to perform multi-granularity grouping according to the similarity vector, and groups clients with similarity into the same group, which is beneficial to improving the performance of federated learning in Non-IID scenarios and effectively reduces the amount of parameters transmitted using the hybrid expert method when the number of clients is large.

[0092] In this embodiment, the client performs low-rank hybrid expert setting and loads several groups of low-rank matrices to obtain expert LoRA, including:

[0093] The client downloads M in-group weighted average low-rank matrices The local pre-trained model of the client remains unchanged, the local parameter matrix A of the client is copied M times, and the M downloaded group weighted average low-rank matrices are used Replace the local parameter matrix B to obtain M The matrix, in turn The matrix is ​​labeled as the i-th expert LoRA, and M expert LoRAs are obtained.

[0094] In this embodiment, the client fine-tunes the expert gating parameters through the low-rank hybrid expert and uploads them to the server, including:

[0095] The client freezes the pre-trained model and expert LoRA. Freezing expert LoRA includes freezing parameter matrix A and parameter matrix Fine-tune the expert gating parameter G of client n, and upload the trained expert gating parameter G to the server after fine-tuning. For the expert gating of the hybrid expert model, the calculation formula for the allocation probability of the i-th expert LoRA is:

[0096]

[0097] Among them, x is the input of expert gating, G i is the expert gating parameter corresponding to the i-th expert LoRA, e is a natural constant, and the softmax() function is used to calculate the allocation probability of the expert gating unit;

[0098] By formula Calculate the total output of the model O(x).

[0099] It should be noted that the client only needs to fine-tune the expert gating parameters for a small number of rounds, otherwise the client model is prone to overfitting.

[0100] In this embodiment, the server performs weighted averaging on the expert gating parameters corresponding to each client to obtain an average expert gating parameter, including:

[0101] The server performs a weighted average of the expert gating parameters corresponding to the client according to the amount of data received from the client to obtain the average expert gating parameter The average expert gating parameter and expert LoRA are loaded into the global model of the server, and the global model is based on the average expert gating parameters Reasoning with expert LoRA.

[0102] For example, the data set owned by the nth client is set to D n , where the total data set The proportion of the total data set occupied by the nth client is Let the gating parameter of the nth client be G (n) , through the formula Calculate weighted average gating parameters

[0103] It should be noted that when there is a new client, only the expert LoRA after grouping and the expert gating parameters after weighted average are needed to be downloaded. It is used to initialize and can be used for normal reasoning of the global model.

[0104] Part of the data in the above formula is calculated by removing the dimension and taking its numerical value. The formula is a formula closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.

[0105] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A communication efficient federated learning multi-granularity grouping fine-tuning method, characterized in that: include: S1: Build a pre-trained model of the client through the LoRA low-rank parameter matrix; wherein the LoRA low-rank parameter matrix includes parameter matrix A and parameter matrix B; S2: The client performs asymmetric fine-tuning on the parameter matrix B and uploads the parameter matrix B to the server; S3: Calculate the similarity vector between each client; S4: The server performs multi-granularity grouping and calculates the weighted average low-rank matrix within the group; S5: The client performs low-rank hybrid expert setting and loads several sets of low-rank matrices to obtain expert LoRA; S6: The client fine-tunes the expert gating parameters through low-rank hybrid experts and uploads them to the server; S7: The server performs weighted averaging on the expert gating parameters corresponding to each client to obtain an average expert gating parameter, and uses it for reasoning of the global model.

2. According to claim 1, a method for fine-tuning multi-granularity grouping with efficient communication of federated learning is characterized in that: The pre-training model of the client is constructed by using the LoRA low-rank parameter matrix, including: All clients use the same pre-trained model as the base model for subsequent fine-tuning and freeze the pre-trained model parameter matrix. And add the LoRA low-rank parameter matrix; among them, the LoRA low-rank parameter matrix includes the parameter matrix and the parameter matrix R is a real number, r is a low-rank dimension, d in is the input dimension of the pre-trained model parameter matrix, d out is the output dimension of the pre-trained model parameter matrix.

3. According to claim 2, a method for fine-tuning multi-granularity grouping with efficient communication of federated learning is characterized in that: The client performs asymmetric fine-tuning on the parameter matrix B and uploads the parameter matrix B to the server, including: X1: All clients use the same LoRA initialization method. After initialization, the LoRA parameters of all clients are the same. X2: Each client uses a local data set to train a pre-trained model. During training, parameter matrix A needs to be frozen and only parameter matrix B needs to be trained. X3: The client uploads the trained parameter matrix B to the server, and parameter matrix A does not need to be uploaded.

4. The method for fine-tuning multi-granularity grouping with efficient communication of federated learning according to claim 2, characterized in that: The calculating of the similarity vectors between the clients includes: Set the number of clients to N, and the pre-trained model of each client has H layers, B n,h Denotes the parameter matrix B of the h-th layer of the n-th client, then the similarity score calculation formula of the parameter matrix B corresponding to the h-th layer of the n+1-th client and the h-th layer of the n-th client is: After calculating the layer_score (n+1, n, h) of a total of h layers, the calculation formula for the similarity client_score between the n+1th client and the nth client is as follows: The similarity vector α of the nth client n for: α n =[client_score(n,1),client_score(n,2),…,client_score(n,n)]; Where n = 1, 2, ..., N, N is the total number of clients; h = 1, 2, ..., H, H is the total number of layers of the pre-trained model in the client.

5. The method for fine-tuning multi-granularity grouping with efficient communication of federated learning according to claim 4, characterized in that: The server performs multi-granularity grouping and calculates a weighted average low-rank matrix within the group, including: Use the similarity vector and K-Means algorithm to group the parameter matrix B into multiple granularities to obtain M groups, which are marked as i. Perform weighted averaging on all parameter matrices in group i according to the amount of data corresponding to the client to obtain a weighted average low-rank matrix within the group. And sent to the client; where i = 1, 2, ..., M, M is the total number of groups; the multi-granularity grouping methods include: Client granularity grouping: Calculate the average vector of the similarity vectors of all layers of the pre-trained model in the client to obtain the client granularity similarity vector, and then use the K-Means algorithm to divide the client granularity similarity vector into M groups; Model block granularity grouping: calculate the average vector of the similarity vectors of all layers in the same block of the same client to obtain the similarity vector of the model block granularity, and then use the K-Means algorithm to divide the similarity vector of the model block granularity into M groups; Model layer granularity grouping: directly use the K-Means algorithm to divide the similarity vectors of the model layer granularity into M groups.

6. The method for fine-tuning multi-granularity grouping with efficient communication of federated learning according to claim 5, characterized in that: The client performs low-rank mixed expert setting and loads several groups of low-rank matrices to obtain expert LoRA, including: The client downloads M in-group weighted average low-rank matrices The local pre-trained model of the client remains unchanged, the local parameter matrix A of the client is copied M times, and the M downloaded group weighted average low-rank matrices are used Replace the local parameter matrix B to obtain M The matrix, in turn The matrix is ​​labeled as the i-th expert LoRA, and M expert LoRAs are obtained.

7. The method for fine-tuning multi-granularity grouping with efficient communication of federated learning according to claim 6, characterized in that: The client fine-tunes the expert gating parameters through the low-rank hybrid expert and uploads the parameters to the server, including: The client freezes the pre-trained model and expert LoRA. Freezing expert LoRA includes freezing parameter matrix A and parameter matrix Fine-tune the expert gating parameter G of client n, and upload the trained expert gating parameter G to the server after fine-tuning. For the expert gating of the hybrid expert model, the calculation formula for the allocation probability of the i-th expert LoRA is: Among them, x is the input of expert gating, G i is the expert gating parameter corresponding to the i-th expert LoRA, e is a natural constant, and the softmax() function is used to calculate the allocation probability of the expert gating unit; By formula Calculate the total output of the model O(x).

8. The method for fine-tuning multi-granularity grouping with efficient communication of federated learning according to claim 7, characterized in that: The server performs weighted averaging on the expert gating parameters corresponding to each client to obtain an average expert gating parameter, including: The server performs a weighted average of the expert gating parameters corresponding to the client according to the amount of data received from the client to obtain the average expert gating parameter The average expert gating parameter and expert LoRA are loaded into the global model of the server, and the global model is based on the average expert gating parameters Reasoning with expert LoRA.

Images