Transform federal learning acceleration method and system based on personalized attention

Abstract

The invention discloses a Transform federal learning acceleration method and system based on personalized attention, and relates to artificial intelligence. The scheme is provided for solving the problems that in the prior art, the communication cost is high, and the model individuation capacity is poor under the data heterogeneous condition. The method mainly comprises the steps that a local lightweight Transform model is built at each client side, personalized row vectors and shared global column vectors are utilized, a projection matrix is obtained through calculation in a mode that elements are multiplied one by one, and then a complete Transform model is obtained. The method has the advantages that the projection personalized vector and the global shared vector are constructed and utilized, and personalized representation of the projection matrix is achieved. The global shared vector is aggregated at the server, so that the global shared vector contains attention-related information of more client data, and a client model is helped to extract key features of the data more accurately. According to the method, only the global shared vector with less parameter quantity needs to be transmitted, and the data quantity needing to be transmitted in each time of uploading and downloading of the model is reduced, so that the communication cost is reduced, and the more efficient communication efficiency is realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a Transformer federated learning acceleration method and system based on personalized attention. Background Art

[0002] Federated learning is a distributed machine learning framework with privacy protection and secure encryption technology. It aims to enable decentralized participants to collaborate on machine learning model training without disclosing private data to other participants. In federated learning, each client trains a model using its own local data and only sends model updates to the server. In this way, federated learning overcomes a series of problems such as data privacy and communication overhead. As the pioneer of federated learning, compared with traditional distributed training, FedAvg《McMahan B, Moore E, Ramage D, et al. Communication-efficient learning of deep networks from decentralized data[C]. Proceedings of the Artificial Intelligence and Statistics. 2017: 1273-1282.》 communicates after multiple rounds of local model training iterations to achieve efficient distributed training. FedProx "Li T, Sahu AK, Zaheer M, et al. Federated optimization in heterogeneous networks [J]. Machine Learning and Systems, 2020, 2: 429-450." imposes a global constraint on the client training loss to limit the local update direction and improve the convergence of the model. However, the heterogeneity of the data distribution of each edge node is still a key factor restricting the development of federated learning. Under the condition of non-independent and identically distributed data distribution, it is very difficult to learn a unified global model that performs well on various data distributions. Therefore, in order to solve these problems, personalized federated learning came into being as an extension of federated learning. It works by learning a personalized model for each client instead of a single global model, while still benefiting from joint training. In order to solve the problem of model heterogeneity, pFedMe "T DinhC, Tran N, Nguyen J. Personalized federated learning with moreau envelopes [C]. Proceedings of Advances in Neural Information Processing Systems. 2020, 33: 21394-21405." adds a proximal regularization term to deal with the client drift problem caused by statistical heterogeneity, which helps to separate personalized model optimization from global model learning.Considering that robustness and fairness are competing constraints in statistical heterogeneous networks, in order to ease the relationship between robustness and fairness, in Ditto "Li T, Hu S, Beirami A, et al. Ditto: Fair and robust federated learning through personalization [C]. Proceedings of the International Conference on Machine Learning. 2021: 6357-6368.", each client learns its additional personalized model with proximal terms and obtains information from the downloaded global model. In order to optimize the personalized model aggregation for clients with heterogeneous data, pFedLA "MaX, Zhang J, Guo S, et al. Layer-wised model aggregation for personalized federated learning [C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2022: 10092-10101." applies a dedicated hypernetwork to each client on the server and trains them to generate aggregation weights for each model layer of different clients, which can effectively utilize the inter-user similarities between non-iid data clients.

[0003] However, most federated learning frameworks are based on convolutional neural networks, which usually focus on high-frequency local patterns. With the development of deep learning, the Transformer model in the field of natural language processing, as a widely used backbone network, has been applied to the application scenarios of federated learning to achieve high-performance distributed training based on the self-attention mechanism. Considering that the projection matrix in Transformer is a key parameter affecting Transformer, FedTP "Li H, Cai Z, Wang J, et al. FedTP: Federated learning by transformer personalization [J]. IEEE Transactions on Neural Networks and Learning Systems, 2023: 13426-13440" uses learnable personalized embedding vectors and hypernetworks to generate personalized projection matrices, which are spliced ​​with the remaining weighted average aggregated networks to obtain a complete personalized model for local training. However, the performance of this method depends on the design of the hypernetwork structure, and because the Transformer model has more parameters and its communication cost is higher, it is difficult for this method to simultaneously take into account the personalized performance and communication efficiency of the model. Summary of the invention

[0004] The purpose of the present invention is to provide a Transformer federated learning acceleration method and system based on personalized attention to solve the problems existing in the above-mentioned prior art, save communication costs, and better adapt to local distribution.

[0005] The Transformer federated learning acceleration method based on personalized attention described in the present invention comprises the following steps:

[0006] S1. Use Dirichlet distribution to divide the data set into different clients in a non-independent and identically distributed manner;

[0007] S2. Each client builds a local lightweight Transformer model, uses personalized row vectors and shared global column vectors, and calculates the projection matrix by multiplying elements one by one, thereby obtaining a complete Transformer model.

[0008] S3. Each client model is trained locally on a local dataset;

[0009] S4. Each client uploads the projection shared column vector and each layer parameter to the server;

[0010] S5. The server aggregates the shared column vectors and layer parameters from different clients to update these parameters;

[0011] S6. The server transmits the updated shared column vector and the parameters of each layer to each client. Each client initializes according to the transmitted result and then repeats steps S3 to S5 until the model converges.

[0012] S7. After the model is updated, verify it on the test set and take the average value.

[0013] The personalized attention-based Transformer federated learning acceleration system described in the present invention includes a server and multiple clients connected to the server for communication, and uses the method to perform federated learning.

[0014] The Transformer federated learning acceleration method and system based on personalized attention described in the present invention have the following advantages:

[0015] 1. Construct a projection personalized vector and a global shared vector to realize the personalized representation of the projection matrix. The personalized vector is optimized and updated based on the specific data distribution of each client, and is always retained on each client without being aggregated on the server, so that it retains more client data distribution information. The global shared vector is aggregated on the server so that it contains more attention-related information of the client data, helping the client model to extract key data features more accurately.

[0016] 2. By using projected personalized vectors and global sharing, a lightweight representation of the projection matrix is ​​realized. Compared with directly transmitting the attention projection matrix, the present invention only needs to transmit a global shared vector with fewer parameters, which reduces the amount of data that needs to be transmitted in each upload and download of the model, thereby reducing the communication cost and achieving more efficient communication efficiency. Due to the reduction in communication costs, the model can complete an aggregate update in a shorter time, thereby speeding up the training process of the federated learning model and achieving both personalized and efficient federated learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of the federated learning acceleration method described in the present invention.

[0018] Figure 2 It is a schematic diagram of local model training of the federated learning acceleration method described in the present invention. DETAILED DESCRIPTION

[0019] Federated learning has a wide range of application scenarios, especially in areas with high requirements for data privacy, security, and distributed computing. In the healthcare field, by training models locally in each hospital and sharing only model parameters rather than data, patient privacy is protected while improving the generalization ability of the model, such as disease prediction and medical image analysis. In the financial services field, banks and financial institutions are also restricted by data privacy and regulations and cannot freely share customer information. Federated learning can help different financial institutions collaborate to jointly improve the performance of credit scoring, anti-fraud detection, and risk management models without exposing customer data. In the field of autonomous driving, autonomous vehicles need to learn driving behavior from a large amount of sensor data. The data collected locally by each vehicle is highly private. Through federated learning, automakers can collaboratively train models without sharing specific data to improve the accuracy of vehicle perception and decision-making. In the field of education, educational institutions can use federated learning to build personalized learning systems to improve student learning outcomes. Without directly sharing student data, the model can use data from different regions and schools to build personalized teaching models.

[0020] like Figure 1 As shown, the Transformer federated learning acceleration method based on personalized attention in the present invention includes the following steps:

[0021] S1. Use Dirichlet distribution to divide the data and divide the data set to each client in a non-independent and identically distributed manner.

[0022] S2. Each client builds a local lightweight Transformer model, uses personalized row vectors and shared global column vectors, and calculates the projection matrix by multiplying elements one by one, and then obtains the complete Transformer model. The specific steps are as follows:

[0023] S2.1. Initialize the projection personalized row vector of each Transformer and a shared global column vector The superscripts represent the number of communication aggregation rounds and the number of iterations in each round respectively; the subscripts n and l represent the nth client and the lth layer respectively; q, k, and v identify the query vector, key vector, and value vector respectively.

[0024] S2.2. Construct the projection matrix based on the projected personalized row vector in S2.1 and the shared global column vector:

[0025]

[0026] in Represents the multiplication of two vector elements.

[0027] S2.3. All layers Construct parameter W, Construct parameter G, and each parameter constructs a set θ.

[0028] S3. Each client model is trained locally on a local dataset, such as Figure 2 As shown, the specific steps are:

[0029] S3.1. For the (t+1)th iteration in r communication rounds, sample data, calculate the network forward propagation, and for the lth Transformer block, according to and calculate The values ​​are as follows:

[0030]

[0031] Where LN(·) represents the layer normalization function, Represents the input of the lth layer.

[0032] S3.2. According to Calculate self-attention:

[0033]

[0034] Where Softmax(·) represents the softmax function.

[0035] S3.3. According to as well as Compute the attention output:

[0036] Right now

[0037] S3.4. For H attention heads, the attention output corresponding to each attention head Splice and use the dimension conversion matrix Reduce the concatenated dimension to the input Dimensions and add shortcut operations:

[0038]

[0039] S3.5. Input the multilayer perceptron to get the output of the lth Transformer block:

[0040]

[0041] Among them, MLP l (·) represents the multilayer perceptron at layer l.

[0042] S3.6. Constructing the loss function based on the complete forward propagation process of the network And update the parameters according to the loss function:

[0043]

[0044] Where α represents the learning rate.

[0045] S4. Each client uploads the projection shared column vector and each layer parameter to the server.

[0046] S5. The server aggregates the shared column vectors and layer parameters from different clients to update these parameters:

[0047]

[0048] in Represents the dataset of the nth client.

[0049] S6. The server transmits the updated shared column vector and the parameters of each layer to each client, and each client initializes according to the results of the transmission. Then, steps S3 to S5 are repeated until the model converges.

[0050] S7. After the model is updated, verify it on the test set and take the average value.

[0051] The personalized attention-based Transformer federated learning acceleration system described in the present invention includes a server and multiple clients connected to the server for communication, and uses the method to perform federated learning.

[0052] For those skilled in the art, various other corresponding changes and deformations can be made according to the technical solutions and concepts described above, and all of these changes and deformations should fall within the protection scope of the claims of the present invention.

Claims

1. Transformer federated learning acceleration method based on personalized attention, characterized by: The following steps are involved: S1. Use Dirichlet distribution to divide the data set into different clients in a non-independent and identically distributed manner; S2. Each client builds a local lightweight Transformer model, uses personalized row vectors and shared global column vectors, and calculates the projection matrix by multiplying elements one by one, thereby obtaining a complete Transformer model. S3. Each client model is trained locally on a local dataset; S4. Each client uploads the projection shared column vector and each layer parameter to the server; S5. The server aggregates the shared column vectors and layer parameters from different clients to update these parameters; S6. The server transmits the updated shared column vector and the parameters of each layer to each client. Each client initializes according to the transmitted result and then repeats steps S3 to S5 until the model converges. S7. After the model is updated, verify it on the test set and take the average value.

2. The Transformer federated learning acceleration method based on personalized attention according to claim 1 is characterized in that: The specific steps of step S2 are: S2.

1. Initialize the projection personalized row vector of each Transformer and a shared global column vector The superscripts represent the number of communication aggregation rounds and the number of iterations in each round respectively; the subscripts n and l represent the nth client and the lth layer respectively, and q, k, and v represent the query vector, key vector, and value vector respectively; S2.

2. Construct the projection matrix based on the projected personalized row vector in S2.1 and the shared global column vector: in Represents the multiplication of two vector elements; S2.

3. All layers Construct parameter W, Construct parameter G, and each parameter constructs a set θ.

3. The Transformer federated learning acceleration method based on personalized attention according to claim 2 is characterized in that: The specific steps of step S3 are: S3.

1. For the (t+1)th iteration in r communication rounds, sample data, calculate the network forward propagation, and for the lth Transformer block, according to and calculate value: Where LN(·) represents the layer normalization function, Represents the input of the lth layer; S3.

2. According to Calculate self-attention: Where Softmax(·) represents the softmax function; S3.

3. According to as well as Compute the attention output: Right now S3.

4. For H attention heads, the attention output corresponding to each attention head Splice and use the dimension conversion matrix Reduce the concatenated dimension to the input Dimensions and add shortcut operations: S3.

5. Input the multilayer perceptron to get the output of the lth Transformer block: Among them, MLP l (·) represents the multilayer perceptron of the lth Transformer block; S3.

6. Constructing the loss function based on the complete forward propagation process of the network And update the parameters according to the loss function: Where α represents the learning rate.

4. The Transformer federated learning acceleration method based on personalized attention according to claim 3 is characterized in that: The updating of step S5 is specifically as follows: in Represents the dataset of the nth client.

5. The Transformer federated learning acceleration method based on personalized attention according to claim 4 is characterized in that: The initialization process of step S6 is as follows:

6. A Transformer federated learning acceleration system based on personalized attention, comprising a server and a plurality of clients communicating with the server, characterized in that: Federated learning is performed using the method as described in any one of claims 1 to 5.

Images