Personalized federal learning method

Abstract

The invention discloses a personalized federal learning method, which is applied to a target training system, the target training system comprises a server and a plurality of participating clients, and the method comprises the following steps: training a to-be-trained model by each participating client according to local training data, updating the aggregated shared weight matrix and the updated private weight matrix obtained in the previous communication round to obtain an updated shared weight matrix and an updated private weight matrix corresponding to the current communication round, and sending the updated shared weight matrix to the server; the server side aggregates the updated shared weight matrix sent by each participating client side to obtain an aggregated shared weight matrix corresponding to the current communication round, and issues the aggregated shared weight matrix to each participating client side to continue iterative updating; and obtaining a target shared weight matrix and a target private weight matrix of the corresponding to-be-trained model by each participating client until an iteration termination condition is met. According to the method, the model can learn helpful global knowledge while the local model specificity of the client is kept.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep learning, and particularly relates to a personalized federated learning method. BACKGROUND

[0002] With the enhancement of data privacy protection awareness and the increasing strictness of data protection regulations in various countries, federated learning emerges as a technology that can realize multi-party collaborative training model under the premise of protecting data privacy, which enables participants to jointly build high-quality models without directly sharing raw data, effectively solving the data island problem.

[0003] However, in the actual deployment process of federated learning, communication cost often becomes a key bottleneck restricting its performance, and efficient communication strategies that reduce communication rounds and reduce the communication volume of each round can significantly reduce bandwidth demand and total training time, making federated learning more feasible in resource-constrained environments. At the same time, due to the heterogeneity of the data distribution of participants, traditional global shared models are difficult to meet the individualized needs of all clients. SUMMARY

[0004] In order to solve the above problems existing in the prior art, the application provides a personalized federated learning method.

[0005] The technical problem to be solved by the application is solved by the following technical scheme: The application provides a personalized federated learning method applied to a target training system, the target training system comprising a server and a plurality of participating clients, the method comprising: Each participating client trains a to-be-trained model according to local training data, updates the aggregated shared weight matrix obtained in the last communication round and the private weight matrix, obtains the updated shared weight matrix and the updated private weight matrix corresponding to the current communication round, and sends the updated shared weight matrix to the server; the weight matrix of the to-be-trained model corresponding to the current communication round comprises the updated shared weight matrix and the updated private weight matrix; The server aggregates the updated shared weight matrix sent by each participating client to obtain the aggregated shared weight matrix corresponding to the current communication round, and delivers it to each participating client for further iterative updating; Until the iteration termination condition is reached, each participating client obtains the target shared weight matrix and the target private weight matrix of the corresponding to-be-trained model.

[0006] The application provides a personalized federated learning method, which directly updates the weight of the decomposed matrix in a compression mode based on matrix decomposition, effectively avoiding information loss caused by quantization or matrix decomposition of the trained model parameters. In addition, the application does not cause training instability caused by quantization and high computational cost caused by matrix decomposition. In addition, compared with the existing method, the federated learning strategy of the application can learn helpful global knowledge while maintaining the specificity of the local model of the client, thereby avoiding the overfitting of the model of the participating client to the local data, causing the client drift problem, and reducing the convergence efficiency and final performance of the model.

[0007] The application will be further described in detail below with reference to the accompanying drawings and the application. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is a flowchart of a personalized federated learning method provided by an embodiment of the application; Figure 2A and Figure 2B is a process diagram of experiment one of a personalized federated learning method provided by an embodiment of the application; Figure 3A and Figure 3B is a process diagram of experiment two of a personalized federated learning method provided by an embodiment of the application. DETAILED DESCRIPTION

[0009] The application will be further described in detail below with reference to the accompanying drawings and the application.

[0010] An embodiment of the application provides a personalized federated learning method, which is applied to a target training system, and the target training system comprises a server and a plurality of participating clients. Referring to Figure 1 , the method comprises the following steps: First, the parameters of the target training system and the trained model are initialized in the initial communication round, and the parameters include the number of participating clients , the maximum communication round , the number of local updates , the batch size (Batch Size), the preset decomposition factor , the initial learning rate , the preselected optimizer, the preselected trained model, the preselected loss function, etc.

[0011] The specific steps of the initial communication round comprise: S101. The server decomposes the initial weight matrix of the model to be trained according to the preset decomposition factor to obtain the first decomposed weight matrix and the second decomposed weight matrix, and sends them to each participating client.

[0012] Optionally, step S101 may specifically include: The server determines whether the preset decomposition factor is a factor of the initial matrix dimension of the initial weight matrix of the model to be trained. If so, the preset decomposition factor is used as the target matrix dimension to decompose the initial weight matrix and obtain the first decomposed weight matrix and the second decomposed weight matrix. If not, the maximum value is determined from the set of candidate values ​​that are less than the preset decomposition factor and are factors of the initial matrix dimension, and the maximum value is used as the target matrix dimension to decompose the initial weight matrix and obtain the first decomposed weight matrix and the second decomposed weight matrix.

[0013] The first decomposition weight matrix has the same matrix dimension as the target matrix, while the second decomposition weight matrix has a matrix dimension determined based on the initial matrix dimension and the target matrix dimension.

[0014] For example, if a preset decomposition factor is used It is the initial matrix dimension (Taking row dimension as an example) the factor, then the initial matrix dimension The target matrix dimension is decomposed into the first decomposition weight matrix. (This refers to the row dimension, which is equal to the column dimension) Matrix dimension of the second decomposition weight matrix (This refers to the row dimension); otherwise, select As the dimension of the target matrix, ,Right now It is less than The largest The factors, then The dimension of the target matrix decomposed into the first decomposition weight matrix (This refers to the row dimension, which is equal to the column dimension) Matrix dimension of the second decomposition weight matrix (This refers to the row dimension).

[0015] The following is a decomposition example: Assume an initial weight matrix. The initial matrix dimension is Preset decomposition factor The first decomposition weight matrix obtained by decomposition is... The dimension of the target matrix is Second decomposition weight matrix The matrix dimension is The number of parameters decreased by about 75% after decomposition.

[0016] Assuming an initial weight matrix The initial matrix dimension of , the preset decomposition factor , the target matrix dimension of is 4, so the first decomposition weight matrix obtained by decomposition has a target matrix dimension of , the matrix dimension of the second decomposition weight matrix is 32 , and the parameter quantity after decomposition is reduced by about 93%.

[0017] Suppose , the value of or is as follows: suppose , , then , When the value of increases gradually, the parameter quantity of the model gradually reduces, and the required communication cost also gradually reduces.

[0018] Optionally, the initial weight matrix can be represented as: , wherein represents the total number of network layers included in the to-be-trained model, .

[0019] The weight matrix of each layer in the to-be-trained model can be represented as:

[0020] , wherein represents the weight matrix of each network layer in the to-be-trained model, , represents the first decomposition weight matrix of each network layer, , represents the second decomposition weight matrix of each network layer, , , , represents the Kronecker product.

[0021] Exemplarily, the specific formula of the Kronecker product is as follows: .

[0022] S102, each participating client trains the to-be-trained model according to the local training data, updates the first decomposition weight matrix and the second decomposition weight matrix, and obtains an updated shared weight matrix and an updated private weight matrix corresponding to the initial communication round.

[0023] Exemplarily, each participating client The weight matrix of the to-be-trained model is initialized locally. Since the present embodiment adopts pre-decomposition, the first decomposed weight matrix and the second decomposed weight matrix are directly initialized, which is equivalent to initializing the initial weight matrix . For brevity, the first decomposed weight matrix and the second decomposed weight matrix are denoted as and

[0024] respectively in the following. Each participating client uses the local training data to train the to-be-trained model, updates the first decomposed weight matrix and the second decomposed weight matrix (the communication round is denoted as ), and obtains the updated shared weight matrix and the updated private weight matrix corresponding to the initial communication round.

[0025] It should be noted that one communication round can include the server-participating client-server.

[0026] Then, the updated shared weight matrix corresponding to the initial communication round is uploaded to the server. The server aggregates the updated shared weight matrices sent by each participating client, obtains the aggregated shared weight matrix corresponding to the initial communication round, and distributes it to each participating client for iterative update in the next round.

[0027] S20, each participating client updates the aggregated shared weight matrix and the updated private weight matrix obtained in the last communication round according to the local training data, obtains the updated shared weight matrix and the updated private weight matrix corresponding to the current communication round, and sends the updated shared weight matrix to the server.

[0028] The weight matrix of the to-be-trained model corresponding to the current communication round includes the updated shared weight matrix and the updated private weight matrix.

[0029] Optionally, step S20 can specifically include: Each participating client trains the to-be-trained model with the local training data. In the forward propagation process, each layer of the network performs two-dimensional convolution on the input data by using the updated private weight matrix of the corresponding last communication round to output an intermediate result, and performs one-dimensional convolution on the intermediate result by using the aggregated shared weight matrix of the corresponding last communication round to output the output result of each layer of the network. The input data of the first layer of the to-be-trained model is the local training data, and the output result of the last layer is the prediction result of the to-be-trained model. During backpropagation, based on the prediction results, the label data in the local training data, and the preset loss function, the aggregated shared weight matrix and the updated private weight matrix obtained in the previous communication round are updated to obtain the updated shared weight matrix and the updated private weight matrix corresponding to the current communication round.

[0030] Optionally, the output of each layer of the network can be represented as:

[0031] in, This represents the output of each layer of the network. This represents the input data for each layer of the network. Indicates the first The first participating client The aggregated shared weight matrix corresponding to the layer network, Indicates the first The first participating client The updated private weight matrix corresponding to the layer network. This represents a two-dimensional convolution operation. This represents a multi-scale one-dimensional convolution operation.

[0032] For example, when the model performs forward propagation, it is not necessary to recover the model output. This allows for the completion of a one-dimensional convolution operation. Assume the input data for this network layer is... The shape is , Indicates the number of points. The dimension (number of channels) of the feature at each point, and the current weight matrix of the model to be trained. The shape is The output data of this layer of the network The shape is Assuming The shape is , shape First use For input data Perform a two-dimensional convolution to obtain the intermediate result. The shape is Change its shape to Then use right Perform multiple one-dimensional convolutions to obtain the output data. The shape is Its shape can be changed to , and use The result obtained by performing one-dimensional convolution is the same, therefore, there is no need to use... reconstruction .

[0033] According to the number of local updates and the batch size determines the number of forward propagations, and after the forward propagation is completed, the backward propagation is performed. Specifically, based on the prediction result of the to-be-trained model, the label data in the local training data, and the preset loss function, the aggregated shared weight matrix obtained in the last communication round and the updated private weight matrix are updated to obtain the updated shared weight matrix and the updated private weight matrix corresponding to the current communication round.

[0034] S20, the server aggregates the updated shared weight matrix sent by each participating client to obtain the aggregated shared weight matrix corresponding to the current communication round, and distributes it to each participating client for further iteration and update.

[0035] Exemplarily, when the participating client completes the local model update, the updated shared weight matrix is uploaded to the server . In order to cope with the case of non-independent and identically distributed data, the updated private weight matrix is saved as a private parameter in the local of the participating client , and is uploaded to the server as a shared parameter. By learning global knowledge from other participating clients through multi-party model aggregation, the to-be-trained model can better recognize a smaller number of classes or even classes that have never been seen. However, because it does not participate in aggregation, more local knowledge can be retained, so that the local specific data can be better processed.

[0036] Optionally, the aggregated shared weight matrix can be represented as:

[0037] wherein, denotes the aggregated shared weight matrix, denotes the updated shared weight matrix sent by the i th participating client, t denotes the communication round between the participating client and the server, denotes the total number of participating clients, . .

[0038] S30, until the iteration termination condition is reached, each participating client obtains the target shared weight matrix and the target private weight matrix of the corresponding to-be-trained model.

[0039] Exemplarily, after the current communication round reaches the maximum communication round , the participating client The updated shared weight matrix is no longer uploaded to the server, and each participating client updates the obtained aggregated shared weight matrix and the updated private weight matrix using local training data to obtain the target shared weight matrix of the corresponding to-be-trained model and the target private weight matrix , and saves the target shared weight matrix and the target private weight matrix as the individualized model weight of the participating client locally, for model inference in the future.

[0040] The personalized federated learning method provided by the application directly updates the decomposed matrix based on the compression method based on matrix decomposition, effectively avoiding the information loss caused by quantization or matrix decomposition of the model parameters of the completed training. In addition, the application also does not have the training instability caused by quantization and the calculation cost caused by matrix post-decomposition. In addition, compared with the existing method, which may cause the model of the participating client to overfit to the local data, causing the client drift problem, and reducing the convergence efficiency and final performance of the model, the federated learning strategy of the application can learn the helpful global knowledge while maintaining the specificity of the local model of the client.

[0041] The personalized federated learning method provided by the application is further described below through simulation experiments.

[0042] Point cloud classification and point cloud partial segmentation are used as application scenarios in the simulation experiment. Point cloud classification aims to semantically label the entire three-dimensional point cloud object and assign it a single class label, thereby realizing the identification and understanding of the object or scene. Point cloud partial segmentation is a more fine-grained point cloud analysis method, which aims to divide the point cloud of a single three-dimensional object into multiple meaningful components and assign each point a part-level label, thereby revealing the internal structure and functional areas of the object.

[0043] The data sets used in this experiment are ModelNet40 and ShapeNet. ModelNet40 is a widely used point cloud classification data set, which contains 40 object categories. The data amount of the training set and the test set is 9843 and 2468, respectively. ShapeNet is a widely used point cloud partial segmentation data set, which contains 16 object categories and 50 part categories. Each object is divided into 2-5 parts, and the data amount of the training set and the test set is 13998 and 2874, respectively.

[0044] The point cloud processing model used in the experiment is PointNet++ (Qi C R, Yi L, Su H, et al. Pointnet++: Deep hierarchical feature learning on point sets in a metricspace [J]. Advances in neural information processing systems, 2017, 30.), and the network structure is not modified. The feature extractor of the model is stacked by multiple set abstraction modules (SA). The SA module mainly performs sampling, grouping, one-dimensional convolution and maximum pooling operations. The feature extractor is followed by a fully connected layer, which is used to calculate the probability of the sample in each category. Matrix decomposition only acts on the one-dimensional convolution layer of the SA module and the fully connected layer, deforms the convolution kernel into a matrix, and then represents it in the form of two matrix Kronecker products.

[0045] The loss function used in the experiment of point cloud classification and partial segmentation task is negative log-likelihood loss, and the specific formula is as follows:

[0046] Wherein, represents the number of samples, represents the index of the true value category of the th sample, represents the prediction probability of the th sample for .

[0047] Experiment 1: The experimental setting of this experiment is the independent and identically distributed case of multiple parties. Only the global model can handle such cases, without the need to add personalized strategies.

[0048] The number of participating clients in this experiment is , the global maximum communication round is , the local update times is , the batch size is , the preset decomposition factor is , the initial learning rate is , the optimizer is Adam, and the model to be trained is PointNet++.

[0049] In terms of local data settings, the training set in the data set is divided into 20 participating clients, and the local training data of each participating client is obtained by uniformly sampling the training set, so that the data distribution of all participating clients is approximately the same to simulate the case of multi-party data independent and identically distributed, as shown in Figure 2A , which shows the training set data distribution when the multi-party data is independent and identically distributed (X axis represents the category, Y axis represents the participating party, and Z axis represents the sample number). The test set is saved in the server for testing the performance of the global aggregated model. After each aggregation operation is completed, the global aggregated model is used to infer the test set data to verify the model performance.

[0050] Referring to Figure 2B , a communication diagram of federated learning without personalized strategy is shown. The initial weight matrix is decomposed into a first decomposition weight matrix and a second decomposition weight matrix according to a preset decomposition factor. When the personalized training strategy is not used, the server initializes the model and sends the first decomposition weight matrix and the second decomposition weight matrix to all clients.

[0051] The participating clients receive and use the local training data to train the model. During the forward propagation of the training, one two-dimensional convolution and multiple one-dimensional convolutions are used to replace the original one-dimensional convolution, so that the initial weight matrix does not need to be restored to reduce the amount of calculation. After the participating clients complete the local training, the updated shared weight matrix and the updated private weight matrix are uploaded to the server.

[0052] The server receives and performs aggregation operation, so as to obtain the updated aggregated shared weight matrix and the aggregated private weight matrix , and then is sent to all participating clients again. If the termination condition is met, the process is stopped; otherwise, the previous steps are repeated.

[0053] Experiment 2: The experimental setting of this experiment is the case of multi-party data non-independent and identically distributed. The global model cannot be used to process the data of each participating party in a targeted manner, so it is necessary to add a personalized model training strategy.

[0054] In this experiment, the number of participating clients is , the global maximum communication round is , the local update number is , the batch size is , the preset decomposition factor is , and the initial learning rate is Optimizer: Adam, Model: PointNet++, Number of points in each sample: 1024 (ModelNet40) and 2048 (ShapeNet).

[0055] The dataset is divided into 20 participating clients, and the local training data of each participating client is obtained by sampling the total dataset according to the Dirichlet distribution. The purpose is to make the data amount and data distribution of participating clients non-uniform to simulate the case of multi-party data non-independent and identically distributed in real scenarios. Referring to Figure 3A , the training set data distribution when multi-party data is non-independent and identically distributed is shown (X-axis represents category, Y-axis represents participating party, and Z-axis represents sample number).

[0056] First, the training set and the test set are merged, and the samples of each category are divided into training set and test set in the ratio of 7:3. Then, the Dirichlet distribution is sampled, and the distribution parameters of the Dirichlet distribution are obtained. The random number matrix with the shape of , represents the proportion of the th user's th category sample in all th category samples. The random number matrix is used to complete the division of each user's training set and test set.

[0057] Referring to Figure 3B , the communication diagram of federated learning with personalized strategy is shown. According to the preset decomposition factor, the initial weight matrix is decomposed into the first decomposition weight matrix and the second decomposition weight matrix . The participating client initializes , and then uses the local training data to train the model. In the forward propagation process of training, one two-dimensional convolution and multiple one-dimensional convolutions are used to replace the original one-dimensional convolution, so that the initial weight matrix does not need to be restored to reduce the amount of calculation. After the participating client completes local training, the updated shared weight matrix is uploaded to the server, and the private weight matrix is saved locally to further reduce the communication amount.

[0058] The server receives the updated and performs aggregation operation to obtain the updated aggregated shared weight matrix , and then the updated aggregated shared weight matrix The result is sent to all participating clients again. If the termination condition is met, the process is stopped; otherwise, the previous step is repeated.

[0059] It is to be understood that the terms "first", "second", and the like, used in the description and in the claims, are used to describe different objects and do not imply a particular order or sequence. The use of data "A" before data "B" in this description and in the claims is not a description of chronological order, except when specifically stated to be "chronological order". Embodiments of the application could be practiced in a chronologically different order than as described in this specification without departing from the scope or spirit of the application. The description and the claims should not be understood as requiring a particular sequence of steps or sequence of actions in order to carry out the application.

[0060] In the description of the specification, the use of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like is intended to indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearance of such phrases in various places in the specification are not necessarily all referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the purview of one of ordinary skill in the art to effect such feature, structure, or characteristic in connection with a different embodiment, unless contextally contraindicated.

[0061] Although the application has been described herein with respect to particular embodiments thereof, those skilled in the art will readily appreciate that other changes, substitutions, and alterations can be made hereto without departing from the application. In the description of the application, the word "comprising" does not exclude the presence of elements or steps other than those listed in a claim. "A" or "an" shall not exclude more than one, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. The reference signs in the claims should not be construed as limiting the scope of the application.

[0062] The above description is further detailed in connection with specific preferred embodiments of the application, and should not be construed as limiting the application to only these descriptions. Those skilled in the art will readily understand that a number of simple modifications and adaptations can be made without departing from the concept of the application, and all such modifications and adaptations are intended to be within the scope of the application.

Claims

1. A personalized federated learning method, characterized in that, Applied to a target training system, the target training system including a server and multiple participating clients, the method includes: Each participating client trains the model to be trained based on local training data, updates the aggregated shared weight matrix and the updated private weight matrix obtained in the previous communication round, obtains the updated shared weight matrix and the updated private weight matrix corresponding to the current communication round, and sends the updated shared weight matrix to the server; the weight matrix corresponding to the current communication round of the model to be trained includes the updated shared weight matrix and the updated private weight matrix. The server aggregates the updated shared weight matrices sent by each participating client to obtain the aggregated shared weight matrix corresponding to the current communication round, and sends it to each participating client for continued iterative updates. Until the iteration termination condition is met, each participating client obtains the target shared weight matrix and target private weight matrix of the model to be trained.

2. The personalized federated learning method according to claim 1, characterized in that, The method further includes: The server decomposes the initial weight matrix of the model to be trained according to the preset decomposition factor to obtain the first decomposed weight matrix and the second decomposed weight matrix, and sends them to each of the participating clients. Each participating client trains the model to be trained based on local training data, updates the first decomposed weight matrix and the second decomposed weight matrix, and obtains the updated shared weight matrix and the updated private weight matrix corresponding to the initial communication round.

3. The personalized federated learning method according to claim 2, characterized in that, The server decomposes the initial weight matrix of the model to be trained according to a preset decomposition factor, obtaining a first decomposed weight matrix and a second decomposed weight matrix, including: The server determines whether the preset decomposition factor is a factor of the initial matrix dimension of the initial weight matrix of the model to be trained. If so, the preset decomposition factor is used as the target matrix dimension to decompose the initial weight matrix, resulting in a first decomposed weight matrix and a second decomposed weight matrix. If not, then from the set of candidate values ​​that are smaller than the preset decomposition factor and are factors of the initial matrix dimension, determine the maximum value as the target matrix dimension, decompose the initial weight matrix to obtain the first decomposed weight matrix and the second decomposed weight matrix. Wherein, the matrix dimension of the first decomposition weight matrix is ​​the target matrix dimension, and the matrix dimension of the second decomposition weight matrix is ​​determined according to the initial matrix dimension and the target matrix dimension.

4. The personalized federated learning method according to claim 3, characterized in that, The initial weight matrix is ​​represented as follows: ,in, This indicates the total number of network layers in the model to be trained. ; The weight matrix of each layer in the model to be trained is represented as follows: in, This represents the weight matrix of each layer in the model to be trained. , This represents the first decomposition weight matrix of each layer of the network. , This represents the second decomposition weight matrix of each layer of the network. , , , It represents the Kronecker product.

5. The personalized federated learning method according to claim 4, characterized in that, Each participating client trains the model to be trained based on local training data, updates the aggregated shared weight matrix and the updated private weight matrix obtained in the previous communication round, and obtains the updated shared weight matrix and the updated private weight matrix corresponding to the current communication round, including: Each participating client trains the model to be trained using local training data. During the forward propagation process, each network layer performs a two-dimensional convolution on the input data using the updated private weight matrix from the previous communication round, outputting intermediate results; and performs a one-dimensional convolution on the intermediate results using the aggregated shared weight matrix from the previous communication round, outputting the output results of each network layer; the input data of the first network layer of the model to be trained is the local training data, and the output result of the last network layer is the prediction result of the model to be trained; During backpropagation, based on the prediction results, the label data in the local training data, and the preset loss function, the aggregated shared weight matrix and the updated private weight matrix obtained in the previous communication round are updated to obtain the updated shared weight matrix and the updated private weight matrix corresponding to the current communication round.

6. The personalized federated learning method according to claim 5, characterized in that, The aggregated shared weight matrix is ​​represented as follows: in, This represents the aggregated shared weight matrix. Indicates the first The update shared weight matrix sent by each participating client, where t represents the communication round between the participating client and the server. This indicates the total number of participating clients. .

7. The personalized federated learning method according to claim 6, characterized in that, The output of each layer of the network is represented as follows: in, This represents the output of each layer of the network. This represents the input data for each layer of the network. Indicates the first The first participating client The aggregated shared weight matrix corresponding to the layer network, Indicates the first The first participating client The updated private weight matrix corresponding to the layer network. This represents a two-dimensional convolution operation. This represents a multi-scale one-dimensional convolution operation.

Images