Distributed federated learning communication compression scheme based on model splitting

Abstract

The invention relates to the field of machine learning, in particular to a distributed federated learning communication compression scheme based on model splitting, and aims to reduce network transmission gradient communication traffic by compressing a model parameter matrix in a parameter updating process. According to the scheme, the method comprises the following steps: for a preset model parameter x, selecting a specified edge device to participate in training, and initializing data and a loss function; for the parameters transmitted to the edge device in the previous step, the edge device calculates a newest model parameter matrix by using local data of the edge device; and for the model parameter matrix calculated in the previous step, the edge device compresses the model parameter matrix by using a model splitting method and converting the model parameter matrix into a triple. The method is used in the field of federated learning in distributed machine learning, parameter optimization of the deep neural network can be realized under the condition that the network transmission bit amount is compressed, and meanwhile, the final model accuracy and relatively low communication rounds can be kept.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and in particular to a distributed federated learning communication compression scheme based on model splitting. Background Art

[0002] In recent years, deep learning has become an important method for solving many machine learning problems, especially with the rapid development of deep neural networks, which are applied to computer vision, natural language processing and other fields. With the increase in the amount of training data and the limitation of the computing power of a single device, distributed machine learning is widely used to train these large-scale deep neural networks, which greatly shortens the training time.

[0003] Federated learning is a popular and efficient distributed machine learning paradigm and framework. It follows the parameter server-worker server architecture of traditional distributed machine learning. It can distribute the model training calculation process to each data owner, allowing them to perform training in a local secure environment, and then submit the training model parameters to the central server for aggregation to obtain a high-quality global model. This cycle repeats until the model converges.

[0004] In some cases, the communication capabilities of the devices involved in federated learning vary greatly, which causes communication overhead to become the main bottleneck in the training process, where the communication time of parameters far exceeds the calculation time. Therefore, communication compression is a crucial necessity in federated learning. At present, there are many communication compression schemes in distributed federated learning scenarios. The most commonly used ones are the schemes for compressing the parameter matrix of the model and the schemes for compressing the communication of the gradient matrix of the model. However, since the above schemes require complicated calculations, the actual effect is generally average, resulting in the loss of accuracy and the reduction of convergence rate. Summary of the invention

[0005] The purpose of the present invention is to provide a communication compression solution based on model splitting to address the technical defects and problems of the above-mentioned existing technologies, which aims to solve the problems of precision loss and reduced convergence rate caused by the existing traditional communication compression algorithms for large neural networks.

[0006] To achieve the above object, the present invention provides the following technical solution: a distributed federated learning communication compression solution based on model splitting, comprising the following steps:

[0007] Step 1: For the preset model parameter x, select the specified edge device to participate in the training, and initialize the data and loss function:

[0008] In which a global server specifies multiple edge devices to participate in the model training task. Each edge device i locally stores the data that will be used to train the model parameters. The distribution of this data is specified by the initial global server. This scheme only considers the case where the data of all edge devices are independent and identically distributed;

[0009] The global server initializes the model parameters, training loss function, and training hyperparameters, distributes the initialized model parameters to each edge device, and specifies the loss function for each edge device.

[0010] Step 2: The edge device calculates the updated gradient value matrix:

[0011] Based on the distributed gradient descent algorithm, each edge device prepares to calculate the parameter matrix of the current round through its local data and the received model parameters;

[0012] Based on the received loss function and model parameters, each edge device randomly selects a certain amount of local data and uses the back-propagation algorithm to calculate the latest gradient value of the model parameters received in the current round using the selected local data. Where i represents the number of the edge device, represents the model parameter value received in the current round k, f i represents the received model loss function, Represents the back-propagation algorithm, which calculates model parameters based on its local data And the loss function f i The current round gradient value, finally passed To calculate the parameter matrix, where α represents the learning rate;

[0013] Step 3: The edge device uses the model splitting method to compress the calculated parameter matrix:

[0014] Based on the parameter matrix calculated in step 2, this step will compress it using model splitting.

[0015] Each edge device calculates a split matrix based on the parameter matrix in step 2. This splitting scheme randomly splits the entire parameter matrix into two parts, where most of the values ​​at most positions of one part of the matrix are 0, while very few of the values ​​at most positions of the other part of the matrix are 0. Finally, the sparse matrix with very few positions of the matrix being 0 is retained, and the other part of the matrix is ​​discarded. At this time, the model parameters can be transmitted in the network using the sparse matrix transmission scheme, which reduces the amount of communication in the entire federated learning process.

[0016] Step 4: The edge device communicates with the global server and updates parameters:

[0017] The sparse parameter matrix calculated by each edge device in step 3 and obtained after model splitting will be transmitted to the global server through the network; once the global server receives the sparse model parameter matrices of all edge devices in the current round, the global server will add these parameter matrices together as a new round of updates.

[0018] After the global server completes the update operation, it will select the latest model parameters:

[0019] Option (1): If the current model parameters reach the accuracy of the model parameters preset in step 1, the algorithm is terminated and the current latest model parameters are output.

[0020] Select solution (2): If the current model parameters fail to reach the accuracy of the model parameters preset in step 1, the global server will redistribute the latest model parameters to all edge devices, and then the edge devices will execute step 2 again, and so on, until the global server completes the update operation in step 4 in a certain round and selects solution (1).

[0021] Preferably, the higher the compression rate hyperparameter setting of the model splitting selected in step 3, the lower the network communication traffic that can be achieved will be, but this will also result in a later determination of solution (1) in the final step 4. However, under the same communication compression rate, this solution can achieve a higher final model accuracy than the existing communication compression method based on sparse matrices, and the number of communication rounds required to achieve the target accuracy is lower.

[0022] Preferably, the amount of local data selected in step 2 of the present scheme will also affect the efficiency of the algorithm. The more local data is selected, the higher the model accuracy that the algorithm can ultimately achieve, and the fewer communication rounds it takes to achieve the target accuracy, but the resulting time taken by the back propagation algorithm in step 2 to calculate the result is also lower.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] The solution of the present invention optimizes the distributed gradient descent algorithm in federated learning based on model splitting, and can ensure that the model can achieve the final accuracy and shorter communication rounds while compressing the communication bandwidth. Since each device only processes a part of the parameters, the optimization calculations performed on the device will also be less, thereby reducing the computing load of the device. Another advantage of not uploading all model parameters during the federated learning process is that the data privacy performance of edge devices is enhanced. The federated learning communication compression algorithm based on model splitting does not introduce other means of privacy computing. It only relies on uploading incomplete model parameters to increase the difficulty for attackers to try to restore the attacked training set and model through gradients or model parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the model splitting process of the present invention;

[0026] Figure 2 It is a schematic diagram of the implementation process of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The present invention will be further described below in conjunction with the accompanying drawings:

[0029] See also Figure 2 , for a given neural network model to be trained, the following steps are used to optimize the model parameters while compressing the communication volume:

[0030] Step 1: For the preset model parameter x, select the specified edge device to participate in the training, and initialize the data and loss function.

[0031] Step 2: For the parameters transmitted to the edge device in step 1, the edge device calculates the model parameter matrix using its local data.

[0032] Step 3: For the model parameter matrix calculated in step 2, the edge device uses the model splitting method to sparse the model parameter matrix.

[0033] Step 4: The global server uses the sparse model parameter matrix compressed in step 3 to update the global parameters.

[0034] The detailed implementation of each step of this plan is as follows:

[0035] Step 1: The global server specifies multiple edge devices to participate in the model training task. Each edge device i has a local storage of the data dataset that will be used to train the model parameters. i At the same time, the global server initializes the model parameters x, the training loss function f(x), and the training hyperparameter information, where the training hyperparameter information includes the gradient descent learning rate α of each edge device, and distributes all these parameters to each edge device.

[0036] Step 2: The current communication round is recorded as k. Each edge device i will randomly select local data dataset i ∈Dataset i, for model parameters transmitted through the global server And the loss function f i , to calculate the current loss value After the loss value is calculated, each edge device uses the back propagation algorithm to calculate the model parameters for the current round k. The gradient matrix Finally, the model parameter moment is calculated by the gradient descent algorithm.

[0037] In step 3, each edge device uses the model splitting algorithm to calculate the sparse model parameter matrix. During the network transmission of the sparse model parameter matrix, it is converted into a triple table for transmission, thereby reducing the communication transmission volume of each edge device.

[0038] In order to describe our pooling scheme more concisely, the maximum model split and average model split in the following algorithm will be represented by the Split operator.

[0039] When each edge device completes step 2 to obtain the gradient value matrix Then, the local gradient descent learning rate α of the edge device initialized in step 1 and the model parameters transmitted from the global server are used. To calculate the model parameter matrix through the gradient descent algorithm

[0040] The core point of this step is to record the current communication round as k, and each edge device i has the model parameter matrix obtained in step 2. Using model splitting for compression The compressed model parameter matrix is ​​used express.

[0041] It should be noted that the role of model splitting is to sparse the model parameter matrix to be uploaded in step 4. This step can reduce the number of bits occupied by the uploaded model parameter matrix to alleviate the extension of training time caused by high-load communication when training large neural networks. Since each device only processes a part of the parameters, the optimization calculations performed on the device will also be less, thereby reducing the computing load of the device. Another advantage of not uploading all model parameters during federated learning is that it enhances the data privacy performance of edge devices. The federated learning communication compression algorithm based on model splitting does not introduce other privacy computing methods. It only relies on uploading incomplete model parameters to increase the difficulty for attackers to try to restore the attacked training set and model through gradients or model parameters. Compared with other communication compression methods, our operator can achieve better model accuracy and lower communication rounds at the same communication compression rate, while saving the computing power of edge devices and protecting the model security of each edge device.

[0042] In step 4, each edge device uploads the sparse model parameter matrix calculated in step 3 to the global server to aggregate and update the parameters.

[0043] The process of aggregation and parameter update is as follows:

[0044] Once all edge devices have calculated the sparse model parameter matrix Uploaded to the global server, the global server will sum these sparse model parameter matrices, which is formally expressed by the following formula This step can update the global parameters and obtain the global parameters x for the next round k+1 After this step, the global server will go through a selection step:

[0045] Selection step (1): If the current model parameters reach the accuracy of the model parameters preset in step 1, the algorithm terminates and outputs the current latest model parameters x end , which is the current local optimal value of the global model parameter in the current global server

[0046] Select step (2): If the current model parameters fail to reach the accuracy of the model parameters preset in step 1, the global server will redistribute the latest model parameters to all edge devices, and then the edge devices will execute step 2 again, and so on, until the global server completes the update operation in step 4 in a certain round and selects step (1).

[0047] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0048] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A distributed federated learning communication compression scheme based on model splitting, characterized by: The following steps are involved: Step 1: For the preset model parameter x, select the specified edge device to participate in the training, and initialize the data and loss function: In which a global server specifies multiple edge devices to participate in the model training task. Each edge device i locally stores the data that will be used to train the model parameters. The distribution of this data is specified by the initial global server. This scheme only considers the case where the data of all edge devices are independent and identically distributed; The global server initializes the model parameters, training loss function, and training hyperparameters, distributes the initialized model parameters to each edge device, and specifies the loss function for each edge device. Step 2: The edge device calculates the updated gradient value matrix: Based on the distributed gradient descent algorithm, each edge device prepares to calculate the parameter matrix of the current round through its local data and the received model parameters; Based on the received loss function and model parameters, each edge device randomly selects a certain amount of local data and uses the back-propagation algorithm to calculate the latest gradient value of the model parameters received in the current round using the selected local data. Where i represents the number of the edge device, represents the model parameter value received in the current round k, f i represents the received model loss function, Represents the back-propagation algorithm, which calculates model parameters based on its local data And the loss function f i The current round gradient value, finally passed To calculate the parameter matrix, where α represents the learning rate; Step 3: The edge device uses the model splitting method to compress the calculated parameter matrix: Based on the parameter matrix calculated in step 2, this step will compress it using model splitting. Each edge device calculates a split matrix based on the parameter matrix in step 2. This splitting scheme randomly splits the entire parameter matrix into two parts, where most of the values ​​at most positions of one part of the matrix are 0, while very few of the values ​​at most positions of the other part of the matrix are 0. Finally, the sparse matrix with very few positions of the matrix being 0 is retained, and the other part of the matrix is ​​discarded. At this time, the model parameters can be transmitted in the network using the sparse matrix transmission scheme, which reduces the amount of communication in the entire federated learning process. Step 4: The edge device communicates with the global server and updates parameters: The sparse parameter matrix calculated by each edge device in step 3 and obtained after model splitting will be transmitted to the global server through the network; Once the global server receives the sparse model parameter matrices of all edge devices in the current round, the global server will add up these parameter matrices as a new round of updates. After the global server completes the update operation, it will select the latest model parameters: Option (1): If the current model parameters reach the accuracy of the model parameters preset in step 1, the algorithm is terminated and the current latest model parameters are output. Select solution (2): If the current model parameters fail to reach the accuracy of the model parameters preset in step 1, the global server will redistribute the latest model parameters to all edge devices, and then the edge devices will execute step 2 again, and so on, until the global server completes the update operation in step 4 in a certain round and selects solution (1).

2. According to a distributed federated learning communication compression scheme based on model splitting according to claim 1, it is characterized by: The higher the compression rate hyperparameter of the model splitting selected in step 3 is set, the lower the network communication traffic that can be achieved will be, but it will also result in a later determination of solution (1) in the final step 4. However, under the same communication compression rate, this solution can achieve a higher final model accuracy than the existing communication compression method based on sparse matrices, and the number of communication rounds required to achieve the target accuracy is lower.

3. According to a distributed federated learning communication compression scheme based on model splitting according to claim 1, it is characterized by: The amount of local data selected in step 2 of the present solution will also affect the efficiency of the algorithm. The more local data is selected, the higher the model accuracy that the algorithm can ultimately achieve, and the fewer communication rounds it takes to achieve the target accuracy. However, the time it takes for the back propagation algorithm in step 2 to calculate the result is also shorter.

Images