Adaptive federated learning method for heterogeneous computing devices

Abstract

The present invention discloses an adaptive federated learning method for devices with heterogeneous computing power, belonging to the technical field of federated learning. The present invention selects a global model for a federated learning task matched by a server, wherein each user device can independently select trainable local model parameters according to its own computing power during each round of iteration, so that user devices with weaker computing power spend less training time, thereby improving the model training speed of federated learning; the technical solution of the present invention allows users with poor computing power to train only a part of the model parameters, reducing local computing overhead, so that a larger-scale global model can be adopted; and solves the impact of differences in the computing power of user devices on federated learning.

Description

Technical Field

[0001] The present invention relates to the field of federated learning technology, and in particular to an adaptive federated learning method for heterogeneous computing devices. Background Art

[0002] In recent years, with the rapid development of machine learning technology, its application in fields such as natural language processing and computer vision has become increasingly widespread. Training high-performance machine learning models requires a large amount of high-quality training data. The data generated by user devices such as mobile devices and IoT devices in their daily activities is large in scale and rich in variety, which is conducive to training machine learning models. Traditional centralized machine learning aggregates data from user devices to servers to complete model training. However, this approach faces two major challenges. First, due to geographical dispersion, large data volumes, or legal restrictions, the data held by user devices is difficult to communicate efficiently. Second, user device data often contains private information of users, which owners of private data are reluctant to disclose to the outside world. These issues have severely restricted the application of traditional machine learning technology in distributed environments and privacy-sensitive fields.

[0003] Federated learning, a privacy-preserving distributed machine learning paradigm, effectively addresses the aforementioned issues. It allows different user devices to collaboratively train a global model by sharing model updates rather than the original data itself, while maintaining local data storage. Specifically, under the direction of a server, each user device trains a model on local data and periodically uploads updated model parameters to the server. This ensures the security and privacy of user device data while enabling cross-device machine learning model training.

[0004] In actual application scenarios, factors such as the model, parameters, and working conditions of user devices may vary significantly, resulting in different computing power among different user devices, i.e., computing power heterogeneity. Users with poor computing power have slow computing speeds and may not be able to properly train models with large parameter sizes. In scenarios where user devices have heterogeneous computing power, traditional federated learning methods have the following problems: (1) User devices with poor computing power cannot train large-scale models, thereby limiting the types of global models available. (2) When all user devices train models of the same size, some user devices with poor computing power will spend more time training the model. User devices with strong computing power have to wait for other users to complete training after uploading model updates, thereby slowing down the entire federated learning training process. Summary of the Invention

[0005] In view of the above problems, the present invention provides an adaptive federated learning method for devices with heterogeneous computing power. In the technical solution of the present invention, user devices autonomously select trainable model parameters in each round based on their own computing power, allowing user devices with weaker computing power to spend less training time, thereby improving the model training speed of federated learning. The technical solution of the present invention allows users with poor computing power to train only a portion of the model parameters, reducing local computing overhead and thus enabling the use of a larger global model. The method of the present invention allows user devices to determine the trainable parameters of the local model in real time based on the current local computing power, thus addressing the impact of differences in user device computing power on federated learning.

[0006] The present invention provides an adaptive federated learning method for heterogeneous computing devices, comprising:

[0007] Step S1: Select the global model of the federated learning task matched by the server and initialize the parameters to obtain the original global model; broadcast the original global model to each device user;

[0008] Step S2: let k = 1. When k = 1, it indicates the initial iteration.

[0009] Step S3: obtaining a global model received by each user equipment in the kth iteration;

[0010] The global model received by each user device in the kth iteration is used as the corresponding local model;

[0011] Step S4: Obtain the computing power of the local model corresponding to each user device in the kth iteration, and determine the trainable parameters and non-trainable parameters of the local model corresponding to each user device based on the computing power of the local model corresponding to each user device;

[0012] Step S5: Let i=1. When i=1, it represents the first user equipment in the k-th iteration;

[0013] Step S6: Use the stochastic gradient descent method to update the i-th user device A in the k-th iteration i,k Corresponding local model B i,k The trainable parameters of the i-th user device A in the k-th iteration are obtained i,k Updated trainable parameters;

[0014] Step S7: Obtain the i-th user equipment A in the k-th iteration i,k The corresponding non-trainable parameters;

[0015] The i-th user equipment A in the k-th iteration i,k The corresponding non-trainable parameters and updated trainable parameters are aggregated using the federated averaging algorithm to obtain the i-th user device A in the k-th iterationi,k Update parameters of

[0016] Step S8: Based on the i-th user equipment A in the k-th iteration i,k The updated local model B′ of the i-th user device in the k-th iteration is obtained by updating the parameters i,k , and record;

[0017] Step S9: Determine whether i is greater than or equal to I, where I represents the total number of user devices. If so, obtain the updated local models of I user devices in the kth iteration and proceed to the next step. If not, set i = i + 1 and return to step S5.

[0018] Step S10: obtaining an updated global model in the kth iteration based on the updated local models of I user equipment in the kth iteration;

[0019] Determine whether k is greater than or equal to K, where K represents the total number of iterations. If so, the global model updated in the kth iteration is used as the final global model. If not, set k=k+1 and return to step S2.

[0020] Optionally, the expression of the local model corresponding to each user equipment in step S3 is:

[0021]

[0022] Among them, w i represents the local model of the i-th user device, represents the model parameters of layers 1 to -q in the local model of the i-th user device, represents the model parameters of layers p-q+1 to p in the local model of the i-th user equipment, where is the untrainable parameter of the local model of the i-th user device, The local model for the i-th user device is trainable.

[0023] Optionally, the expression of the kth updated trainable parameter of the i-th user equipment in step S6 is:

[0024]

[0025] in, represents the model parameters of layers p-q+1 to p in the local model of the i-th user device, η is the learning rate, represents the gradient of the trainable parameters of the local model of the i-th user device, is the loss function, D i is the local dataset of the i-th user device, w i represents the local model of the i-th user device.

[0026] Optionally, the local model corresponding to each user equipment has p levels L = {l1, l2, .., l p}, where l p Represents the pth level; the amount of computation required for forward propagation of each level is N forward ={n1, n2, .., n p}, where n p Represents the amount of computation required for the p-th level forward propagation;

[0027] The amount of computation required for back propagation at each level is N backward ={2n1, 2n2, .., 2n p}.

[0028] Optionally, the specific steps of determining the trainable parameters of the local model corresponding to each user equipment in step S4 include:

[0029] Obtain the trainable parameters of each user device and the size of the local dataset of each user device in the kth iteration and the computing power of each user device

[0030] Based on the size of the local dataset of each user device in the kth iteration and the computing power of each user device Calculate the computing speed of each user device Select the user device with the fastest computing speed * All parameters of are used as trainable parameters;

[0031] Based on the local dataset size and computing power of the user device with the fastest computing speed, the trainable parameters of the remaining user devices are obtained.

[0032] Optionally, the expression of the trainable parameters of the local model corresponding to each user device is:

[0033]

[0034] Among them, N q Indicates the amount of local model training computation when the last q layers participate in back propagation, N p Indicates the amount of local model training computation when the last p layers participate in back propagation, is the number of trainable parameter layers for user device i in the kth iteration, Indicates the user device i with the fastest computing speed * The size of the local dataset, Indicates the user device i with the fastest computing speed * The computing power in round k;

[0035] Another object of the present invention is to provide an application of an adaptive federated learning method for heterogeneous computing devices, characterized in that image classification processing is performed based on the final global model described in step S10.

[0036] Compared with the prior art, the present invention has at least the following beneficial effects:

[0037] (1) The technical solution of the present invention allows users with poor computing power to train only a portion of the model parameters, reducing local computing overhead and thus enabling the use of a larger global model;

[0038] (2) In the technical solution of the present invention, the user device independently selects the trainable model parameters in each round according to its own computing power, so that the user device with weaker computing power spends less training time, thereby improving the model training speed of federated learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings are only for purposes of illustrating particular embodiments and are not to be considered limiting of the invention.

[0040] Figure 1 is a schematic diagram of the structure of a global model in an embodiment of the present invention;

[0041] Figure 2 Schematic diagram of the process of an adaptive federated learning method for heterogeneous computing devices in an embodiment of the present invention;

[0042] Figure 3 Schematic diagram of the comparison of experimental results of the embodiment of the present invention and the federated averaging algorithm. DETAILED DESCRIPTION

[0043] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. In addition, the present invention can also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited by the specific embodiments disclosed below.

[0044] A specific embodiment of the present invention, as Figure 1-3 , discloses an adaptive federated learning method for heterogeneous computing devices, such as Figure 1 The system consists of a server and several user devices, each with different computing power. The server and all user devices use the same model structure, and the user devices select the trainable parameters of the local model based on their own computing power.

[0045] Figure 3This is a comparison chart of the experimental results of the method proposed in this invention and the federated averaging algorithm. In practice, the adaptive federated learning method for heterogeneous computing devices is applied to image classification tasks; it includes 1 server and 10 user devices, each of which holds part of the training data from the CIFAR-10 dataset, and the global model is the ResNet-18 model. The computing power of users varies. The curve of the model accuracy of the global model on the test dataset as a function of training time is shown in Figure 2. Figure 3 shown.

[0046] In order to illustrate the effectiveness of the method proposed in the present invention, the above technical solution of the present invention is described in detail below through a specific embodiment. The specific implementation steps are as follows:

[0047] The present invention provides an adaptive federated learning method for heterogeneous computing devices, comprising:

[0048] Step S1: Select the global model of the federated learning task matched by the server and initialize the parameters to obtain the original global model; broadcast the original global model to each device user;

[0049] Step S2: let k = 1. When k = 1, it indicates the initial iteration.

[0050] Step S3: obtaining a global model received by each user equipment in the kth iteration;

[0051] The global model received by each user device in the kth iteration is used as the corresponding local model;

[0052] Step S4: Obtain the computing power of the local model corresponding to each user device in the kth iteration, and determine the trainable parameters and non-trainable parameters of the local model corresponding to each user device based on the computing power of the local model corresponding to each user device;

[0053] It can be understood that the trainable parameters are the trainable parameters and non-trainable parameters of the ResNet-18 model corresponding to the user device with the fastest computing power in the ResNet-18 model received by each user device.

[0054] Step S5: Let i=1. When i=1, it represents the first user equipment in the k-th iteration;

[0055] Step S6: Use the stochastic gradient descent method to update the i-th user device A in the k-th iteration i,k Corresponding local model B i,k The trainable parameters of the i-th user device A in the k-th iteration are obtained i,k Updated trainable parameters;

[0056] Step S7: Obtain the i-th user equipment A in the k-th iterationi,k The corresponding non-trainable parameters;

[0057] The i-th user equipment A in the k-th iteration i,k The corresponding non-trainable parameters and updated trainable parameters are aggregated using the federated averaging algorithm to obtain the i-th user device A in the k-th iteration i,k Update parameters of

[0058] Step S8: Based on the i-th user equipment A in the k-th iteration i,k The updated parameters of the i-th user equipment are used to obtain the updated local model B′ i,k , and record;

[0059] Step S9: Determine whether i is greater than or equal to I, where I represents the total number of user devices. If so, obtain the updated local models of I user devices in the kth iteration and proceed to the next step. If not, set i = i + 1 and return to step S5.

[0060] Step S10: obtaining an updated global model in the kth iteration based on the updated local models of I user equipment in the kth iteration;

[0061] Determine whether k is greater than or equal to K, where K represents the total number of iterations. If so, the global model updated in the kth iteration is used as the final global model. If not, set k=k+1 and return to step S2.

[0062] Another object of the present invention is to provide an application of an adaptive federated learning method for computing power heterogeneous devices in the field of image information processing, and to perform image classification processing based on the final global model described in step S10.

[0063] Preferably, the expression of the local model corresponding to each user equipment in step S3 is:

[0064]

[0065] Among them, w i represents the local model of the i-th user device, represents the model parameters of layers 1 to pq in the local model of the i-th user equipment, represents the model parameters of layers p-q+1 to p in the local model of the i-th user equipment, where is the untrainable parameter of the local model of the i-th user device, The local model for the i-th user device is trainable.

[0066] Preferably, the expression of the kth updated trainable parameter of the i-th user equipment in step S6 is:

[0067]

[0068] in, represents the model parameters of layers p-q+1 to p in the local model of the i-th user device, η is the learning rate, represents the gradient of the trainable parameters of the local model of the i-th user device, is the loss function, D i is the local dataset of the i-th user device.

[0069] The trainable parameters in the local model described in the present invention are located after the non-trainable parameters. Gradient calculation for the trainable parameters does not require backpropagation on all model parameters, thereby reducing the computational overhead of local model training corresponding to each user device.

[0070] It can be understood that the local model corresponding to each user equipment has p levels L = {l1, l2, .., l p}, where l p Represents the pth level; the amount of computation required for forward propagation of each level is N forward ={n1, n2, .., n p}, where n p Represents the amount of computation required for the p-th level forward propagation;

[0071] The amount of computation required for back propagation at each level is N backward ={2n1, 2n2, .., 2n p};

[0072] In the present invention, the back propagation time of each layer of the local model is twice that of the forward propagation. In addition, during the calculation of the forward and backward propagation of the local model, the impact of operations such as nonlinear activation, pooling, and residual connections on the calculation time is small and can be ignored.

[0073] For example, take the ResNet-18 model as an example. In this model, there are 8 residual blocks and 1 single convolutional layer, that is, p = 9. Since the parameter scale of the output layer is much smaller than that of the convolutional layer, its computational overhead can be ignored. The computational cost of forward propagation (unit: GFLOP) s ) is:

[0074] N forward ={0.12,0.23,0.23,0.17,0.23,0.17,0.23,0.17,0.23}

[0075] When the training parameters of the user equipment are all the parameters of the last q layers, the sum of the computational complexity of a single forward propagation and backpropagation is expressed as:

[0076] {N q＝1，2，..，9}={5.34,5.10,4.64,4.18,3.84,3.38,3.04,2.58,2.24}.

[0077] In the present invention, the user equipment only needs to train some parameters of the deep layer of the network, and there is no need to perform back propagation on the shallow layer parameters. That is, the trainable parameters of the user equipment are all the parameters of the last q layers, and the expression is:

[0078] N q =n1+n2+...+n p-q +3n p-q+1 +...+3n p

[0079] Among them, all layers participate in forward propagation, and q is the layer involved in backward propagation.

[0080] Preferably, the specific steps of determining the trainable parameters of the local model corresponding to each user equipment in step S4 include:

[0081] Obtain the trainable parameters of each user device and the size of the local dataset of each user device in the kth iteration The computing power of each user device

[0082] Based on the trainable parameters of each user device in the kth iteration, the size of the local data set of each user device The computing power of each user device The calculation time of each user device is obtained as follows:

[0083]

[0084] in, is the number of trainable parameter layers of user device i in the kth round. r is related to the number of local stochastic gradient descent times of the user device and the batch size of training samples, and can be regarded as a constant.

[0085] In order to reduce the difference in computing time between different user devices, the user device with the fastest computing speed i * The computation time of is set as the lower bound of the computation time of other user devices;

[0086] The calculation time of the preset user device i is greater than or equal to the user device i * The calculation time is expressed as:

[0087]

[0088]

[0089] Simplifying, we can get:

[0090]

[0091] in, It represents the model training computation amount when user device i participates in back propagation at the qth level after the kth round of training. Indicates the user device i with the fastest computing speed * The size of the local dataset, Indicates the user device i with the fastest computing speed * The amount of model training calculation when q layers participate in back propagation after the kth round of training, Indicates the user device i with the fastest computing speed * The computing power in the kth round.

[0092] Based on the size of the local dataset of each user device in the kth iteration and the computing power of each user device Calculate the computing speed of each user device Select the user device with the fastest computing speed * All parameters of are used as trainable parameters;

[0093] Based on the local dataset size and computing power of the user device with the fastest computing speed, the trainable parameters of the remaining user devices are obtained, which are expressed as:

[0094]

[0095]

[0096]

[0097] Among them, N q Indicates the amount of model training calculation when the last q layers participate in back propagation, N p Indicates the amount of model training calculation when the last p layers participate in back propagation, is the number of trainable parameter layers for user device i in the kth iteration, is an integer.

[0098] Exemplarily, in step S4, taking the ResNet-18 model as an example, a single user device can reduce the communication volume by up to about 60% when uploading the model.

[0099] In this method, the trainable parameters in the local model are placed after the non-trainable parameters. This eliminates the need to perform backpropagation on all model parameters to calculate the gradient of the trainable parameters, thereby reducing the computational overhead of model training. During each round of training, all user devices perform the same number of stochastic gradient descent passes on the local dataset.

[0100] The proposed method can achieve higher model accuracy than the federated averaging algorithm at the same training time. Through the method described in this example, the user device can adaptively select the appropriate number of trainable parameters, ensuring good model accuracy while reducing training time overhead.

[0101] In the present invention, when using the backpropagation algorithm to calculate the gradients of the trainable parameters in the local model corresponding to each user device, the gradients of the parameters near the output layer of the local model are calculated first, and the parameters near the input layer are calculated last. Given the number of trainable parameters in the local model, the backpropagation computational complexity is reduced when the trainable parameters are close to the output layer of the local model. When selecting trainable parameters, the user device in the present invention prioritizes parameters near the output layer to reduce local computational overhead.

[0102] In this invention, the user device only uploads local trainable parameters as a model update, not the complete local model. During local training, the user device does not update parameters other than the trainable parameters. Therefore, parameters not uploaded to the server remain unchanged before and after local training. Furthermore, before aggregation, the parameters uploaded by the user device are first supplemented with the non-trainable parameters of the local model from the previous round. Aggregation can then be performed using the federated averaging algorithm.

[0103] In particular, in the present invention, the global model parameters are randomly initialized at the beginning of training. In the first few rounds of training, the non-trainable parameters are excluded from the aggregation to accelerate the model convergence.

[0104] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. An adaptive federated learning method for heterogeneous computing devices, characterized by: include: Step S1: Select the global model of the federated learning task matched by the server and initialize the parameters to obtain the original global model; broadcast the original global model to each device user; Step S2: let k = 1. When k = 1, it indicates the initial iteration. Step S3: obtaining a global model received by each user equipment in the kth iteration; The global model received by each user device in the kth iteration is used as the corresponding local model; Step S4: Obtain the computing power of the local model corresponding to each user device in the kth iteration, and determine the trainable parameters and non-trainable parameters of the local model corresponding to each user device based on the computing power of the local model corresponding to each user device; Step S5: Let i=1. When i=1, it represents the first user equipment in the k-th iteration; Step S6: Use the stochastic gradient descent method to update the i-th user device A in the k-th iteration i,k Corresponding local model B i,k The trainable parameters of the i-th user device A in the k-th iteration are obtained i,k Updated trainable parameters; Step S7: Obtain the i-th user equipment A in the k-th iteration i,k The corresponding non-trainable parameters; The i-th user equipment A in the k-th iteration i,k The corresponding non-trainable parameters and updated trainable parameters are aggregated using the federated averaging algorithm to obtain the i-th user device A in the k-th iteration i,k Update parameters of Step S8: Based on the i-th user equipment A in the k-th iteration i,k The updated local model B′ of the i-th user device in the k-th iteration is obtained by updating the parameters i,k , and record; Step S9: Determine whether i is greater than or equal to I, where I represents the total number of user devices. If so, obtain the updated local models of I user devices in the kth iteration and proceed to the next step. If not, set i = i + 1 and return to step S5. Step S10: obtaining an updated global model in the kth iteration based on the updated local models of I user equipment in the kth iteration; Determine whether k is greater than or equal to K, where K represents the total number of iterations. If so, the global model updated in the kth iteration is used as the final global model. If not, set k=k+1 and return to step S2.

2. The adaptive federated learning method for heterogeneous computing devices according to claim 1, characterized in that: The expression of the local model corresponding to each user equipment in step S3 is: Among them, w i represents the local model of the i-th user device, represents the model parameters of layers 1 to pq in the local model of the i-th user equipment, represents the model parameters of layers p-q+1 to p in the local model of the i-th user equipment, where is the untrainable parameter of the local model of the i-th user device, The local model for the i-th user device is trainable.

3. The adaptive federated learning method for heterogeneous computing devices according to claim 1, characterized in that: The expression of the kth updated trainable parameter of the i-th user equipment in step S6 is: in, represents the model parameters of layers p-q+1 to p in the local model of the i-th user device, η is the learning rate, represents the gradient of the trainable parameters of the local model of the i-th user device, is the loss function, D i is the local dataset of the i-th user device, w i represents the local model of the i-th user device.

4. The adaptive federated learning method for heterogeneous computing devices according to claim 1, characterized in that: The local model corresponding to each user device has p levels L = {l1,l2,..,l p }, where l p Represents the pth level; the amount of computation required for forward propagation of each level is N forward ={n1,n2,..,n p }, where n p Represents the amount of computation required for the p-th level forward propagation; The amount of computation required for back propagation at each level is N backward ={2n1,2n2,..,2n p }.

5. The adaptive federated learning method for heterogeneous computing devices according to claim 1, characterized in that: The specific steps of determining the trainable parameters of the local model corresponding to each user device in step S4 include: Obtain the trainable parameters of each user device and the size of the local dataset of each user device in the kth iteration and the computing power of each user device Based on the size of the local dataset of each user device in the kth iteration and the computing power of each user device Calculate the computing speed of each user device Select the user device with the fastest computing speed * All parameters of are used as trainable parameters; Based on the local dataset size and computing power of the user device with the fastest computing speed, the trainable parameters of the remaining user devices are obtained.

6. The adaptive federated learning method for heterogeneous computing devices according to claim 5, characterized in that: The expression of the trainable parameters of the local model corresponding to each user device is: Among them, N q Indicates the amount of local model training computation when the last q layers participate in back propagation, N p Indicates the amount of local model training computation when the last p layers participate in back propagation, is the number of trainable parameter layers for user device i in the kth iteration, Indicates the user device i with the fastest computing speed * The size of the local dataset, Indicates the user device i with the fastest computing speed * The computing power in the kth round.

7. An application of the adaptive federated learning method for heterogeneous computing devices according to any one of claims 1 to 6, characterized in that: Image classification processing is performed based on the final global model described in step S10.

Images