A communication efficient federated learning method based on model parameter differentiation discarding

Abstract

The application discloses a communication efficient federated learning method based on model parameter differentiation discarding, which comprises the following steps: determining the structure of a local model and initializing; training the initialized local model through a local data set; obtaining the parameter discard rate calculation allowed parameter quantity issued by a server and uploading to the server; performing an aggregation operation on the trained local model through the uploaded parameter quantity to obtain a global model; if the global model does not satisfy a convergence condition, the server updates the initialized local model according to the optimal parameter discard rate issued according to the environmental factors of users; until the convergence condition is satisfied, the optimal global model is output. The application can fully utilize the computing resources and communication resources of users, accelerate the convergence of the global model and improve the generalization performance of the global model. The application can be widely applied to the field of computer artificial intelligence technology as a communication efficient federated learning method based on model parameter differentiation discarding.

Description

Technical Field

[0001] This invention relates to the field of computer artificial intelligence technology, and in particular to a communication-efficient federated learning method based on differential discarding of model parameters. Background Technology

[0002] With the widespread use of IoT devices and the rise of edge computing, a large amount of data is generated on terminal devices or edge devices. Traditional centralized machine learning methods require each user to upload local data to cloud servers to participate in model training. However, centralized methods are difficult to meet the requirements of data privacy and security. In order to avoid uploading a large amount of private data to cloud servers for centralized artificial intelligence model training, researchers have proposed mobile edge computing, which migrates artificial intelligence model training from the cloud to the edge or terminal. At the same time, the continuous improvement of the hardware performance of terminal devices and edge devices makes it possible to train artificial intelligence models on mobile devices. However, due to the limited amount of data that users have, it is not enough to train a high-performance model. To achieve the goals of protecting data privacy and collaborative model training, a distributed machine learning scheme called federated learning has gradually emerged in recent years. It allows users to train machine learning models in a distributed network by sharing only local model updates instead of the original data. Although federated learning brings benefits in terms of data privacy, it comes at the cost of training efficiency. This is because federated learning requires frequent exchange of model parameters between servers and users. In addition, due to differences in geographical distribution and communication conditions between users and servers, communication time can vary greatly even when transmitting models of the same size. In classic federated learning, all users need to synchronize and wait for the slowest user to complete the parameter upload before starting a new round of training, which further reduces communication efficiency. This phenomenon is known as the "faller effect."

[0003] To improve communication efficiency, existing technologies primarily employ user selection mechanisms, meaning only a subset of users are chosen to participate in updating the global model. User-selection-based federated learning methods reduce the number of users participating in global training according to various rules, including random selection, selecting users with shorter training times, or selecting users based on custom utility metrics. While user-selection-based federated learning can effectively improve communication efficiency and mitigate the laggard effect, it inevitably wastes some expensive computational resources. Furthermore, some users remain idle for extended periods, having fewer opportunities to participate in global updates, resulting in a model that cannot well adapt to the data distribution of these users. Essentially, user-selection-based federated learning reduces the amount of data the global model can indirectly access, which is detrimental to global model convergence. More importantly, user selection further exacerbates data heterogeneity and may even lead to the loss of certain data labels, thereby weakening the generalization ability of the global model. Summary of the Invention

[0004] To address the aforementioned technical problems, the present invention aims to provide a communication-efficient federated learning method based on differential model parameter discarding, which can fully utilize the user's computing and communication resources, accelerate global model convergence, and improve the generalization performance of the global model.

[0005] The first technical solution adopted in this invention is: a communication-efficient federated learning method based on model parameter differential discarding, comprising the following steps:

[0006] S1. Determine the structure of the local model and initialize the parameters of the local model. The structure of the local model is a neural network model, and the parameters of the local model are the weight values ​​and bias values ​​of the connections in the neural network model.

[0007] S2. Train the initialized local model using a local dataset, where the local dataset represents data stored by the user on the local device and is private, and cannot be directly transmitted to other users or the central server.

[0008] S3. Obtain the parameter discard rate issued by the server and calculate the number of parameters allowed to be uploaded based on the parameter discard rate;

[0009] S4. Upload the allowed number of parameters to the server;

[0010] S5. Aggregate the trained local model based on the uploaded parameters and output the global model;

[0011] S6. Make a judgment on the global model;

[0012] S7. If it is determined that the global model does not meet the preset convergence condition, the server will send the optimal parameter discard rate and global model according to the user's environmental factors, update the initialized local model, and repeat the above steps S2 to S6 based on the updated local model.

[0013] S8. If the global model is found to meet the preset convergence condition, the optimal global model is output.

[0014] Furthermore, in step S2, the expression for training the initialized local model is as follows:

[0015]

[0016]

[0017] In the above formula, This represents the local model parameters of user n before training in round t. Let η represent the model parameters after training user n in round t, and let η represent the learning rate. f represents the local dataset of user n. j (W) represents the loss value of sample j in model W. This represents the gradient generated by user n during local training.

[0018] Furthermore, step S3 specifically includes:

[0019] S31. Obtain the parameter discard rate and calculate the importance of the parameters of the local model to the preset convergence condition;

[0020] S32. Select the local model parameters with the highest importance to construct a matrix to represent the sparsification position of the model. An element of the matrix with a value of 1 indicates that the parameter at the corresponding position is uploaded, and an element with a value of 0 indicates that the parameter at the corresponding position is not uploaded.

[0021] S33. Calculate the number of neuron channels allowed to be uploaded to each layer of the local model and the corresponding importance index;

[0022] S34. Select the neuron channel with the highest importance index in each layer of the local model and upload the element with a value of 1 in the matrix.

[0023] Furthermore, the specific formula for calculating the importance index of the neuron channels allowed to be uploaded to each layer of the local model is as follows:

[0024]

[0025]

[0026] In the above formula, This represents the importance index value of neuron channel k for user n. This represents the change in model parameters before and after user n's local training in round t.

[0027] Furthermore, in step S5, the expression for the aggregation operation is as follows:

[0028]

[0029] In the above formula, m n W represents the number of samples in the local dataset for user n. t This represents the global model parameters after the aggregation in round t. This indicates the number of parameters that can be uploaded.

[0030] Furthermore, in step S7, the expression for updating the initialized local model is as follows:

[0031]

[0032] In the above formula, This represents the local model of user n before the (t+1)th round of training. Represents the global model. This indicates the location of sparsification in the model.

[0033] Furthermore, the step S7, which involves issuing the optimal parameter discard rate based on the user's environmental factors, specifically includes:

[0034] S71. Calculate the local model upload time for each user under a given parameter drop rate, the local model distribution time for each user under a given parameter drop rate, and the local model training time, and construct the time calculation results;

[0035] S72. Obtain the number of local dataset samples for each user, the distribution of local datasets for each user, and the size of local models for each user, and construct the parameter collection results;

[0036] S73. Integrate the above time calculation results and parameter collection results to solve for the optimal parameter discard rate.

[0037] Furthermore, the specific expression for solving the optimal parameter discard rate is as follows:

[0038]

[0039]

[0040]

[0041] In the above formula, D max δ represents the maximum parameter sparsity rate, and A represents the penalty factor. server This indicates the proportion of parameters requested by the server. U represents the time required for the local model to be trained. n Indicates the size of the user's local model. This represents the discard rate of the nth user in round t. This represents the upload data rate between user n and the server. m represents the downlink data rate between the server and user n. n This represents the number of samples in each user's local dataset, where m represents the total number of samples and C represents the total number of categories. This represents the proportion of user n's c-th category data to user n's local dataset, and U represents the number of parameters in the global model. This represents the loss value generated by user n in the t-th round of training.

[0042] The beneficial effects of the method of this invention are as follows: This invention calculates the number of parameters allowed to be uploaded based on the parameter drop rate issued by the server. The server requires users to upload a certain number of parameters for model aggregation. The goal is to minimize the total training time while ensuring the generalization of the global model. Therefore, for global models that do not meet the preset convergence conditions, an optimization problem needs to be defined. This problem not only needs to reduce local training time and transmission time, i.e., reduce the impact of system heterogeneity, but also needs to consider the contributions of different users, i.e., reduce the impact of data heterogeneity and model heterogeneity. To achieve this goal, the optimization problem is expressed as minimizing the time and regularization term of a global round under the constraint of the number of parameters required for model aggregation. Compared with traditional federated learning methods, this invention uses a differentiated parameter drop method to exchange model parameters, which can enable more users to participate in global training under limited communication resources, while alleviating the fallback effect and ensuring that the global model converges at a faster speed. Attached Figure Description

[0043] Figure 1 This is a flowchart of the steps of an efficient federated learning method for communication based on differential discarding of model parameters according to the present invention.

[0044] Figure 2 This is a schematic diagram illustrating the changes in the user's local model during the communication-efficient federated learning process based on model parameter differential discarding, as described in this invention.

[0045] Figure 3 This is a schematic diagram of the steps in this invention to select the neuronal channel with the highest importance index;

[0046] Figure 4 This is a schematic diagram of the steps in the present invention to solve for the optimal parameter discard rate. Detailed Implementation

[0047] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0048] Reference Figure 1 and Figure 2 This invention provides an efficient federated learning method for communication based on model parameter differential discarding, the method comprising the following steps:

[0049] S10, Model parameter initialization;

[0050] Specifically, each user determines the model structure of their local model and initializes the model parameters. The number of local model parameters for user n is represented by U. nThe number of parameters in the global model is denoted by U, and the number of parameters in the local model is no greater than the number of parameters in the global model, i.e., U0. n ≤U;

[0051] Here, model structure refers to neural network models, including multilayer perceptrons, convolutional neural networks, and recurrent neural networks; model parameters refer to the weights and biases of connections in the neural network.

[0052] S20, Local Model Training;

[0053] Specifically, each user trains a local model using a local dataset, as shown in the following formula:

[0054]

[0055]

[0056] In the above formula, This represents the local model parameters of user n before training in round t. Let η represent the model parameters after training user n in round t, and let η represent the learning rate. f represents the local dataset of user n. j (W) represents the loss value of sample j in model W. This represents the gradient generated by user n during local training;

[0057] Local datasets refer to data stored by users on their local devices. This data is private and cannot be directly transmitted to other users or central servers.

[0058] S30, Upload parameter selection;

[0059] Specifically, each user's dropout rate is determined by parameters assigned by the server. Calculate the allowed number of parameters to be uploaded, and calculate the importance of each parameter of the model for global convergence. Select the most important parameters and form a matrix. Used to indicate the sparsification location of the model, the element takes the value 0 or 1. The 0 element in the code indicates that the parameter at the corresponding position is not uploaded. The 1 in the string indicates the parameter to be uploaded at the corresponding position; therefore, the allowed parameters can be listed using... express;

[0060] S301, reference Figure 3 The number of neurons or channels allowed to upload at each layer of the model is calculated as follows:

[0061]

[0062] In the above formula, N represents the number of neurons or channels allowed to upload data in the l-th layer of the model. l This represents the number of neurons or channels in the l-th layer of the model. This represents the discard rate of the nth user in round t.

[0063] S302. Calculate the importance index of each neuron or channel in the model, as shown in the following formula:

[0064]

[0065] In the above formula, This represents the importance index value of neuron channel k for user n. This represents the change in model parameters before and after user n's local training in round t.

[0066] S303. For each layer l of the model, select the layer with the highest importance index. One neuron or channel is used for uploading.

[0067] S40, Upload model parameters;

[0068] Specifically, each user will select the parameters in step S30. Uploaded to the server.

[0069] S50, global model aggregation;

[0070] Specifically, after receiving all the model parameters uploaded by users, the server performs an aggregation operation on the model and determines whether the convergence condition is met. If it is met, the global model is output; otherwise, step S60 is executed. The specific aggregation formula is shown below:

[0071]

[0072] In the above formula, m n W represents the number of samples in the local dataset for user n. t This represents the global model parameters after the aggregation in round t. This indicates the number of parameters that can be uploaded.

[0073] S60, parameter discard rate allocation;

[0074] Specifically, the server determines the proportion of parameters that each user is allowed to upload in the next round based on factors such as each user's communication environment, training speed, and data quality.

[0075] S601, Reference Figure 4 The model upload time for each user under a given discard rate is calculated as follows:

[0076]

[0077] In the above formula, This represents the model upload time for user n. This represents the upload data rate between user n and the server;

[0078] S602. Calculate the model delivery time for each user under a given discard rate parameter, as shown in the following formula:

[0079]

[0080] In the above formula, This indicates the time when the server sends the model to user n. This represents the downlink data rate between the server and user n;

[0081] S603. Calculate the local training time for each user, as shown in the following formula:

[0082]

[0083] In the above formula, c represents the time it takes for user n to train the local model. n b represents the CPU cycles required for user n to process one data sample. n f represents the batch size used by user n in a local training round. n This represents the CPU frequency of user n.

[0084] S604. Collect the number of samples mn in each user's local dataset, and the total number of samples.

[0085] S605. Collect the distribution of each user's local dataset, the specific expression of which is as follows:

[0086]

[0087] In the above formula, This represents the proportion of user n's c-th category data to user n's local dataset, where C represents the total number of categories;

[0088] It can be viewed as a whole, and the server does not need to know the specific distribution of each category, thereby reducing the leakage of privacy data.

[0089] S606. Collect the size of the local model for each user. The model size for user n is represented by U. n The number of parameters in the global model is represented by U;

[0090] S607. Solve for the optimal parameter sparsity rate. The specific optimization formula is shown below:

[0091]

[0092]

[0093]

[0094] In the above formula, D max δ represents the maximum parameter sparsity rate, and A represents the penalty factor. server This indicates the proportion of parameters requested by the server. U represents the time required for the local model to be trained. n Indicates the size of the user's local model. This represents the discard rate of the nth user in round t. This represents the upload data rate between user n and the server. m represents the downlink data rate between the server and user n. n This represents the number of samples in each user's local dataset, where m represents the total number of samples and C represents the total number of categories. This represents the proportion of user n's c-th category data to user n's local dataset, and U represents the number of parameters in the global model. This represents the loss value generated by user n in the t-th round of training.

[0095] S70, global model, and parameter discard rate distribution;

[0096] Specifically, the server will use the global model And the parameter sparsity in the next round Distribute to the relevant users;

[0097] S80, local model update;

[0098] Specifically, after receiving the global model from the server, the user updates the local model using the following formula, the expression of which is:

[0099]

[0100] In the above formula, This represents the local model of user n before the (t+1)th round of training. Represents the global model. This indicates the location of sparsification in the model.

[0101] In a specific embodiment of this invention, the server requires users to upload a certain number of parameters for model aggregation. The goal is to minimize the total training time while ensuring the generalization of the global model. Therefore, it is necessary not only to reduce local training time and transmission time (affected by system heterogeneity) but also to consider the contributions of different users (affected by data heterogeneity and model heterogeneity). To achieve this goal, the optimization problem is expressed as minimizing the time and regularization term of a global round, given the constraint of the number of parameters required for model aggregation. Compared with traditional federated learning methods, the method of this invention comprehensively considers the data heterogeneity, model heterogeneity, and system heterogeneity of different users, assigning differentiated parameter dropout rates to different users to achieve optimal overall performance. The above optimization problem is a convex optimization problem, which can be solved using various existing convex optimization solvers, resulting in low computational difficulty.

[0102] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A communication-efficient federated learning method based on differential discarding of model parameters, characterized in that, Includes the following steps: S1. Determine the structure of the local model and initialize the parameters of the local model. The structure of the local model is a neural network model, and the parameters of the local model are the weight values ​​and bias values ​​of the connections in the neural network model. S2. Train the initialized local model using a local dataset, where the local dataset represents data stored by the user on the local device and is private, and cannot be directly transmitted to other users or the central server. S3. Obtain the parameter discard rate issued by the server and calculate the number of parameters allowed to be uploaded based on the parameter discard rate; S4. Upload the allowed number of parameters to the server; S5. Aggregate the trained local model based on the uploaded parameters and output the global model; S6. Make a judgment on the global model; S7. If it is determined that the global model does not meet the preset convergence condition, the server will send the optimal parameter discard rate and global model according to the user's environmental factors, update the initialized local model, and repeat the above steps S2 to S6 based on the updated local model. S8. If the global model is determined to satisfy the preset convergence condition, the optimal global model is output. Step S7, which involves distributing the optimal parameter discard rate based on the user's environmental factors, specifically includes: S71. Calculate the local model upload time for each user under a given parameter drop rate, the local model distribution time for each user under a given parameter drop rate, and the local model training time, and construct the time calculation results; S72. Obtain the number of local dataset samples for each user, the distribution of local datasets for each user, and the size of local models for each user, and construct the parameter collection results; S73. Integrate the above time calculation results and parameter collection results to solve for the optimal parameter discard rate; The specific expression for solving the optimal parameter discard rate is as follows: In the above formula, This represents the maximum parameter sparsity rate. Indicates the penalty factor. This indicates the proportion of parameters requested by the server. This indicates the time required for the local model to be trained. Indicates the size of the user's local model. Indicates the first The user Wheel parameter discard rate, Indicates user The upload data rate between the server and the server Representing server and user Downlink data rate between This indicates the number of samples in each user's local dataset. Represents the total number of samples. Indicates the total number of categories. Indicates user The Data by category accounts for users The proportion of local datasets, This represents the number of parameters in the global model. Indicates user No. The loss value generated during training rounds.

2. The efficient federated learning method for communication based on differential discarding of model parameters as described in claim 1, characterized in that, In step S2, the expression for training the initialized local model is as follows: In the above formula, Indicates the first wheel users Local model parameters before training Indicates the first wheel users The trained model parameters Indicates the learning rate. On behalf of users Local dataset, Representative sample In the model The loss value in Indicates user Gradients generated during local training Indicates user The number of samples in the local dataset.

3. The efficient federated learning method for communication based on differential discarding of model parameters according to claim 2, characterized in that, Step S3 specifically also includes: S31. Obtain the parameter discard rate and calculate the importance of the parameters of the local model to the preset convergence condition; S32. Select the local model parameters with the highest importance to construct a matrix to represent the sparsification position of the model. An element of the matrix with a value of 1 indicates that the parameter at the corresponding position is uploaded, and an element with a value of 0 indicates that the parameter at the corresponding position is not uploaded. S33. Calculate the number of neuron channels allowed to be uploaded to each layer of the local model and the corresponding importance index; S34. Select the neuron channel with the highest importance index in each layer of the local model and upload the element with a value of 1 in the matrix.

4. The efficient federated learning method for communication based on differential discarding of model parameters according to claim 3, characterized in that, The formula for calculating the importance metric of the number of neuron channels allowed to be uploaded to each layer of the local model is shown below: In the above formula, Indicates user neuronal channels Importance index value, Indicates user In the The change in model parameters before and after local training.

5. The efficient federated learning method for communication based on differential discarding of model parameters according to claim 4, characterized in that, In step S5, the expression for the aggregation operation is as follows: In the above formula, Indicates the first The global model parameters after round aggregation, This indicates the number of parameters allowed to be uploaded. This indicates the location of sparsification in the model.

6. The efficient federated learning method for communication based on differential discarding of model parameters according to claim 5, characterized in that, In step S7, the expression for updating the initialized local model is as follows: In the above formula, Indicates user No. The local model before training rounds. This represents the global model.

Images