Federal learning communication method based on entropy driving

Abstract

The invention provides a federated learning communication method based on entropy driving. The method comprises the following implementation steps: initializing a federated learning communication scene; each client performs iterative training on the local model; each client compresses the weight parameter of the local model based on entropy and calculates compensation information; the central server performs information compensation on the compressed parameters; and the central server obtains a federated learning communication result. According to the method, each client performs singular value decomposition on the weight parameter of the local model and compresses the weight parameter based on the matrix after entropy truncation decomposition, so that the compression rate limitation caused by a fixed truncation threshold is avoided, the expenditure of federal learning communication is reduced, and the communication efficiency is improved. And through the compensation information calculated through the difference matrix of the local model parameters and the compression parameters of the client, an area with relatively large information loss after compression can be corrected, the defect that potential information loss is neglected due to excessive attention to reduction of the transmission parameter quantity is avoided, and the model still has relatively high precision after parameter compression is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning and relates to an entropy-driven federated learning communication method, which can be applied to edge computing, smart Internet of Things, autonomous driving and other fields. Background Art

[0002] Federated learning communication is the process of collaborative training between the central server and the client through the transmission of model parameters during the federated learning process. Specifically, the central server collaborates with multiple clients in a distributed manner. The clients train the model using local data and upload the updated model parameters to the central server. The server integrates the parameter updates of all clients through an aggregation algorithm, generates a global model and sends it to each client until the maximum communication round is reached. Through this distributed communication mechanism, federated learning can use decentralized data to collaboratively train high-performance global models while protecting data privacy.

[0003] During the federated learning communication process, the client needs to periodically upload local model parameters to the central server to achieve global model aggregation. However, with the exponential growth of the number of layers in deep neural network models, the scale of parameters required to be transmitted in a single communication has exceeded the order of millions, resulting in severe transmission congestion on edge devices in bandwidth-constrained, high-latency communication environments. Especially in mobile deployment scenarios, due to the competition for hardware resources between the device computing unit and the communication module, large-scale parameter transmission will not only cause peak occupancy of network bandwidth, but also cause timing conflicts between local training and parameter upload. However, the existing federated learning communication methods have a large amount of data when transmitting parameters between the client and the central server, resulting in high communication costs. Therefore, how to effectively reduce the communication cost in federated learning has become one of the urgent problems to be solved in the current field of federated learning.

[0004] In order to reduce the communication overhead problem in federated learning, for example, Yue Niu et al. published a paper titled "Federated Learning of LargeModels at the Edge via Principal Sub-Model Training" in the journal Transactions on Machine Learning Research in 2022, proposing a federated learning communication method PriSM based on principal-sub-models. This method first initializes the global model through a central server, then iteratively trains the client local model, and then performs singular value decomposition on the client local model parameters. The importance sampling strategy is used to generate each client sub-model by truncating the singular value size with a fixed threshold, thereby achieving compression of the client model parameters, and uploading the sub-model parameters to the central server for aggregation until the maximum communication round is reached and the training is ended. This method effectively reduces the amount of parameter transmission from the client to the central server. However, due to the use of a fixed truncation threshold, the generated sub-model lacks flexibility and is difficult to adapt to the information distribution of different convolutional layers, which may result in limited compression rate and still large parameter transmission. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and propose an entropy-driven federated learning communication method to solve the technical problem of large transmission volume caused by limited parameter compression rate in the prior art.

[0006] To achieve the above object, the technical solution adopted by the present invention includes the following steps:

[0007] (1) Initialize the federated learning communication scenario:

[0008] Initialize a federated learning communication scenario including a central server and K client devices, train the global model constructed by the central server for t iterations, with a maximum number of iterations of T, and set t = 0; where K ≥ 2;

[0009] (2) Each client iteratively trains the local model:

[0010] Each client uses the global model sent by the central server as the local model and iteratively trains it to obtain the weight parameters of each client's local model in this iteration.

[0011] (3) Each client compresses the weight parameters of the local model based on entropy and calculates compensation information:

[0012] Each client calculates the local model parameters based on entropy Compress and pass the compressed parameters Calculate compensation information Will and Send to the central server;

[0013] (4) The central server performs information compensation on the compressed parameters:

[0014] The central server passes compensation information Compressed parameters for each client Perform information compensation to obtain compensated parameters

[0015] (5) The central server obtains the federated learning communication results:

[0016] The central server aggregates the parameters of the K clients after compensation and obtains the aggregation result W of the tth iteration. t * , and determine whether t = T is true. If so, get the trained global model parameter W t * , otherwise, let t=t+1 and execute step (2).

[0017] Compared with the prior art, the present invention has the following advantages:

[0018] 1. Each client of the present invention performs singular value decomposition on the weight parameters of the local model and truncates the decomposed matrix based on entropy to achieve compression of the weight parameters. It can accurately identify and retain parameters that contribute greatly to model performance, while eliminating redundant or low-contribution parameters, avoiding the defect of limited compression rate caused by the use of fixed truncation threshold compression in the prior art, reducing the transmission amount of parameters, and thereby reducing the overhead of federated learning communication.

[0019] 2. The present invention performs information compensation on the compressed weight parameters through the compensation information calculated by the difference matrix between the client local model parameters and the compression parameters. It can identify the areas with large information loss in the compression process and realize the correction of information loss in the compression process. It avoids the defect of the prior art that over-focuses on reducing the amount of transmission parameters while ignoring the limitations of potential information loss, and ensures that the model still has high accuracy after parameter compression. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart for implementing the present invention. DETAILED DESCRIPTION

[0021] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0022] Reference Figure 1 , the present invention comprises the following steps:

[0023] Step 1) Initialize the federated learning communication scenario:

[0024] Initialize a federated learning communication scenario including a central server and K client devices, train the global model constructed by the central server for t iterations, with a maximum number of iterations of T, and set t = 0; where K ≥ 2;

[0025] In this embodiment, K=10, T=100, and the global model includes a convolutional neural network structure of a feature extraction module and a classification module; the feature extraction module includes 8 cascaded residual blocks, each residual block consists of 2 convolutional layers, 1 normalization layer and 1 activation function layer, and the input of the residual block is jump-connected to the activation function layer, and the total number of convolutional layers included in all residual blocks is L=16; the classification module includes 1 global average pooling layer and 1 fully connected layer.

[0026] Step 2) Each client iteratively trains the local model:

[0027] Each client uses the global model sent by the central server as the local model and iteratively trains it to obtain the weight parameters of each client's local model in this iteration. The implementation steps are:

[0028] (2a) Initialize the number of local iterations to u, the maximum number of local iterations to U = 20, U ≥ 2, and the dataset used for client iterative training is CIFAR-10, which contains 60,000 32 × 32 pixel RGB color images. The number of target categories in the dataset is C = 10, covering ten categories such as airplanes, cars, and birds. Each category contains 5,000 training images and 1,000 test images. The dataset will be evenly divided to each client, and each client receives 5,000 training samples and 1,000 test samples as local data X. k , and the images owned by each client are not repeated, where the training sample set of the kth client includes C = 10 target categories of X k images and their class labels y for each image x x,c , the weight parameter of the u-th local iteration is C≥2,X k ≥2000, and let u=0;

[0029] (2b) The x-th image is used as the input of the feature extraction module for feature extraction. The feature extraction module has 8 residual modules. The first two residual blocks output feature maps of 64 channels, the next two residual blocks output feature maps of 128 channels, the next two residual blocks output feature maps of 256 channels, and the last two residual blocks output feature maps of 512 channels. Each residual block in the feature extraction module has two convolution layers that perform convolution operations on the input feature map to extract features. The normalization layer and the activation function layer perform nonlinear transformation on the features. The input of the residual block is added to the features after nonlinear transformation to obtain the feature map of the x-th training sample. The feature map of the extracted x-th training sample is classified. The global average pooling layer of the classification module is first used for downsampling to reduce the spatial dimension of the final feature map to 1×1, generating a column vector of dimension 512, and then inputting it into the fully connected layer to output a one-dimensional vector of length 10 to obtain the predicted probability of each category. The one with the largest probability is the final classification result of the x-th image.

[0030] (2c) Use the cross entropy loss function and use the classification results of each image Calculate the loss value of the local model Then, stochastic gradient descent is used, For local model parameters Update to get the local model parameters of this local iteration Loss value and the local model parameters of this local iteration The calculation formulas are:

[0031]

[0032] Among them, η = 0.01 is the learning rate, is the gradient operator, is the loss function L k About local parameters gradient.

[0033] (2d) Determine whether u=U. If so, obtain the local model of the k-th client for the t-th global training. Otherwise, let u=u+1 and execute step (2b).

[0034] Step 3) Each client compresses the weight parameters of the local model based on entropy and calculates compensation information:

[0035] Each client calculates the local model parameters based on entropy Compress and pass the compressed parameters Calculate compensation information Will and Sent to the central server, where each client calculates the local model parameters based on entropy Compression is performed, including the steps of singular value decomposition and matrix truncation, specifically:

[0036] (3a) Each client has local model parameters The weight of the lth convolutional layer in Perform singular value decomposition to obtain the left singular value matrix A diagonal matrix consisting of R singular values and the right singular value matrix

[0037]

[0038] Among them, svd(·) is the singular value decomposition;

[0039] Singular value decomposition weights Decompose into left singular value matrix Diagonal Matrix and the right singular value matrix in and are orthogonal matrices, representing the orthogonal basis of the input feature space and the output feature space, respectively. and Each column vector of defines a direction in the feature space, and the diagonal matrix Contains singular values, which reflect the importance of weights in different directions in the feature space; Decompose into and Can better analyze The information distribution and structural characteristics in the image can be used to identify the main feature directions and redundant information based on the structural characteristics;

[0040] (3b) Through each client local model parameters The weight of the lth convolutional layer in Calculate the kernel entropy of the fth filter and kernel sparsity and through and Calculate the average information richness of the lth layer

[0041]

[0042] Among them, ∑ is the accumulation operation, avg(·) is the average operation, and |·| is the absolute value operation; is the client local model parameter The information richness of the f-th filter of the l-th layer; dm(w l,f ) is the density index of each filter, which measures the information density near the fth filter in the lth layer; the nearest neighbor matrix d l is a matrix that measures the correlation between every two filters in the lth layer; d l,f is the nearest neighbor matrix constructed for each filter in the lth convolutional layer; ω is used to weigh the kernel entropy and kernel sparsity Information Richness The relative contribution hyperparameter, in this embodiment, ω = 1;

[0043] Among them, nuclear entropy Weighed The information distribution diversity of the fth filter in the lth convolutional layer in its neighborhood. The smaller the kernel entropy value, the less information there is in the neighborhood of the filter. The larger the kernel entropy value, the more information there is in the neighborhood, which contributes more to the model performance. Reflects The redundancy of the lth convolutional layer parameters, The larger the value, the more redundant the parameters are. The smaller the value, the less redundant the parameter is and the higher the validity of the parameter is. and kernel sparsity Respectively reflects the characteristics of information in the filter neighborhood from the perspective of information volume and redundancy, and the average information richness Combination and And through the balance between the amount of information and redundancy of the hyperparameter ω, the importance of the parameters in the convolutional layer can be more comprehensively measured from the amount of information and redundancy, providing theoretical support for the parameter compression of the model;

[0044] (3c) Through the diagonal matrix The rth singular value a r Calculate the number of singular values ​​retained by the lth convolutional layer And through R t k ,l For the left singular value matrix Truncate to keep only the first Left singular value matrix of columns Diagonal Matrix Truncate to keep only the first Before departure Diagonal matrix of columns Right singular value matrix Truncate to keep only the first Right singular value matrix of rows

[0045] Realize Compression, get the compressed parameters in The calculation formula is:

[0046]

[0047] in, and Respectively and The truncation result, Γ l is the local model parameter The effective information measure of the lth convolutional layer;

[0048] Γ l The measure that can reflect the effective information in the lth convolutional layer is because it is obtained through the diagonal matrix The sum of all singular values ​​in and the average information richness The obtained singular values ​​reflect the importance of the weight matrix in different directions, so it can dynamically measure the effective information in the convolution layer; the number of singular values ​​is retained is through the l Subtract the diagonal matrices one by one Each singular value of until Γ l <=0. The reason for this is that the singular values ​​of the diagonal matrix are distributed from large to small. By subtracting each singular value in turn, the larger singular values ​​will be counted first. If several larger singular values ​​can basically represent the main information of the convolutional layer, then the remaining singular values ​​can be truncated; this process ensures the dynamic adjustment of the truncation threshold so that it can adapt to the information distribution of different convolutional layers. This method can significantly reduce the number of parameters.

[0049] In this embodiment, the nearest neighbor matrix d l It is based on the client's local training parameters The weight of the first layer Constructed, the weight of the fth filter in this convolutional layer is When the other filters f + When it is located in the κ nearest neighbor interval of the fth filter of the convolutional layer, let otherwise, Get the nearest neighbor matrix d constructed by the fth filter l,f ,||·|| means finding the Euclidean distance;

[0050] In this embodiment, the compensation information is calculated The formula is:

[0051]

[0052] Among them, norm1 (·) indicates the calculation of l 1 Norm; t k Represents the client local model parameters After compression with the client The difference matrix of Er; β is a hyperparameter that controls the ratio of compensation parameters; Sort(·) is a t k The elements in the array are sorted in descending order; Retain(·) is based on the ratio of the hyperparameter β, from the sorted Er t k The previous large parameter values ​​are retained and recorded in the compensation information In this embodiment, β=0.1.

[0053] Since the compressed parameters Parameters before compression There are differences between them. When some of the parameters have large differences, the accuracy of the model will be significantly reduced. It is to record these locations and parameter values ​​with large differences, and to The parameters with larger errors are corrected to reduce the accuracy loss caused by parameter compression.

[0054] Step 4) The central server performs information compensation on the compressed parameters:

[0055] The central server passes compensation information Compressed parameters for each client Perform information compensation to obtain compensated parameters The implementation method is:

[0056] The central server passes compensation information Non-zero parameters The compressed parameters The corresponding position in Replace to achieve information compensation, and get Compensation results in The compensation result is

[0057] Step 5) The central server obtains the federated learning communication results:

[0058] Parameters after the central server compensates K clients Perform federated aggregation to obtain the aggregation result W of the tth iteration t * , and determine whether t = T is true. If so, get the trained global model parameter Wt * , otherwise, let t=t+1 and execute step (2).

[0059] In this embodiment, the parameters after compensation for K clients are Perform federated aggregation, the aggregation formula is:

[0060]

[0061] The following is a further description of the technical effects of the present invention in conjunction with the simulation results:

[0062] 1. Simulation conditions and contents:

[0063] The hardware platform of the simulation experiment is: a Xeon(R) Platinum 8255C CPU with a main frequency of 2.50GHz*96, a 256GB memory, and an NVIDIAA100 SXM4 80GB graphics processor. The software platform is: an Ubuntu 18.04.5 operating system, python 3.8.16, and pytorch 1.12.1.

[0064] The parameter transmission amount and classification accuracy of the present invention and the prior art are simulated and compared, and the results are shown in Table 1.

[0065] The parameter transmission volume is defined as the total amount of parameters uploaded by all clients to the central server; the classification accuracy uses the overall accuracy OA as the evaluation indicator, and its calculation formula is as follows:

[0066]

[0067] Among them, TP represents the number of true positive samples, TN represents the number of true negative samples, FP represents the number of false positive samples, and FN represents the number of false negative samples. By calculating the OA value, the performance of the optimized federated learning algorithm on correctly classified and misclassified samples can be comprehensively evaluated.

[0068] 2. Analysis of simulation results:

[0069] Table 1

[0070] Method Type Parameter transfer amount Classification accuracy OA Prior art 6.696MB 81.32％ The present invention 5.864MB 82.19％

[0071] As can be seen from Table 1, compared with the prior art, the parameter transmission amount of the present invention is reduced by 0.832MB, and the classification accuracy is improved by 0.87%.

Claims

1. A federated learning communication method based on entropy driving, characterized in that: The steps include: (1) Initialize the federated learning communication scenario: Initialize a federated learning communication scenario including a central server and K client devices, train the global model constructed by the central server for t iterations, with a maximum number of iterations of T, and set t = 0; where K ≥ 2; (2) Each client iteratively trains the local model: Each client uses the global model sent by the central server as the local model and iteratively trains it to obtain the weight parameters of each client's local model in this iteration. (3) Each client compresses the weight parameters of the local model based on entropy and calculates compensation information: Each client calculates the local model parameters based on entropy Compress and pass the compressed parameters Calculate compensation information Will and Send to the central server; (4) The central server performs information compensation on the compressed parameters: The central server passes compensation information Compressed parameters for each client Perform information compensation to obtain compensated parameters (5) The central server obtains the federated learning communication results: The central server aggregates the parameters of the K clients after compensation and obtains the aggregation result W of the tth iteration. t * , and determine whether t = T is true. If so, get the trained global model parameter W t * , otherwise, let t=t+1 and execute step (2).

2. The method according to claim 1, characterized in that The global model described in step (1) adopts a convolutional neural network model including a cascaded feature extraction module and a classification module; the feature extraction module includes a plurality of cascaded residual blocks, each residual block includes at least two stacked convolutional layers, a normalization layer and an activation function layer, and the input of the residual block is jump-connected with the activation function layer, the total number of convolutional layers included in all residual blocks is L, each convolutional layer includes F filters, F ≥ 3; the classification module includes a stacked global average pooling layer and a fully connected layer.

3. The method according to claim 2, characterized in that Each client in step (2) iteratively trains the local model, and the implementation steps are as follows: (2a) Initialize the number of local iterations to u, the maximum number of local iterations to U, U ≥ 2, and the training sample set of the kth client includes X of C target categories. k images and their class labels y for each image x x,c , the weight parameter of the u-th local iteration is C≥2,X k ≥2000, and let u=0; (2b) The feature extraction module extracts features from the xth image to obtain a feature map of the xth image; the classification module classifies the extracted xth feature map to obtain a classification result of the xth feature map. (2c) Use the cross entropy loss function and use the classification results of each image Calculate the loss value of the local model Then, stochastic gradient descent is used, For local model parameters Update to get the local model parameters of this local iteration (2d) Determine whether u=U. If so, obtain the local model of the k-th client for the t-th global training. Otherwise, let u=u+1 and execute step (2b).

4. The method according to claim 3, characterized in that: The loss value described in step (2c) and the local model parameters of this local iteration The calculation formulas are: Where η is the learning rate, is the gradient operator, is the loss function L k About local parameters gradient.

5. The method according to claim 1, characterized in that Each client in step (3) calculates the local model parameters based on entropy Compression is performed, including the steps of singular value decomposition and matrix truncation, specifically: (3a) Each client has local model parameters The weight of the lth convolutional layer in Perform singular value decomposition to obtain the left singular value matrix A diagonal matrix consisting of R singular values and the right singular value matrix Among them, svd(·) is the singular value decomposition; (3b) Through each client local model parameters The weight of the lth convolutional layer in Calculate the kernel entropy of the fth filter and kernel sparsity and through and Calculate the average information richness of the lth layer Among them, ∑ is the accumulation operation, avg(·) is the average operation, and |·| is the absolute value operation; is the client local model parameter The information richness of the f-th filter of the l-th layer; dm(w l,f ) is the density index of each filter, which measures the information density near the fth filter in the lth layer; the nearest neighbor matrix d l is a matrix that measures the correlation between every two filters in the lth layer; d l,f is the nearest neighbor matrix constructed for each filter in the lth convolutional layer; ω is used to weigh the kernel entropy and kernel sparsity Information Richness The relative contribution of hyperparameters; (3c) Through the diagonal matrix The rth singular value a r Calculate the number of singular values ​​retained by the lth convolutional layer and through For the left singular value matrix Diagonal Matrix and the right singular value matrix Perform matrix truncation respectively to achieve Compression, get the compressed parameters in The calculation formula is: in, and Respectively and The truncation result, Γ l is the local model parameter The effective information measure of the l-th convolutional layer.

6. The method according to claim 5, characterized in that The left singular value matrix described in step (3c) Diagonal Matrix and the right singular value matrix Matrix truncation is performed separately, and the truncation method is: pass Will Truncate to keep only the first Left singular value matrix of columns Will Truncate to keep only the first Before departure Diagonal matrix of columns Will Truncate to keep only the first Right singular value matrix of rows 7. The method according to claim 5, characterized in that Compensation information as described in step (3) The calculation formula is: Among them, Retain(·) indicates the operation of retaining parameters, Sort(Er t k ) represents the parameters of the client local model With the compressed parameters The difference matrix The descending operation is performed, β represents the hyperparameter that controls the ratio of the compensation parameter, and norm1(·) represents the 1-norm operation.

8. The method according to claim 1, characterized in that The compressed parameters for each client described in step (4) Information compensation is carried out by: The central server passes compensation information Non-zero parameters The compressed parameters The corresponding position in Replace to achieve information compensation, and get Compensation results in The compensation result is 9. The method according to claim 1, characterized in that: The parameters after compensation for K clients described in step (5) Perform federated aggregation, the aggregation formula is:

Images