A wireless federated learning method based on 1-bit compressed sensing

Abstract

The invention discloses a wireless federated learning method based on 1-bit compressed sensing, including: a base station sends a global model to each user terminal; the user terminal provides data for training locally; the user terminal compares the local model with the sent global model; the user terminal records the amplitude and trend of the local model update in this round; the user terminal dynamically selects the sparsity and the threshold for sparsification according to the amplitude of the local model update; the user terminal sparsifies the trend of the local model update according to the threshold; the user terminal compresses by a 1-bit compressed sensing method; the base station reconstructs the observed signal by a BIHT algorithm and the received sparsity; the base station restores the local model of the user terminal by the received threshold and the reconstructed updated sparsity trend; the base station updates the global model; the base station sends a new global model to each user terminal for a new round of training until convergence is reached. The invention reduces the transmission energy consumption of the user terminal.

Description

Technical Field

[0001] The present invention relates to the technical field of wireless communication of mobile devices, and in particular to a wireless federated learning method based on 1-bit compressed sensing. Background Art

[0002] With the rapid development of artificial intelligence, deep learning neural networks have also achieved rapid development. At the same time, it requires the collection of a large amount of data information to obtain the best learning effect, but this also leads to various privacy leakage problems. Therefore, a distributed learning framework-federated learning was proposed to protect user privacy. Unlike traditional centralized learning methods, federated learning can store data information on the user side. It does not require the collection of user-side data, but only requires the collection of local model data after training on the user side. Since this method can protect the data security of the user side, it can use more user data for model training without leaking privacy. However, since it does not collect all the user's data at one time, but interacts with the user side model in each round of training, there is continuous data communication between the base station and the user side, which puts higher requirements on wireless transmission.

[0003] In federated learning training, the scale of the model is often very large, and the federated learning architecture requires the server and the user to continuously exchange model data information. If we require the user to upload all the data in the entire model to the server through wireless transmission in each round of training, it is very difficult. There are several reasons: 1. Under the premise of ensuring communication quality, the user side constantly sending a large amount of model data will generate a huge transmission energy consumption, and the users who provide data for federated learning are often mobile users with a huge base, among which users of portable mobile devices such as mobile phones account for a considerable proportion. For the battery life of such small portable devices, continuously sending a huge amount of model data information will cause a great energy burden on users; 2. From the perspective of learning efficiency, when there is a large number of data sets, the federated learning model also needs to be converged by a sufficient number of rounds and updates. Even if the user side can upload the model data accurately, if the communication delay cannot be guaranteed to be low, it will also affect the training of the federated learning model: 3. Since wireless transmission is mostly used for model upload, we have to consider the communication overhead problem. The smaller the total amount of communication data, the smaller the communication overhead. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a wireless federated learning method based on 1-bit compressed sensing, which reduces the amount of information uploaded by the user end in a reliable way, thereby reducing the communication overhead of the user end while ensuring the effectiveness of model training.

[0005] The present invention adopts the following technical solutions to solve the above technical problems:

[0006] A wireless federated learning method based on 1-bit compressed sensing proposed in the present invention includes the following steps:

[0007] Step 1: Initialize the number of iterations t=1. The base station initializes the global model to G(ω0) and sends G(ω0) to each user terminal.

[0008] Step 2: The base station uses the global model G(ω t-1 ) is sent to each user terminal, and each user terminal performs the t-th learning and training locally using the local data to obtain an updated local model;

[0009] Step 3: During the tth learning and training, each user terminal will update the local model G i (ω t ) is expressed as a one-dimensional column vector;

[0010] Step 4: During the tth learning training, each user terminal participating in the training compares G(ω t-1 ) and G i (ω t ), and obtain the update amplitude D of the local model of the i-th user terminal during the t-th learning training i (ω t ), the trend of the local model update of the i-th user side during the t-th learning training S i (ω t );

[0011] Step 5: During the tth learning training, the i-th user terminal dynamically selects the pair S i (ω t ) is the sparsity K of the sparse i，t And the threshold th i,t ;

[0012] Step 6: During the tth learning training, the i-th user terminal selects the threshold th i,t Trends in local model updates i (ω t ) to obtain the sparse trend of local model update

[0013] Step 7: During the t-th learning training, each user terminal selects the Gaussian random measurement matrix A as the sensing matrix and compresses it using the 1-bit compressed sensing method. Get the observed signal y i (ω t );

[0014] Step 8: During the tth learning training, each user terminal observes the signal y i (ω t ), sparsity K i,t and threshold th i,t Send to base station;

[0015] Step 9: During the tth learning training, the base station receives the observation signal y i (ω t ), select the sensing matrix A and sparsity K i,t , use the BIHT algorithm to observe the signal y i (ω t ) to reconstruct the signal and obtain the sparse trend of the reconstructed local model update The base station and the user end share the matrix A;

[0016] Step 10: During the t-th learning training, the base station updates the sparse trend according to the reconstructed local model G(ω t-1 ) and threshold th i,t Restore the local model after the t-th learning and training on the user side to obtain the restored local model G′ i (ω t );

[0017] Step 11: During the t-th learning training, the base station obtains the restored local model G′ of all user terminals i (ω t ) and then average these restored local models to obtain the global model G(ω t );

[0018] Step 12: The base station converts the new global model G(ω t ) is sent to the user end for a new round of learning and training, setting t=t+1, and returning to step 2 until the learning and training process reaches convergence.

[0019] As a further optimization scheme of the wireless federated learning method based on 1-bit compressed sensing described in the present invention, in step 2, G(ω t-1 ) is sent to each user terminal by broadcasting.

[0020] As a further optimization scheme of the wireless federated learning method based on 1-bit compressed sensing described in the present invention, step 4 is as follows: During the t-th learning training, each user terminal participating in the training compares G(ω t-1 ) and G i (ω t ) in each parameter, G i (ω t ) minus G(ω t-1) The absolute value of the difference is recorded as the amplitude D of the local model update of the i-th user terminal during the t-th learning training i (ω t ), the positive and negative signs of the difference are recorded as the trend S of the local model update of the i-th user terminal during the t-th learning training i (ω t ).

[0021] As a further optimization scheme of the wireless federated learning method based on 1-bit compressed sensing described in the present invention, the method of step 5 is as follows:

[0022] Setting parameter α t ∈(0.4,0.8), set the sparsification parameters to the first sparsification parameter p1∈(4%,6%) and the second sparsification parameter p2∈(9%,11%), respectively, and record D i (ω t ) contains N data, then let the first sparsity during the t-th learning training The second sparsity Record D i (ω t ) The value of the N data is the first The large value is The numerical value is The large value is like Setting the Threshold Sparsity Otherwise, set the threshold to Sparsity Where [·] indicates rounding.

[0023] As a further optimization scheme of the wireless federated learning method based on 1-bit compressed sensing described in the present invention, in step 6, the sparseness method is as follows:

[0024] D i (ω t ) and th i，t For comparison, if D i (ω t ) is greater than th i,t , then retain S i (ω t ), otherwise S i (ω t ) is set to 0.

[0025] As a further optimization scheme of the wireless federated learning method based on 1-bit compressed sensing described in the present invention, in step 10, the recovery method is as follows:

[0026] Read For every value in If the nth value in is 1, then G(ω t-1 ) plus the nth value in i，t ;if If the nth value in is -1, then G(ω t-1 ) minus the nth value in i,t ;if If the nth value in is 0, then G(ω t-1 ) to operate on the nth value in ).

[0027] As a further optimization scheme of the wireless federated learning method based on 1-bit compressed sensing described in the present invention, in step 4,

[0028] D i (ω t )=abs(G i (ω t )-G(ω t-1 ))

[0029] S i (ω t )=sign(G i (ω t )-G(ω t-1 ));

[0030] Where abs(·) means taking the absolute value, and sign(·) means taking the sign information.

[0031] As a further optimization scheme of the wireless federated learning method based on 1-bit compressed sensing described in the present invention, in step 7,

[0032]

[0033] As a further optimization scheme of the wireless federated learning method based on 1-bit compressed sensing described in the present invention, in step 9

[0034]

[0035] in, To use the BIHT algorithm, K i,t The observed signal is reconstructed for the input sparsity, where K i,t Represents the sparse trend of the local model update obtained by the i-th user after the sparse operation The sparsity of .

[0036] As a further optimization scheme of the wireless federated learning method based on 1-bit compressed sensing described in the present invention, in step 11,

[0037] G(ω t ) = FL_mean(G′ i (ω t ));

[0038] Among them, FL_mean(G′ i (ω t )) means for all G′ i (ω t ) takes the bitwise average.

[0039] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0040] The present invention reduces the amount of model data that needs to be uploaded by the user end by introducing a dynamic threshold for sparseness and a 1-bit compressed sensing method, converts the model data into a type that is easier to transmit, reduces the amount of model data that needs to be transmitted at the user end, reduces the transmission energy consumption and model training cost at the user end, and can well approximate the lossless transmission effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of the wireless federated learning method based on 1-bit compressed sensing of the present invention.

[0042] Figure 2 This is a schematic diagram of the relationship between the wireless federated learning base station and the user based on 1-bit compressed sensing of the present invention.

[0043] Figure 3 This is a simulation diagram of the training effect of the wireless federated learning method based on 1-bit compressed sensing of the present invention.

[0044] Figure 4 This is a communication overhead simulation diagram of the wireless federated learning method based on 1-bit compressed sensing of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] The present invention adopts the BIHT algorithm for signal reconstruction in 1-bit compressed sensing. Since the signal to be processed in this paper belongs to symbol data, the core steps of the BIHT algorithm for symbol data are introduced below:

[0047] In the following process: x t represents the x value of iteration t, A T represents the transposed matrix of matrix A, η k(v) means keeping the values ​​of the K elements with the largest magnitude in vector v unchanged and setting the other elements to 0, and sign(·) means taking the sign function, that is,

[0048]

[0049] Initialize x 0 =0 N×1 .

[0050] Input control parameter a in the algorithm to control the gradient descent step size, measurement matrix A∈R M×N , measurement vector y = sign(Ax)∈B M , signal sparsity K, maximum number of iterations nIter;

[0051] Loop the following steps nIter times;

[0052] to a t Iterate

[0053] a t =x t-1 +aA T (y-sign(Ax t-1 ));

[0054] x t Iterate

[0055] x t =η K (a t );

[0056] Output

[0057] To address the above issues, we proposed a solution based on the premise that the model transmission is digital signal transmission, that is, to optimize the source coding at the source.

[0058] like Figure 1 As shown, the wireless federated learning method based on 1-bit compressed sensing of the present invention includes the following steps:

[0059] (1) Initialization iteration number t = 1, the base station initializes the global model and sends it to each user end.

[0060] (2) The base station broadcasts the global model obtained from the t-1th learning and training to each user terminal. Each user terminal performs the tth learning and training locally using local data to obtain an updated local model.

[0061] (3) During the tth learning and training, each user terminal represents the updated local model in the form of a one-dimensional column vector;

[0062] (4) During the t-th learning and training, each user terminal participating in the training compares the global model and the local model, subtracts the global model from the local model, and records the absolute value of the difference as the magnitude of the local model update of the i-th user terminal during the t-th learning and training, and records the positive and negative signs of the difference as the trend of the local model update of the i-th user terminal during the t-th learning and training;

[0063] (5) During the t-th learning training, the i-th user terminal dynamically selects the sparsity and threshold for sparsely updating the trend of the local model;

[0064] (6) During the t-th learning training, the i-th user terminal performs sparseness on the trend of the local model update according to the selected threshold to obtain the sparse trend of the local model update;

[0065] (7) During the t-th learning training, each user terminal selects an appropriate Gaussian random measurement matrix as the sensing matrix, and compresses the sparse trend of the local model update through the 1-bit compressed sensing method to obtain the observation signal;

[0066] (8) During the tth learning training, each user terminal sends the observation signal, sparsity and threshold to the base station;

[0067] (9) During the t-th learning training, the base station selects the sensing matrix and sparsity according to the received observation signal, uses the BIHT algorithm to reconstruct the observation signal, and obtains the sparse trend of the reconstructed local model update; the sensing matrix is ​​shared by the base station and the user end;

[0068] (10) During the t-th learning and training, the base station restores the local model of the user end after the t-th learning and training according to the sparse trend of the reconstructed local model update, the global model obtained in the t-1th round, and the threshold to obtain the restored local model;

[0069] (11) During the t-th learning and training, after the base station obtains the restored local models of all user terminals, it averages these restored local models to obtain the global model of the t-th learning and training;

[0070] (12) The base station sends the new global model to the user end for a new round of learning and training. Let t = t + 1, and return to the above-mentioned step (2) in the specific implementation method until the learning and training process reaches convergence.

[0071] like Figure 2 As shown, in the wireless federated learning method based on 1-bit compressed sensing of the present invention, the relationship between the base station and each user terminal includes the following situations:

[0072] (1) The base station obtains the global model and sends it to each user terminal at the same time;

[0073] (2) In this round of training, the user end participating in the training trains a local model locally and transmits it back to the base station through 1-bit compressed sensing;

[0074] (3) The base station decodes and reconstructs the observed signal to obtain the restored local model, and updates the global model, ending this round of training.

[0075] (4) If the training reaches convergence, it stops; otherwise, a new round of training starts.

[0076] like Figure 3 As shown, in the wireless federated learning method based on 1-bit compressed sensing of the present invention, experiments are conducted on the handwriting data set MNIST, and the standard federated learning method and the wireless federated learning method based on 1-bit compressed sensing that can reconstruct lossless signals are compared. The training effect of the wireless federated learning method based on 1-bit compressed sensing of the present invention can well approximate the process of lossless signal reconstruction, and can also well approximate the standard federated learning method.

[0077] like Figure 4 As shown, in the wireless federated learning method based on 1-bit compressed sensing of the present invention, experiments were conducted on the handwriting data set MNIST. Compared with the standard federated learning method, when trained to the same level, the wireless federated learning method based on 1-bit compressed sensing of the present invention reduced the total size of data uploaded by the user end by about 5.5 times.

[0078] The above description is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a technician familiar with the technical field within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention.

Claims

1. A wireless federated learning method based on 1-bit compressed sensing, characterized in that: The following steps are involved: Step 1: Initialize the number of iterations t=1. The base station initializes the global model to G(ω0) and sends G(ω0) to each user terminal. Step 2: The base station uses the global model G(ω t-1 ) is sent to each user terminal, and each user terminal performs the t-th learning and training locally using the local data to obtain an updated local model; Step 3: During the tth learning and training, each user terminal will update the local model G i (ω t ) is expressed as a one-dimensional column vector; Step 4: During the tth learning training, each user terminal participating in the training compares G(ω t-1 ) and G i (ω t ), and obtain the update amplitude D of the local model of the i-th user terminal during the t-th learning training i (ω t ), the trend of the local model update of the i-th user side during the t-th learning training S i (ω t ); Step 5: During the tth learning training, the i-th user terminal dynamically selects the pair S i (ω t ) is the sparsity K of the sparse i,t And the threshold th i,t ; Step 6: During the tth learning training, the i-th user terminal selects the threshold th i,t Trends in local model updates i (ω t ) to obtain the sparse trend of local model update Step 7: During the t-th learning training, each user terminal selects the Gaussian random measurement matrix A as the sensing matrix and compresses it using the 1-bit compressed sensing method. Get the observed signal y i (ω t ); Step 8: During the tth learning training, each user terminal observes the signal y i (ω t ), sparsity K i,t and threshold th i,t Send to base station; Step 9: During the tth learning training, the base station receives the observation signal y i (ω t ), select the sensing matrix A and sparsity K i,t , use the BIHT algorithm to observe the signal y i (ω t ) to reconstruct the signal and obtain the sparse trend of the reconstructed local model update The base station and the user end share the matrix A; Step 10: During the t-th learning training, the base station updates the sparse trend according to the reconstructed local model G(ω t-1 ) and threshold th i,t Restore the local model after the user's t-th learning and training to obtain the restored local model G′ i (ω t ); Step 11: During the tth learning training, the base station obtains the restored local model G′ of all user terminals i (ω t ) and then average these restored local models to obtain the global model G(ω t ); Step 12: The base station converts the new global model G(ω t ) is sent to the user end for a new round of learning and training, setting t = t + 1, and returning to step 2 until the learning and training process reaches convergence; The method for step 5 is as follows: Setting parameter α t ∈(0.4,0.8), set the sparsification parameters to the first sparsification parameter p1∈(4%,6%) and the second sparsification parameter p2∈(9%,11%), respectively, and record D i (ω t ) contains N data, then let the first sparsity during the t-th learning training The second sparsity Record D i (ω t ) The value of the N data is the first The large value is The numerical value is The large value is like Setting the Threshold Sparsity Otherwise, set the threshold to Sparsity Where [·] indicates rounding; In step 10, the recovery method is as follows: Read For every value in If the nth value in is 1, then G(ω t-1 ) plus the nth value in i,t ;if If the nth value in is -1, then G(ω t-1 ) minus the nth value in i,t ;if If the nth value in is 0, then G(ω t-1 ) to operate on the nth value in ).

2. According to claim 1, a wireless federated learning method based on 1-bit compressed sensing is characterized in that: In step 2, G(ω t-1 ) is sent to each user terminal by broadcasting.

3. According to claim 1, a wireless federated learning method based on 1-bit compressed sensing is characterized in that: Step 4 is as follows: During the tth learning training, each user terminal participating in the training compares G(ω t-1 ) and G i (ω t ) in each parameter, G i (ω t ) minus G(ω t-1 ) The absolute value of the difference is recorded as the amplitude D of the local model update of the i-th user terminal during the t-th learning training i (ω t ), the positive and negative signs of the difference are recorded as the trend S of the local model update of the i-th user terminal during the t-th learning training i (ω t ).

4. The wireless federated learning method based on 1-bit compressed sensing according to claim 1, characterized in that: In step 6, the sparseness method is as follows: D i (ω t ) and th i,t For comparison, if D i (ω t ) is greater than th i,t , then retain S i (ω t ), otherwise S i (ω t ) is set to 0.

5. The wireless federated learning method based on 1-bit compressed sensing according to claim 1, characterized in that: In step 4, D i (oh t )=abs(G i (oh t )-G(ω t-1 )) S i (oh t )=sign(G i (oh t )-G(ω t-1 )); Where abs(·) means taking the absolute value, and sign(·) means taking the sign information.

6. The wireless federated learning method based on 1-bit compressed sensing according to claim 1, characterized in that: In step 7, 7. The wireless federated learning method based on 1-bit compressed sensing according to claim 1, characterized in that: In step 9, in, To use the BIHT algorithm, K i,t The observed signal is reconstructed for the input sparsity, where K i,t Represents the sparse trend of the local model update obtained by the i-th user after the sparse operation The sparsity of .

8. The wireless federated learning method based on 1-bit compressed sensing according to claim 1, characterized in that: In step 11, G(ω t )=FL_mean(G′ i (ω t )); Among them, FL_mean(G′ i (ω t )) means for all G′ i (ω t ) takes the bitwise average.

Images