A federated learning uplink compression and quantization joint design method and system

Abstract

The application discloses a kind of federal learning uplink compression quantization joint design method and system, the method includes: obtaining communication resource quantity and user to be uploaded parameter;Introduce the optimization problem with constraint, based on communication resource quantity condition constraint, according to the user to be uploaded parameter, the optimization problem is solved, and sparse budget and quantization bit number are obtained;According to sparse budget, the user to be uploaded parameter is sparsified;According to quantization bit number, the user to be uploaded parameter after sparsification is randomly and uniformly quantized and compressed;Again upload to server and decompress.The system includes: acquisition module, solving module, sparsification module, compression module and decompression module.Through using the application, by minimizing the compression uplink transmission parameter error, the compression error can be further reduced under the same communication overhead.The application is a kind of federal learning uplink compression quantization joint design method and system, and can be widely applied to federal learning uplink compression technical field.

Description

Technical Field

[0001] This invention relates to the field of federated learning uplink compression technology, and in particular to a federated learning uplink compression quantization co-design method and system. Background Technology

[0002] Artificial intelligence (AI) technology is rapidly developing, driven by data. Federated learning is a distributed learning framework where a central server broadcasts a global model. User-side models are then trained on local datasets, and the updated parameters are sent back to the server. The server then aggregates these parameters, repeating this process until convergence. A major challenge of federated learning is communication overhead. This overhead refers to the significant communication costs incurred from the large number of parameters uploaded and sent during the process. Since federated learning typically involves one server supporting multiple users, uplink communication resources are often scarce. It is necessary for servers to compress parameters using sparsity and quantization strategies, which can greatly reduce the communication overhead of the entire federated learning process. Among the existing algorithms for uplink compression in federated learning, some only use sparsity strategies, some only use quantization strategies, and some combine sparsity and quantization strategies. However, under the given communication resource constraints, there are many feasible ways to combine sparsity and quantization strategies. Instead of jointly considering the theoretical analysis of sparsity and quantization strategies to select the optimal combination method, they are only empirically selected from multiple feasible methods. There is no consideration of organically combining sparsity and quantization strategies for joint design of sparsity and quantization. Summary of the Invention

[0003] To address the aforementioned technical problems, the present invention aims to provide a federated learning uplink compression quantization joint design method and system, which can further reduce compression error under the same communication overhead by minimizing the compression uplink transmission parameter error.

[0004] The first technical solution adopted in this invention is: a federated learning uplink compression and quantization joint design method, comprising the following steps:

[0005] Obtain the quantity of communication resources and the parameters to be uploaded by the user;

[0006] A constrained optimization problem is introduced. Based on the constraint of the quantity of communication resources, the optimization problem is solved according to the parameters to be uploaded by the user, and the sparse budget and quantization bit number are obtained.

[0007] The user's upload parameters are sparsified according to the sparse budget to obtain the sparsified user upload parameters.

[0008] Based on the quantization bit depth, the sparsified user upload parameters are randomly and uniformly quantized to obtain the compressed user upload parameters.

[0009] The compressed user-uploaded parameters are uploaded to the server for decompression and federated learning aggregation.

[0010] Furthermore, the acquisition of the number of communication resources and the user's parameters to be uploaded specifically includes:

[0011] The number of communication resources refers to the communication resources that the user can use to upload parameters to the server in this round.

[0012] The parameter to be uploaded by the user is the model difference vector that the user is preparing to upload, wherein the model difference vector is the change in the model parameters of the user before and after local training in one round of federated learning.

[0013] Furthermore, the step of introducing a constrained optimization problem, based on the constraint of the quantity of communication resources, and solving the optimization problem according to the user's upload parameters to obtain the sparse budget and quantization bit depth, specifically includes:

[0014] Copy the parameters to be uploaded by the user, i.e., the model difference vector, to construct a vector copy;

[0015] Sort the vectors in the vector copy according to the descending order of the absolute values ​​of the elements to obtain the sorted vector;

[0016] By introducing a constrained optimization problem, the objective function corresponding to the constrained optimization problem is iteratively solved using sorted vectors to obtain different solution values.

[0017] Select the solution result corresponding to the objective function with the minimum solution value, and output the sparse budget and quantization bit depth.

[0018] Furthermore, the expression for the constrained optimization problem is as follows:

[0019]

[0020] stS(B+log2d)=C

[0021] B∈{2,3,…,32}

[0022] In the above formula, S represents the sparse budget, B represents the quantization bits, C represents the number of communication bits available to the user, d represents the length of the model difference vector, and g S Let represent the sparsified model difference vector, and Δ represent the difference between the maximum and minimum absolute values ​​of the S non-zero values ​​in the sparsified model difference vector.

[0023] Furthermore, the step of sparsifying the user's upload parameters according to the sparse budget to obtain the sparsified user upload parameters specifically includes:

[0024] The user's upload parameters are sparsified according to the sparse budget, that is, all elements except the S elements with the largest absolute values ​​in the model difference vector are set to zero, resulting in the sparsified user upload parameters.

[0025] Furthermore, the step of randomly and uniformly quantizing the sparsified user upload parameters according to the quantization bit depth to obtain the compressed user upload parameters specifically includes:

[0026] The number of quantization levels is determined based on the number of quantization bits.

[0027] Based on the number of quantization levels and the maximum and minimum values ​​of the non-zero elements in the sparsified user upload parameters, the non-zero elements in the sparsified user upload parameters are subjected to min-max normalization, so that the range of their absolute values ​​is normalized to be greater than or equal to zero and less than or equal to the number of quantization levels minus one, thus obtaining the normalized vector.

[0028] The normalized vector is then subjected to random rounding and denormalization processes to obtain the compressed user-uploaded parameters.

[0029] The second technical solution adopted in this invention is: a federated learning uplink compression and quantization joint design system, comprising:

[0030] The acquisition module is used to obtain the number of communication resources and the parameters to be uploaded by the user;

[0031] The solution module is used to introduce a constrained optimization problem. Based on the constraint of the quantity of communication resources, the optimization problem is solved according to the parameters to be uploaded by the user, and the sparse budget and quantization bit number are obtained.

[0032] The sparsification module is used to sparsify the user's upload parameters according to the sparsity budget, so as to obtain the sparsified user upload parameters.

[0033] The compression module is used to randomly and uniformly quantize the sparsified user upload parameters according to the quantization bit depth to obtain the compressed user upload parameters.

[0034] The decompression module is used to upload the compressed user-uploaded parameters to the server for decompression and to complete federated learning aggregation.

[0035] The beneficial effects of the method and system of this invention are as follows: By obtaining the difference vector between the number of communication resources and the model to be uploaded by the user, this invention further introduces and solves a constrained optimization problem. Given the number of communication resources, it can design the optimal sparse budget and quantization bit depth based on the characteristics of the parameters to be uploaded, and perform joint sparse quantization compression on the uploaded parameters, so that the compression error can be minimized. Thus, under the same communication overhead, the compression error can be further reduced, and the compression performance and training performance can be improved. Attached Figure Description

[0036] Figure 1 This is a flowchart of the steps of a federated learning uplink compression quantization joint design method according to the present invention;

[0037] Figure 2 This is a structural block diagram of a federated learning uplink compressed quantization joint design system according to the present invention;

[0038] Figure 3 This is a schematic diagram comparing the accuracy of a general uplink compression algorithm in federated learning and a joint design algorithm model using this invention in a specific embodiment of the present invention;

[0039] Figure 4 This is a schematic diagram comparing the mean square error of the general uplink compression algorithm in federated learning and the compressed mean square error using the joint design algorithm of this invention in a specific embodiment of the present invention. Detailed Implementation

[0040] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0041] This invention provides a sparse quantization joint design algorithm for uplink compression in federated learning. The following specific example illustrates the implementation of this disclosure, and simulation results are given to demonstrate the effectiveness of the invention. The relevant settings for the specific embodiment are as follows: the dataset uses the SVHN image digit recognition dataset; the model structure to be trained is MobileNet-v2, with a model length d = 2270846; the communication resource is C = 545000 bits; the optimizer uses SGD, with a learning rate set to 0.005, SGD momentum set to 0, and the training set batch size set to 64. Each user traverses the training set 3 times in one round of training; the total number of users is 100, and the SVHN dataset is randomly and evenly distributed among these 100 users; the number of users scheduled in each round is 10. Since the sparse quantization joint design algorithm provided by this invention is applied to the uplink compression of each user scheduled in each round of federated learning, this embodiment only takes the uplink compression of one user in one round as an example for illustration. The final effect diagram is a comparison of the model training of each user scheduled in each round using the sparse quantization joint design algorithm of this invention, as well as a comparison of the compression error.

[0042] Reference Figure 1 This invention provides a federated learning uplink compressed quantization joint design method, which includes the following steps:

[0043] S1. Obtain the number of communication resources C = 545000 and the parameters to be uploaded by the user;

[0044] Specifically, the number of communication resources refers to the communication resources that the user can use in this round, in bits; the parameters to be uploaded by the user are the model difference vector that the user is preparing to upload, specifically, the change in model parameters before and after local training in one round of federated learning, which is a one-dimensional vector with the same length as the model parameters.

[0045] S2. Introduce a constrained optimization problem. Based on the constraint of the quantity of communication resources, i.e. the number of communication bits available to the user, solve the optimization problem according to the parameters to be uploaded by the user to obtain the sparse budget and the number of quantization bits.

[0046] Specifically, the input communication resources and model difference vector are used to create a copy of the model difference vector. This copy is then sorted in descending order of the absolute values ​​of its elements to obtain a sorted vector. Using this sorted vector, the objective function corresponding to different quantization bits B and sparse budget S is iteratively calculated. The different values ​​of are then taken, and the minimum objective function value corresponding to the quantization bit depth B and sparse budget S is taken as the result. The sparse budget and quantization bit depth can be obtained by solving the following constrained optimization problem, the expression of which is:

[0047]

[0048] stS(B+log2d)=C

[0049] B∈{2,3,…,32}

[0050] In the above formula, S represents the sparse budget, B represents the quantization bits, C represents the number of communication bits available to the user, d represents the length of the model difference vector, and g S Let represent the sparsified model difference vector, and Δ represent the difference between the maximum and minimum absolute values ​​of the S non-zero values ​​in the sparsified model difference vector.

[0051] Its pseudocode is as follows:

[0052] Input: The sorted vector gsorted, with communication bits C = 545000;

[0053] Output: Optimal quantization bits B * Optimal sparse budget S * ;

[0054] (1) d ← length of gsorted;

[0055] (2)f * ←0;

[0056] (3)B * ←2;

[0057] (4)

[0058] (5) Loop B goes from 2 to 32;

[0059] (6)

[0060] (7)

[0061] (8)

[0062] (9) Condition if f <f * ;

[0063] (10)f * ←f;

[0064] (11)B * ←B;

[0065] (12)S * ←S;

[0066] (13) Termination condition statement;

[0067] (14) End the loop statement;

[0068] Assume that the optimal quantization bit depth and sparse budget obtained in this round of calculation are B = 8 bits and S = 18166, respectively.

[0069] S3. Sparsify the user's upload parameters according to the sparse budget to obtain the sparsified user upload parameters.

[0070] Specifically, based on the sparse budget S = 18166 calculated in the above steps, the model difference vector is sparsified. The sparse budget S is the number of non-zero elements retained in the model difference vector, that is, setting all elements except the S elements with the largest absolute values ​​in the model difference vector to zero, specifically setting all elements except the 18166 elements with the largest absolute values ​​in the model difference vector to zero, resulting in the sparsified model difference vector g. S .

[0071] S4. Randomly and uniformly quantize the sparsified user upload parameters according to the quantization bit depth to obtain the compressed user upload parameters.

[0072] Specifically, based on the quantization bit depth B = 8 bits calculated according to the above steps, the sparsified model difference vector g is... S The 18166 non-zero elements are randomly and uniformly quantized to obtain a compressed model difference vector after joint sparsification quantization. The pseudocode is as follows:

[0073] Input: Sparse budget S = 18166, quantization bits B = 8 bits, sparsified model difference vector g S ;

[0074] Output: Compressed model difference vector after joint sparsification quantization

[0075] (1)g←|g s |;

[0076] (2) gmax ← the maximum value of the non-zero elements in g;

[0077] (3) gmin ← the minimum value of the non-zero elements in g;

[0078] (4)

[0079] (5) Loop i: the position of all non-zero elements in g;

[0080] (6)

[0081] (7)

[0082] (8) r ← a random number uniformly distributed on [0,1];

[0083] (9) Conditions;

[0084] (10) If r <p:

[0085] (11) Otherwise:

[0086] (12) Termination condition statement;

[0087] (13) End the loop statement;

[0088] (14)g←(gmax-gmin)g * +gmin;

[0089] (15) / / Multiply corresponding elements;

[0090] The algorithm outputs a compressed model difference vector that has undergone joint sparsification quantization. This refers to the compressed model difference vector obtained by using S=18166 sparsification and B=8-bit random uniform quantization, under the condition that the communication resources are limited to C=545000 bits.

[0091] S5. Upload the compressed user-uploaded parameters to the server for decompression and complete federated learning aggregation.

[0092] The simulation experiment of this invention is shown below:

[0093] This example describes the sparse quantization joint design algorithm for uplink compression in federated learning provided by this invention. Under the communication resource constraint of C=545000, without using the joint design algorithm provided by this invention, only S=22708 sparsity and B=2-bit quantization are used, resulting in worse training performance and higher compression error in each round compared to using the joint design algorithm provided by this invention. Using the joint design algorithm provided by this invention, the lowest compression error can be calculated. Experimental results are shown in [reference needed]. Figure 3 and Figure 4 As shown;

[0094] It is important to note that when using only S=22708 sparsity and B=2-bit quantization without employing the joint design algorithm of this invention, S=22708 sparsity and B=2-bit quantization are used for every user in every round. However, when using the joint sparsity and quantization design algorithm of this invention, the algorithm calculates the optimal sparsity budget and quantization bit depth based on the input model difference vector. Therefore, the optimal sparsity budget and quantization bit depth may differ for each user in each round. The S=18166 sparsity and B=8-bit sparsity described in the above implementation is merely an example for one user in one round.

[0095] Reference Figure 2 A federated learning uplink compression and quantization joint design system, comprising:

[0096] The acquisition module is used to obtain the number of communication resources and the parameters to be uploaded by the user;

[0097] The solution module is used to introduce a constrained optimization problem. Based on the constraint of the quantity of communication resources, the optimization problem is solved according to the parameters to be uploaded by the user, and the sparse budget and quantization bit number are obtained.

[0098] The sparsification module is used to sparsify the user's upload parameters according to the sparsity budget, so as to obtain the sparsified user upload parameters.

[0099] The compression module is used to randomly and uniformly quantize the sparsified user upload parameters according to the quantization bit depth to obtain the compressed user upload parameters.

[0100] The decompression module is used to upload the compressed user-uploaded parameters to the server for decompression and to complete federated learning aggregation.

[0101] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0102] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A federated learning uplink compression and quantization joint design method, characterized in that, Includes the following steps: Obtain the quantity of communication resources and the parameters to be uploaded by the user; A constrained optimization problem is introduced. Based on the constraint of the quantity of communication resources, the optimization problem is solved according to the parameters to be uploaded by the user, and the sparse budget and quantization bit number are obtained. The user's upload parameters are sparsified according to the sparse budget to obtain the sparsified user upload parameters. Based on the quantization bit depth, the sparsified user upload parameters are randomly and uniformly quantized to obtain the compressed user upload parameters. The compressed user-uploaded parameters are uploaded to the server for decompression and federated learning aggregation. The specific expression for the constrained optimization problem is as follows: stS(B+log2d)=C B∈{2,3,…,32} In the above formula, S represents the sparse budget, B represents the quantization bits, C represents the number of communication bits available to the user, d represents the length of the model difference vector, and g S Let represent the sparsified model difference vector, and Δ represent the difference between the maximum and minimum absolute values ​​of the Q non-zero values ​​in the sparsified model difference vector.

2. The federated learning uplink compression and quantization joint design method according to claim 1, characterized in that, The acquisition of the number of communication resources and the parameters to be uploaded by the user specifically includes: The number of communication resources refers to the communication resources that the user can use to upload parameters to the server in this round. The parameter to be uploaded by the user is the model difference vector that the user is preparing to upload, wherein the model difference vector is the change in the model parameters of the user before and after local training in one round of federated learning.

3. The federated learning uplink compression and quantization joint design method according to claim 2, characterized in that, The introduced constrained optimization problem, based on the constraint of communication resource quantity, solves the optimization problem according to the user's upload parameters to obtain the sparse budget and quantization bit depth. This step specifically includes: Copy the parameters to be uploaded by the user, i.e., the model difference vector, to construct a vector copy; Sort the vectors in the vector copy according to the descending order of the absolute values ​​of the elements to obtain the sorted vector; By introducing a constrained optimization problem, the objective function corresponding to the constrained optimization problem is iteratively solved using sorted vectors to obtain different solution values. Select the solution result corresponding to the objective function with the minimum solution value, and output the sparse budget and quantization bit depth.

4. The federated learning uplink compression and quantization joint design method according to claim 3, characterized in that, The step of sparsifying the user's upload parameters according to the sparse budget to obtain the sparsified user upload parameters specifically includes: The user's upload parameters are sparsified according to the sparse budget, that is, all elements except the S elements with the largest absolute values ​​in the model difference vector are set to zero, resulting in the sparsified user upload parameters.

5. The federated learning uplink compression and quantization joint design method according to claim 4, characterized in that, The step of randomly and uniformly quantizing the sparsified user upload parameters according to the quantization bit depth to obtain the compressed user upload parameters specifically includes: The number of quantization levels is determined based on the number of quantization bits. Based on the number of quantization levels and the maximum and minimum values ​​of the non-zero elements in the sparsified user upload parameters, the non-zero elements in the sparsified user upload parameters are subjected to min-max normalization, so that the range of their absolute values ​​is normalized to be greater than or equal to zero and less than or equal to the number of quantization levels minus one, thus obtaining the normalized vector. The normalized vector is then subjected to random rounding and denormalization processes to obtain the compressed user-uploaded parameters.

6. A federated learning uplink compression and quantization joint design system, characterized in that, A method for executing a federated learning uplink compressed quantization joint design method as described in claim 1 includes the following modules: The acquisition module is used to obtain the number of communication resources and the parameters to be uploaded by the user; The solution module is used to introduce a constrained optimization problem. Based on the constraint of the quantity of communication resources, the optimization problem is solved according to the parameters to be uploaded by the user, and the sparse budget and quantization bit number are obtained. The sparsification module is used to sparsify the user's upload parameters according to the sparsity budget, so as to obtain the sparsified user upload parameters. The compression module is used to randomly and uniformly quantize the sparsified user upload parameters according to the quantization bit depth to obtain the compressed user upload parameters. The decompression module is used to upload the compressed user-uploaded parameters to the server for decompression and to complete federated learning aggregation.

Images