A two-way adaptive gradient compression method and system for federated learning

Abstract

The application provides a kind of two-way adaptive gradient compression method and system for federated learning.The method comprises: server adaptively determines the quantization bit width and compression rate of this round of training, and transmits gradient or model parameter to each client;Client calculates the gradient of this round according to the relevant parameters and compresses, predicts the convergence speed of the next round of training, uploads the compressed gradient and convergence speed prediction value to the server;The server decompresses, aggregates and recompresses the compressed gradient of each client, calculates the global convergence speed prediction value according to the convergence speed prediction value uploaded by each client, updates the global model and sends the queue.The method of the application improves the communication efficiency while ensuring the model accuracy by adaptive gradient compression rate and quantization bit width;Design the encoding method in sparsification and quantization and the novel two-way gradient compression strategy, reduce the communication overhead, increase the number of clients that can participate in each round, and alleviate the bandwidth bottleneck in large-scale federated learning.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and in particular to a bidirectional adaptive gradient compression method and system for federated learning. Background Technology

[0002] With the rapid development of deep learning, people are paying increasing attention to data privacy. Federated learning, as a new distributed learning paradigm, enables multiple parties to build a joint model without exposing their private training data. However, in the training of large-scale federated learning systems, the transmission of model parameters and gradients not only leads to long communication times and low transmission efficiency, but also limits the number of clients participating in federated learning. To overcome these problems, a common method is to compress the transmitted gradients.

[0003] Gradient compression is divided into gradient quantization and gradient sparsity. Gradient quantization refers to reducing the number of bits in the gradient data to transmit gradient data more quickly. Common gradient quantization methods include 1-bit SGD, Q-SGD, ECQ-SGD, and Sign-SGD. However, this strategy reduces the precision of the gradient data, thereby reducing training accuracy. Gradient sparsity refers to selectively transmitting a portion of the gradient during gradient transmission to reduce communication time. Common gradient sparsity methods include TopK, DGC, RedSync, and Gaussian KSGD. However, this strategy may result in important gradients not being transmitted, reducing model accuracy. This strategy also requires additional transmission of gradient location information, reducing the data compression ratio.

[0004] Furthermore, current gradient sparsity techniques are only used for compressing uplink data, which cannot effectively alleviate bandwidth constraints in large-scale federated learning. Summary of the Invention

[0005] Embodiments of the present invention provide a bidirectional adaptive gradient compression method and system for federated learning, so as to effectively increase the number of clients supported by the federated learning model in a network bandwidth-constrained environment.

[0006] To achieve the above objectives, the present invention adopts the following technical solution.

[0007] According to one aspect of the present invention, a bidirectional adaptive gradient compression method for federated learning is provided, comprising:

[0008] The server adaptively determines the quantization bit width and compression ratio for this round of training, selects the clients participating in this round of training, and transmits gradients or model parameters to each client.

[0009] The client updates the local model based on the relevant parameters sent by the server; calculates and compresses the gradient for the current round based on the local dataset; predicts the convergence speed of the next round of training based on the training situation; and uploads the compressed gradient and the predicted convergence speed to the server.

[0010] The server decompresses, aggregates, and recompresses the compression gradients uploaded by each client, calculates the global convergence rate prediction based on the convergence rate predictions uploaded by each client, and updates the global model.

[0011] The server updates the sending queue, and this round ends.

[0012] Preferably, the server adaptively determines the quantization bit width and compression ratio for this round of training, selects the clients participating in this round of training, and transmits gradients or model parameters to each client, including:

[0013] During federated learning, the server adaptively determines the quantization bit width and compression ratio for this round of training, and obtains the predicted convergence speed b for the (t+1)th round. t q t+1 Let q represent the quantization bit width of the gradient in the (t+1)th round. t Let q represent the quantization bit width of the gradient in the t-th round, σ represent the settable hyperparameter, and q represent the quantization bit width of the gradient in the t-th round. max This represents the maximum quantization bit width of the gradient; when b t <σ and q t max At that time, q t+1 =q t +1, otherwise q t+1 =q t ;r t+1 Let γt+1 represent the compression ratio of the gradient in the (t+1)th round, and γ1 and γ2 represent configurable hyperparameters; using the formula r t+1 =γ1*(b t ) 2 +γ2 calculates the compression ratio of the gradient in the (t+1)th round;

[0014] The server selects the clients participating in this round of training. t Let c represent the number of clients participating in training in round t; if network congestion occurs in round t-1, c t =c t-1 / 2; If no network congestion occurs in round t-1, c t =c t-1 +1; The server determines the quantity c. t Randomly select clients to participate in this round of training;

[0015] The server transmits gradients or model parameters to the clients participating in the current training round.

[0016] ​Preferably, the server transmits gradients or model parameters to the clients participating in the current training round, including:

[0017] If it is the first round of training, the server transmits the initial compression ratio, initial quantization bit width, and initial model parameters to the clients participating in this round of training;

[0018] If it is not the first round of training, t represents the round number of this training session, and e k Let t represent the number of training epochs in the latest session for the k-th client, and m represent a configurable hyperparameter; the server calculates the number of epochs since the last training session for each client participating in the current training epoch. k For the k-th client, if te k ≤m, the server retrieves te from the sending queue from back to front. k Multiple gradient rounds are sent to the client, along with the adaptive compression ratio, adaptive quantization bit width, and multiple gradient rounds; if te k >m, the server sends the adaptive compression ratio, adaptive quantization bit width, and global model parameters for this round to the client.

[0019] Preferably, the client updates its local model based on the relevant parameters sent by the server; calculates and compresses the gradient for the current round based on the local dataset; predicts the convergence speed for the next round of training based on the training results; and uploads the compressed gradient and the predicted convergence speed to the server, including:

[0020] If the client receives global model parameters from the server, the client will set the received global model parameters as local model parameters.

[0021] If the client receives multiple gradients from the server, the client converts each value of the position encoding information of each gradient sent by the server from decimal to binary and puts them into the position mask list in order; the index of the element with a value of 1 in the position mask list is used as the decoded position information; the client restores the quadruple information of each gradient sent by the server into a value information list; and the decoded position information and value information list are converted into a gradient list.

[0022] The transformed gradients from multiple rounds are summed bitwise to obtain the aggregated gradient result. The local model parameters are then added to the aggregated gradients to obtain the updated local model parameters. The gradient list is calculated as follows:

[0023]

[0024]

[0025] y i =sign(G i )*(G i) max *Gq i / q t

[0026]

[0027]

[0028] in, Represents a list of location masks. Representative value information list, Represents a gradient list, where t represents the t-th training iteration, s represents the s-th gradient received by the client, and l represents the gradient of the l-th layer of the model; sign(G i ) represents the gradient G i The symbol, Gq i Represents gradient G i The quantized value, q t This represents the quantization bit width.

[0029] The formulas for calculating gradient aggregation and model update are as follows:

[0030]

[0031]

[0032] in, This represents the parameters of the l-th layer of the client model at round t.

[0033] The client performs local training based on global model parameters and the local dataset, calculating the gradient for the current round. Based on the training progress, the client predicts the convergence rate for the next training round. The client's loss value for round t is expressed as... The level value of the client's loss in round t is represented as The trend value of the client's loss in round t is represented as: Where t represents the training round, and k represents the kth client, using the formula... Calculate the level value of the loss in round t, where α1 represents the level weight, using the formula... Calculate the trend value of the loss in round t, where α2 represents the trend value weight; [The last part, "the trend value of the loss in round t," appears to be a typo and can be omitted.] As a predicted value of the convergence speed of the k-th client in the (t+1)-th round of training;

[0034] The client compresses the gradient and uploads the compressed gradient and convergence rate prediction to the server.

[0035] Preferably, the client performs gradient compression, including:

[0036] g t Let v be the gradient of the t-th round.t Let G be the momentum of the t-th round. t This is expressed as the cumulative gradient in round t; the gradient in round t is expressed using the formula v. t =v t-1 +β·g t Momentum correction is performed, where β is a configurable hyperparameter; G is used to adjust momentum. t =G t-1 +v t Perform gradient accumulation;

[0037] r t Let sc represent the compression ratio sent by the server in round t. l Let represent the number of cumulative gradients in layer l during round t; the client sorts the gradients of each layer of the model from largest to smallest absolute value, and then calculates the gradients of the (r)th layer. t *sc l The absolute value of each gradient is used as the threshold for the gradient of the l-th layer; in the gradients of each layer of the model, gradients that are non-zero and whose absolute value is greater than the threshold are selected, and the value information and position information of the selected gradients are obtained.

[0038] The client converts the location information into a list of location masks. If the mask is sent in gradients, the corresponding position in the mask list will have a value of 1; otherwise, it will have a value of 0. The location mask list is then compressed into location-encoded information using bitwise operations.

[0039] For the gradient value information at the corresponding position, q t This represents the quantization bit width sent by the server in round t. This represents the maximum value of the cumulative gradient of the l-th layer after filtering. This represents the value of the i-th cumulative gradient in the l-th layer after filtering. Let represent the quantized value of the i-th cumulative gradient in the l-th layer after filtering, where sign(x) represents the sign of the value x, and abs(x) represents the absolute value of the value x; Represented using 32-bit binary code; q t Represented using Elias γ code; calculate the sign of the i-th cumulative gradient in the l-th layer after filtering. Represented using 1-bit binary code; using the formula The quantized and filtered i-th accumulated gradient of the l-th layer is converted into an Elias gamma code; a quadruple is then used. Encodes the value information of the gradient of the l-th layer;

[0040] The client uses the location encoding information and value information as the result of gradient compression.

[0041] Preferably, the server decompresses, aggregates, and recompresses the compression gradients uploaded by each client, calculates the global convergence rate prediction value based on the convergence rate prediction values ​​uploaded by each client, and updates the global model, including:

[0042] The server converts each value of the position encoding information sent by each client from decimal to binary and puts them into a position mask list in order. The index of the element with a value of 1 in the position mask list is used as the decoded position information. The server restores the quadruples of gradient information sent by the server in each round into a value information list. The decoded position information and value information list are then converted into a gradient list.

[0043] The gradients from each client are weighted and averaged according to the size of the local dataset to obtain the aggregated gradient result. The calculation of the aggregated gradient is as follows:

[0044]

[0045] in, n represents the gradient of the l-th layer of the model for the k-th client in the t-th round of training. k This represents the size of the local dataset of the k-th client;

[0046] The server uses the same gradient sparsity and quantization methods as the client to compress the gradient aggregation results, resulting in compressed gradients.

[0047] The server calculates all clients. The weighted average b t The global convergence speed prediction value is used as the convergence speed prediction value for the (t+1)th round, where t represents the training round and k represents the kth client. The calculation of the global convergence speed prediction value is as follows:

[0048]

[0049] Where, n k This represents the size of the local dataset of the k-th client;

[0050] The server decompresses the compressed gradient, adds the global model parameters to the decompressed gradient, and obtains the updated global model.

[0051] Preferably, the server updates the sending queue, including:

[0052] The server updates the sending queue, and compresses gradients into the queue; t represents the number of training epochs in this round, e k Let t represent the latest training round number of the k-th client, T represent the period for cleaning the sending queue, and n represent the length of the sending queue; when t%T=0, n-max{te} in the sending queue will be removed. k} elements are dequeued.

[0053] According to another aspect of the present invention, a bidirectional adaptive gradient compression system for federated learning is provided, comprising: a server and a client;

[0054] The server is used to adaptively determine the quantization bit width and compression ratio for this round of training, select the clients participating in this round of training, and transmit gradients or model parameters to each client.

[0055] The client is used to update the local model according to the relevant parameters sent by the server; calculate and compress the gradient of the current round according to the local dataset; predict the convergence speed of the next round of training according to the training situation; and upload the compressed gradient and the predicted convergence speed to the server.

[0056] The server is also used to decompress, aggregate, and recompress the compression gradients uploaded by each client, calculate the global convergence rate prediction value based on the convergence rate prediction value uploaded by each client, and update the global model and the sending queue.

[0057] Preferably, the server includes:

[0058] The client manager is used for client selection and adaptive compression parameter configuration, and uses a send queue to store downlink transmission gradients from multiple rounds.

[0059] Memory is used for momentum correction and gradient accumulation of gradients;

[0060] A compressor, used for the compression and decompression of gradients;

[0061] Aggregator, used for aggregating uphill gradients;

[0062] The client includes:

[0063] Memory is used for momentum correction and gradient accumulation of gradients;

[0064] A compressor, used for the compression and decompression of gradients;

[0065] Aggregator, used for aggregating downhill gradients.

[0066] According to another aspect of the present invention, a computer product is provided, comprising a computer program, which, when run on one or more processors, implements the bidirectional adaptive gradient compression method for federated learning as described above, the method comprising:

[0067] The server adaptively determines the quantization bit width and compression ratio for this round of training, selects the clients participating in this round of training, and transmits gradients or model parameters to each client.

[0068] The client updates the local model based on the relevant parameters sent by the server; calculates and compresses the gradient for the current round based on the local dataset; predicts the convergence speed of the next round of training based on the training situation; and uploads the compressed gradient and the predicted convergence speed to the server.

[0069] The server decompresses, aggregates, and recompresses the compression gradients uploaded by each client, and calculates the global convergence rate prediction value based on the convergence rate prediction value uploaded by each client.

[0070] The server updates the global model and the sending queue.

[0071] As can be seen from the technical solutions provided by the embodiments of the present invention above, the method and system of the present invention improve communication efficiency while ensuring model accuracy by adaptive gradient compression rate and quantization bit width; designing sparsity and quantization encoding methods and novel bidirectional gradient compression strategies reduces communication overhead, increases the number of clients supported by the federated learning model in network bandwidth-constrained environments, and alleviates the bandwidth bottleneck in large-scale federated learning.

[0072] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0073] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 A flowchart illustrating a bidirectional adaptive gradient compression method for federated learning provided in an embodiment of the present invention;

[0075] Figure 2 This is a schematic diagram illustrating the implementation principle of a bidirectional adaptive gradient compression method for federated learning provided in an embodiment of the present invention.

[0076] Figure 3 This is a flowchart illustrating gradient compression as described in an embodiment of the present invention. Detailed Implementation

[0077] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0078] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0079] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0080] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0081] Example 1

[0082] The specific processing flow of a bidirectional adaptive gradient compression method for federated learning provided in this embodiment of the invention is as follows: Figure 1 As shown in the diagram, the implementation schematic is as follows: Figure 2 As shown, the processing steps include the following:

[0083] Step S10: The server's client manager adaptively determines the quantization bit width and compression rate for this round of training, selects the clients participating in this round of training, and transmits gradient or model parameters to each client.

[0084] Step S20: The client participating in this round of training updates its local model according to the relevant parameters sent by the server; calculates the gradient of this round based on the local dataset, and predicts the convergence speed of the next round of training based on the training situation; performs gradient compression using the client memory and compressor; and uploads the compressed gradient and the predicted convergence speed to the server.

[0085] Step S30: The server receives the compressed gradient and convergence speed prediction values ​​uploaded by each client; the server compressor and aggregator are used to decompress and aggregate the gradients respectively; the server memory and compressor are used to compress the gradients; the global convergence speed prediction value is calculated based on the convergence speed prediction values ​​uploaded by each client.

[0086] Step S40: The server updates the global model and puts the compressed gradient of this round into the sending queue for later use. This round ends.

[0087] Specifically, step S10 above includes:

[0088] S11: During federated learning, the server's client manager adaptively determines the quantization bit width and compression ratio for this round of training. Obtain the predicted convergence speed b for round t+1. t ;q t+1 Let q represent the quantization bit width of the gradient in the (t+1)th round, σ represent a configurable hyperparameter, and q represent the quantization bit width of the gradient in the (t+1)th round. max This represents the maximum quantization bit width of the gradient; when b t <σ and q t max At that time, q t+1 =q t +1, otherwise q t+1 =q t ;r t+1 Let γt+1 represent the compression ratio of the gradient in the (t+1)th round, and γ1 and γ2 represent configurable hyperparameters; using the formula r t+1 =γ1*(b t ) 2 +γ2 calculates the compression ratio of the gradient in round t+1.

[0089] S12: Select the clients to participate in this round of training. t Let c represent the number of clients participating in training in round t; if network congestion occurs in round t-1, c t =c t-1 / 2; If no network congestion occurs in round t-1, c t =c t-1 +1; The server determines the quantity c. t The clients participating in this round of training are randomly selected.

[0090] S13: The server transmits gradients or model parameters to the clients participating in this round of training. Specifically, this includes the following steps:

[0091] If it is the first round of training, the server transmits the initial compression ratio, initial quantization bit width, and initial model parameters to the clients participating in this round of training;

[0092] If it is not the first round of training, t represents the round number of this training session, and e​k Let t represent the number of training epochs in the latest session for the k-th client, and m represent a configurable hyperparameter; the server calculates the number of epochs since the last training session for each client participating in the current training epoch. k For the k-th client, if te j ≤m, the server retrieves te from the sending queue from back to front. k Multiple gradient rounds are sent to the client, along with the adaptive compression ratio, adaptive quantization bit width, and multiple gradient rounds; if te k >m, the server sends the adaptive compression ratio, adaptive quantization bit width, and global model parameters for this round to the client.

[0093] Specifically, step S20 above includes: the client participating in this round of training updates its local model according to the relevant parameters sent by the server; calculates the gradient for this round based on the local dataset, and predicts the convergence speed of the next round of training based on the training situation; performs gradient compression using the client memory and compressor; and uploads the compressed gradient and predicted values ​​to the server, including:

[0094] S21: If the client receives global model parameters, the client will set the received global model parameters as local model parameters;

[0095] If the client receives gradients from multiple rounds, the client converts each value of the position encoding information of each round of gradients sent by the server from decimal to binary and puts them into the position mask list in order; the index of the element with a value of 1 in the position mask list is used as the decoded position information; the client restores the quadruple information of each round of gradients sent by the server into a value information list; and the decoded position information and value information list are converted into a gradient list.

[0096] The transformed gradients from multiple rounds are summed bitwise to obtain the aggregated gradient result. The local model parameters are then obtained and added to the aggregated gradients to obtain the updated local model. The gradient list is calculated as follows:

[0097]

[0098]

[0099] y i =sign(G i )*(G i ) max *Gq i / q t

[0100]

[0101]

[0102] in, Represents a list of location masks. Representative value information list, Represents a gradient list, where t represents the t-th training iteration, s represents the s-th gradient received by the client, and l represents the gradient of the l-th layer of the model; sign(G i ) represents the gradient G i The symbol, Gq i Represents gradient G i The quantized value, q t This represents the quantization bit width.

[0103] The formulas for calculating gradient aggregation and model update are as follows:

[0104]

[0105]

[0106] in, This represents the parameters of the l-th layer of the client model at round t.

[0107] S22: Calculate the gradient for this round based on the local dataset, and predict the convergence speed for the next round of training based on the training progress. This includes the following steps:

[0108] Perform local training based on the local dataset and calculate the gradient for this round.

[0109] Predict the convergence speed of the next training round based on the training results, including: the client's loss value in round t is expressed as... The level value of the client's loss in round t is represented as The trend value of the client's loss in round t is represented as: Where t represents the training round, and k represents the k-th client. Using the formula... Calculate the level value of the loss in round t, where α1 represents the level weight. Using the formula... Calculate the trend value of the loss in round t, where α2 represents the trend value weight; [The last part, "the trend value of the loss in round t," appears to be a typo and can be omitted.] This is used as a predicted value for the convergence speed of the k-th client in the (t+1)-th training round.

[0110] S23: Gradient compression using client-side memory and a compressor. Specifically, this includes:

[0111] g t Let v be the gradient of the t-th round. t Let G be the momentum of the t-th round. t This is expressed as the cumulative gradient in round t; the gradient in round t is expressed using the formula v. t =v t-1 +β·gt Momentum correction is performed, where β is a configurable hyperparameter; G is used to adjust momentum. t =G t-1 +v t Perform gradient accumulation;

[0112] r t Let sc represent the compression ratio sent by the server in round t. l Let represent the number of cumulative gradients in layer l during round t; sort the gradients of each layer of the model in descending order of absolute value, and then sort the gradients of the (r)th layer... t *sc l The absolute value of each gradient is used as the threshold for the gradient of the l-th layer. In the gradients of each layer of the model, gradients that are non-zero and whose absolute value is greater than the threshold are selected to obtain the value information and position information of the selected gradients.

[0113] For location information, it is first converted into a location mask list. Specifically, if the gradient is sent, the corresponding position in the mask list is 1, and if it is not sent, it is 0. Then, the location mask list is compressed into location encoded information using bitwise operations. Specifically, the location mask list is traversed from front to back, and every 64 mask information is converted from binary to decimal unsigned long integer.

[0114] For the gradient value information at the corresponding position, q t This represents the quantization bit width sent by the server in round t. This represents the maximum value of the cumulative gradient of the l-th layer after filtering. This represents the value of the i-th cumulative gradient in the l-th layer after filtering. Let represent the quantized value of the i-th cumulative gradient in the l-th layer after filtering, where sign(x) represents the sign of the value x, and abs(x) represents the absolute value of the value x; Represented using 32-bit binary code; q t Represented using Elias γ code; calculate the sign of the i-th cumulative gradient in the l-th layer after filtering. Represented using 1-bit binary code; using the formula The quantized and filtered i-th accumulated gradient of the l-th layer is converted into an Elias gamma code; a quadruple is then used. Encodes the value information of the gradient of the l-th layer;

[0115] The location encoding information and value information are used as the result of gradient compression. Figure 3 This is a flowchart illustrating gradient compression as described in an embodiment of the present invention.

[0116] S24: Upload the compressed gradient and predicted values ​​to the server.

[0117] Specifically, step S30 includes: the server receiving compressed gradients and convergence rate predictions uploaded by each client; decompressing and aggregating the gradients using the server compressor and aggregator respectively; compressing the gradients using the server-side memory and compressor; and calculating the global convergence rate prediction. Specifically, this includes:

[0118] S31: Use the server compressor and aggregator to decompress and aggregate the gradients respectively.

[0119] The server converts each value of the position encoding information sent by each client from decimal to binary and puts them into a position mask list in order; the index of the element with a value of 1 in the position mask list is used as the decoded position information; the server restores the quadruples of gradient information sent in each round into a value information list; and the decoded position information and value information list are converted into a gradient list.

[0120] The gradients from each client are weighted and averaged according to the size of the local dataset to obtain the aggregated gradient result. The calculation of the aggregated gradient is as follows:

[0121]

[0122] in, n represents the gradient of the l-th layer of the model for the k-th client in the t-th round of training. k This represents the size of the local dataset of the k-th client.

[0123] S32: Compress the gradient aggregation results using the same gradient sparsification and quantization methods as the client to obtain compressed gradients.

[0124] S33: Calculate all clients The weighted average b t The global convergence rate prediction is used as the convergence rate prediction for the (t+1)th round, where t represents the training round and k represents the kth client. The global convergence rate prediction is calculated as follows:

[0125]

[0126] Where, n k This represents the size of the local dataset of the k-th client.

[0127] Based on the predicted global convergence speed, determine whether the next round of the model is in a critical period. If it is in a critical period, it is necessary to reduce the sparsity and the number of quantization bits to improve communication efficiency while ensuring accuracy.

[0128] Specifically, step S40 above includes:

[0129] S41: The server updates the sending queue, and compresses gradients into the queue; t represents the number of training rounds in this round, ek Let t represent the latest training round number of the k-th client, T represent the period for cleaning the sending queue, and n represent the length of the sending queue; when t%T=0, n-max{te} in the sending queue will be removed. k} elements are dequeued.

[0130] S42: Use the server compressor to decompress the compressed gradient, add the global model parameters to the decompressed gradient, and obtain the updated global model.

[0131] Example 2

[0132] This invention also provides a bidirectional adaptive gradient compression system for federated learning, the system components of which include: a server and a client;

[0133] The server is used to adaptively determine the quantization bit width and compression ratio for this round of training, select the clients participating in this round of training, and transmit gradients or model parameters to each client.

[0134] The client is used to update the local model according to the relevant parameters sent by the server; calculate and compress the gradient of the current round according to the local dataset; predict the convergence speed of the next round of training according to the training situation; and upload the compressed gradient and the predicted convergence speed to the server.

[0135] The server is also used to decompress, aggregate, and recompress the compression gradients uploaded by each client, calculate the global convergence rate prediction value based on the convergence rate prediction value uploaded by each client, and update the global model and the sending queue.

[0136] Specifically, the aforementioned servers include:

[0137] The client manager is used for client selection and adaptive compression parameter configuration, and uses a send queue to store downlink transmission gradients from multiple rounds.

[0138] Memory is used for momentum correction and gradient accumulation of gradients;

[0139] A compressor, used for the compression and decompression of gradients;

[0140] Aggregator, used for aggregating uphill gradients.

[0141] The above clients include:

[0142] Memory is used for momentum correction and gradient accumulation of gradients;

[0143] A compressor, used for the compression and decompression of gradients;

[0144] Aggregator, used for aggregating downhill gradients.

[0145] The specific process of performing bidirectional adaptive gradient compression for federated learning using the system of this invention is similar to that of the aforementioned method embodiments, and will not be repeated here.

[0146] Example 3

[0147] This invention provides a computer program (product), including a computer program that, when run on one or more processors, implements the bidirectional adaptive gradient compression method for federated learning as described above, the method comprising:

[0148] The server adaptively determines the quantization bit width and compression ratio for this round of training, selects the clients participating in this round of training, and transmits gradients or model parameters to each client.

[0149] The client updates the local model based on the relevant parameters sent by the server; calculates and compresses the gradient for the current round based on the local dataset; predicts the convergence speed of the next round of training based on the training situation; and uploads the compressed gradient and the predicted convergence speed to the server.

[0150] The server decompresses, aggregates, and recompresses the compression gradients uploaded by each client, and calculates the global convergence rate prediction value based on the convergence rate prediction value uploaded by each client.

[0151] The server updates the global model and the sending queue.

[0152] In summary, this invention proposes an adaptive gradient compression method and system. This invention can utilize the training patterns of deep models to determine the compression ratio of gradient sparsification and the bit width of gradient quantization in each round, improving communication efficiency while ensuring model convergence and accuracy. It designs encoding methods in sparsification and quantization to further reduce communication overhead, increase the number of clients that can participate in each round, and alleviate bandwidth bottlenecks in large-scale federated learning. Furthermore, it designs a novel bidirectional gradient compression strategy that allows bidirectional gradient compression to be achieved even when only a portion of clients participate in training in each round, further increasing the number of clients supported by the federated learning model in network bandwidth-constrained environments.

[0153] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0154] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0155] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0156] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A bidirectional adaptive gradient compression method for federated learning, characterized in that, include: The server adaptively determines the quantization bit width and compression ratio for this round of training, selects the clients participating in this round of training, and transmits gradients or model parameters to each client. The client updates its local model based on the relevant parameters sent by the server; The gradient for the current round is calculated and compressed based on the local dataset. The convergence speed of the next round of training is predicted based on the training situation. The compressed gradient and the predicted convergence speed are then uploaded to the server. The server decompresses, aggregates, and recompresses the compression gradients uploaded by each client, calculates the global convergence rate prediction based on the convergence rate predictions uploaded by each client, and updates the global model. The server updates the sending queue, and this round ends.

2. The method according to claim 1, characterized in that, The server adaptively determines the quantization bit width and compression ratio for this round of training, selects the clients participating in this round of training, and transmits gradients or model parameters to each client, including: During federated learning, the server adaptively determines the quantization bit width and compression ratio for this round of training, and obtains the predicted convergence speed b for the (t+1)th round. t q t+1 Let q represent the quantization bit width of the gradient in the (t+1)th round. t Let q represent the quantization bit width of the gradient in the t-th round, σ represent the settable hyperparameter, and q represent the quantization bit width of the gradient in the t-th round. max This represents the maximum quantization bit width of the gradient; when b t <σ and q t <q max At that time, q t+1 =q t +1, otherwise q t+1 =q t ;r t+1 Let γt+1 represent the compression ratio of the gradient in the (t+1)th round, and γ1 and γ2 represent configurable hyperparameters; using the formula r t+1 =γ1*(b t ) 2 +γ2 calculates the compression ratio of the gradient in the (t+1)th round; The server selects the clients participating in this round of training. t Let c represent the number of clients participating in training in round t; if network congestion occurs in round t-1, c t =c t-1 / 2; If no network congestion occurs in round t-1, c t =c t-1 +1; The server determines the quantity c. t Randomly select clients to participate in this round of training; The server transmits gradients or model parameters to the clients participating in the current training round.

3. The method according to claim 2, characterized in that, The server transmits gradients or model parameters to the clients participating in this round of training, including: If it is the first round of training, the server transmits the initial compression ratio, initial quantization bit width, and initial model parameters to the clients participating in this round of training; If it is not the first round of training, t represents the round number of this training session, and e k Let represent the number of training epochs in the latest session for the k-th client, and m represent a configurable hyperparameter; the server calculates the number of epochs since the last training session for each client participating in the current training epoch. k For the k-th client, if te k ≤m, the server retrieves te from the sending queue from back to front. k Multiple gradient rounds are sent to the client, along with the adaptive compression ratio, adaptive quantization bit width, and multiple gradient rounds; if te k If the value is greater than m, the server will send the adaptive compression ratio, adaptive quantization bit width, and global model parameters for this round to the client.

4. The method according to claim 2 or 3, characterized in that, The client updates its local model based on the relevant parameters sent by the server; calculates and compresses the gradient for the current round based on the local dataset; predicts the convergence speed of the next round of training based on the training results; and uploads the compressed gradient and the predicted convergence speed to the server, including: If the client receives global model parameters from the server, the client will set the received global model parameters as local model parameters. If the client receives multiple gradients from the server, the client converts each value of the position encoding information of each gradient sent by the server from decimal to binary and puts them into the position mask list in order; the index of the element with a value of 1 in the position mask list is used as the decoded position information; the client restores the quadruple information of each gradient sent by the server into a value information list; and the decoded position information and value information list are converted into a gradient list. The transformed gradients from multiple rounds are summed bitwise to obtain the aggregated gradient result. The local model parameters are then added to the aggregated gradients to obtain the updated local model parameters. The gradient list is calculated as follows: y i =sign(G i )*(G i ) max *Gq i / q t in, Represents a list of location masks. Representative value information list, Represents a gradient list, where t represents the t-th training iteration, s represents the s-th gradient received by the client, and l represents the gradient of the l-th layer of the model; sign(G i ) represents the gradient G i The symbol, Gq i Represents gradient G i The quantized value, q t This represents the quantization bit width. The formulas for calculating gradient aggregation and model update are as follows: in, This represents the parameters of the l-th layer of the client model at round t. The client performs local training based on global model parameters and the local dataset, calculating the gradient for the current round. Based on the training progress, the client predicts the convergence rate for the next training round. The client's loss value for round t is expressed as... The level value of the client's loss in round t is represented as: The trend value of the client's loss in round t is represented as: Where t represents the training round, and k represents the kth client, using the formula... Calculate the level value of the loss in round t, where α1 represents the level weight, using the formula... Calculate the trend value of the loss in round t, where α2 represents the trend value weight; [The last part, "the trend value of the loss in round t," appears to be a typo and can be omitted.] As a predicted value of the convergence speed of the k-th client in the (t+1)-th round of training; The client compresses the gradient and uploads the compressed gradient and convergence rate prediction to the server.

5. The method according to claim 4, characterized in that, The client-side gradient compression includes: g t Let v be the gradient of the t-th round. t Let G be the momentum of the t-th round. t This is expressed as the cumulative gradient in round t; the gradient in round t is expressed using the formula v. t =v t-1 +β·g t Momentum correction is performed, where β is a configurable hyperparameter; G is used to adjust momentum. t =G t-1 +v t Perform gradient accumulation; r t Let sc represent the compression ratio sent by the server in round t. l Let represent the number of cumulative gradients in layer l during round t; the client sorts the gradients of each layer of the model from largest to smallest absolute value, and then calculates the gradients of the (r)th layer. t *sc l The absolute value of each gradient is used as the threshold for the gradient of the l-th layer; in the gradients of each layer of the model, gradients that are non-zero and whose absolute value is greater than the threshold are selected, and the value information and position information of the selected gradients are obtained. The client converts the location information into a list of location masks. If the mask is sent in gradients, the corresponding position in the mask list will have a value of 1; otherwise, it will have a value of 0. The location mask list is then compressed into location-encoded information using bitwise operations. For the gradient value information at the corresponding position, q t This represents the quantization bit width sent by the server in round t. This represents the maximum value of the cumulative gradient of the l-th layer after filtering. This represents the value of the i-th cumulative gradient in the l-th layer after filtering. Let represent the quantized value of the i-th cumulative gradient in the l-th layer after filtering, where sign(x) represents the sign of the value x, and abs(x) represents the absolute value of the value x; Represented using 32-bit binary code; q t Represented using Elias γ code; calculate the sign of the i-th cumulative gradient in the l-th layer after filtering. Represented using 1-bit binary code; using the formula The quantized and filtered i-th accumulated gradient of the l-th layer is converted into an Elias gamma code; a quadruple is then used. Encodes the value information of the gradient of the l-th layer; The client uses the location encoding information and value information as the result of gradient compression.

6. The method according to claim 5, characterized in that, The server decompresses, aggregates, and recompresses the compression gradients uploaded by each client, calculates the global convergence rate prediction based on the convergence rate predictions uploaded by each client, and updates the global model, including: The server converts each value of the position encoding information sent by each client from decimal to binary and puts them into a position mask list in order. The index of the element with a value of 1 in the position mask list is used as the decoded position information. The server restores the quadruples of gradient information sent in each round into a value information list. The decoded position information and value information list are then converted into a gradient list. The gradients from each client are weighted and averaged according to the size of the local dataset to obtain the aggregated gradient result. The calculation of the aggregated gradient is as follows: in, n represents the gradient of the l-th layer of the model for the k-th client in the t-th round of training. k This represents the size of the local dataset of the k-th client; The server uses the same gradient sparsity and quantization methods as the client to compress the gradient aggregation results, resulting in compressed gradients. The server calculates all clients. The weighted average b t The global convergence speed prediction value is used as the convergence speed prediction value for the (t+1)th round, where t represents the training round and k represents the kth client. The calculation of the global convergence speed prediction value is as follows: Where, n k This represents the size of the local dataset of the k-th client; The server decompresses the compressed gradient, adds the global model parameters to the decompressed gradient, and obtains the updated global model.

7. The method according to claim 6, characterized in that, The server update sending queue includes: The server updates the sending queue, and compresses the gradients into the queue; t represents the number of training epochs in this round, e k Let t represent the latest training round number of the k-th client, T represent the period for cleaning the sending queue, and n represent the length of the sending queue; when t%T=0, n-max{te} in the sending queue will be removed. k } elements are dequeued.

8. A bidirectional adaptive gradient compression system for federated learning, characterized in that, include: Server and client; The server is used to adaptively determine the quantization bit width and compression ratio for this round of training, select the clients participating in this round of training, and transmit gradients or model parameters to each client. The client is used to update the local model according to the relevant parameters sent by the server; calculate and compress the gradient of the current round according to the local dataset; predict the convergence speed of the next round of training according to the training situation; and upload the compressed gradient and the predicted convergence speed to the server. The server is also used to decompress, aggregate, and recompress the compression gradients uploaded by each client, calculate the global convergence rate prediction value based on the convergence rate prediction value uploaded by each client, and update the global model and the sending queue.

9. The system according to claim 8, characterized in that, The server includes: The client manager is used for client selection and adaptive compression parameter configuration, and uses a send queue to store downlink transmission gradients from multiple rounds. Memory is used for momentum correction and gradient accumulation of gradients; A compressor, used for the compression and decompression of gradients; Aggregator, used for aggregating uphill gradients; The client includes: Memory is used for momentum correction and gradient accumulation of gradients; A compressor, used for the compression and decompression of gradients; Aggregator, used for aggregating downhill gradients.

10. A computer product comprising a computer program, said computer program, when run on one or more processors, for implementing the bidirectional adaptive gradient compression method for federated learning as described above, the method comprising: The server adaptively determines the quantization bit width and compression ratio for this round of training, selects the clients participating in this round of training, and transmits gradients or model parameters to each client. The client updates its local model based on the relevant parameters sent by the server; The gradient for the current round is calculated and compressed based on the local dataset. The convergence speed of the next round of training is predicted based on the training situation. The compressed gradient and the predicted convergence speed are then uploaded to the server. The server decompresses, aggregates, and recompresses the compression gradients uploaded by each client, and calculates the global convergence rate prediction value based on the convergence rate prediction value uploaded by each client. The server updates the global model and the sending queue.

Images