A federated learning acceleration method based on adaptive quantization, device and medium

Abstract

The application discloses a federated learning acceleration method based on adaptive quantization, equipment and medium, wherein the method comprises the following steps: initializing a global model of a server and the same quantization level of each client; the client downloads the global model and the quantization level from the server, initializes a local model, carries out local training according to the initialized model, and calculates the update amount of the local model; the client calculates the empirical risk of the local model, quantizes the update amount of the local model, uploads the empirical risk and the quantized model update amount to the server; the server determines the quantization level of the next round according to the empirical risk uploaded by the client; the server carries out model aggregation, and downloads the quantization level and the aggregated model to the client until the model converges or reaches the upper limit of the communication cost. The federated learning based on adaptive quantization judges the required upload quantization level of different clients according to the local expected risk in the training process of different clients, and realizes adaptive calculation of the quantization level.

Description

TECHNICAL FIELD

[0001] The present application relates to deep learning, federated learning and quantization technology, and in particular to a federated learning acceleration method based on adaptive quantization, a device and a medium. BACKGROUND

[0002] Federated learning (FL) as a distributed machine learning framework allows multiple participants to collaboratively train a model without directly sharing data, effectively protecting the privacy and data security of users. The concept of federated learning and its classic model FedAvg were proposed. FedAvg achieves the goal of federated learning by uploading only the gradient or model update. With the gradual development of federated learning, many optimization algorithms have emerged, such as FedProx as an improved method, which introduces a regularization term during each local training to suppress the difference between local data and the global model, effectively improving the performance of the model.

[0003] Although federated learning has made some progress, it still faces the problems of communication overhead and computational efficiency when there are many clients. To solve this problem, quantization technology is introduced into federated learning. By quantizing the gradient or model update to a lower precision representation, the size of the transmitted data can be significantly reduced, the communication cost can be compressed, and to some extent the consumption of storage and computing resources can be reduced, thereby making federated learning more efficient. As a classic algorithm that combines quantization with distributed training, QSGD uses a random fixed-point quantizer to quantize the model gradient to achieve efficient decentralized distributed training. For quantization level q, the normalized update quantity is quantized to one of the q partitions in [0, 1]. QSGD quantizes the gradient vector according to the given quantization level and encodes it using Elias encoding. The encoded gradient is used for communication. Based on the quantization idea of QSGD, FedPAQ proposes a federated learning quantization method using the same quantization method. It quantizes the model update of multiple iterations at the local level to reduce the upload cost of federated learning. To further adjust the quantization level, DAdaQuant proposes a time-client adaptive federated learning quantization method. For the time dimension, DAdaQuant dynamically adjusts the global quantization level according to the global loss in the training process. When the loss increases compared to the previous round, the global quantization level is increased. Then, for the spatial dimension, an optimization objective is constructed to minimize the sum of the quantization levels of different clients while ensuring the quantization error, thereby obtaining adaptive client quantization levels. However, this method of obtaining personalized quantization levels from global quantization levels does not consider the learning of each client model in the hypothesis space, making it difficult for the client to upload the model to adjust the model according to the training of the client data set. This makes it difficult for clients with poor training conditions to obtain appropriate quantization levels and thus obtain a server aggregation model with high expression ability to affect subsequent model training. SUMMARY

[0004] To at least partially solve one of the technical problems existing in the prior art, the purpose of the present application is to provide a federated learning acceleration method based on adaptive quantization, equipment and medium.

[0005] The first technical solution adopted by the present application is:

[0006] A federated learning acceleration method based on adaptive quantization, comprising the following steps:

[0007] Initialize the global model M of the server 0 and the same quantization level of each client where subscript n represents the nth client, n∈{1,2,…,N}, N is the total number of clients;

[0008] The client downloads the global model and quantization level from the server, initializes the local model, performs local training based on the initialized model, and calculates the update amount of the local model;

[0009] The client calculates the empirical risk of the local model, quantifies the update amount of the local model, and uploads the empirical risk and quantified model update amount to the server;

[0010] The server determines the next round of quantitative level and the local model required to participate in the aggregation based on the empirical risk uploaded by the client;

[0011] The server aggregates the model and transmits the quantization level and aggregated model to the client, so that the client can update the local model and upload the empirical risk and model update amount until the model converges or reaches the communication cost limit.

[0012] Furthermore, the client downloads the global model and quantization level from the server, initializes the local model, performs local training based on the initialized model, and calculates the update amount of the local model, including:

[0013] For the rth communication round (a communication round includes each client downloading and initializing the model, performing local training, and uploading the model to the server for aggregation), the nth client downloads the global model M from the server r-1 and quantitative level

[0014] The nth client initializes the local model

[0015] The client uses the local data set, based on the initialized model, and completes the local model training according to the given training algorithm to obtain the model

[0016] Based on local model and model Calculate the model update amount:

[0017] Furthermore, the client calculates the empirical risk of the local model, quantifies the update amount of the local model, and uploads the empirical risk and the quantified model update amount to the server, including:

[0018] Each client uses a local training model Perform a forward propagation on the complete training set and calculate the empirical risk of the training set

[0019] The quantization level determined by each client based on the previous communication round The update amount of the local model is quantized using a random fixed-point quantizer Q(·,·) to obtain and upload the quantized model update and the empirical risk to the server. and upload the quantized model update and the empirical risk to the server.

[0020] Further, the empirical risk The calculation formula is as follows:

[0021]

[0022] In the formula, S n represents the number of samples, (x n,i , y n,i ) represents the i-th sample, and l(·) and f(·,·) represent the loss function and the neural network respectively.

[0023] Further, the server determines the quantization level of the next round according to the empirical risk uploaded by the client, including:

[0024] The server calculates the full-precision model saves the empirical risk value of each client in the current round and the corresponding full-precision model;

[0025] If r = 1, the quantization level is not changed

[0026] If r > 1, the server calculates the ratio of the current communication round empirical risk to the previous round empirical risk represents the relative change rate of the loss value;

[0027] Set the empirical risk threshold T1, and update the quantization level according to the relationship between and T1 to obtain a new quantization level

[0028] Further, the calculation formula of the new quantization level is as follows:

[0029]

[0030] In the formula, is the quantization level determined in the previous communication round; q max is the full-precision corresponding quantization level.

[0031] Further, the server aggregates the model and downloads the quantization level and the aggregated model to the client, including:

[0032] If r ≤ 2, the server determines the client model

[0033]

[0034] If r>2, set the client model selection threshold T2 to satisfy T1<T2, and the server determines the client model according to the relationship between T2 and T1; when T1<T2, , use the client model of the previous two rounds as the client model of the current round , use the full-precision model restored by using the model update amount uploaded in the current round as the client model of the current round , use the full-precision model restored by using the model update amount uploaded in the current round as the client model of the current round

[0035] According to the model , model aggregation is performed to obtain an aggregated model M r ;

[0036] The server downloads the aggregated model M r to each client, and downloads the quantization level

[0037] Further, the calculation formula of the aggregated model M r is as follows:

[0038]

[0039] In the formula, S n and S j both represent the number of samples.

[0040] The second technical solution adopted by the present application is:

[0041] An electronic device, comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the federated learning acceleration method based on adaptive quantization as described above.

[0042] The third technical solution adopted by the present application is:

[0043] A computer readable storage medium, the storage medium storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the federated learning acceleration method based on adaptive quantization as described above.

[0044] The fourth technical solution adopted by the present application is:

[0045] A computer program product or computer program comprises computer instructions stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned method.

[0046] The beneficial effects of the present application are: the present application is based on federated learning of adaptive quantization, according to the local expected risk in the training process of different clients, to judge the required upload quantization level of different clients, realize the adaptive calculation of quantization level; and according to the local expected risk to judge the model quality of the client in the previous round, so as to further determine the required client model participating in model aggregation under the current round, realize the high generalization representation of global model. The method realizes the global model with higher performance under the condition of the same communication transmission amount. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing part of the embodiments of the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the premise.

[0048] Figure 1 is a step flow chart of a federated learning acceleration method based on adaptive quantization in an embodiment of the present application;

[0049] Figure 2 is a schematic diagram of client quantization level calculation and server model aggregation in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application. For the step numbers in the following embodiments, they are only set for the convenience of explanation and description, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0051] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, front, back, left, right and the like, is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0052] In the description of the present application, one or more is meant to be one or more, more than two is meant to be two or more, greater than, less than, more than, etc. are understood to exclude the number, above, below, within, etc. are understood to include the number. If the first, second is described, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the sequence of indicated technical features.

[0053] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0054] In the scenario of distributed model training with different image data for different clients, the problem of high communication cost of federated learning is overcome, and the present application proposes a federated learning acceleration scheme based on adaptive quantization, which dynamically adjusts the upload quantization level of each client model update according to the training situation of each client model, and finally realizes a global model of the server with higher generalization performance at a lower communication cost, realizing efficient distributed training of image recognition models.

[0055] Embodiment 1

[0056] As shown in Figure 1 and Figure 2 The present embodiment provides a federated learning acceleration method based on adaptive quantization, comprising the following steps:

[0057] S1, initializing a global model M of a server 0 and the same quantization level as each client Wherein n∈{1,2,…,N}, N is the total number of clients.

[0058] S2, the client downloads the global model and the quantization level from the server, initializes the local model, performs local training according to the initialized model, and calculates the update amount of the local model.

[0059] For the rth communication round, the nth client downloads the global model M r-1 from the server and initializes the local model The client downloads the quantization level The client performs local training according to the initialized model to obtain a model and calculates an update amount of the local model

[0060] Specifically, step S2 includes the following steps:

[0061] S2.1, for the rth communication round (one communication round includes that each client downloads and initializes a model, performs local training, and uploads the model to the server for aggregation), the client downloads a global model M from the server r-1 and a quantization level

[0062] S2.2, the nth client initializes a local model n∈{1,2,…,N}.

[0063] S2.3, the client uses a local data set, according to the initialized model, and according to a given training algorithm, completes the model training to obtain

[0064] S2.4, the client calculates the model update amount:

[0065] S3, the client calculates the empirical risk of the local model, quantizes the update amount of the local model, and uploads the empirical risk and the quantized model update amount to the server.

[0066] Each client uses a local training model performs a forward propagation on the training set to calculate the empirical risk of the training set Each client uses a quantization level determined according to the previous communication round uses a quantizer Q(·,·) to quantize the local model update amount to obtain and uploads the empirical risk and the quantized model update amount to the server.

[0067] Exemplarily, step S3 specifically includes the following steps:

[0068] S3.1, each client uses a local training model performs a forward propagation on the complete training set to calculate the empirical risk of the training set where S n represents the number of samples (x n,i , y n,i ) represents the ith sample, and l(·) and f(·,·) represent a loss function and a neural network, respectively.

[0069] S3.2, each client uses a quantization level determined according to the previous communication round The local model update is quantized using the quantizer Q(·,·) to obtain and the empirical risk as well as the quantized model update are uploaded to the server.

[0070] S4. The server determines the quantization level for the next round and the local models required to participate in the aggregation based on the empirical risk uploaded by the clients.

[0071] As an optional implementation, step S4 specifically includes the following steps:

[0072] S4.1. The server calculates the full-precision model based on the quantized updates of each client and saves the empirical risk values of each client in the current round as well as the corresponding full-precision models.

[0073] S4.2. If r = 1, the quantization level remains unchanged

[0074] S4.3. If r > 1, the server calculates the ratio of the empirical risk in the current communication round to the empirical risk in the previous round which represents the relative change rate of the loss value.

[0075] S4.4. Let the empirical risk threshold be T1, and update the quantization level based on the relationship between and T1 to obtain whose mapping relationship is as follows:

[0076]

[0077] S5. The server performs model aggregation and downloads the quantization level and the aggregated model to the clients, enabling the clients to update their local models and upload the empirical risk and model updates until the model converges or reaches the communication cost limit.

[0078] If r ≤ 2, the server determines the client models based on the quantized updates uploaded by the clients in the current round and performs aggregation to obtain M r ; if r > 2, set the client model selection threshold T2 such that T1 < T2, and the server determines the client models based on the relationship between and T2 and performs model aggregation to obtain M r . The server downloads the aggregated model M r and the quantization level, and repeats S2 - S4 until the model converges or reaches the communication cost limit.

[0079] Specifically, step S5 specifically includes the following steps:

[0080] S5.1, if r<=2, the server determines the client model according to the current round of client upload quantization update amount

[0081]

[0082] S5.2, if r>2, set the client model selection threshold T2 to satisfy T1<T2, the server determines the client model according to the relationship between and T2, when , the first two rounds of client model are used as the client model of the current round , the full-precision model restored using the model update amount uploaded in the current round is used as the client model of the current round as the client model of the current round

[0083] S5.3, according to , the model aggregation is performed to obtain M r :

[0084]

[0085] S5.4, the server downloads the aggregated model M r to each client, and simultaneously downloads the quantization level

[0086] S5.5, repeat S2-S4 until the model converges or the communication cost upper limit is reached.

[0087] In summary, the present application is based on adaptive quantization federated learning, according to the local expected risk in the training process of different clients, to judge the upload quantization level required by different clients, to realize adaptive calculation of quantization level; and according to the local expected risk to judge the model quality of the client in the previous round, so as to further determine the client model required for participating in model aggregation in the current round, to realize high generalization representation of the global model. This method realizes obtaining a higher performance global model under the same communication transmission amount.

[0088] Embodiment 2

[0089] The embodiment of the application also provides an electronic device, which comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to realize a federated learning acceleration method based on adaptive quantization as shown in Figure 1 .

[0090] It can be understood that the memory can include a random access memory (RAM) and can also include a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function, instructions for implementing various method embodiments described above, etc.; and the data storage area can store data created according to the use of the server, etc.

[0091] The processor can include one or more processing cores. The processor connects various parts within the entire server through various interfaces and lines, executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor can be integrated with a combination of one or more of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes operating systems and application programs, etc.; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but can be implemented by a separate chip.

[0092] Since the electronic device is an electronic device corresponding to the federated learning acceleration method based on adaptive quantization of the embodiments of the present application, and the principle of solving problems by the electronic device is similar to that of the method, the implementation of the electronic device can be referred to the implementation process of the above-mentioned method embodiments, and the repeated parts will not be described here.

[0093] Embodiment 3

[0094] The embodiments of the present application also provide a computer readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by a processor to implement the federated learning acceleration method based on adaptive quantization as shown in Figure 1 ​

[0095] Those skilled in the art can understand that all or part of the steps in the above-mentioned various methods of the embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable storage medium, including Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other medium that can be used to carry or store data which can be read by a computer.

[0096] Since the storage medium is a storage medium corresponding to the federated learning acceleration method based on adaptive quantization of the embodiments of the present application, and the principle of solving the problem of the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiments, and the repeated parts will not be described again.

[0097] Embodiment 4

[0098] In some possible implementation manners, various aspects of the method of the embodiments of the present application can also be implemented in the form of a program product, which includes program codes for causing a computer device to execute the steps of the federated learning acceleration method based on adaptive quantization according to various exemplary embodiments of the present application described above in the specification when the program product is run on the computer device. Among them, the executable computer program code or "code" for executing various embodiments can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (for example, Transact-SQL), Perl, or in various other programming languages.

[0099] It should be understood that various aspects of the application can be implemented in hardware, software, firmware or a combination of them. In the above embodiments, various steps or methods can be implemented in software or firmware which is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combination, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application-specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field-programmable gate arrays (FPGA), and so forth.

[0100] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. Furthermore, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0101] The above embodiments are only for the purpose of illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made according to the essence of the present application should be covered within the protection scope of the present application.

Claims

1. A federated learning acceleration method based on adaptive quantization, characterized in that, The method comprises the following steps: initializing a global model of a server and a same quantization level of each client; the client downloads the global model and the quantization level from the server, initializes a local model, performs local training according to the initialized model, and calculates an update amount of the local model; the client calculates an experience risk of the local model, quantizes the update amount of the local model, and uploads the experience risk and the quantized model update amount to the server; the server determines a quantization level of the next round according to the experience risk uploaded by the client; the server performs model aggregation, and downloads the quantization level and the aggregated model to the client, so that the client updates the local model and uploads the experience risk and the model update amount until the model converges or reaches a communication cost upper limit; the client downloads the global model and the quantization level from the server, initializes a local model, performs local training according to the initialized model, and calculates an update amount of the local model, comprising: For the rth communication round, the nth client downloads the global model M from the server r-1 and quantization levels The subscript n represents the client sequence number, n ∈ {1, 2, …, N}, N is the total number of clients; The nth client initializes the local model The client utilizes the local dataset, based on the initialized model, to complete local model training according to a given training algorithm, to obtain a model According to the local model and the model The model update amount is calculated: the client calculates an experience risk of the local model, quantizes the update amount of the local model, and uploads the experience risk and the quantized model update amount to the server, comprising: Clients utilize local trained models One forward pass over the full training set, computing the empirical risk on the training set quantization levels determined according to a previous communication round quantize the update of the local model using a stochastic quantizer Q(·,·) to obtain and upload the empirical risk and the quantized model update to the server; Empirical risk The formula for the calculation is: In the formula, S n represents the number of samples, (x n,i ,y n,i ) represents the i-th sample, and l(·) and f(·,·) represent the loss function and the neural network, respectively.

2. The federated learning acceleration method based on adaptive quantization according to claim 1, characterized in that, the server determines a quantization level of the next round according to the experience risk uploaded by the client, comprising: The server calculates a full-precision model according to the model update quantity of each client after quantization Save the experience risk value of each client in the current round And the corresponding full-precision model; If r = 1, then the quantization level is not changed If r>1, the server calculates the ratio of the current communication round experience risk and the previous round experience risk represents the relative change rate of the loss value; Set an empirical risk threshold T1, according to Update the quantization level according to the relationship with T1 to obtain a new quantization level 3. The federated learning acceleration method based on adaptive quantization according to claim 2, characterized in that, New quantization levels The calculation formula is as follows: wherein quantization level determined for the previous communication turn; q max is the full-precision corresponding quantization level.

4. The federated learning acceleration method based on adaptive quantization according to claim 1, characterized in that, the server performs model aggregation, and downloads the quantization level and the aggregated model to the client, comprising: If r≤2, the server determines the client model based on the client upload quantization update amount of the current round If r>2, set a client model selection threshold T2 satisfying T1<T2, and the server determines the client model according to the relationship between T2; when the first two rounds of client models are used as the client model of the current round when the full-precision model restored using the model update uploaded by the current round is used as the client model of the current round According to the model A model aggregation is performed to obtain an aggregated model M r ; The server transmits the aggregated model M r to each client while transmitting the quantization level 5. The federated learning acceleration method based on adaptive quantization according to claim 4, characterized in that, The aggregated model M r The calculation formula is: In the formula, S n , S j each represents the number of samples.

6. An electronic device, comprising: The electronic device comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to realize the method of any one of claims 1 to 5.

7. A computer readable storage medium characterized in that, The storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to realize the method of any one of claims 1 to 5.

Images