A Federated Learning Method and System Based on Low-Precision Integer Representation

Abstract

The present invention provides a federated learning method under low-precision integer representation, comprising: a server initializes a global floating-point model and selects a terminal; quantizes the global floating-point model into an integer global integer model; the server sends the global integer model to the terminal; the terminal trains the acquired global integer model using training data to obtain an integer low-precision model; the terminal transmits the low-precision model to the server; the server dequantizes the global integer model and the low-precision model, and updates the global floating-point model using the dequantized global integer model and the dequantized low-precision model; repeats the steps after the above initialization until the global floating-point model converges or reaches a preset number of training times; the present invention solves the energy consumption problem of terminal floating-point operations by designing a neural network training algorithm with full integer representation and solves the accuracy problem of low-precision federated learning by designing a low-precision model aggregation algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a federated learning method and system under low-precision integer representation. Background Art

[0002] Federated learning enables multiple devices to collaboratively learn a shared prediction model while keeping all training data on the device, allowing AI applications to be deployed while protecting user privacy. Figure 1 During the entire federated learning process, each device downloads the latest global model from the cloud server and improves the model based on the data stored locally, and then aggregates these model updates to the cloud server in an encrypted manner. The cloud server averages all uploaded updates to improve the global model and transmits them back to the corresponding terminal to continue updating until convergence.

[0003] However, the inventors found that the above-mentioned federated learning system is limited by wireless network transmission and low-power terminal training, which causes the following serious problems: First, due to the limited wireless network bandwidth and the amount of data transmitted by the neural network model, federated learning training is a very time-consuming process. For example, a report released by Google revealed that it took 3,000 rounds and 5 days to train an input method prediction model on millions of smartphones; secondly, because the current mobile terminals are not friendly enough to support neural network training, frequent high-precision floating-point operations will seriously consume the power of mobile terminals. Summary of the invention

[0004] In view of this, the present invention aims to provide a solution that can overcome at least one of the above-mentioned defects of the prior art.

[0005] On the one hand, the present invention provides a federated learning method under low-precision integer representation, comprising the following steps: S1, a server initializes a global floating-point model and its training, and selects at least one terminal; S2, quantizes the global floating-point model into an integer global integer model; S3, the server sends the global integer model to the terminal; S4, the terminal obtains the global integer model, and trains the obtained global integer model using training data to obtain an integer low-precision model; S5, the terminal transmits the low-precision model to the server; S6, the server dequantizes the global integer model and the low-precision model, and updates the global floating-point model using the dequantized global integer model and the dequantized low-precision model; and S7, repeats steps S2 to S6 until the global floating-point model converges or reaches a preset number of training times.

[0006] In one embodiment, S1 includes: S11, obtaining a task, and the server randomly sets a global floating-point model based on the task; S12, the server selects terminals to participate in training; S13, the server specifies unified parameters for the model training of each terminal, and sends the parameters to each terminal.

[0007] In one embodiment, S2 includes: quantizing the weight coefficients in the global floating-point model into integers to obtain a global integer model.

[0008] In one embodiment, S4 includes: the terminal acquires the global integer model, and obtains the low-precision model through stochastic gradient descent training.

[0009] In one embodiment, the low-precision model is obtained by the stochastic gradient descent method, including: quantizing the training data to obtain quantized training data and corresponding labels, and using the quantized training data to perform forward propagation and back propagation in sequence to obtain the locally updated low-precision model; wherein, during the forward propagation process, the quantized training data is used as the input of the first layer of the neural network, the input of the current layer of the neural network is multiplied by the weight coefficient of the global integer model to obtain the output of the current layer, and then the forward intermediate result of the output is passed to the next layer of the neural network as input, and the processing of each layer of the neural network is the same, and then the low-precision loss is calculated based on the label and the forward intermediate result of the output of the last layer of the neural network; during the back propagation process, the gradient g is calculated based on the low-precision loss. A , using the quantized training data X and the gradient g as the input of the last layer in the neural network A Determine the model parameter update g of the last layer of neural network W =X*g A , using the model parameter update amount g W Update the weight coefficient W of the last layer of the global model, using the gradient g A The weight coefficient W of the last layer of the global model determines the loss value g of the previous layer of the current network X =g A *W, then the loss value g X As the low-precision loss of the current layer, it is passed to the previous layer. The processing of each layer of the neural network is the same until the calculation of the parameters in each layer of the neural network is completed. By updating the model parameter g W The low-precision model is obtained by aggregation.

[0010] In one embodiment, during the forward propagation, before the step of multiplying the input of the current layer neural network by the weight coefficient of the global integer model, the input of the current layer neural network and the weight coefficient of the global integer model are transformed by shifting and numerical rounding operations to obtain a lower precision integer representation; during the backward propagation, the gradient g is transformed by shifting and numerical rounding operations. A Transform so that the gradient g A Represented as an integer with lower precision. In one embodiment, the S6 includes: S61, the server dequantizes the global integer model into a first floating-point model; S62, the server dequantizes the low-precision model into a second floating-point model; S63, the server subtracts the corresponding weight coefficient in the first floating-point model from the weight coefficient in the second floating-point model to obtain the weight update amount of each terminal local training model, and obtains the weight update amount of the global floating-point model by weighted averaging based on the weight update amount; S64, the server adds the weight update amount of the global floating-point model to the corresponding weight coefficient in the global floating-point model to update the global floating-point model.

[0011] Another aspect of the present invention provides a federated learning system under low-precision integer representation, including: a server, used to initialize a global floating-point model and its training, and select at least one terminal; also used to quantize the initialized or updated global floating-point model into an integer global integer model, send the global integer model to the terminal, dequantize the global integer model and the low-precision model sent by the at least one terminal, update the global floating-point model using the dequantized global integer model and the dequantized low-precision model, return to execute the operation of quantizing the global floating-point model into an integer global integer model and subsequent operations, until the global floating-point model converges or reaches a preset number of training times; at least one terminal, used to train the global integer model sent by the server, obtain an integer low-precision model, and send it to the server.

[0012] In one embodiment, the server is used to quantize the weight coefficients in the global floating point model into integers.

[0013] In one embodiment, the terminal obtains the low-precision model by training through a stochastic gradient descent method.

[0014] In one embodiment, the terminal obtains the low-precision model by using a stochastic gradient descent method, including:

[0015] The method of obtaining the low-precision model by the stochastic gradient descent method includes: quantizing the training data to obtain quantized training data and corresponding labels, and using the quantized training data to perform forward propagation and back propagation in sequence to obtain the locally updated low-precision model; wherein, during the forward propagation process, the quantized training data is used as the input of the first layer of the neural network, the input of the current layer of the neural network is multiplied by the weight coefficient of the global integer model to obtain the output of the current layer, and then the forward intermediate result of the output is passed to the next layer of the neural network as the input, and the processing of each layer of the neural network is the same, and then the low-precision loss is calculated according to the forward intermediate result of the label and the output of the last layer of the neural network; during the back propagation process, the gradient g is calculated according to the low-precision loss. A , using the quantized training data X and the gradient g as the input of the last layer in the neural network A Determine the model parameter update g of the last layer of neural network W =X*g A , using the model parameter update amount g W Update the weight coefficient W of the last layer of the global model, using the gradient g A The weight coefficient W of the last layer of the global model determines the loss value g of the previous layer of the current network X =g A *W, and then the loss value g X As the low-precision loss of the current layer, it is passed to the previous layer. The processing of each layer of the neural network is the same until the calculation of the parameters in each layer of the neural network is completed. By updating the model parameter g W The low-precision model is obtained by aggregation.

[0016] In one embodiment, during the forward propagation, before the step of multiplying the input of the current layer neural network by the weight coefficient of the global integer model, the input of the current layer neural network and the weight coefficient of the global integer model are transformed by shifting and numerical rounding operations to obtain a lower precision integer representation; during the backward propagation, the gradient g is transformed by shifting and numerical rounding operations. A Transform so that the gradient g A Represented as an integer with lower precision.

[0017] Another aspect of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the federated learning method under low-precision integer representation as described in any one of the above items when executing the computer program.

[0018] Another aspect of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the federated learning method under low-precision integer representation as described in any of the above items are implemented.

[0019] It can be seen from the above technical scheme that in the present invention, the global floating-point model and its training are initialized by the server, and at least one terminal is selected; the global floating-point model is then quantized into an integer global integer model; the server then sends the global integer model to the terminal; the terminal obtains the global integer model, and trains the obtained global integer model with training data to obtain an integer low-precision model; the terminal transmits the low-precision model to the server; the server dequantizes the global integer model and the low-precision model, and updates the global floating-point model with the dequantized global integer model and the dequantized low-precision model. Point model; and repeat the steps after the above initialization until the global floating-point model converges or reaches a preset number of training times; in the above processing, the global floating-point model is quantized into a global integer model represented by a full integer. On the one hand, the entire training process of the terminal is integer operation, and does not involve floating-point operation, which greatly reduces the energy consumption of terminal operations. On the other hand, the amount of data transmitted by the model between the server and the terminal is greatly reduced; secondly, by designing a low-precision model aggregation algorithm for quantization error compensation and adding the integer global integer model to the iterative process, the accuracy deviation of low-precision federated learning can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The following drawings are only used to illustrate and explain the present invention, and do not limit the scope of the present invention:

[0021] Figure 1 A schematic diagram of the federated learning process under floating point representation in the related art of this field is shown;

[0022] Figure 2 A schematic diagram of the process of federated learning under floating point representation in one embodiment of the present invention is shown;

[0023] Figure 3 A schematic diagram of a federated learning algorithm under low-precision integer representation in one embodiment of the present invention is shown;

[0024] Figure 4 A schematic diagram of a low-precision training process in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0026] In the drawings, the thickness, size and shape of objects have been slightly exaggerated for ease of explanation. The drawings are only examples and are not drawn strictly to scale.

[0027] It should also be understood that the terms "comprises", "including", "having", "includes" and / or "comprising", when used in this specification, indicate the presence of the stated features, wholes, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, parts and / or combinations thereof. In addition, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire listed features rather than modifying the individual elements in the list. In addition, when describing embodiments of the present application, "may" is used to mean "one or more embodiments of the present application". And, the term "exemplary" is intended to refer to an example or illustration.

[0028] Unless otherwise defined, all terms (including technical terms and scientific terms) used in this article have the same meaning as commonly understood by ordinary technicians in the field to which this application belongs. It should also be understood that terms (such as terms defined in commonly used dictionaries) should be interpreted as having the same meaning as their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined in this article.

[0029] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] Figure 2 A schematic diagram of a method flow of an embodiment of the present invention is shown, comprising the following steps: comprising the following steps:

[0031] S1, the server initializes the global floating point model and its training, and selects at least one terminal;

[0032] S2, a global integer model that quantizes the global floating-point model into an integer;

[0033] S3, the server sends the global integer model to the terminal;

[0034] S4, the terminal obtains the global integer model, and trains the obtained global integer model using the training data to obtain a low-precision integer model;

[0035] S5, the terminal transmits the low-precision model to the server;

[0036] S6, the server dequantizes the global integer model and the low-precision model, and updates the global floating-point model using the dequantized global integer model and the dequantized low-precision model; and

[0037] S7, repeat steps S2 to S6 until the global floating point model converges or reaches a preset number of training times.

[0038] The following is a specific example of the federated learning method under low-precision integer representation of the present application.

[0039] S1, the server initializes the global floating point model and its training, and selects at least one terminal.

[0040] refer to Figure 3 In the implementation manner provided by the present invention, S1 includes the following steps:

[0041] S11, obtain the task, and the server randomly sets a global floating-point model based on the task; S12, the server selects terminals to participate in the training; S13, the server specifies uniform parameters for the model training of each terminal, and sends the parameters to each terminal.

[0042] In some implementations, the cloud server may be a BBU-pool. Optionally, in a wireless access network, the server may also be a centralized server in the cloud of a Cloud server.

[0043] In some embodiments, the terminal device may be an interactive terminal commonly used in the art that supports conversation or processing with a computer server, such as a mobile phone. The method of selecting a terminal to participate in training in this application may be based on the existing method of selecting a terminal for federated learning, such as a random selection method.

[0044] In some implementations, the global floating-point number model may be a BP neural network model or a model related to nlp natural language processing, which is not limited in this document.

[0045] In the embodiment provided by the present invention, the weight coefficient W in the global floating point model is represented by a floating point number.

[0046] In some implementations, the specified parameters may be batch size, local epochs, and learning rate, etc., which are not limited in this document.

[0047] S2, quantizes the global floating-point model into a global integer model of integer type.

[0048] In the implementation provided herein, the weight coefficient W in the global floating-point model is represented by a floating-point number. Specifically, in S2, W in the global floating-point model is quantized so that W is represented by an integer.

[0049] In the embodiment provided by the present invention, although quantizing the weight coefficient from floating point number to integer will lose a certain degree of precision, in terms of energy consumption, the energy consumption generated by the terminal training the global integer model of integer will be greatly reduced.

[0050] In some implementations, the quantization method may be performed using various existing quantization methods, such as the BinnaryConnect algorithm flow, BNN (Boundary Network Node) and other methods, which are not limited in this document.

[0051] S3, the server sends the global integer model to the terminal.

[0052] In some implementations, the terminal can obtain the domain name of the server and then connect to the server according to the domain name to obtain the required resources, which is not limited in this article. After the server connects to the selected terminal, the server uses the established connection to send the global integer model to the terminal, specifically, to send the weight coefficient of the global integer model to the terminal.

[0053] S4, the terminal obtains the global integer model, and uses the training data to train the obtained global integer model to obtain a low-precision integer model.

[0054] refer to Figure 4 In the embodiment provided by the present invention, after the current terminal obtains the global integer model, the current global integer model is trained by the stochastic gradient descent method to obtain a low-precision model. During the entire training process, the processing performed by the terminal is all integer processing.

[0055] In the embodiment provided by the present invention, the process of training the global integer model by each terminal participating in the training includes processing of several layers of neural networks, wherein each layer of neural network performs the same processing on the global integer model and the training data. The process of training the terminal using a multi-layer neural network is described in detail below.

[0056] Specifically, in an embodiment of the present invention, the global integer model training process performed on each terminal is the same, and the global integer model training process of the terminal may include the following steps:

[0057] S41: In the terminal participating in the training, training data (X, Y) is stored, and the training data is quantized by the terminal to obtain quantized training data for model training.

[0058] In some implementations, the quantization method may be performed using a BinnaryConnect algorithm flow, a BNN (Boundary Network Node) method, etc., which is not limited herein.

[0059] In addition, when quantizing the training data, all the training data may be quantized to obtain quantized training data, or the training data may be sampled first and then the sampling results may be quantized to obtain quantized training data.

[0060] In the embodiment provided by the present invention, the locally updated low-precision model is obtained by sequentially performing forward propagation and back propagation using quantized training data.

[0061] Specifically, during the forward propagation process, the quantized training data (X, Y) is used as the input of the first layer of the neural network, the input of the current layer of the neural network is multiplied by the weight coefficient of the global integer model to obtain the output of the current layer, and then the forward intermediate result of the output is passed to the next layer of the neural network as input. The processing of each layer of the neural network is the same, and then the low-precision loss is calculated based on the label Y in the quantized training data and the forward intermediate result of the output of the last layer of the neural network.

[0062] During the back propagation process, the gradient g is calculated based on the low-precision loss A , using the quantized training data (X, Y) of the input of the last layer of the neural network and the gradient g A Determine the model parameter update g of the last layer of neural network W =X*g A , using the model parameter update amount g W Update the weight coefficient W of the last layer of the global model, using the gradient g A The weight coefficient W of the last layer of the global model determines the loss value g of the previous layer of the current network X =g A *W, and then the loss value g X As the low-precision loss of the current layer, it is passed to the previous layer. The processing of each layer of the neural network is the same until the calculation of the parameters in each layer of the neural network is completed. By updating the model parameter g W Aggregation results in a low-precision model.

[0063] Reference again Figure 4 Specifically, the input of the current layer of neural network is multiplied by the weight coefficient of the global integer model to obtain the matrix intermediate result A, which is then transformed into a lower precision representation; the gradient loss g is determined using the transformed matrix intermediate result Y Y, f is the low precision loss, see formula (1) for details.

[0064]

[0065] The above processing includes a lower precision transformation operation, that is, transforming the data from a high-precision representation to a low-precision representation. In the forward propagation process, before the step of multiplying the input of the current layer neural network with the weight coefficient of the global integer model, the input of the current layer neural network and the weight coefficient of the global integer model are transformed by shifting and numerical rounding operations to obtain a lower precision integer representation; in the backward propagation process, the gradient g is transformed by shifting and numerical rounding operations. A Transform so that the gradient g A The above processing may be, for example, a conversion from a 32-bit representation to an 8-bit representation. The specific low-precision conversion process may be implemented in various existing ways, such as Figure 4 The figure shows the transformation of low-precision representation through shift and value rounding operations. The original number of bits (e.g., 32 bits) and the target number of bits (e.g., 8 bits) when performing a lower-precision transformation can be determined by comprehensive consideration of multiple angles such as operation accuracy, cost, and energy consumption.

[0066] S5, the terminal transmits the low-precision model to the server;

[0067] S6, the server dequantizes the global integer model and the low-precision model, and updates the global floating-point model using the dequantized global integer model and the dequantized low-precision model.

[0068] Specifically, in order to save data transmission between the terminal and the server and reduce the amount of data calculation when the terminal performs model training, the floating-point model of federated learning is converted into an integer model and sent to the terminal for training. However, it is obvious that the integer model reduces the model accuracy. In order to minimize the accuracy loss after quantizing the floating-point model, in the embodiment of the present invention, S6 may include the following steps:

[0069] S61, the server dequantizes the global integer model into a first floating point model;

[0070] S62, the server dequantizes the low-precision model into a second floating-point model;

[0071] The inverse quantization process is the opposite operation of the quantization process, and the specific inverse quantization process can be performed in various existing ways.

[0072] S63, the server subtracts the corresponding weight coefficient in the first floating-point model from the weight coefficient in the second floating-point model to obtain the weight update amount of each terminal's local training model, and obtains the weight update amount of the global floating-point model by weighted average based on the weight update amount;

[0073] Wherein, each floating point model includes a plurality of weight coefficients, and the first floating point model is derived from the global integer model, and the second floating point model is derived from the low-precision model obtained by training the global integer model, so the neural network structures of the two are the same, and the weight coefficients are also corresponding. In this step, the corresponding weight coefficients of the two floating point models are subtracted to obtain the update amount of the corresponding weight.

[0074] S64, the server adds the weight update amount of the global floating-point number model to the corresponding weight coefficient in the global floating-point number model to update the global floating-point number model.

[0075] In some implementations, independent steps are used to implement multiple standard inverse quantization processes respectively, and then the corresponding output is selected according to the inverse quantization type of the input bitstream. For example, the inverse quantization method that can support both MPEG-2 and AVS is: first determine whether the inverse quantization type is MPEG-2 or AVS, then select the respective inverse quantization processes according to the type, obtain the inverse quantization coefficients after inverse quantization and output them.

[0076] In some implementations, the number of terminals participating in the training may be one or more, and the weight update amount of the global floating-point model is based on the weight update amount of the terminal local training model obtained by training each terminal participating in the training.

[0077] After completing all the above steps, the global floating-point model is updated once in the current round, and the steps after the above initialization (i.e., S2-S6) are repeated to perform the next round of iterative update on the global integer model obtained by the update of this round until the global floating-point model converges or reaches the preset training round.

[0078] In summary, in the present invention, a global floating-point model and its training are initialized by a server, and at least one terminal is selected; the global floating-point model is then quantized into an integer global integer model; the server then sends the global integer model to the terminal; the terminal obtains the global integer model, and uses training data to train the obtained global integer model to obtain an integer low-precision model; the terminal transmits the low-precision model to the server; the server dequantizes the global integer model and the low-precision model, and updates the global floating-point model using the dequantized global integer model and the dequantized low-precision model; and the steps after the above initialization are repeated until the global floating-point model converges or reaches a preset number of training times; in the above processing, the global floating-point model is quantized into a global integer model represented by a full integer, which reduces the energy consumption of terminal operations; secondly, by designing a low-precision model aggregation algorithm for quantization error compensation and adding the integer global integer model to the iterative process, the accuracy deviation after quantization can be reduced.

[0079] Another aspect of the present invention provides a federated learning system under low-precision integer representation, including: a server, used to initialize a global floating-point model and its training, and select at least one terminal; also used to quantize the initialized or updated global floating-point model into an integer global integer model, send the global integer model to the terminal, dequantize the global integer model and the low-precision model sent by at least one terminal, update the global floating-point model using the dequantized global integer model and the dequantized low-precision model, return to perform the operation of quantizing the global floating-point model into an integer global integer model and its subsequent operations, until the global floating-point model converges or reaches a preset number of training times; at least one terminal, used to train the global integer model sent by the server, obtain an integer low-precision model, and send it to the server. The system design is based on the same idea as the method, and this article will not repeat it.

[0080] In some implementations, the server may be a BBU-pool. Optionally, in a wireless access network, the server may also be a centralized server in the cloud of a Cloud server.

[0081] In some implementations, the terminal may be an interactive terminal commonly used in the art that supports conversation or processing with a computer server, and this document will not limit this.

[0082] Another aspect of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the federated learning method under the low-precision integer representation as described above are implemented. The design of the electronic device is based on the same idea as the method, and will not be elaborated in this article.

[0083] Another aspect of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the federated learning method under the low-precision integer representation as described above are implemented. The design of the computer program is based on the same idea as the method, and will not be repeated in this article.

[0084] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A federated learning method under low-precision integer representation, characterized in that: The following steps are involved: S1, the server initializes the global floating point model and its training, and selects at least one terminal; S2, quantizing the global floating-point model into a low-precision global integer model; S3, the server sends the global integer model to the terminal; S4, the terminal obtains the global integer model, and trains the obtained global integer model using training data to obtain a low-precision integer model; S5, the terminal transmits the low-precision model to the server; S6, the server dequantizes the global integer model and the low-precision model, and updates the global floating-point model using the dequantized global integer model and the dequantized low-precision model; and S7, repeat steps S2 to S6 until the global floating point model converges or reaches a preset number of training times.

2. The method according to claim 1, characterized in that The S1 includes: S11, obtaining a task, and the server randomly sets a global floating point number model based on the task; S12, the server selects a terminal to participate in training; S13, the server specifies uniform parameters for model training of each terminal, and sends the parameters to each terminal.

3. The method according to claim 1, characterized in that The S2 includes: The weight coefficients in the global floating-point model are quantized into integers to obtain a global integer model.

4. The method according to claim 1, characterized in that The S4 includes: The terminal obtains the global integer model and obtains the low-precision model through stochastic gradient descent training.

5. The method according to claim 4, characterized in that The obtaining of the low-precision model by the stochastic gradient descent method comprises: Quantizing the training data to obtain quantized training data and corresponding labels, and using the quantized training data to perform forward propagation and back propagation in sequence to obtain a locally updated low-precision model; Wherein, in the process of the forward propagation, the quantized training data is used as the input of the first layer of the neural network, the input of the current layer of the neural network is multiplied by the weight coefficient of the global integer model to obtain the output of the current layer, and then the output forward intermediate result is passed to the next layer of the neural network as the input, and the processing of each layer of the neural network is the same, and then the low-precision loss is calculated according to the forward intermediate result of the output of the label and the last layer of the neural network; During the back propagation process, the gradient g is calculated based on the low-precision loss A , using the quantized training data X and the gradient g as the input of the last layer in the neural network A Determine the model parameter update g of the last layer of neural network W =X*g A , using the model parameter update amount g W Update the weight coefficient W of the last layer of the global shaping model, using the gradient g A The weight coefficient W of the last layer of the global integer model determines the loss value g of the previous layer of the current network. X =g A *W, and then the loss value g X As the low-precision loss of the current layer, it is passed to the previous layer. The processing of each layer of the neural network is the same until the calculation of the parameters in each layer of the neural network is completed. By updating the model parameter g W The low-precision model is obtained by aggregation.

6. The method according to claim 5, characterized in that In the process of the forward propagation, before the step of multiplying the input of the current layer neural network by the weight coefficient of the global integer model, the input of the current layer neural network and the weight coefficient of the global integer model are transformed by shifting and numerical rounding operations to obtain a lower precision integer representation; During the back propagation process, the gradient g is shifted and rounded off. A Transform so that the gradient g A Represented as an integer with lower precision.

7. The method according to claim 1, characterized in that The S6 includes: S61, the server dequantizes the global integer model into a first floating point model; S62, the server dequantizes the low-precision model into a second floating-point model; S63, the server subtracts the corresponding weight coefficient in the first floating-point model from the weight coefficient in the second floating-point model to obtain a weight update amount of each terminal local training model, and obtains a weight update amount of the global floating-point model by weighted averaging based on the weight update amount; S64, the server adds the weight update amount of the global floating-point number model to the corresponding weight coefficient in the global floating-point number model to update the global floating-point number model.

8. A federated learning system with low-precision integer representation, characterized in that: include: The server is used to initialize the global floating point model and its training, and select at least one terminal; Also used for quantizing the initialized or updated global floating-point model into an integer global integer model, sending the global integer model to the terminal, dequantizing the global integer model and the low-precision model sent by the at least one terminal, updating the global floating-point model using the dequantized global integer model and the dequantized low-precision model, returning to perform the operation of quantizing the global floating-point model into an integer global integer model and subsequent operations thereof, until the global floating-point model converges or reaches a preset number of training times; At least one terminal is used to train the global integer model sent by the server using training data to obtain a low-precision integer model and send it to the server.

9. The system according to claim 8, characterized in that include: The server is used for quantizing the weight coefficients in the global floating point number model into integers.

10. The system according to claim 8, characterized in that The terminal obtains the low-precision model through stochastic gradient descent training.

11. The system according to claim 10, characterized in that The obtaining of the low-precision model by the stochastic gradient descent method comprises: Quantizing the training data to obtain quantized training data and corresponding labels, and using the quantized training data to perform forward propagation and back propagation in sequence to obtain a locally updated low-precision model; Wherein, in the process of the forward propagation, the quantized training data is used as the input of the first layer of the neural network, the input of the current layer of the neural network is multiplied by the weight coefficient of the global integer model to obtain the output of the current layer, and then the output forward intermediate result is passed to the next layer of the neural network as the input, and the processing of each layer of the neural network is the same, and then the low-precision loss is calculated according to the forward intermediate result of the output of the label and the last layer of the neural network; During the back propagation process, the gradient g is calculated based on the low-precision loss A , using the quantized training data X and the gradient g as the input of the last layer in the neural network A Determine the model parameter update g of the last layer of neural network W =X*g A , using the model parameter update amount g W Update the weight coefficient W of the last layer of the global shaping model, using the gradient g A The weight coefficient W of the last layer of the global integer model determines the loss value g of the previous layer of the current network. X =g A *W, and then the loss value g X As the low-precision loss of the current layer, it is passed to the previous layer. The processing of each layer of the neural network is the same until the calculation of the parameters in each layer of the neural network is completed. By updating the model parameter g W The low-precision model is obtained by aggregation.

12. The system according to claim 11, characterized in that In the process of the forward propagation, before the step of multiplying the input of the current layer neural network by the weight coefficient of the global integer model, the input of the current layer neural network and the weight coefficient of the global integer model are transformed by shifting and numerical rounding operations to obtain a lower precision integer representation; in the process of the backward propagation, the gradient g is transformed by shifting and numerical rounding operations. A Transform so that the gradient g A Represented as an integer with lower precision.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the federated learning method under low-precision integer representation as described in any one of claims 1 to 7 are implemented.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the federated learning method under low-precision integer representation as described in any one of claims 1 to 7 are implemented.

Images