A federated training acceleration method based on an unreliable transmission protocol

Abstract

The application relates to the technical field of machine learning and computer networks, and discloses a federated training acceleration method based on an unreliable transmission protocol, which solves how to maximize the receiving rate of a complete layer under uncertain packet loss by designing a congestion feedback-based unreliable layer transmission algorithm; and guarantees the model precision of the federated training under parameter loss by designing a federated freezing training framework based on layer parameters. The federated training acceleration method based on the unreliable transmission protocol designs a wearable federated training acceleration technology based on the unreliable transmission protocol. The key idea is to directly ignore the network-side lost parameters instead of performing complex detection and retransmission operations on the network transmission protocol, so that the end-to-end network throughput is greatly improved; then, based on the parameter redundancy characteristics of machine learning training, the receiving end replaces the network packet loss with the previous round of parameters, and adopts a layer freezing training technology to realize high-precision training of the federated learning under parameter loss.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical fields of machine learning and computer network, in particular to a federated training acceleration method based on unreliable transmission protocol. BACKGROUND

[0002] In recent years, with the development of wearable technology, people can track their activities through wearable devices such as smart phones, wristbands and smart patches to understand their health status. However, due to the privacy problem caused by the centralized storage of personal health data on the cloud server, the willingness of users to share health data is greatly reduced. In particular, recent privacy protection regulations have increased the difficulty of learning artificial intelligence models by centralizing user data on cloud servers. Federated learning (FL) is currently one of the most effective methods to achieve privacy-preserving machine learning across a large number of mobile devices. Applying FL on a large number of wearable devices can achieve accurate and timely health analysis without sharing sensitive health data. First, wearable devices achieve efficient device-side training by offloading training tasks to paired smartphones. Then, the global model parameters are aggregated through the cloud server to obtain a high-precision model. Finally, a lightweight model is placed on the phone or wearable device for accurate and timely inference.

[0003] However, the present inventors have found that the cloud collaborative training framework is limited by the uncertainty of packet loss in mobile networks, which causes the following serious problems: First, the upload bandwidth of mobile networks is often very low, which prolongs the convergence time of federated training, for example: the upload bandwidth of 4G cellular networks is usually less than 10 Mbps, and it takes more than a day to train a lightweight MobileNet; second, existing communication bottleneck optimization techniques are often based on the unreasonable assumption of fixed bandwidth, using parameter compression or training optimization methods, which have a certain degree of loss of training accuracy. SUMMARY

[0004] (I) Technical problems solved

[0005] In view of the deficiencies in the prior art, the present application provides a federated training acceleration method based on unreliable transmission protocol, which has the advantages of quickly obtaining high-precision models for user health and safety, and solves the problems raised in the above background technology.

[0006] (II) Technical solutions

[0007] The present application provides the following technical solutions: a federated training acceleration method based on unreliable transmission protocol, which solves how to maximize the reception rate of complete layers under uncertain packet loss by designing a congestion feedback-based unreliable layer transmission algorithm; secondly, by designing a federated frozen training framework based on layer parameters, the model accuracy of federated training is guaranteed without loss under parameter loss.

[0008] Mainly comprising the following steps:

[0009] S1, server initialization;

[0010] S2, server model parameter distribution;

[0011] S3, terminal layer frozen training local model;

[0012] S4, terminal model parameter upload update;

[0013] S5, server aggregation terminal model to obtain global updated model.

[0014] Preferably, the server initialization:

[0015] The cloud server first randomly initializes a global floating point model for a certain federated learning task, and the model is updated by the cloud and the terminal participating in federated learning training;

[0016] The cloud server randomly selects the terminal participating in federated training according to the strategy of traditional federated learning;

[0017] The cloud server specifies unified parameters for model training on each device, such as batch size, local epochs and learning rate.

[0018] Preferably, the server distributes the model parameters:

[0019] The cloud server detects the network congestion state based on the delay gradient and packet loss rate, and obtains the current state: S1 congestion occurs and lasts for a period of time, S2 congestion does not occur and will not occur in a short time.

[0020] The cloud server caches the model parameters according to the layer structure and data size information in the sending queue.

[0021] The transmission controller of the cloud server selects the layer with the most matched data size according to the current congestion state: if the congestion state is S1, the layer with larger data is sent; if the congestion state is S2, the layer with smaller data is sent.

[0022] Preferably, the terminal layer freezes the training local model:

[0023] The terminal receives the layer distributed by the cloud server, and since in the network condition with data loss, the terminal can only receive part of the complete layer, the parameters of the corresponding position in the last round of training are used to fill the missing layer;

[0024] After the missing layers are filled, these parameter layers are frozen, meaning they only participate in the forward propagation process; the parameters of these layers are not updated during the backpropagation process to prevent these old parameters from affecting the global model update.

[0025] Preferably, the terminal uploads model parameter updates:

[0026] After training is complete, the terminal sends the updated parameters of this round to the server using the same congestion feedback layer transport protocol. These parameters only include the layers updated during backpropagation in training.

[0027] Preferably, the server aggregation terminal model obtains a globally updated model:

[0028] The server checks the layer information of the received terminal parameters and aggregates and updates the overlapping layer parameters between the terminal parameters using a traditional weighted average method.

[0029] The layer parameters lost by the terminal are not included in the global model update. However, if all parameters for a layer are lost, the server will replace them with the parameters from the previous update.

[0030] Compared with existing technologies, this invention provides a federated training acceleration method based on an unreliable transmission protocol, which has the following beneficial effects:

[0031] 1. This paper presents a wearable federated training acceleration method based on an unreliable transmission protocol. The key idea is to directly ignore the parameters lost on the network side, rather than performing complex detection and retransmission operations on the network transmission protocol, thereby significantly improving end-to-end network throughput. Then, based on the parameter redundancy characteristics of machine learning training, the receiving end replaces network packet loss with parameters from the previous round, and employs a layer-freezing training technique to achieve high-precision training of federated learning even with parameter loss.

[0032] 2. This federated training acceleration method based on unreliable transmission protocols develops an efficient layer-freezing federated training system and effectively solves the following two core technical challenges: First, by designing an unreliable layer transmission algorithm based on congestion feedback, it addresses how to maximize the reception rate of the complete layer under uncertain packet loss; second, by designing a federated freezing training framework based on layer parameters, it ensures that the accuracy of the federated training model remains unaffected even with parameter loss. Attached Figure Description

[0033] Fig. 1 This is a schematic diagram of the wearable federated training framework based on an unreliable transmission protocol according to the present invention.

[0034] Fig. 2 This is a schematic diagram of the layer parameter transmission protocol for congestion feedback in this invention;

[0035] Fig. 3 Flowchart of the wearable federated training algorithm based on the non-reliable transmission protocol of the present application. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0037] Please refer to Figs. 1-3 The present application designs a wearable federated training acceleration technology based on a non-reliable transmission protocol. The key idea is to directly ignore these network-side lost parameters, instead of complex detection and retransmission operations on the network transmission protocol, thereby greatly improving the end-to-end network throughput; then, based on the parameter redundancy characteristics of machine learning training, the receiving end replaces the network packet loss with the previous round of parameters, and uses the layer freezing training technology to realize high-precision training of federated learning under parameter loss. In addition, the present application develops an efficient layer freezing federated training system, and practically solves the following two core technical challenges: first, by designing a non-reliable layer transmission algorithm based on congestion feedback, how to maximize the reception rate of complete layers under uncertain packet loss is solved; second, by designing a federated freezing training framework based on layer parameters, the model accuracy of federated training under parameter loss is guaranteed without loss.

[0038] Our system mainly contains the following steps: (1) server initialization, (2) server model parameter distribution, (3) terminal layer freezing training local model, (4) terminal model parameter upload update, (5) server aggregation of terminal model to obtain global updated model.

[0039] 1. Server initialization:

[0040] 1.1) The cloud server first randomly initializes a global floating-point number model for a certain federated learning task, and this model is updated by the cloud and the terminals participating in federated learning training.

[0041] 1.2) The cloud server randomly selects terminals participating in federated training in accordance with the strategy of traditional federated learning.

[0042] 1.3) The cloud server specifies uniform parameters for model training on each device, such as batch size, local epochs, and learning rate, etc.

[0043] 2. Server model parameter distribution:

[0044] 2.1) Cloud server detects network congestion state based on delay gradient and packet loss rate, and obtains the current state: S1 congestion occurs and lasts for a period of time, S2 congestion does not occur and will not occur in a short time.

[0045] 2.2) Cloud server caches model parameters according to the structure of each layer and its data size information in the sending queue.

[0046] 2.3) The transmission controller of the cloud server selects the layer with the most matched data size according to the current congestion state: if the congestion state is S1, send the layer with larger data size; if the congestion state is S2, send the layer with smaller data size.

[0047] 3. Freeze the local model of the terminal layer:

[0048] 3.1) The terminal receives the layer sent by the cloud server. Due to the network condition with data loss, the terminal can only receive part of the complete layer, and the parameters of the corresponding position in the last round of training are used to fill in the missing layer.

[0049] 3.2) After filling in the missing layer, these parameter layers are frozen, that is, they only participate in the forward propagation process. The parameters of these layers will not be updated in the back propagation process to prevent these old parameters from affecting the global model update.

[0050] 4. Terminal uploads model parameter updates:

[0051] 4.1) After training, the terminal sends the updated parameters of this round to the server using the same congestion feedback layer transmission protocol as 2.3), and these parameters only include the layers updated by back propagation in 3.2).

[0052] 5. Server aggregates terminal model to obtain globally updated model:

[0053] 5.1) The server checks the layer information of the received terminal parameters, and updates the cross-layer parameters between the terminal parameters according to the traditional weighted average method.

[0054] 5.2) The terminal missing layer parameters do not participate in updating the global model, but if all parameters of this layer are missing, the server replaces them with the last round of updated parameters of this layer.

Claims

1. A federated training acceleration method based on a non-reliable transport protocol, characterized in that: The first is to solve how to maximize the reception rate of complete layers under uncertain packet loss by designing a non-reliable layer transmission algorithm based on congestion feedback; the second is to ensure the model accuracy of federated training without loss under parameter loss by designing a federated frozen training framework based on layer parameters. The method mainly comprises the following steps: S1, server initialization; S2, server model parameter distribution; S3, terminal layer frozen training of local model; S4, terminal model parameter uploading; S5, server aggregation of terminal model to obtain globally updated model; Server initialization: The cloud server first randomly initializes a global floating-point model for a certain federated learning task, which is updated by the cloud and terminals participating in federated learning training; The cloud server randomly selects terminals participating in federated training according to the traditional federated learning strategy; The cloud server specifies unified parameters for model training on each device; Server model parameter distribution: The cloud server detects the network congestion state based on the delay gradient and packet loss rate to obtain the current state: S1, congestion occurs and lasts for a period of time, S2, no congestion occurs and will not occur in a short period of time; The cloud server caches the model parameters according to the layer structure and data size information in the sending queue; The cloud server's transmission controller selects the layer with the most matched data size according to the current congestion state: if the congestion state is S1, the layer with larger data size is sent; if the congestion state is S2, the layer with smaller data size is sent; Terminal layer frozen training of local model: The terminal receives the layer distributed by the cloud server, and since it can only receive part of the complete layer under the network condition with data loss, the terminal fills in the missing layer using the parameters of the corresponding position in the last round of training; After filling in the missing layer, these parameter layers are frozen, that is, they only participate in the forward propagation process; these layers of parameters are not updated in the backward propagation process to prevent these old parameters from affecting the global model update.

2. The federated training acceleration method based on an unreliable transport protocol according to claim 1, wherein: Terminal model parameter uploading: After training, the terminal sends the updated parameters of this round to the server using the same congestion feedback layer transmission protocol, and these parameters only include the layers updated by the backward propagation in the training.

3. The federated training acceleration method based on an unreliable transport protocol according to claim 1, wherein: Server aggregation of terminal model to obtain globally updated model: The server checks the layer information of the received terminal parameters, and updates the cross-layer parameters between the terminal parameters according to the traditional weighted average method; The terminal's missing layer parameters do not participate in updating the global model, but if all parameters of this layer are missing, the server replaces them with the parameters of the last round of updates.

Images