Aggregation method based on heterogeneous federated learning

Abstract

An aggregation method based on heterogeneous federated learning, comprising data preprocessing; constructing a network framework; constructing an Alexnet network; training the Alexnet network; and testing the Alexnet network. In the process of training the Alexnet network, the gradient aggregation strategy and the dynamic adjustment of the learning rate are optimized, the training process of the model is accelerated, the model converges to the ideal state faster, and different sources of data are allowed to participate in the model training, breaking the data island, promoting the sharing and utilization of data, and optimizing the utilization of computing resources by allocating appropriate batch size and learning rate according to the actual computing capacity of the participants, avoiding resource waste.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of federated learning, and particularly relates to an aggregation method based on heterogeneous federated learning. BACKGROUND

[0002] With the development of AI technology, the business volume in the industry is increasing, and the performance of a single machine cannot meet the increasing business volume. Multiple machines are needed to cope with large-scale application scenarios. Such application scenarios promote distributed learning to the focus. Nowadays, distributed deep learning is developing more and more widely used in real scenarios. We can buy a CPU at a low price and perform tasks on the terminal, which can share the computing pressure of the overall system and improve the throughput of the overall system. However, the traditional distributed learning faces serious problems of data security and privacy protection. To solve this problem, we hope to get a feasible solution that enables the system to use each other's data more efficiently and accurately, and meets the privacy protection and data security requirements.

[0003] Therefore, federated learning emerges as the times require. Federated learning is a process of joint modeling by using data of other parties in machine learning. Without sharing data resources, each party can perform joint training of data and establish a shared machine learning model under the premise of protecting data privacy and meeting legal and regulatory requirements. This also enables multiple participants to continue machine learning while protecting data privacy and meeting legal and regulatory requirements, and solves the data island problem.

[0004] However, federated learning also faces certain challenges. In the entire federated learning framework, we cannot develop a unified standard to regulate the computing power, network environment, transmission delay and other factors of the participants, which also causes the diversity of the participants. This problem is called heterogeneous federated learning. The problems existing in heterogeneous federated learning include that the network environments of different clients are different, causing large differences in training model time and negatively affecting the timeliness of the entire system; and different computing powers of different participants cause insufficient accuracy of the trained model in the same time, which negatively affects the accuracy of the global model. The existing Fedavg method can efficiently aggregate model parameters of all participants and obtain a relatively average global model, but it does not consider the problem of heterogeneous participants. The current ABS method mainly considers completing the training of the global model on time and ignores the differences between heterogeneous participants, and does not achieve the ideal effect. SUMMARY

[0005] The technical problems to be solved by the present application are to overcome the shortcomings of the prior art and provide an aggregation method based on heterogeneous federated learning, which can narrow the gap between heterogeneous participants, quickly converge the model and improve the generalization performance.

[0006] The technical solution adopted to solve the above technical problems is an aggregation method based on heterogeneous federated learning, comprising the following steps:

[0007] Step 1. Data preprocessing

[0008] After obtaining the data set and normalizing, divide it into training set and test set according to the proportion;

[0009] Step 2. Construct network framework

[0010] Determine the number N of distributed participants of the network framework, and allocate the data in the training set to each participant according to the data distribution Non_IID assumption;

[0011] Step 3. Construct Alexnet network

[0012] The Alexnet network model is composed of network block 1, network block 2, network block 3, network block 4, network block 5 and network block 6 connected in sequence, wherein the network block 1, network block 2 and network block 5 are each composed of a convolutional layer, an activation function unit and a maximum pooling layer connected in sequence; the network block 3 and the network block 4 are each composed of a convolutional layer and an activation function unit connected in sequence; the network block 6 is composed of three fully connected layers and two activation function units, and each two fully connected layers are connected through an activation function unit;

[0013] Step 4. Train Alexnet network

[0014] Step 4.1. Each participant sends the respective data into the Alexnet network for training to obtain the respective network model, and the initial learning rate of each participant is α;

[0015] Step 4.2. Each participant uploads the gradient parameters obtained in the training process to the center server, the center server aggregates the gradient parameters of all participants, calculates the global gradient, and updates the global model;

[0016] The method for the center server to aggregate the gradient parameters of all participants is:

[0017] The center server sets a batch size threshold, divides the participants with training data volume > batch size threshold into class A, and divides the participants with training data volume ≤ batch size threshold into class a, calculates the average modulus length of the two classes of participants according to the gradient parameters of the two classes of participants respectively, and obtains the ratio β = m avgs / m avgb , m avgsis the average modulus of all participants in class A, m avgb is the average modulus of all participants in class A; the gradient parameters of the participants in class A are updated by multiplying the gradient parameters of the participants in class A by the multiplier beta, the gradient parameters of the participants in class A remain unchanged, and the aggregation is completed;

[0018] Step 4.3. The central server issues the updated global model as the initial model for the next training.

[0019] Step 4.4. Repeat steps 4.1-4.3 until the network model converges, and save the parameters of the network model.

[0020] Step 5. Test the Alexnet network

[0021] Input the test set into the Alexnet network for testing, calculate and record the accuracy ACC.

[0022] As a preferred technical solution, the formula for calculating the global gradient in step 4.2 is:

[0023]

[0024] In the formula, G g is the global gradient, g i is the gradient parameter of the i-th participant, i∈[1,N].

[0025] As a preferred technical solution, in step 1, the data set is an image data set or a speech data set or a text semantic data set.

[0026] As a preferred technical solution, in step 2, the network framework is the pytorch framework.

[0027] As a preferred technical solution, in step 4.1, the initial learning rate is all alpha = 0.01.

[0028] As a preferred technical solution, in step 4.1, the method for each participant to input their respective data into the Alexnet network for training is the mini-batch gradient descent method.

[0029] The beneficial effects of the present application are as follows:

[0030] The present application can accelerate the training process of the model, make the model converge to the ideal state faster, and allow different sources of data to participate in model training, breaking the data silos, promoting the sharing and utilization of data, and optimizing the utilization of computing resources, avoiding resource waste.

[0031] The distributed training method of the application reduces the dependence on the central server, improves the robustness and fault tolerance of the application system, protects the data privacy and compliance of the participants while training the global model, allows a large number of participants to join, supports large-scale deployment, and is widely used in the fields of mobile device learning, medical care, Internet of Things, intelligent manufacturing, etc. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a flowchart of the aggregation method based on heterogeneous federated learning of the application.

[0033] Figure 2 is a schematic diagram of the heterogeneous federated learning of the application. DETAILED DESCRIPTION

[0034] The application will be further described in detail below in combination with the drawings and examples, but the application is not limited to the following embodiments.

[0035] Example 1

[0036] In the aggregation method based on heterogeneous federated learning of the present embodiment, the following steps are included: Figure 1 , 2 In the aggregation method based on heterogeneous federated learning of the present embodiment, the following steps are included:

[0037] Step 1. Data preprocessing

[0038] Take the CIFAR-10 dataset, which contains 60,000 pictures with a size of 32*32 pixels, normalize the pixel values of the CIFAR-10 dataset to [0, 1], and divide the dataset into a training set and a test set in a ratio of 5:1;

[0039] Step 2. Construct network framework

[0040] Determine the network framework as pytorch framework, the number of distributed participants of the network framework N = 10, and according to the data distribution Non_IID hypothesis, allocate data in the training set to each participant;

[0041] Step 3. Construct Alexnet network

[0042] The Alexnet network model is composed of network block 1, network block 2, network block 3, network block 4, network block 5, and network block 6 connected in sequence, wherein the network block 1, the network block 2, and the network block 5 are each composed of a convolutional layer, an activation function unit, and a max-pooling layer connected in sequence; the network block 3 and the network block 4 are each composed of a convolutional layer and an activation function unit connected in sequence; and the network block 6 is composed of three fully connected layers and two activation function units, and each two fully connected layers are connected through an activation function unit;

[0043] Step 4. Training the Alexnet network

[0044] Step 4.1. Each participant inputs respective data into the Alexnet network and performs local training according to the mini-batch gradient descent method to obtain respective network models, and the initial learning rate a of each participant is 0.01, and the initial batch size is {B1, B2, B3…B 10}, wherein B1=B2=B3=B4=B5=128, B6=B7=B8=B9=B 10 10=4, and the training ends after 5 iterations e.

[0045] Step 4.2. Each participant uploads the gradient parameters obtained during the training process to the central server, the central server aggregates the gradient parameters of all participants, and calculates the global gradient G according to g G i is the global gradient, g i is the gradient parameter of the i-th participant, i∈[1,10], and the global model is updated.

[0046] The method for the central server to aggregate the gradient parameters of all participants is as follows:

[0047] The central server sets a batch size threshold, divides the participants whose training data volume is greater than the batch size threshold into class A, and divides the participants whose training data volume is less than or equal to the batch size threshold into class a, calculates the average module length of the participants in the two classes respectively according to the gradient parameters of the participants in the two classes, and obtains a rate β=m avgs / m avgb , m avgs is the average module length of all participants in class a, and m avgb is the average module length of all participants in class A; the gradient parameters of the participants in class A are updated by multiplying the gradient parameters of the participants in class A by the rate β, the gradient parameters of the participants in class a remain unchanged, and the aggregation is completed.

[0048] Step 4.3. The central server issues the updated global model as the initial model for the next training.

[0049] Step 4.4. Repeat steps 4.1-4.3 until the network model converges, and save the parameters of the network model.

[0050] Step 5. Testing the Alexnet network

[0051] Input the test set into the Alexnet network for testing, calculate and record the accuracy ACC.

[0052] Example 2

[0053] The data set of the aggregation method based on the heterogeneous federated learning in this embodiment is a voice data set, and other steps are the same as those in Embodiment 1.

[0054] Embodiment 3

[0055] The data set of the aggregation method based on the heterogeneous federated learning in this embodiment is a text semantic data set, and other steps are the same as those in Embodiment 1.

[0056] Experiment

[0057] In order to verify the beneficial effects of the present application, the inventors compared the method of Embodiment 1 with the prior art FedAvg method and ABS method, wherein the accuracy values are as shown in Table 1.

[0058] Table 1 Accuracy values of Embodiment 1, FedAvg method and ABS method

[0059] Test method Evaluation function F1 (ACC value) Example 1 87.23％ FedAvg method 85.17％ ABS method 86.05％

[0060] As can be seen from Table 1, the accuracy ACC value of the aggregation method based on the heterogeneous federated learning of the present application is 2.42% higher than the accuracy ACC value of the FedAvg method and 1.37% higher than the accuracy ACC value of the ABS method. It shows that the network model of the present application has better generalization ability and can better capture the potential patterns and rules of data, and performs well in processing the data and computing power differences of different participants.

Claims

1. An aggregation method based on heterogeneous federated learning, characterized in that, The method comprises the following steps: Step 1. Data preprocessing After obtaining the data set and normalizing, the data set is divided into a training set and a test set in proportion; Step 2. Building a network framework Determine the number N of distributed participants of the network framework, and distribute the data in the training set to each participant according to the Non-IID assumption of the data distribution; Step 3. Building an Alexnet network The Alexnet network model is composed of network block 1, network block 2, network block 3, network block 4, network block 5 and network block 6 connected in sequence, wherein the network block 1 and the network block 2 and the network block 5 are each composed of a convolutional layer, an activation function unit and a maximum pooling layer connected in sequence; the network block 3 and the network block 4 are each composed of a convolutional layer and an activation function unit connected in sequence; the network block 6 is composed of three fully connected layers and two activation function units, and each two fully connected layers are connected through an activation function unit; Step 4. Training the Alexnet network Step 4.

1. Each participant inputs the respective data into the Alexnet network for training to obtain a respective network model, and the initial learning rate of each participant is α; Step 4.

2. Each participant uploads the gradient parameters obtained in the training process to the center server, the center server aggregates the gradient parameters of all participants, calculates the global gradient, and updates the global model; The method for the center server to aggregate the gradient parameters of all participants is: The center server sets a batch size threshold, divides participants with training data amount > batch size threshold into class A, divides participants with training data amount ≤ batch size threshold into class a, respectively calculates average module length of participants in the two classes according to gradient parameters of the participants in the two classes, and obtains a ratio β = m avgs / m avgb , m avgs is average module length of all participants in class a, m avgb is average module length of all participants in class A; the gradient parameters of participants in class A are updated by multiplying the gradient parameters of participants in class A by the ratio β, the gradient parameters of participants in class a remain unchanged, and aggregation is completed. Step 4.

3. The center server issues the updated global model as the initial model for the next training; Step 4.

4. Repeat steps 4.1-4.3 until the network model converges, and save the parameters of the network model; Step 5. Test the Alexnet network The test set is input into the Alexnet network for testing, and the accuracy ACC is calculated and recorded.

2. The aggregation method based on heterogeneous federated learning according to claim 1, characterized in that, The formula for calculating the global gradient in step 4.2 is: where G g is the global gradient, g i is the gradient parameter of the i-th participant, i ∈ [1, N].

3. The aggregation method based on heterogeneous federated learning according to claim 1, characterized in that: In step 1, the data set is an image data set or a voice data set or a text semantic data set.

4. The aggregation method based on heterogeneous federated learning according to claim 1, characterized in that: In step 2, the network framework is a pytorch framework.

5. The aggregation method based on heterogeneous federated learning according to claim 1, characterized in that: In step 4.1, the initial learning rate α is 0.

01.

6. The aggregation method based on heterogeneous federated learning according to claim 1, characterized in that: In step 4.1, the method for each participant to input the respective data into the Alexnet network for training is a mini-batch gradient descent method.

Images