Aggregation method based on heterogeneous federated learning

By setting batch size thresholds and dynamically adjusting the learning rate in heterogeneous federated learning, the problem of model training inconsistency caused by participants' computing power differences is solved, and faster model convergence and higher generalization performance are achieved, suitable for large Data sharing and privacy protection for large-scale distributed systems.

CN118966285AActive Publication Date: 2024-11-15SHAANXI NORMAL UNIV
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411058578.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-15
Estimated Expiration
2044-08-02

AI Technical Summary

Technical Problem

Differences in participants' computing power and network environment in heterogeneous federated learning lead to inconsistent model training time, affecting system timeliness and model accuracy. Existing methods such as Fedavg and Adapting Batch Size fail to effectively solve these problems.

Method used

By setting the batch size threshold in the central server, the participants are divided into different categories, and the gradient parameter magnification is calculated and applied for model aggregation, the learning rate is dynamically adjusted, the gradient aggregation strategy is optimized to adapt to the computing power of different participants, and the Alexnet network is adopted. Distributed training with the pytorch framework.

Benefits of technology

Accelerate the model training process, improve model convergence speed, improve generalization performance, optimize resource utilization, protect data privacy, and support large-scale deployment and diversity participants.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118966285A_ABST
    Figure CN118966285A_ABST
Patent Text Reader

Abstract

The invention discloses an aggregation method based on heterogeneous federated learning. The method comprises the steps of data preprocessing; constructing a network framework; an Alexnet network is constructed; the Alexnet network is trained; and the Alexnet network is tested. In the Alexnet network training process, through the optimized gradient aggregation strategy and dynamic adjustment of the learning rate, the training process of the model can be accelerated, the model is enabled to converge to an ideal state more quickly, data from different sources are allowed to participate in model training, a data island is broken, data sharing and utilization are promoted, and the training efficiency of the Alexnet network is improved. The appropriate batch size and learning rate can be allocated according to the actual computing power of the participant, the utilization of computing resources is optimized, and resource waste is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of federated learning, and specifically relates to an aggregation method based on heterogeneous federated learning. Background Art

[0002] With the rapid development of AI technology, the business volume in the industry is getting bigger and bigger. To cope with the increasing business volume, the performance of one machine is no longer enough. Multiple machines are needed to cope with large-scale application scenarios. This application scenario has prompted distributed learning to become the focus. Today, the development of distributed deep learning is increasingly widely used in real-world scenarios. We can buy a CPU at a low price and perform tasks on the terminal. It can not only share the computing pressure of the overall system, but also improve the throughput of the overall system. However, traditional distributed learning faces a series of serious problems such as data security and privacy protection. To solve this problem, we hope to obtain a feasible solution that allows the system to use their respective data more efficiently and accurately, and meet privacy protection and data security.

[0003] Thus, federated learning came into being. Federated learning is a process in which each participant can use the data of other parties to jointly model during machine learning. All parties do not need to share data resources, that is, the data does not leave the local area, and data joint training is carried out to establish a shared machine learning model. This also enables multiple participants to continue machine learning under the premise of protecting data privacy and meeting legal and compliance requirements, solving the problem of data islands.

[0004] However, federated learning also faces certain challenges. In the entire federated learning framework, we cannot formulate a unified standard to specify the computing power, network environment, transmission delay and other factors of the participants, which leads to the diversity of participants; this problem is called heterogeneous federated learning. Problems in heterogeneous federated learning include different network environments of different clients, which leads to large differences in the time of training models, which has a negative impact on the timeliness of the entire system; for example, different participants have different computing powers, and the accuracy of the models trained in the same time is not enough, which has a negative impact on the accuracy of the global model. The existing Fedavg method can efficiently aggregate the model parameters of all participants and obtain a more average global model, but it still does not consider the problem of heterogeneous participants. The current Adapting Batch Size (ABS) method is more concerned with completing the training of the global model on time and ignores the differences between heterogeneous participants, and has not achieved the desired effect. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide an aggregation method based on heterogeneous federated learning that narrows the gap between heterogeneous participants, can quickly converge the model, and improves generalization performance.

[0006] The technical solution adopted to solve the above technical problems is: an aggregation method based on heterogeneous federated learning, including the following steps:

[0007] Step 1. Data preprocessing

[0008] After obtaining and normalizing the data set, divide it into training set and test set in proportion;

[0009] Step 2. Build the network framework

[0010] Determine the number of distributed participants N of the network framework, and assign data from the training set to each participant based on the Non_IID assumption of data distribution;

[0011] Step 3. Build the Alexnet network

[0012] The Alexnet network model is composed of network block 1, network block 2, network block 3, network block 4, network block 5, and network block 6 connected in sequence. The network block 1, network block 2, and network block 5 are all composed of convolutional layers, activation function units, and maximum pooling layers connected in sequence; the network block 3 and network block 4 are all composed of convolutional layers and activation function units connected in sequence; the network block 6 is composed of 3 fully connected layers and 2 activation function units, and every two fully connected layers are connected through activation function units;

[0013] Step 4. Train the Alexnet network

[0014] Step 4.1. Each participant sends their own data into the Alexnet network for training to obtain their own network model. The initial learning rate of each participant is α;

[0015] Step 4.2. Each participant uploads the gradient parameters obtained during the training process to the central server. The central server aggregates the gradient parameters of all participants, calculates the global gradient, and updates the global model.

[0016] The method by which the central server aggregates the gradient parameters of the participants is:

[0017] The central server sets the batch size threshold, divides the participants whose training data volume > the batch size threshold into class A, and divides the participants whose training data volume ≤ the batch size threshold into class A. The average modulus length of the two types of participants is calculated according to the gradient parameters of the two types of participants, and the multiplier β = m is obtained. avgs / m avgb , m avgsis the average modulus length of all participants in class a, m avgb is the average modulus of all participants in class A; the gradient parameters of participants in class A are updated by multiplying them by the multiplication factor β, while the gradient parameters of participants in class A remain unchanged, thus completing the aggregation;

[0018] Step 4.3. The central server sends the updated global model as the initial model for the next training;

[0019] Step 4.4. Repeat steps 4.1 to 4.3 until the network model converges and save the parameters of the network model;

[0020] Step 5. Test the Alexnet network

[0021] Input the test set into the Alexnet network for testing, and calculate and record the accuracy ACC.

[0022] As a preferred technical solution, the formula for calculating the global gradient in step 4.2 is:

[0023]

[0024] In the formula, G g is the global gradient, g i is the gradient parameter of the i-th participant, i∈[1,N].

[0025] As a preferred technical solution, in step 1, the data set is an image data set or a speech data set or a text semantic data set.

[0026] As a preferred technical solution, in step 2, the network framework is a pytorch framework.

[0027] As a preferred technical solution, in step 4.1, the initial learning rate is α=0.01.

[0028] As a preferred technical solution, in step 4.1, the method by which each participant sends their respective data into the Alexnet network for training is a mini-batch gradient descent method.

[0029] The beneficial effects of the present invention are as follows:

[0030] The present invention can accelerate the model training process through optimized gradient aggregation strategy and dynamic adjustment of learning rate, making the model converge to the ideal state faster, and allowing data from different sources to participate in model training, breaking data silos, promoting data sharing and utilization, and being able to allocate appropriate batch size and learning rate according to the actual computing power of participants, thereby optimizing the utilization of computing resources and avoiding resource waste.

[0031] The distributed training method of the present invention reduces the dependence on the central server and improves the robustness and fault tolerance of the application system. Since the present invention does not require data to leave the local device, it protects the data privacy and compliance of participants while training the global model, allows a large number of participants to join, supports large-scale deployment, and is widely promoted and used in mobile device learning, healthcare, Internet of Things, smart manufacturing and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of the aggregation method based on heterogeneous federated learning of the present invention.

[0033] Figure 2 It is a schematic diagram of heterogeneous federated learning of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples, but the present invention is not limited to the following embodiments.

[0035] Example 1

[0036] exist Figure 1 , 2 The aggregation method based on heterogeneous federated learning in this embodiment includes the following steps:

[0037] Step 1. Data preprocessing

[0038] Take the CIFAR-10 dataset, which contains 60,000 images with a size of 32*32 pixels. Normalize the pixel values ​​of the CIFAR-10 dataset to [0,1] and divide it into a training set and a test set in a ratio of 5:1.

[0039] Step 2. Build the network framework

[0040] The network framework is determined to be the pytorch framework, the number of distributed participants in the network framework is N=10, and the data in the training set is allocated to each participant based on the Non_IID hypothesis of data distribution;

[0041] Step 3. Build the Alexnet network

[0042] The Alexnet network model is composed of network block 1, network block 2, network block 3, network block 4, network block 5, and network block 6 connected in sequence. The network block 1, network block 2, and network block 5 are all composed of convolutional layers, activation function units, and maximum pooling layers connected in sequence; the network block 3 and network block 4 are all composed of convolutional layers and activation function units connected in sequence; the network block 6 is composed of 3 fully connected layers and 2 activation function units, and every two fully connected layers are connected through activation function units;

[0043] Step 4. Train the Alexnet network

[0044] Step 4.1. Each participant sends their own data to the Alexnet network and performs local training according to the mini-batch gradient descent method to obtain their own network model. The initial learning rate α of the participants is 0.01, and the initial batch size is {B1, B2, B3…B 10}, where B1=B2=B3=B4=B5=128, B6=B7=B8=B9=B 10 =4, and the training ends after iteration e=5 rounds;

[0045] Step 4.2. Each participant uploads the gradient parameters obtained during the training process to the central server, and the central server aggregates the gradient parameters of the participants. Calculate the global gradient, G g is the global gradient, g i is the gradient parameter of the i-th participant, i∈[1,10], and updates the global model;

[0046] The method by which the central server aggregates the gradient parameters of all participants is:

[0047] The central server sets the batch size threshold, divides the participants whose training data volume > the batch size threshold into class A, and divides the participants whose training data volume ≤ the batch size threshold into class A. The average modulus length of the two types of participants is calculated according to the gradient parameters of the two types of participants, and the multiplier β = m is obtained. avgs / m avgb , m avgs is the average modulus length of all participants in class a, m avgb is the average modulus of all participants in class A; the gradient parameters of participants in class A are updated by multiplying them by the multiplication factor β, while the gradient parameters of participants in class A remain unchanged, thus completing the aggregation;

[0048] Step 4.3. The central server sends the updated global model as the initial model for the next training;

[0049] Step 4.4. Repeat steps 4.1 to 4.3 until the network model converges and save the parameters of the network model;

[0050] Step 5. Test the Alexnet network

[0051] Input the test set into the Alexnet network for testing, and calculate and record the accuracy ACC.

[0052] Example 2

[0053] The data set of the aggregation method based on heterogeneous federated learning in this embodiment is a speech data set, and the other steps are the same as those in Embodiment 1.

[0054] Example 3

[0055] The data set of the aggregation method based on heterogeneous federated learning in this embodiment is a text semantic data set, and the other steps are the same as those in Embodiment 1.

[0056] experiment

[0057] In order to verify the beneficial effects of the present invention, the inventors used the method of Example 1 to compare with the FedAvg method and the ABS method in the prior art, wherein the accuracy values ​​are shown in Table 1.

[0058] Table 1 Accuracy values ​​of Example 1, FedAvg method and ABS method

[0059] Test Method Evaluation function F1 (ACC value) Example 1 87.23% FedAvg Method 85.17% ABS Method 86.05%

[0060] As can be seen from Table 1, the accuracy ACC value of the aggregation method based on heterogeneous federated learning of the present invention is 2.42% higher than the accuracy ACC value of the FedAvg method, and 1.37% higher than the accuracy ACC value of the ABS method. This shows that the network model of the present invention has better generalization ability, can better capture the potential patterns and laws of data, and performs well in processing data and computing power differences of different participants.

Claims

1. An aggregation method based on heterogeneous federated learning, characterized in that: The following steps are involved: Step 1. Data preprocessing After obtaining and normalizing the data set, divide it into training set and test set in proportion; Step 2. Build the network framework Determine the number of distributed participants N in the network framework, and assign data from the training set to each participant based on the Non_IID assumption of data distribution; Step 3. Build the Alexnet network The Alexnet network model is composed of network block 1, network block 2, network block 3, network block 4, network block 5, and network block 6 connected in sequence. The network block 1, network block 2, and network block 5 are all composed of convolutional layers, activation function units, and maximum pooling layers connected in sequence; the network block 3 and network block 4 are all composed of convolutional layers and activation function units connected in sequence; the network block 6 is composed of 3 fully connected layers and 2 activation function units, and every two fully connected layers are connected through activation function units; Step 4. Train the Alexnet network Step 4.

1. Each participant sends their own data into the Alexnet network for training to obtain their own network model. The initial learning rate of each participant is α; Step 4.

2. Each participant uploads the gradient parameters obtained during the training process to the central server. The central server aggregates the gradient parameters of all participants, calculates the global gradient, and updates the global model. The method by which the central server aggregates the gradient parameters of the participants is: The central server sets the batch size threshold, divides the participants whose training data volume > the batch size threshold into class A, and divides the participants whose training data volume ≤ the batch size threshold into class A. The average modulus length of the two types of participants is calculated according to the gradient parameters of the two types of participants, and the multiplier β = m is obtained. avgs / m avgb , m avgs is the average modulus length of all participants in class a, m avgb is the average modulus of all participants in class A; the gradient parameters of participants in class A are updated by multiplying them by the multiplication factor β, while the gradient parameters of participants in class A remain unchanged, thus completing the aggregation; Step 4.

3. The central server sends the updated global model as the initial model for the next training; Step 4.

4. Repeat steps 4.1 to 4.3 until the network model converges and save the parameters of the network model; Step 5. Test the Alexnet network Input the test set into the Alexnet network for testing, and calculate and record the accuracy ACC.

2. The aggregation method based on heterogeneous federated learning according to claim 1 is characterized in that: The formula for calculating the global gradient in step 4.2 is: In the formula, G g is the global gradient, g i is the gradient parameter of the i-th participant, i∈[1,N].

3. The aggregation method based on heterogeneous federated learning according to claim 1 is characterized in that: In step 1, the data set is an image data set, a speech data set, or a text semantic data set.

4. The aggregation method based on heterogeneous federated learning according to claim 1 is characterized in that: In step 2, the network framework is the pytorch framework.

5. The aggregation method based on heterogeneous federated learning according to claim 1 is characterized in that: In step 4.1, the initial learning rate is α=0.

01.

6. The aggregation method based on heterogeneous federated learning according to claim 1 is characterized in that: In step 4.1, the method by which each participant sends their respective data into the Alexnet network for training is the mini-batch gradient descent method.

Citation Information

Patent Citations

  • Differentiated noise adding method and system in federated learning gradient exchange

    CN111260061A

  • Flow classification method and system based on federated learning

    CN111865815A

  • Federal learning classification model training method based on depth deterministic strategy gradient

    CN115062710A

  • Federal learning optimization method and system based on reinforcement learning

    CN116523079A

  • Edge federal learning training method and system for heterogeneous data

    CN116611535A