Efficient segmentation federated edge learning method combining equipment clustering and advanced gradient aggregation, program, equipment and storage medium

Abstract

The invention belongs to the technical field of segmentation federated learning, and particularly relates to an efficient segmentation federated edge learning method combining equipment clustering and advanced gradient aggregation, a program, equipment and a storage medium. According to the model segmentation algorithm based on equipment clustering designed by the invention, the equipment with similar capabilities is allocated to the same group, and all the equipment in the group adopt the same model segmentation strategy, so that the intermediate server only needs to store one server-side model, and the load of the intermediate server is reduced. According to the local model aggregation algorithm based on gradient aggregation in advance designed by the invention, the intermediate server only needs to store one server-side model, so that the memory consumption is greatly reduced, the communication delay caused by equipment instability and model synchronization between equipment is relieved, and the model training efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of segmented federated learning, and specifically relates to an efficient segmented federated edge learning method, program, device and storage medium combining device clustering and early gradient aggregation. Background Art

[0002] In recent years, with the widespread application of technologies such as the Internet of Things and 5G, more and more lightweight smart terminal devices are connected to the Internet and generate a large amount of data. How to efficiently process this data and realize cross-device edge collaborative computing has become an important challenge. Split federated learning combines the advantages of split learning and federated learning, and can achieve efficient collaborative training between resource-constrained devices. The hierarchical learning architecture based on end-edge-cloud introduces an intermediate server layer to reduce the training delay caused by the frequent exchange of model segmentation layer results between devices and remote cloud servers in the end-cloud architecture. However, the heterogeneity of resources between devices and device instability (such as battery exhaustion, network disconnection, etc.) bring new challenges to the hierarchical learning architecture. Summary of the invention

[0003] The object of the present invention is to provide an efficient segmentation federated edge learning method, program, device and storage medium combining device clustering and early gradient aggregation.

[0004] An efficient segmentation federated edge learning method combining device clustering and early gradient aggregation includes the following steps:

[0005] Step 1: The central server executes the model segmentation algorithm based on device clustering, groups all devices, and formulates an adaptive model segmentation strategy for each group;

[0006] The central server is connected to multiple groups of intermediate servers, each intermediate server is connected to multiple groups of devices, and each group of devices is only connected to one group of intermediate servers;

[0007] Step 2: Each device uses local data to perform client model forward propagation and uploads the model segmentation layer activation values ​​and corresponding labels to the intermediate server connected to it;

[0008] Step 3: Each intermediate server executes the local model aggregation algorithm based on the advance gradient aggregation, updates the server-side model, and returns the aggregated gradient to each device connected to it;

[0009] Step 4: Each device uses the aggregated gradient to perform client model backpropagation;

[0010] Step 5: After each device and the intermediate server complete the traversal of local data, they upload their own models to the central server;

[0011] Step 6: The central server performs global average aggregation on the models uploaded by all devices and intermediate servers to obtain a global model; and divides the global model according to the model segmentation strategy of each group, and returns the corresponding model part to each device and intermediate server;

[0012] Step 7: All devices and intermediate servers update their own models based on the partial global model returned by the central server;

[0013] Step 8: Repeat steps 2 to 7 until the global model converges, completing this round of federated learning.

[0014] Furthermore, in step 1, the central server executes a model segmentation algorithm based on device clustering, specifically:

[0015] Step 1.1: The central server executes the K-Means clustering algorithm based on the device capabilities to group devices with similar performance into the same group;

[0016] Step 1.2: For each group, the central server randomly selects a device, performs model pre-training on the segmented models according to the model segmentation strategy, obtains the model pre-training time and load corresponding to each model segmentation strategy; selects the model segmentation strategy with the smallest time consumption and load as the model segmentation strategy for the group;

[0017] Step 1.3: Repeat step 1.2 until an adapted model segmentation strategy is developed for each group.

[0018] Furthermore, in step 3, the intermediate server executes a local model aggregation algorithm based on early gradient aggregation, specifically:

[0019] Step 3.1: The intermediate server receives and concatenates the model segmentation layer activation values ​​uploaded by the connected devices within the aggregation time window to generate an input activation matrix;

[0020] Step 3.2: The intermediate server performs forward and backward propagation of the server-side model according to the input activation matrix, calculates the server-side model gradient, matches the corresponding gradient according to the model segmentation layer activation value uploaded by each device, and performs gradient average aggregation to obtain the aggregated gradient;

[0021] Step 3.3: The intermediate server uses the aggregated gradient to update the server-side model and returns it to each device connected to it, while dynamically adjusting the aggregation time window in the next iterative calculation.

[0022] Furthermore, in step 3.1, the activation matrix is ​​input for:

[0023]

[0024] in, Indicates the activation value of the model segmentation layer uploaded by the i-th group of devices connected to the m-th group of intermediate servers in the t-th iteration calculation of this round of federated learning, i=1,2,...,I m , I m Indicates the total number of devices connected to the mth group of intermediate servers.

[0025] Furthermore, the server-side model gradient in step 3.2 for:

[0026]

[0027] in, It represents the matrix composed of the training data labels of each device connected to the mth group of intermediate servers in the tth iteration calculation of this round of federated learning; It represents the predicted value obtained by the server-side model training of the mth group of intermediate servers in the tth iteration calculation of this round of federated learning; It represents the weight of the server-side model of the mth group of intermediate servers in the tth iteration calculation of this round of federated learning.

[0028] Furthermore, in step 3.2, the corresponding gradient is matched according to the activation value of the model segmentation layer uploaded by each device, which is specifically expressed as:

[0029]

[0030] in, It represents the gradient assigned to the i-th device connected to the m-th group of intermediate servers in the t-th iteration calculation of this round of federated learning;

[0031] Perform gradient average aggregation to obtain the aggregated gradient

[0032]

[0033] Furthermore, the method for dynamically adjusting the aggregation time window in the next iterative calculation in step 3.3 is specifically as follows:

[0034]

[0035] in, It represents the waiting time of the aggregated time window of the mth group of intermediate servers in the t+1th iteration calculation of this round of federated learning; It represents the time taken by the i-th group of devices connected to the m-th group of intermediate servers to upload the activation values ​​of the model segmentation layer in the t-th iteration calculation of this round of federated learning; 0≤β≤1, which is used to weigh the longest upload time and the average upload time. When β=1, the m-th group of intermediate servers will wait for all devices to upload the activation values ​​of the model segmentation layer before performing server-side model training.

[0036] A computer device / equipment / system comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned efficient segmentation federated edge learning method combining device clustering and early gradient aggregation.

[0037] A computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of the above-mentioned efficient segmentation federated edge learning method combining device clustering and early gradient aggregation.

[0038] A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned efficient segmentation federated edge learning method combining device clustering and early gradient aggregation.

[0039] The beneficial effects of the present invention are:

[0040] The model segmentation algorithm based on device clustering designed by the present invention allocates devices with similar capabilities to the same group, and all devices in the group adopt the same model segmentation strategy, so that the intermediate server only needs to save one server-side model, reducing the load on the intermediate server. The local model aggregation algorithm based on early gradient aggregation designed by the present invention enables the intermediate server to only save one server-side model, thereby greatly reducing memory consumption, alleviating communication delays caused by device instability and model synchronization between devices, and thus improving model training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a framework diagram of the present invention.

[0042] Figure 2 This is a comparison result diagram of the model accuracy of the present invention, Hier-SFL, Hier-RSFL and Hier-PSL.

[0043] Figure 3 This is a comparison result of the time required for one round of model training of the present invention, Hier-SFL, Hier-RSFL and Hier-PSL.

[0044] Figure 4 This is a comparison result of the time required for the present invention to achieve the target accuracy with Hier-SFL, Hier-RSFL and Hier-PSL.

[0045] Figure 5 This is a comparison chart of memory usage between the present invention and Hier-SFL during model training.

[0046] Figure 6 It is a pseudo code diagram of the model segmentation algorithm based on device clustering in the present invention.

[0047] Figure 7 It is a pseudo code diagram of the local model aggregation algorithm based on early gradient aggregation in the present invention. DETAILED DESCRIPTION

[0048] The present invention is further described below in conjunction with the accompanying drawings.

[0049] The present invention relates to an efficient segmented federated edge learning method, program, device and storage medium that combines device clustering and advance gradient aggregation in edge learning, in combination with the currently widely used segmented federated learning. The present invention first assigns devices with similar capabilities to the same group, and all devices in the group adopt the same model segmentation strategy, so that the intermediate server only needs to save one server-side model, reducing the load of the intermediate server; then, the intermediate server adopts the advance gradient aggregation method to alleviate the training delay caused by device instability and model synchronization between devices, thereby improving the efficiency of model training.

[0050] An efficient segmentation federated edge learning method combining device clustering and early gradient aggregation includes the following steps:

[0051] Step 1: The central server executes the model segmentation algorithm based on device clustering, groups all devices, and formulates an adaptive model segmentation strategy for each group;

[0052] The central server is connected to multiple groups of intermediate servers, each intermediate server is connected to multiple groups of devices, and each group of devices is only connected to one group of intermediate servers;

[0053] Step 1.1: The central server executes the K-Means clustering algorithm based on the device capabilities to group devices with similar performance into the same group;

[0054] Step 1.2: For each group, the central server randomly selects a device, performs model pre-training on the segmented models according to the model segmentation strategy, obtains the model pre-training time and load corresponding to each model segmentation strategy; selects the model segmentation strategy with the smallest time consumption and load as the model segmentation strategy for the group;

[0055] Step 1.3: Repeat step 1.2 until an adapted model segmentation strategy is developed for each group;

[0056] Step 2: Each device uses local data to perform client model forward propagation and uploads the model segmentation layer activation values ​​and corresponding labels to the intermediate server connected to it;

[0057] Step 3: Each intermediate server executes the local model aggregation algorithm based on the advance gradient aggregation, updates the server-side model, and returns the aggregated gradient to each device connected to it;

[0058] Step 3.1: The intermediate server receives and concatenates the model segmentation layer activation values ​​uploaded by the connected devices within the aggregation time window to generate the input activation matrix

[0059]

[0060] in, Indicates the activation value of the model segmentation layer uploaded by the i-th group of devices connected to the m-th group of intermediate servers in the t-th iteration calculation of this round of federated learning, i=1,2,...,I m , I m Indicates the total number of devices connected to the mth group of intermediate servers;

[0061] Step 3.2: The intermediate server performs forward and backward propagation of the server-side model based on the input activation matrix and calculates the server-side model gradient Match the corresponding gradient according to the model segmentation layer activation value uploaded by each device And perform gradient average aggregation to get the aggregate gradient

[0062]

[0063] in, It represents the matrix composed of the training data labels of each device connected to the mth group of intermediate servers in the tth iteration calculation of this round of federated learning; It represents the predicted value obtained by the server-side model training of the mth group of intermediate servers in the tth iteration calculation of this round of federated learning; represents the weight of the server-side model of the mth group of intermediate servers in the tth iteration calculation of this round of federated learning;

[0064]

[0065] in, It represents the gradient allocated to the i-th group of devices connected to the m-th group of intermediate servers in the t-th iteration calculation of this round of federated learning;

[0066] Perform gradient average aggregation to obtain the aggregated gradient

[0067]

[0068] Step 3.3: The intermediate server uses the aggregated gradient to update the server-side model and returns it to each device connected to it, while dynamically adjusting the aggregation time window in the next iterative calculation;

[0069]

[0070] in, It represents the waiting time of the aggregated time window of the mth group of intermediate servers in the t+1th iteration calculation of this round of federated learning; Indicates the time it takes for the i-th group of devices connected to the m-th group of intermediate servers to upload the model segmentation layer activation values ​​in the t-th iteration calculation of this round of federated learning; 0≤β≤1, which is used to weigh the longest upload time and the average upload time. When β=1, the m-th group of intermediate servers will wait for all devices to upload the model segmentation layer activation values ​​before performing server-side model training;

[0071] Step 4: Each device uses the aggregated gradient to perform client model backpropagation;

[0072] Step 5: After each device and the intermediate server complete the traversal of local data, they upload their own models to the central server;

[0073] Step 6: The central server performs global average aggregation on the models uploaded by all devices and intermediate servers to obtain a global model; and divides the global model according to the model segmentation strategy of each group, and returns the corresponding model part to each device and intermediate server;

[0074] Step 7: All devices and intermediate servers update their own models based on the partial global model returned by the central server;

[0075] Step 8: Repeat steps 2 to 7 until the global model converges, completing this round of federated learning.

[0076] The overall framework diagram of the present invention is as follows: Figure 1 The present invention is compared with a partitioned federated learning algorithm (Hier-SFL) that does not consider device heterogeneity, a partitioned federated learning algorithm (Hier-RSFL) that considers device heterogeneity, and a parallel partitioned federated learning algorithm (Hier-PSL) that considers training efficiency, and applies these methods to a hierarchical architecture.

[0077] In order to verify the effectiveness of the present invention, the present invention compares the model accuracy with the above three methods under the same model and data conditions. The experimental results are as follows: Figure 2The present invention can achieve a model accuracy similar to that of other methods, while significantly fewer rounds are required for model convergence.

[0078] In order to compare the time required for different model training methods to complete a round of model training, each method performs 10 rounds of model training and repeats 10 times to take the average value to ensure the accuracy of the experimental results. The experimental results are shown in Figure 3 The average time for the present invention to complete a round of model training is 25.44s, while Hier-SFL, Hier-RSFL and Hier-PSL require 41.47s, 37.97s and 36.76s respectively.

[0079] To further verify the effectiveness of the present invention in improving model training efficiency, the time required for each method to reach the target accuracy was recorded. The experimental results are as follows: Figure 4 As shown. The present invention takes 190.88 seconds to reach the target accuracy, while Hier-SFL, Hier-RSFL and Hier-PSL take 272.79 seconds, 257.01 seconds and 255.47 seconds respectively. This is because the present invention effectively reduces the communication delay caused by device heterogeneity and client model synchronization through device similarity clustering and gradient aggregation method, thereby improving the model training efficiency.

[0080] In order to verify the effectiveness of the present invention in reducing the memory usage of the intermediate server, the present invention is compared with Hier-SFL in performing 10 rounds of model training with the same number of clients. The experimental results are as follows: Figure 5 The memory usage rate of the intermediate server of the present invention is much lower than that of Hier-SFL, because the intermediate server of the present invention only needs to save one server-side model, thereby greatly reducing memory consumption.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An efficient segmentation federated edge learning method combining device clustering and early gradient aggregation, characterized in that: The following steps are involved: Step 1: The central server executes the model segmentation algorithm based on device clustering, groups all devices, and formulates an adaptive model segmentation strategy for each group; The central server is connected to multiple groups of intermediate servers, each intermediate server is connected to multiple groups of devices, and each group of devices is only connected to one group of intermediate servers; Step 2: Each device uses local data to perform client model forward propagation and uploads the model segmentation layer activation values ​​and corresponding labels to the intermediate server connected to it; Step 3: Each intermediate server executes the local model aggregation algorithm based on the advance gradient aggregation, updates the server-side model, and returns the aggregated gradient to each device connected to it; Step 4: Each device uses the aggregated gradient to perform client model backpropagation; Step 5: After each device and the intermediate server complete the traversal of local data, they upload their own models to the central server; Step 6: The central server performs global average aggregation on the models uploaded by all devices and intermediate servers to obtain a global model; and divides the global model according to the model segmentation strategy of each group, and returns the corresponding model part to each device and intermediate server; Step 7: All devices and intermediate servers update their own models based on the partial global model returned by the central server; Step 8: Repeat steps 2 to 7 until the global model converges, completing this round of federated learning.

2. The efficient segmentation federated edge learning method combining device clustering and early gradient aggregation according to claim 1 is characterized by: In step 1, the central server executes a model segmentation algorithm based on device clustering, specifically: Step 1.1: The central server executes the K-Means clustering algorithm based on the device capabilities to group devices with similar performance into the same group; Step 1.2: For each group, the central server randomly selects a device, performs model pre-training on the segmented models according to the model segmentation strategy, obtains the model pre-training time and load corresponding to each model segmentation strategy; selects the model segmentation strategy with the smallest time consumption and load as the model segmentation strategy for the group; Step 1.3: Repeat step 1.2 until an adapted model segmentation strategy is developed for each group.

3. The method of efficient segmentation federated edge learning combining device clustering and early gradient aggregation according to claim 1, characterized in that: In step 3, the intermediate server executes a local model aggregation algorithm based on early gradient aggregation, specifically: Step 3.1: The intermediate server receives and concatenates the model segmentation layer activation values ​​uploaded by the connected devices within the aggregation time window to generate an input activation matrix; Step 3.2: The intermediate server performs forward and backward propagation of the server-side model according to the input activation matrix, calculates the server-side model gradient, matches the corresponding gradient according to the model segmentation layer activation value uploaded by each device, and performs gradient average aggregation to obtain the aggregated gradient; Step 3.3: The intermediate server uses the aggregated gradient to update the server-side model and returns it to each device connected to it, while dynamically adjusting the aggregation time window in the next iterative calculation.

4. The efficient segmentation federated edge learning method combining device clustering and early gradient aggregation according to claim 3 is characterized by: The activation matrix is ​​entered in step 3.1 for: in, Indicates the activation value of the model segmentation layer uploaded by the i-th group of devices connected to the m-th group of intermediate servers in the t-th iteration calculation of this round of federated learning, i=1,2,...,I m , I m Indicates the total number of devices connected to the mth group of intermediate servers.

5. The method of efficient segmentation federated edge learning combining device clustering and early gradient aggregation according to claim 4, characterized in that: The server-side model gradient in step 3.2 for: in, It represents the matrix composed of the training data labels of each device connected to the mth group of intermediate servers in the tth iteration calculation of this round of federated learning; It represents the predicted value obtained by the server-side model training of the mth group of intermediate servers in the tth iteration calculation of this round of federated learning; It represents the weight of the server-side model of the mth group of intermediate servers in the tth iteration calculation of this round of federated learning.

6. The efficient segmentation federated edge learning method combining device clustering and early gradient aggregation according to claim 5, characterized in that: In step 3.2, the corresponding gradient is matched according to the activation value of the model segmentation layer uploaded by each device, which is specifically expressed as: in, It represents the gradient assigned to the i-th device connected to the m-th group of intermediate servers in the t-th iteration calculation of this round of federated learning; Perform gradient average aggregation to obtain the aggregated gradient 7. The efficient segmentation federated edge learning method combining device clustering and early gradient aggregation according to claim 6, characterized in that: The method for dynamically adjusting the aggregation time window in the next iterative calculation in step 3.3 is as follows: in, It represents the waiting time of the aggregated time window of the mth group of intermediate servers in the t+1th iteration calculation of this round of federated learning; It represents the time taken by the i-th group of devices connected to the m-th group of intermediate servers to upload the activation values ​​of the model segmentation layer in the t-th iteration calculation of this round of federated learning; 0≤β≤1, which is used to weigh the longest upload time and the average upload time. When β=1, the m-th group of intermediate servers will wait for all devices to upload the activation values ​​of the model segmentation layer before performing server-side model training.

8. A computer device / equipment / system comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images