An efficient segmentation federated edge learning method combining device clustering and early gradient aggregation, program, device and storage medium

Abstract

The present application belongs to the technical field of split federated learning, and particularly relates to an efficient split federated edge learning method combining device clustering and early gradient aggregation, a program, a device and a storage medium. The model splitting algorithm based on device clustering designed by the present application allocates devices with similar capabilities to the same group, and all devices in the group adopt the same model splitting strategy, so that the intermediate server only needs to save one server-side model, thereby reducing the load of the intermediate server. The local model aggregation algorithm based on early gradient aggregation designed by the present application makes the intermediate server only need to save one server-side model, thereby greatly reducing the memory consumption, alleviating the communication delay caused by device instability and model synchronization between devices, and thereby improving the model training efficiency.

Description

Technical Field

[0001] This invention belongs to the field of segmented federated learning technology, specifically relating to an efficient segmented federated edge learning method, program, device, and storage medium that combines device clustering and pre-gradient aggregation. Background Technology

[0002] In recent years, with the widespread application of technologies such as the Internet of Things (IoT) and 5G, an increasing number of lightweight smart terminal devices have connected to the Internet, generating massive amounts of data. How to efficiently process this data and achieve cross-device edge collaborative computing has become a significant challenge. Segmentation-based federated learning combines the advantages of segmentation learning and federated learning, enabling efficient collaborative training among resource-constrained devices. A layered learning architecture based on the edge-cloud architecture introduces an intermediate server layer to reduce training latency caused by frequent exchanges of model segmentation layer results between devices and remote cloud servers. However, resource heterogeneity between devices and device instability (such as battery depletion or network connection loss) present new challenges to the layered learning architecture. Summary of the Invention

[0003] The purpose of this invention is to provide an efficient segmentation federated edge learning method, program, device, and storage medium that combines device clustering and pre-gradient aggregation.

[0004] An efficient segmentation federated edge learning method combining device clustering and pre-gradient aggregation includes the following steps:

[0005] Step 1: The central server executes a device clustering-based model segmentation algorithm to group all devices and formulate an appropriate model segmentation strategy for each group;

[0006] The central server is connected to multiple intermediate servers, each intermediate server is connected to multiple groups of devices, and each group of devices is connected to only one set of intermediate servers.

[0007] Step 2: Each device uses local data to perform client model forward propagation and uploads the model segmentation layer activation values ​​and corresponding labels to the intermediate server connected to it;

[0008] Step 3: Each intermediate server executes a local model aggregation algorithm based on pre-aggregated gradients, updates the server-side model, and returns the aggregated gradients to each device connected to it;

[0009] Step 4: Each device performs backpropagation of the client model using aggregated gradients;

[0010] Step 5: After each device and intermediate server have completed the traversal of local data, they upload their respective models to the central server;

[0011] Step 6: The central server performs global average aggregation on the models uploaded by all devices and intermediate servers to obtain the global model; and splits the global model according to the model splitting strategy of each group, and returns the corresponding model parts to each device and intermediate server.

[0012] Step 7: All devices and intermediate servers update their respective models based on the partial global model returned by the central server;

[0013] Step 8: Repeat steps 2 to 7 until the global model converges, completing this round of federated learning.

[0014] Furthermore, in step 1, the central server executes a device clustering-based model segmentation algorithm, specifically as follows:

[0015] Step 1.1: The central server executes the K-Means clustering algorithm based on the device capabilities to group devices with similar performance into the same group;

[0016] Step 1.2: For each group, the central server randomly selects a device, performs model pre-training on the segmented models according to the model segmentation strategy, and obtains the time consumption and load of model pre-training corresponding to each model segmentation strategy; selects the model segmentation strategy with the least time consumption and load as the model segmentation strategy for that group.

[0017] Step 1.3: Repeat step 1.2 until a suitable model segmentation strategy is developed for each group.

[0018] Furthermore, in step 3, the intermediate server executes a local model aggregation algorithm based on pre-gradient aggregation, specifically as follows:

[0019] Step 3.1: The intermediate server receives and splices the model segmentation layer activation values ​​uploaded by the connected devices within the aggregation time window to generate the input activation matrix;

[0020] Step 3.2: The intermediate server performs forward and backward propagation of the server-side model based on the input activation matrix, calculates the gradient of the server-side model, matches the corresponding gradient based on the activation values ​​of the model segmentation layer uploaded by each device, and performs gradient averaging and aggregation to obtain the aggregated gradient.

[0021] Step 3.3: The intermediate server updates the server-side model using aggregated gradients and returns the results to the connected devices, while dynamically adjusting the aggregation time window in the next iteration calculation.

[0022] Furthermore, in step 3.1, the activation matrix is ​​input. for:

[0023]

[0024] in, This represents the activation value of the model segmentation layer uploaded by the i-th device connected to the m-th intermediate server in the t-th iteration of this federated learning round, where i = 1, 2, ..., I. m I m This represents the total number of devices connected to the m-th intermediate server.

[0025] Furthermore, in step 3.2, the server-side model gradient... for:

[0026]

[0027] in, This represents the matrix consisting of the training data labels of each device connected to the m-th intermediate server in the t-th iteration of this round of federated learning. This represents the predicted value obtained by the server-side model training of the m-th intermediate server in the t-th iteration of this round of federated learning. This represents the weight of the server-side model of the m-th intermediate server in the t-th iteration of this round of federated learning.

[0028] Furthermore, in step 3.2, the gradient is matched according to the activation value of the model segmentation layer uploaded by each device, specifically as follows:

[0029]

[0030] in, This represents the gradient assigned to the i-th device connected to the m-th intermediate server in the t-th iteration of this round of federated learning.

[0031] Perform gradient-averaged aggregation to obtain the aggregate gradient.

[0032]

[0033] Furthermore, the method for dynamically adjusting the aggregation time window in the next iteration calculation in step 3.3 is as follows:

[0034]

[0035] in, This represents the waiting time of the aggregation time window of the m-th intermediate server in the (t+1)-th iteration of this round of federated learning. This represents the time taken for the i-th group of devices connected to the m-th group of intermediate servers to upload the model segmentation layer activation values ​​in the t-th iteration of this round of federated learning; 0≤β≤1, used to balance the longest upload time and the average upload time. When β=1, the m-th group of intermediate servers will wait for all devices to upload the model segmentation layer activation values ​​before executing server-side model training.

[0036] A computer device / apparatus / system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described efficient segmentation federated edge learning method combining device clustering and pre-gradient aggregation.

[0037] A computer-readable storage medium having a computer program / instructions thereon that, when executed by a processor, implements the steps of the above-described efficient segmentation federated edge learning method combining device clustering and pre-gradient aggregation.

[0038] A computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the above-described efficient segmentation federated edge learning method combining device clustering and pre-gradient aggregation.

[0039] The beneficial effects of this invention are as follows:

[0040] The device clustering-based model segmentation algorithm designed in this invention assigns devices with similar capabilities to the same group, and all devices within the group use the same model segmentation strategy. This allows the intermediate server to store only one copy of the server-side model, reducing its load. The local model aggregation algorithm based on pre-gradient aggregation designed in this invention also requires the intermediate server to store only one copy of the server-side model, significantly reducing memory consumption and mitigating communication latency caused by device instability and inter-device model synchronization, thereby improving model training efficiency. Attached Figure Description

[0041] Figure 1 This is a framework diagram of the present invention.

[0042] Figure 2 This is a graph showing the accuracy comparison results of the present invention with Hier-SFL, Hier-RSFL, and Hier-PSL models.

[0043] Figure 3 This is a comparison chart showing the time required for one round of model training of the present invention with that of Hier-SFL, Hier-RSFL, and Hier-PSL.

[0044] Figure 4 This is a comparison chart showing the time required for the present invention to achieve the target accuracy with Hier-SFL, Hier-RSFL, and Hier-PSL.

[0045] Figure 5 This is a graph showing the comparison of memory usage between the present invention and Hier-SFL during model training.

[0046] Figure 6 This is a pseudocode diagram of the device clustering-based model segmentation algorithm in this invention.

[0047] Figure 7 This is a pseudocode diagram of the local model aggregation algorithm based on advance gradient aggregation in this invention. Detailed Implementation

[0048] The present invention will now be further described with reference to the accompanying drawings.

[0049] This invention relates to an efficient segmentation federated edge learning method, program, device, and storage medium that combines device clustering and pre-gradient aggregation, specifically for edge learning and incorporates the widely used segmentation federated learning approach. First, devices with similar capabilities are grouped together, and all devices within a group employ the same model segmentation strategy. This allows the intermediate server to store only one copy of the server-side model, reducing its load. Then, the intermediate server uses pre-gradient aggregation to mitigate training latency caused by device instability and inter-device model synchronization, thereby improving model training efficiency.

[0050] An efficient segmentation federated edge learning method combining device clustering and pre-gradient aggregation includes the following steps:

[0051] Step 1: The central server executes a device clustering-based model segmentation algorithm to group all devices and formulate an appropriate model segmentation strategy for each group;

[0052] The central server is connected to multiple intermediate servers, each intermediate server is connected to multiple groups of devices, and each group of devices is connected to only one set of intermediate servers.

[0053] Step 1.1: The central server executes the K-Means clustering algorithm based on the device capabilities to group devices with similar performance into the same group;

[0054] Step 1.2: For each group, the central server randomly selects a device, performs model pre-training on the segmented models according to the model segmentation strategy, and obtains the time consumption and load of model pre-training corresponding to each model segmentation strategy; selects the model segmentation strategy with the least time consumption and load as the model segmentation strategy for that group.

[0055] Step 1.3: Repeat step 1.2 until a suitable model segmentation strategy is developed for each group;

[0056] Step 2: Each device uses local data to perform client model forward propagation and uploads the model segmentation layer activation values ​​and corresponding labels to the intermediate server connected to it;

[0057] Step 3: Each intermediate server executes a local model aggregation algorithm based on pre-aggregated gradients, updates the server-side model, and returns the aggregated gradients to each device connected to it;

[0058] Step 3.1: Within the aggregation time window, the intermediate server receives and concatenates the activation values ​​of the model segmentation layer uploaded by its connected devices to generate the input activation matrix.

[0059]

[0060] in, This represents the activation value of the model segmentation layer uploaded by the i-th device connected to the m-th intermediate server in the t-th iteration of this federated learning round, where i = 1, 2, ..., I. m I m This represents the total number of devices connected to the m-th intermediate server.

[0061] Step 3.2: The intermediate server performs forward and backward propagation of the server-side model based on the input activation matrix, and calculates the gradient of the server-side model. The corresponding gradient is matched based on the activation values ​​of the model segmentation layer uploaded by each device. Then perform gradient averaging aggregation to obtain the aggregated gradient.

[0062]

[0063] in, This represents the matrix consisting of the training data labels of each device connected to the m-th intermediate server in the t-th iteration of this round of federated learning. This represents the predicted value obtained by the server-side model training of the m-th intermediate server in the t-th iteration of this round of federated learning. This represents the weight of the server-side model of the m-th intermediate server in the t-th iteration of this round of federated learning.

[0064]

[0065] in, This represents the gradient assigned to the i-th group of devices connected to the m-th group of intermediate servers in the t-th iteration of this round of federated learning.

[0066] Perform gradient-averaged aggregation to obtain the aggregate gradient.

[0067]

[0068] Step 3.3: The intermediate server updates the server-side model using aggregated gradients and returns the results to the devices connected to it, while dynamically adjusting the aggregation time window in the next iteration calculation.

[0069]

[0070] in, This represents the waiting time of the aggregation time window of the m-th intermediate server in the (t+1)-th iteration of this round of federated learning. This represents the time taken for the i-th group of devices connected to the m-th group of intermediate servers to upload the model segmentation layer activation values ​​in the t-th iteration of this round of federated learning; 0≤β≤1, used to balance the longest upload time and the average upload time. When β=1, the m-th group of intermediate servers will wait for all devices to upload the model segmentation layer activation values ​​before executing server-side model training.

[0071] Step 4: Each device performs backpropagation of the client model using aggregated gradients;

[0072] Step 5: After each device and intermediate server have completed the traversal of local data, they upload their respective models to the central server;

[0073] Step 6: The central server performs global average aggregation on the models uploaded by all devices and intermediate servers to obtain the global model; and splits the global model according to the model splitting strategy of each group, and returns the corresponding model parts to each device and intermediate server.

[0074] Step 7: All devices and intermediate servers update their respective models based on the partial global model returned by the central server;

[0075] Step 8: Repeat steps 2 to 7 until the global model converges, completing this round of federated learning.

[0076] The overall framework diagram of the present invention is as follows: Figure 1 As shown, this invention compares with Hier-SFL (Hier-SFL), Hier-RSFL (Hier-RSFL), and Hier-PSL (Hier-PSL), which considers training efficiency, and applies these methods to a hierarchical architecture.

[0077] To verify the effectiveness of this invention, its model accuracy was compared with the three methods mentioned above under the same model and data conditions. The experimental results are as follows: Figure 2As shown, this invention achieves model accuracy comparable to other methods, while requiring significantly fewer rounds for model convergence.

[0078] To compare the time required to complete one round of model training for different methods, each method was trained for 10 rounds, and the average was taken to ensure the accuracy of the experimental results. The experimental results are as follows: Figure 3 As shown, the average time for this invention to complete one round of model training is 25.44s, while Hier-SFL, Hier-RSFL, and Hier-PSL require 41.47s, 37.97s, and 36.76s, respectively.

[0079] To further verify the effectiveness of this invention in improving model training efficiency, the time required for each method to reach the target accuracy was recorded. The experimental results are as follows: Figure 4 As shown, this invention achieves the target accuracy in 190.88 seconds, while Hier-SFL, Hier-RSFL, and Hier-PSL require 272.79 seconds, 257.01 seconds, and 255.47 seconds, respectively. This is because this invention effectively reduces communication latency caused by heterogeneous devices and client model synchronization through device similarity clustering and gradient aggregation methods, thereby improving model training efficiency.

[0080] To verify the effectiveness of this invention in reducing intermediate server memory usage, a comparison was made with Hier-SFL after 10 rounds of model training, under the same number of clients. The experimental results are as follows: Figure 5 As shown, the memory usage of the intermediate server in this invention is much lower than that of Hier-SFL because the intermediate server in this invention only needs to store one copy of the server-side model, thereby significantly reducing memory consumption.

[0081] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An efficient segmentation federated edge learning method combining device clustering and pre-gradient aggregation, characterized in that, Includes the following steps: Step 1: The central server executes a device clustering-based model segmentation algorithm to group all devices and formulate an appropriate model segmentation strategy for each group; The central server is connected to multiple intermediate servers, each intermediate server is connected to multiple groups of devices, and each group of devices is connected to only one set of intermediate servers. Step 2: Each device uses local data to perform client model forward propagation and uploads the model segmentation layer activation values ​​and corresponding labels to the intermediate server connected to it; Step 3: Each intermediate server executes a local model aggregation algorithm based on pre-aggregated gradients, updates the server-side model, and returns the aggregated gradients to each device connected to it; Step 3.1: The intermediate server receives and splices the model segmentation layer activation values ​​uploaded by the connected devices within the aggregation time window to generate the input activation matrix; Step 3.2: The intermediate server performs forward and backward propagation of the server-side model based on the input activation matrix, calculates the gradient of the server-side model, matches the corresponding gradient based on the activation values ​​of the model segmentation layer uploaded by each device, and performs gradient averaging and aggregation to obtain the aggregated gradient. in, This represents the gradient assigned to the i-th device connected to the m-th intermediate server in the t-th iteration of this round of federated learning. This represents the activation value of the model segmentation layer uploaded by the i-th device connected to the m-th intermediate server in the t-th iteration of this federated learning round, where i = 1, 2, ..., I. m , For the gradient of the server-side model; Perform gradient-averaged aggregation to obtain the aggregated gradient. Step 3.3: The intermediate server updates the server-side model using aggregated gradients and returns the updated data to the connected devices, while dynamically adjusting the aggregation time window for the next iteration. m This represents the total number of devices connected to the m-th intermediate server. Step 4: Each device performs backpropagation of the client model using aggregated gradients; Step 5: After each device and intermediate server have completed the traversal of local data, they upload their respective models to the central server; Step 6: The central server performs global average aggregation on the models uploaded by all devices and intermediate servers to obtain the global model; and splits the global model according to the model splitting strategy of each group, and returns the corresponding model parts to each device and intermediate server. Step 7: All devices and intermediate servers update their respective models based on the partial global model returned by the central server; Step 8: Repeat steps 2 to 7 until the global model converges, completing this round of federated learning.

2. The efficient segmentation federated edge learning method combining device clustering and pre-gradient aggregation as described in claim 1, characterized in that: In step 1, the central server executes a device clustering-based model segmentation algorithm, specifically: Step 1.1: The central server executes the K-Means clustering algorithm based on the device capabilities to group devices with similar performance into the same group; Step 1.2: For each group, the central server randomly selects a device, performs model pre-training on the segmented models according to the model segmentation strategy, and obtains the time consumption and load of model pre-training corresponding to each model segmentation strategy; selects the model segmentation strategy with the least time consumption and load as the model segmentation strategy for that group. Step 1.3: Repeat step 1.2 until a suitable model segmentation strategy is developed for each group.

3. The efficient segmentation federated edge learning method combining device clustering and pre-gradient aggregation as described in claim 1, characterized in that: In step 3.1, the activation matrix is ​​input. for:

4. The efficient segmentation federated edge learning method combining device clustering and pre-gradient aggregation as described in claim 1, characterized in that: The server-side model gradient in step 3.2 for: in, This represents the matrix consisting of the training data labels of each device connected to the m-th intermediate server in the t-th iteration of this round of federated learning. This represents the predicted value obtained by the server-side model training of the m-th intermediate server in the t-th iteration of this round of federated learning. This represents the weight of the server-side model of the m-th intermediate server in the t-th iteration of this round of federated learning.

5. The efficient segmentation federated edge learning method combining device clustering and pre-gradient aggregation according to claim 1, characterized in that: The method for dynamically adjusting the aggregation time window in the next iteration calculation in step 3.3 is as follows: in, This represents the waiting time of the aggregation time window of the m-th intermediate server in the (t+1)-th iteration of this round of federated learning. This represents the time taken for the device in group i connected to group m intermediate servers to upload the activation values ​​of the model segmentation layer in the t-th iteration of this round of federated learning; 0≤β≤1, used to balance the longest upload time and the average upload time. When β=1, the group m intermediate servers will wait for all devices to upload the activation values ​​of the model segmentation layer before executing server-side model training.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 5.

8. A computer program product comprising computer instructions, characterized in that: When executed by a processor, the computer instructions implement the steps of the method according to any one of claims 1 to 5.

Images