An efficient heterogeneous federated learning method for industrial Internet of Things

Abstract

The present invention provides an efficient heterogeneous federated learning method for industrial Internet of Things, belonging to the field of federated learning technology. The neural network model is divided into an edge feature extractor and a global classifier; the edge feature extractor is deployed on the edge device, and the global classifier is deployed on the server; the edge device is also integrated with an edge classifier, which is combined with the edge feature extractor to form an edge model; the server aggregates the weight parameters of the edge feature extractor in this round of edge training based on the synchronization time, and updates the global model; at the same time, the running time of each edge device is calculated through the running time model; the specification parameters of each edge model are customized according to the running time and the synchronization time; the updated global model and edge model specification parameters are sent to the corresponding edge device; each edge device generates an edge model for the next round of edge training according to the sent global model and model specification parameters. The above steps realize heterogeneous federated learning with both efficiency and effectiveness.

Description

Technical Field

[0001] The present invention relates to the technical field of federated learning, and in particular to an efficient heterogeneous federated learning method for industrial Internet of Things. Background Art

[0002] With the rapid development of key technologies such as deep learning, edge computing, and 5G, the Industrial Internet of Things is rising rapidly. The importance of data privacy protection has become increasingly prominent. In order to achieve efficient and secure data mining to process the massive data generated by industrial Internet of Things devices, federated learning has come into being. As a privacy-preserving distributed machine learning solution, federated learning can achieve collaborative learning without sharing local data. In federated learning, industrial Internet of Things edge devices can independently train models based on local data, while collaboratively building a global model through servers, and using synchronous or asynchronous methods for efficient global training. However, when applying federated learning to industrial Internet of Things edge computing, system heterogeneity is a core challenge that cannot be ignored. There are significant differences in the hardware resources and computing power of industrial Internet of Things edge devices. This heterogeneity leads to significant imbalance in local training time, further exacerbating the laggard problem. Specifically, the edge model weight aggregation process needs to wait for the weaker performance devices to complete training, which not only increases the idle time of high-performance devices, but also significantly prolongs the overall training time, especially in the synchronous training mechanism. System heterogeneity affects the training efficiency of the global model and reduces the convergence accuracy of the model. In addition, resource consumption (such as CPU, network bandwidth, memory, and energy consumption) is particularly critical in resource-constrained federated edge learning. Since the resource conditions of different devices vary, how to optimize the balance between model accuracy, training efficiency, and resource consumption under resource constraints has become a complex and challenging problem.

[0003] Regarding the impact of system heterogeneity on training efficiency and convergence performance; in the prior art, asynchronous training methods are often used to reduce the idle time of devices to improve training efficiency; however, this method may cause the global model to tilt towards high-performance devices, thereby reducing convergence accuracy. In addition, asynchronous updates may cause multiple versions of the global model to exist in the same round, further affecting the stability of training and the accuracy of the final model. In order to alleviate the problem of system heterogeneity, existing research has made some progress in improving synchronization strategies. One strategy is to use a kick-out mechanism to exclude devices with slower training from most aggregations, thereby accelerating the training cycle. However, this method has certain risks, that is, if the excluded devices contain critical information, it may cause the global model to lose important knowledge. Another strategy is to allow laggard devices to participate in aggregation, but by customizing models for devices to reduce the difference in training time between devices, thereby achieving synchronous aggregation. However, designing such a model compression or customization method based on device resource heterogeneity requires a trade-off between model accuracy, training efficiency and resource consumption, which is very difficult.

[0004] Therefore, an efficient heterogeneous federated learning method for IIoT is needed. Summary of the invention

[0005] In view of this, the present invention provides an efficient heterogeneous federated learning method for the industrial Internet of Things, which uses a heterogeneous federated learning collaborative framework and a resource-aware adaptive edge model customization strategy to optimize the model size according to the resource capabilities of the industrial Internet of Things edge devices. At the same time, considering the complementary advantages of the cloud and the edge, some models with high memory intensiveness and high computing requirements are allocated to the resource-rich cloud environment, while the remaining computing tasks are allocated to the edge devices for execution, thereby achieving heterogeneous federated learning with both efficiency and effectiveness.

[0006] To this end, the present invention provides the following technical solutions:

[0007] An efficient heterogeneous federated learning method for industrial Internet of Things, including:

[0008] Split the neural network model into an edge feature extractor and a global classifier;

[0009] The edge feature extractor is deployed on the edge device, and the global classifier is deployed on the server;

[0010] The edge device is also integrated with an edge classifier, and the edge classifier is combined with the edge feature extractor to form an edge model;

[0011] The server aggregates the weight parameters of the edge feature extractor in this round of edge training based on the synchronization time, and updates the global model; at the same time, calculates the running time of each edge device through the running time model; customizes the specification parameters of each edge model according to the running time and the synchronization time; and sends the updated global model and edge model specification parameters to the corresponding edge device;

[0012] Each edge device generates an edge model for the next round of edge training based on the issued global model and model specification parameters.

[0013] Further, calculating the running time of each edge device through the running time model includes:

[0014] Get the computational workload and communication volume of this round of edge training;

[0015] Based on the computing workload and communication volume of this round of edge training combined with the computing bandwidth and communication bandwidth of the edge device, the computing time and communication time of this round of edge training are obtained;

[0016] The sum of the computation time and the communication time is taken as the running time.

[0017] Furthermore, the synchronization time includes:

[0018] The time required for the edge device to perform a round of local updates is used as the initial synchronization time;

[0019] All edge devices are grouped and clustered according to the initial synchronization time to obtain several edge device groups;

[0020] The longest running time in each edge device group is used as the synchronization time of the edge device group.

[0021] Furthermore, the specification parameters of each edge model are customized according to the running time and the synchronization time, including:

[0022] When the synchronization time of an edge device group is less than a set threshold, the number of iterations of the edge model in the group is increased to the preset threshold;

[0023] When the synchronization time of the edge device group is greater than the set threshold, the running time of each edge device is calculated through the running time model, and the total number of neurons of the edge model within the edge device group is customized according to the running time.

[0024] Furthermore, the customizing the total number of neurons of the edge model in the edge device group according to the running time includes:

[0025] Formulate the edge model specification parameter customization process as a priority-based neuron selection optimization problem;

[0026] Taking maximizing the contribution of the selected neurons under resource constraints as the goal, modeling the priority-based neuron selection optimization problem to obtain a priority-based neuron selection optimization model;

[0027] The constraints of the priority-based neuron selection optimization model include: energy consumption and memory of edge devices;

[0028] The total number of neurons in the neural network model is obtained by combining the running time and the priority-based neuron selection optimization model.

[0029] Furthermore, the server aggregates the weight parameters of the edge feature extractor to update the global model, including:

[0030] Train a global classifier based on knowledge distillation;

[0031] Aggregating the weight parameters of the edge feature extractor to form a global feature extractor through a baseline-first weighted aggregation method;

[0032] The trained global classifier and global feature extractor are combined to construct a global model.

[0033] Furthermore, the baseline-first weighted aggregation method includes:

[0034] The ratio of the total number of edge model neurons to the local iteration factor is defined as the benchmark value of the edge model;

[0035] The benchmark value of the edge model is used as the weight for aggregation of each edge feature extractor.

[0036] Furthermore, the training of the global classifier based on the knowledge distillation method includes:

[0037] Calculate the divergence between the teacher model’s posterior probability and the student model’s predicted probability;

[0038] A global classifier function is constructed based on the divergence between the posterior probability of the teacher model and the predicted probability of the student model.

[0039] Advantages and positive effects of the present invention:

[0040] The present invention customizes a suitable model size according to the computing and communication capabilities of each edge device through a runtime estimation model based on device heterogeneity, thereby minimizing the training time difference and alleviating the laggard problem caused by system heterogeneity.

[0041] The present invention divides the neural network into a feature extractor and a global classifier through split federated learning, and deploys them on edge devices and servers respectively. At the same time, a lightweight classifier is integrated into the feature extractor to form a fully trainable edge model, reducing the memory and computing burden of resource-constrained devices.

[0042] The present invention utilizes the device grouping strategy to set the group synchronization time, customizes the model for slower devices, increases the number of iterations for faster devices, and optimizes the training efficiency of the global model.

[0043] The present invention utilizes feature maps and soft labels uploaded by edge devices and optimizes the global classifier through a knowledge distillation method, thereby protecting privacy and improving the convergence performance of asynchronous training.

[0044] The present invention adopts a baseline-priority weighted aggregation scheme to dynamically adjust weight distribution according to device level, avoid global model training biased towards high-performance devices, and effectively improve the convergence and accuracy of the model.

[0045] The present invention optimizes the neuron selection strategy by combining energy consumption and memory constraints in the design, thereby achieving a balance between model accuracy, training efficiency and resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 It is a method framework diagram in an embodiment of the present invention;

[0048] Figure 2 4 is a flowchart of federated learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0050] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0051] The present invention provides an efficient heterogeneous federated learning method for industrial Internet of Things, comprising:

[0052] 1. Adaptive edge model customization: The server conducts a comprehensive analysis based on the heterogeneity of edge device resources, builds an analysis model that perceives device capabilities, and dynamically evaluates the model size that is suitable for each device. A lightweight model is tailored for the edge device based on its computing power, memory limitations, and network bandwidth. The customized model is distributed to each edge device through the federated learning server for local training, taking into account both training efficiency and resource optimization.

[0053] 2. Synchronous edge training: The edge device uses model segmentation technology to segment the feature extractor from the global model and train it in combination with a lightweight local classifier. While ensuring data privacy, the edge device uses local data to efficiently train the customized model. After training, the edge device only uploads the weight parameters of the feature extractor to the server for aggregation, reducing communication overhead and improving privacy protection.

[0054] 3. Asynchronous server training based on knowledge distillation: The server uses knowledge distillation technology to optimize the training process of the global classifier for the feature maps and soft labels uploaded by heterogeneous devices. By integrating the feature information of multiple devices in an asynchronous manner, the training delay problem caused by device heterogeneity is effectively solved, while ensuring the robustness and generalization performance of the global classifier.

[0055] 4. Global model aggregation: The server adopts a baseline-first weighted aggregation strategy to aggregate the feature extractor weights uploaded by edge devices. During the aggregation process, the heterogeneity of devices and their differences in contribution to the model are fully considered to achieve efficient and robust global model updates. Finally, the optimized feature extractor is merged with the global classifier to generate the latest global model.

[0056] The present invention adopts segmented federated learning, combining the model segmentation advantage of segmented learning with the parallel training principle of federated learning. Split into two components: Feature Extractor , and the global classifier , deployed on edge devices and servers respectively; the edge device integrates a lightweight classifier for the feature extractor , thus forming a new, fully trainable edge model. Since the fully connected layers of the convolutional neural network consume a lot of memory, the feature extractor is trained on the edge device and the classifier is trained on the server; after completing the local training, the edge device will update , transmitted to the server for aggregation.

[0057] During local updates, the edge device asynchronously uploads Extracted feature maps and its corresponding inference results. Then, the server uses Training as input features The model segmentation method of the present invention defines the optimization target as the edge model and server model Collaborative optimization:

[0058]

[0059]

[0060]

[0061] in, and denote the loss functions of the edge model and the server model respectively; , , Represent the model weights of the edge feature extractor, edge classifier and global classifier respectively. Representation feature extractor Output feature map.

[0062] In this embodiment, 1. Adaptive edge model customization includes:

[0063] 1) Determine the edge model size based on resources;

[0064] In order to reduce the running time of lagging devices and ultimately achieve training acceleration; first, a resource-aware running time estimation model is established and used to determine the specific edge model size for each heterogeneous device. This embodiment focuses on the heterogeneity of edge device computing power and communication capabilities, which mainly affect the running time of local training. The neural network model in this embodiment is based on a convolutional neural network, and the computational workload of each round of edge training is estimated through a neuron-level analysis method. and traffic , the formula is as follows:

[0065]

[0066]

[0067] Where: I and J represent the number of convolutional layers and fully connected layers contained in the convolutional neural network respectively; and Represent the number of neurons in the convolutional layer and the fully connected layer respectively; and Represents the filter size and output feature map dimension of the i-th convolutional layer; Indicates the number of model training times; Indicates data bandwidth.

[0068] According to the computing power and communication capacity of the edge device, the running time estimation model is expressed as the computing time and communication time The sum of:

[0069]

[0070] in: and are the computing bandwidth and communication bandwidth of the device, respectively, which are the key factors affecting the time dimension of device heterogeneity. Therefore, given the training time T, a specific model size can be estimated for each heterogeneous edge device , represents the total number of neurons.

[0071] 2) Edge model customization and optimization;

[0072] Based on the determined edge model size ,A resource-constrained edge model customization method is proposed to create edge models.,In order to achieve model aggregation, a specific neuron training method is proposed.

[0073] The model customization process is formulated as a priority-based neuron selection optimization problem:

[0074]

[0075] in, Represents the contribution of the neuron, quantifying the impact of the neuron on the model update, calculated by the magnitude of the change in the current training cycle. is the set of neurons selected for the next local training cycle, is the number of neurons in the i-th layer. method to ensure the convergence of the global model update. , maximizes the total contribution of the selected neurons. Therefore, choose ,in .

[0076] Considering the resource constraints, the device imposes different restrictions on resources. Therefore, the convolutional neural network at the neuron level is used to calculate the consumption model to obtain the relationship between the neuron selection strategy and resource consumption. In edge devices, with energy consumption and memory as the main constraints, the corresponding optimization problem can be restated as:

[0077]

[0078]

[0079] in, is the contribution of neurons under resource constraints; and Respectively represent the energy consumption and memory usage of local training; and They are the energy consumption and memory limitations of edge devices in federated learning; is a set of neurons selected only by contribution, is the remaining set of neurons. First select neurons to ensure the basic convergence of the global model update. Then, the problem is transformed into choosing of The remaining neurons are .

[0080] The contribution of each neuron under resource constraints is defined as:

[0081]

[0082] in, is the normalized neuron contribution, and It is located in Layer of neurons The normalized energy and memory of and Calculated. Indicates Energy consumption of the layer; Indicates The memory usage of the layer, L represents the total number of layers in the network. , It is the computation consumption weight determined by different actual resource constraints of edge devices.

[0083] 2. Synchronous edge training strategy:

[0084] The adaptive edge model customization solution customizes models of different sizes for different devices, thereby minimizing the difference in training time and alleviating the laggard problem. However, optimizing only for lagging devices may introduce the following problems:

[0085] 1) Due to data heterogeneity, lagging devices may have unique and critical information. Limiting them to use partial models may cause the global model to be biased towards faster devices, thereby missing important data.

[0086] 2) Considering the limited capabilities of most edge devices, using only a small number of faster devices as synchronization baselines may significantly affect the overall training time and convergence time of the global model.

[0087] Therefore, in order to identify the appropriate synchronization reference time and facilitate model customization, the devices are grouped and clustered by the runtime, which is defined as the time required for the local device to perform a round of local updates. l ,grade 2 ,…,grade k}, specify the slowest running time for each category group as the synchronization time {st 1 ,st 2 …,st k}, used to alternately perform certain stages of local training. In this embodiment, grade is selected in round i k As the synchronization benchmark, the synchronization time At the same time, all grades with similar running time k The device trains the full model without model compression. At the start of an update, the server syncs based on the sync reference time, customizing the model for slower devices while increasing the number of local iterations for faster devices, but with limits to avoid overfitting.

[0088] 3. Asynchronous server training based on knowledge distillation;

[0089] The knowledge distillation method is used to train a lightweight student model by extracting knowledge from the teacher model (a model with high prediction accuracy). The student model is trained to match the softmax distribution of the teacher model. Given the teacher model and student models , whose functions are and , the traditional knowledge distillation method is achieved by minimizing the output objective function of the hidden layer or softmax layer of the network:

[0090]

[0091] To enhance The generalization performance of the teacher model is used in this embodiment. To guide the training. The predicted value of the student model The predicted value of the teacher model The matching relationship between them is measured by Kullback-Leibler (KL) divergence, and the formula is:

[0092]

[0093] in, , , is the temperature hyperparameter of softmax. Therefore, the overall loss function is defined as:

[0094]

[0095] in: is the cross entropy loss between the predicted value and the correct label; is the trade-off coefficient.

[0096] In this embodiment, the server is updated only by training the classifier of the global model on the feature map. To protect privacy, the edge device only sends the extracted features and soft labels, not the original data or the real labels, thereby reducing the possibility of data reconstruction attacks. This embodiment utilizes the abstract characteristics of high-level features, which contain less sensitive information than the original data, thereby reducing privacy risks.

[0097] Therefore, the global classifier training based on the knowledge distillation method in this embodiment is as follows:

[0098]

[0099] in: and are the soft labels generated by the edge device and the classifier, respectively.

[0100] In addition, the sample training strategy has a significant impact on model convergence. In particular, when the model is trained on one dataset and then transferred to a new dataset, the model can quickly adapt to the new categories. However, this adaptation weakens the accuracy of the model for previously trained categories, resulting in a rapid decrease in test accuracy. To avoid such problems, this implementation ensures a balanced distribution of labels in each training batch during global classifier training. Specifically, the global classifier is trained using feature maps and soft labels asynchronously transmitted from the local device, treating each dataset as an independent training entity. By analyzing the probability distribution of the soft labels, the category with the highest probability is selected to classify the data. Subsequently, a stratified sampling method is used to evenly distribute samples of each category to each training batch, thereby ensuring a balanced representation of all categories. This strategy prevents some categories from being missing for a long time and alleviates the label clustering problem, thereby optimizing the asynchronous training process and improving model accuracy.

[0101] 4. Global Model Aggregation

[0102] The server combines the global feature extractor and the global classifier to construct a global model. The global feature extractor is formed by aggregating the weight parameters of the edge feature extractors, while the global classifier is trained using the feature maps uploaded by the edge devices and the local soft labels. At the same time, this embodiment adopts a baseline-first weighted aggregation scheme to dynamically adjust the relative weight assigned to each device and divide it based on its level. Define the baseline value ,in and Represents the model size factor and local iteration factor for constraining slower devices and faster devices, respectively. In addition, when the level of the device is the baseline in the current training round, The device with the maximum value representing the baseline level will be assigned the largest weight value. The corresponding global aggregation process is defined as follows:

[0103]

[0104] in, In this way, the global model training is dynamically guided and balanced to avoid the global model training being biased towards a subset of faster devices and effectively improve the convergence performance.

[0105] The method of the present invention is further described with a specific application example. In this implementation, an efficient heterogeneous federated learning algorithm is constructed as follows:

[0106]

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An efficient heterogeneous federated learning method for industrial Internet of Things, characterized by: include: Split the neural network model into an edge feature extractor and a global classifier; The edge feature extractor is deployed on the edge device, and the global classifier is deployed on the server; The edge device is also integrated with an edge classifier, and the edge classifier is combined with the edge feature extractor to form an edge model; The server updates the global model by aggregating the weight parameters of the edge feature extractor in this round of edge training based on the synchronization time in the following steps: Train a global classifier based on knowledge distillation; Aggregating the weight parameters of the edge feature extractor to form a global feature extractor through a baseline-first weighted aggregation method; Combine the trained global classifier and global feature extractor to construct a global model; The synchronization time is determined by: The time required for the edge device to perform a round of local updates is used as the initial synchronization time; All edge devices are grouped and clustered according to the initial synchronization time to obtain several edge device groups; The longest running time in each edge device group is used as the synchronization time of the edge device group; At the same time, the running time of each edge device is calculated through the running time model; Customize the specification parameters of each edge model according to the running time and the synchronization time; send the updated global model and edge model specification parameters to the corresponding edge device; Each edge device generates an edge model for the next round of edge training based on the issued global model and model specification parameters.

2. According to claim 1, an efficient heterogeneous federated learning method for industrial Internet of Things is characterized by: The operation time of each edge device is calculated by the operation time model, including: Get the computational workload and communication volume of this round of edge training; Based on the computing workload and communication volume of this round of edge training combined with the computing bandwidth and communication bandwidth of the edge device, the computing time and communication time of this round of edge training are obtained; The sum of the computation time and the communication time is taken as the running time.

3. According to claim 2, an efficient heterogeneous federated learning method for industrial Internet of Things, characterized in that: Customize the specification parameters of each edge model according to the running time and the synchronization time, including: When the synchronization time of an edge device group is less than a set threshold, the number of iterations of the edge model in the group is increased to the preset threshold; When the synchronization time of the edge device group is greater than the set threshold, the running time of each edge device is calculated through the running time model, and the total number of neurons of the edge model within the edge device group is customized according to the running time.

4. According to claim 3, an efficient heterogeneous federated learning method for industrial Internet of Things is characterized by: The method of customizing the total number of neurons of the edge model in the edge device group according to the runtime includes: Formulate the edge model specification parameter customization process as a priority-based neuron selection optimization problem; Taking maximizing the contribution of the selected neurons under resource constraints as the goal, modeling the priority-based neuron selection optimization problem to obtain a priority-based neuron selection optimization model; The constraints of the priority-based neuron selection optimization model include: energy consumption and memory of edge devices; The total number of neurons in the neural network model is obtained by combining the running time and the priority-based neuron selection optimization model.

5. According to claim 1, an efficient heterogeneous federated learning method for industrial Internet of Things, characterized in that: The baseline-first weighted aggregation method comprises: The ratio of the total number of edge model neurons to the local iteration factor is defined as the benchmark value of the edge model; The benchmark value of the edge model is used as the weight for aggregation of each edge feature extractor.

6. According to claim 1, an efficient heterogeneous federated learning method for industrial Internet of Things, characterized in that: The method of training a global classifier based on knowledge distillation includes: Calculate the divergence between the teacher model’s posterior probability and the student model’s predicted probability; A global classifier function is constructed based on the divergence between the posterior probability of the teacher model and the predicted probability of the student model.

Images