A Clustered Federated Multi-Task Learning Method and Device for the Internet of Things

Abstract

This invention provides a clustered federated multi-task learning method and apparatus for the Internet of Things (IoT). By clustering IoT terminal devices, the data distribution within the same cluster becomes more approximate. A federated multi-task learning algorithm is executed within each cluster, with global training and personalized training tasks performed on each IoT terminal device. This achieves data sharing within the cluster while fully utilizing local data from each IoT terminal device for training on personalized tasks, thus efficiently utilizing local data and improving training effectiveness. During local training on each IoT terminal device, the number of training rounds is adjusted based on computing power, fully utilizing the computing resources of each IoT terminal device and improving model training efficiency. Regularization constraints applied to personalized training tasks using the global model effectively prevent overfitting, control the degree of personalization, and improve model quality.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a clustered federated multi-task learning method and apparatus for the Internet of Things. Background Technology

[0002] The Internet of Things (IoT) digitizes and networks everything, enabling efficient information interaction between objects, between objects and people, and between people and their environment. It integrates various information technologies into social behavior through new service models, and is considered a major revolution in the information field. To better utilize the massive amounts of data generated by the IoT, a traditional method for training specific tasks involves collecting data, training it uniformly, and then distributing the trained model to the terminal. However, this process compromises the privacy and security of terminal data, making data transmission vulnerable to attacks and leaks. Collecting data in the cloud also violates user privacy rights. Another traditional method is to train the model on the terminal itself. However, this is limited by the terminal's computing and memory resources, restricting model selection. Furthermore, if the data sample size or quality is insufficient for a single terminal, the trained model may be of low quality.

[0003] In existing technologies, federated learning achieves data sharing while ensuring terminal data privacy. Terminals collaboratively train shared models without leaving their local storage, making it a promising application in the Internet of Things (IoT). However, the IoT environment is complex, and applying federated learning in IoT faces challenges related to heterogeneous devices, data, and models.

[0004] Device heterogeneity refers to the differences in configuration of terminal devices in the Internet of Things (IoT), including hardware and network conditions, leading to variations in terminal computing, storage, and communication capabilities. Device heterogeneity causes problems such as high communication costs, disconnections, and fault tolerance, indirectly affecting model training efficiency or resulting in varying levels of user participation in model training. Data heterogeneity refers to the non-IID (independent and identically distributed) distribution of data from different terminals. Factors such as the terminal's surrounding environment and operating time lead to differences in data distribution. These differences in data distribution from different terminals often cause weight divergence during federated learning, affecting model quality. Model heterogeneity refers to the possibility that different terminals may deploy models with different architectures in federated learning, leading to the failure of the aggregation process in traditional federated learning.

[0005] In IoT scenarios, heterogeneity can negatively impact model quality, training efficiency, and user experience. Therefore, addressing the adverse effects of heterogeneity in IoT scenarios is an urgent problem to be solved. Summary of the Invention

[0006] This invention provides a clustered federated multi-task learning method and apparatus for the Internet of Things (IoT) to eliminate or improve one or more defects in the prior art and solve the adverse effects of IoT device heterogeneity and model heterogeneity on training tasks.

[0007] One aspect of the present invention provides a clustered federated multi-task learning method for the Internet of Things (IoT), the method running on IoT terminal devices, the IoT terminal devices being relayed to macro base stations via micro base stations, wherein the IoT terminal devices are divided into a predetermined number of clusters with the objective of minimizing data distribution differences; the method is used to perform federated multi-task training, each training round including:

[0008] Download the global model of the current cluster to which the current IoT terminal device belongs in this round from the macro base station;

[0009] The local data of the current IoT terminal device is used to perform a global training task on the personalized model in this round. The global training task uses a first loss function to update the parameters and obtain a global update model.

[0010] The global model is trained using local data from current IoT terminal devices. The personalized training uses a second loss function to update the parameters, resulting in an updated personalized model. The second loss function applies regularization constraints to the personalized training task based on the parameters of the global model, in addition to the first loss function.

[0011] The global update model is sent to the macro base station and aggregated with the models obtained by other IoT terminal devices in the current cluster through global training tasks to update the global model of the current cluster.

[0012] In some embodiments, the method employs the k-means algorithm to cluster the IoT terminal devices, wherein the IoT terminal devices are divided into a predetermined number of clusters with the objective of minimizing data distribution differences, including:

[0013] The predetermined number of IoT terminal devices are randomly initialized as centroids to establish corresponding clusters;

[0014] Calculate the distribution distance between the local data of each remaining IoT terminal device and the local data of each centroid, assign it to the cluster corresponding to the centroid with the closest distribution distance, and calculate the distribution mean of the local data of each IoT terminal device in each cluster to update the centroid, until all IoT terminal devices are classified.

[0015] In some embodiments, the method employs the elbow method to determine the preset number of clusters, wherein the IoT terminal device is divided into a preset number of clusters with the goal of minimizing data distribution differences, including...

[0016] Randomly select an IoT terminal device as the cluster center;

[0017] Calculate the distribution distance between the remaining IoT terminal devices and the local data of each existing cluster center, and select the IoT terminal device with the largest distribution distance as the new cluster center.

[0018] Iterate until a preset number of cluster centers are selected.

[0019] In some embodiments, the method further includes determining the preset quantity using the elbow method.

[0020] In some embodiments, the method further adjusts the local training rounds based on the computing power of each IoT terminal device in the cluster, including:

[0021] The latency required for each IoT terminal device to perform one round of local training is calculated using the following formula:

[0022]

[0023] Among them, l k,z |D represents the latency required for IoT terminal device z in the k-th cluster to perform one round of local training. z | represents the amount of local data on the IoT terminal device z, f z This represents the computing power of the IoT terminal device z;

[0024] The latency of the IoT terminal device that takes the longest time to perform one round of local training in each cluster is taken as the longest latency of that cluster. The expression is:

[0025]

[0026] Among them, l k,max This represents the longest latency required for local training among all IoT terminal devices in the k-th cluster;

[0027] The number of rounds of local training for each IoT terminal device in each cluster is adjusted based on the ratio of the latency required for each IoT terminal device to perform one round of local training to the longest latency corresponding to its cluster. The calculation formula is as follows:

[0028]

[0029] Among them, e z This indicates the number of training rounds for the IoT terminal device z locally. This represents the local training round of the IoT terminal device in the k-th cluster that requires the longest latency to perform local training.

[0030] In some embodiments, the second loss function, based on the first loss function and incorporating the parameters of the global model, applies regularization constraints to the personalized training task. The expression for the second loss function is:

[0031]

[0032] Among them, F z,P (·) represents the second loss function, F z,g (·) represents the first loss function, w z D represents the parameters of the personalized model of IoT terminal device z in the nth round. z w represents the local parameters of IoT terminal device z in the nth round. z,i Indicates w z The parameters of the i-th layer, λ represents the parameters of the i-th layer of the global model used by IoT terminal device z in the n-th round. z,i This indicates the control parameters.

[0033] In some embodiments, the global update model is sent to the macro base station and aggregated with the models obtained by other IoT terminal devices in the current cluster performing global training tasks to update the global model of the current cluster. The calculation formula is as follows:

[0034]

[0035] in, p represents the parameters of the global model for the k-th cluster in the (n+1)th round. z This represents the weight of the IoT terminal device z during aggregation. This represents the parameters of the global update model for IoT terminal device z in the nth round.

[0036] In some embodiments, the method further includes:

[0037] When the preset convergence condition is met, the federated multi-task training is stopped; the preset convergence condition is reaching the preset number of training rounds.

[0038] On the other hand, the present invention also provides a clustered federated multi-task learning device for the Internet of Things, including a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the above method.

[0039] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0040] The beneficial effects of the present invention are at least as follows:

[0041] The clustered federated multi-task learning method and apparatus for the Internet of Things (IoT) described in this invention clusters IoT terminal devices to make the data distribution within the same cluster more similar. A federated multi-task learning algorithm is executed in each cluster, and a global training task and a personalized training task are executed at each IoT terminal device. While achieving data sharing within the cluster, the local data of the IoT terminal devices are fully utilized for training on personalized tasks, which efficiently utilizes the local data of each IoT terminal device and improves the training effect.

[0042] Furthermore, during the local training process of each IoT terminal device, the local training rounds are adjusted according to computing power to make full use of the computing resources of each IoT terminal device and improve the model training efficiency.

[0043] Furthermore, using a global model to apply regularization constraints to personalized training tasks can effectively prevent overfitting, control the degree of personalization, and improve model quality.

[0044] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0045] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0046] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings:

[0047] Figure 1 This is a schematic diagram of the Internet of Things (IoT) structure described in this invention.

[0048] Figure 2 This is a schematic diagram of the IoT terminal device clustering process in one embodiment of the present invention.

[0049] Figure 3 This is a schematic diagram of the IoT terminal device clustering process in another embodiment of the present invention.

[0050] Figure 4 This is a flowchart illustrating a clustered federated multi-task learning method for the Internet of Things according to an embodiment of the present invention.

[0051] Figure 5This is a logical schematic diagram of a clustered federated multi-task learning method for the Internet of Things according to another embodiment of the present invention.

[0052] Figure 6 This is a schematic diagram of intra-cluster global model aggregation in a clustered federated multi-task learning method for the Internet of Things according to an embodiment of the present invention.

[0053] Figure 7 The graph shows the change in average accuracy of the FedAvg (Federated Learning) and PCFML (Clustered Federated Multi-Task Learning) algorithms with the number of training epochs when the Ratio is set to 0.7 on the MNIST dataset.

[0054] Figure 8 The graph shows the change in average accuracy of the FedAvg and PCFML algorithms with training epochs when the Ratio is set to 0.9 on the MNIST dataset.

[0055] Figure 9 The graph shows the change in average accuracy of the FedAvg and PCFML algorithms with training epochs when the Ratio is set to 0.7 on the CIFAR10 dataset.

[0056] Figure 10 The graph shows the change in average accuracy of the FedAvg and PCFML algorithms with training epochs when the Ratio is set to 0.5 on the CIFAR10 dataset. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0058] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0059] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0060] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0061] In traditional federated learning methods, some employ knowledge distillation techniques. Clients upload model predictions instead of model parameters, allowing each client to choose its own designed model architecture. The knowledge provided by each client is dynamically aggregated, resulting in a more comprehensive understanding of the clients' model knowledge. After aggregation, the server returns pseudo-label information to the clients instead of the distribution of model predictions from the public data, improving transmission and communication efficiency.

[0062] Some methods utilize blockchain smart contracts to create federated learning tasks. Data samples are loaded from terminal devices and unloaded onto edge computing devices for local model training. The edge computing devices encrypt the locally trained model parameters and upload them to the blockchain. After consensus is reached among the blockchain nodes, a new block is generated. The smart contract aggregates the model parameters and updates the overall model. The smart contract determines whether the model's preset convergence condition has been met. If not, the next round of training begins; if it has, the federated learning task is terminated. The edge computing devices train personalized models based on global model information combined with their own data.

[0063] However, in the Internet of Things (IoT) scenario, the differences in device configuration and local data structure lead to significant heterogeneity among IoT terminal devices. This makes it impossible for traditional federated learning methods in the existing technology to take into account the heterogeneous features of different terminals in terms of data and models.

[0064] Based on this, one aspect of the present invention provides a clustered federated multi-task learning method for the Internet of Things, such as... Figure 1 As shown, the method runs on an IoT terminal device, which is connected to a macro base station via a micro base station relay. The IoT terminal device is divided into a preset number of clusters with the goal of minimizing data distribution differences.

[0065] It should be noted that IoT terminal devices can include mobile phones, computers, vehicles equipped with smart in-vehicle systems, smart home appliances, and other user terminal devices capable of storing and executing programs. Micro base stations are used to establish relays between IoT terminal devices and macro base stations. Each micro base station's MEC server has a certain computing power and covers several IoT terminal devices through connected macro base stations. The MEC server within the macro base station has powerful computing and communication resources.

[0066] In this embodiment, to reduce the heterogeneity of IoT terminal devices at the data level, the similarity of data distribution is evaluated, and IoT terminal devices with similar data distributions are grouped into the same cluster for federated multi-task learning. For example, wearable smart devices used to collect vital sign information may include smartwatches, smart bracelets, smart blood pressure monitors, etc. The data collected by these devices mainly includes parameters such as heart rate, body temperature, blood pressure, and blood oxygen. Due to differences in collection methods, the data structures of smartwatches and smart bracelets are relatively similar, while the data structure collected by smart blood pressure monitors differs somewhat from that of smartwatches and smart bracelets. During the analysis of the vital sign analysis model, different devices require personalized training due to differences in application scenarios, data structures, and analysis objectives. Therefore, on the one hand, to make training more efficient and avoid the model failing to converge due to insufficient local data on the terminal, and on the other hand, to ensure that training considers device and data heterogeneity, this embodiment clusters the IoT terminal devices, grouping smartwatches and smart bracelets into one cluster and smart blood pressure monitors into another.

[0067] Specifically, the method for clustering IoT terminal devices involves determining the similarity of local data distribution. In some embodiments, this method uses the k-means algorithm to cluster the IoT terminal devices, dividing them into a predetermined number of clusters with the goal of minimizing data distribution differences. Figure 2 As shown, steps S101 to S102 are included:

[0068] Step S101: Randomly initialize a preset number of IoT terminal devices as centroids to establish corresponding clusters.

[0069] Step S102: Calculate the distribution distance between the local data of the remaining IoT terminal devices and the local data of each centroid, assign them to the cluster corresponding to the centroid with the closest distribution distance, and calculate the distribution mean of the local data of each IoT terminal device in each cluster to update the centroid, until all IoT terminal devices are classified.

[0070] In this embodiment, the k-means algorithm is used as a clustering algorithm, where k represents the number of categories and means represents the mean. By pre-setting the value of k and the initial centroid of each category, similar data points are divided, and the optimal clustering result is obtained through iterative optimization using the mean of the division. Here, the distribution distance is calculated using Euclidean distance. IoT terminal devices with relatively close Euclidean distances in local data are grouped into the same cluster to address heterogeneity.

[0071] In some embodiments, the method employs the elbow method to determine the preset number of clusters, and the IoT terminal devices are divided into a preset number of clusters with the goal of minimizing data distribution differences, such as... Figure 3 As shown, steps S201 to S203 are included:

[0072] Step S201: Randomly select an IoT terminal device as the cluster center.

[0073] Step S202: Calculate the distribution distance between the remaining IoT terminal devices and the local data of each existing cluster center, and select the IoT terminal device with the largest distribution distance as the new cluster center.

[0074] Step S203: Iterate until a preset number of cluster centers are selected.

[0075] In steps S101-S102, when the centroids are randomly selected, the k-means algorithm has low time complexity due to this random selection method. However, a drawback is that it can result in two centroids being too close together. Steps S201-S203, in the initial stage, select only one IoT terminal device as the cluster center (centroid). By continuously calculating the distances from the remaining points to each cluster center, the IoT terminal device with the largest distance is established as the new cluster center, effectively solving the problem of two cluster centroids being too close.

[0076] In other embodiments, the elbow method can be used to select the inflection point to determine the final number of clusters. The elbow method uses the sum of average errors (SSE) to measure the clustering quality. As the number of cluster categories increases, the decrease in SSE will be sharp, and then tend to level off as the value of k continues to increase. The inflection point is then used as the final number of cluster categories.

[0077] Specifically, after clustering IoT terminal devices according to the distribution of local data, the clustered federated multi-task learning method for IoT performs federated multi-task training. In each training round, such as... Figure 4 and Figure 5 As shown, steps S301 to S304 are included:

[0078] Step S301: Download the global model of the current cluster to which the current IoT terminal device belongs in this round from the macro base station.

[0079] Step S302: Use the local data of the current IoT terminal device to perform a global training task on the global model. The global training task uses the first loss function to update the parameters and obtain a global updated model.

[0080] Step S303: Use the local data of the current IoT terminal device to perform personalized training on the personalized model in this round. The personalized training uses the second loss function to update the parameters and obtain the updated personalized model. The second loss function combines the parameters of the global model with the first loss function to perform regularization constraints on the personalized training task.

[0081] Step S304: Send the global update model to the macro base station and aggregate it with the models obtained by other IoT terminal devices in the current cluster performing global training tasks to update the global model of the current cluster.

[0082] In step S301 of this embodiment, federated multi-task training is performed on each cluster, and each cluster has its own global model. The global model is downloaded to each IoT terminal device within the cluster and trained using the local data of each device. First, in step S302, a global training task is executed within the cluster, using the local data of each IoT terminal device within the cluster for training. Each IoT terminal device is trained according to a first loss function. The trained global update model is returned to the macro base station in step S304 for aggregation and updating of the global model. In some embodiments, the first loss function is the cross-entropy loss function.

[0083] In step S303, the IoT terminal devices within the cluster also perform personalized training tasks based on their own task objectives. In this personalized training, local data is used to train the personalized model of the current IoT terminal in this round. The structure of the personalized model is consistent with the global model, and it is obtained through transfer learning based on the global model in the initial stage. The structures of the personalized model and the global model are set according to the specific task. In the personalized training task, a second loss function is configured according to the specified objective for training. However, to prevent overfitting and control the degree of model personalization, the second loss function introduces the parameters of the global model on top of the first loss function to apply regularization constraints to the personalized training.

[0084] In some embodiments, the second loss function, based on the first loss function, incorporates the parameters of the global model to apply regularization constraints to the personalized training task. The expression for the second loss function is:

[0085]

[0086] Among them, F z,P (·) represents the second loss function, F z,g (·) represents the first loss function, w z D represents the parameters of the personalized model of IoT terminal device z in the nth round. z w represents the local parameters of IoT terminal device z in the nth round. z,i Indicates w z The parameters of the i-th layer, λ represents the parameters of the i-th layer of the global model used by IoT terminal device z in the n-th round. z,i This indicates the control parameters.

[0087] In step S304, the global update model is sent to the macro base station and aggregated with the models obtained by other IoT terminal devices in the current cluster through global training tasks to update the global model of the current cluster. The calculation formula is as follows:

[0088]

[0089] in, p represents the parameters of the global model for the k-th cluster in the (n+1)th round. z This represents the weight of the IoT terminal device z during aggregation. This represents the parameters of the global update model for IoT terminal device z in the nth round.

[0090] In steps S301 to S304, federated multi-task training stops when a preset convergence condition is met; the preset convergence condition is reaching a preset number of training rounds. In some other embodiments, the preset convergence condition may also be stopping when the loss function is less than a set value.

[0091] The time required for each IoT terminal device to perform a round of local training varies due to differences in hardware structure, computational load, and data volume. Considering that the time required for each IoT terminal device to perform a round of local training typically depends on the device that takes the longest, if all IoT terminal devices perform the same number of local training rounds, devices with smaller data volumes or stronger computing power, resulting in shorter local training times, will remain idle for extended periods. To maximize the utilization of computing resources, this embodiment also adjusts the number of local training rounds to prevent wasted computing resources.

[0092] Specifically, in some embodiments, the method further adjusts the local training rounds according to the computing power of each IoT terminal device in the cluster, including steps S401 to S403:

[0093] Step S401: Calculate the latency required for each IoT terminal device to perform one round of local training. The calculation formula is as follows:

[0094]

[0095] Among them, l k,z |D represents the latency required for IoT terminal device z in the k-th cluster to perform one round of local training. z | represents the amount of local data on the IoT terminal device z, f z This represents the computing power of the IoT terminal device z.

[0096] Step S402: Obtain the latency of the IoT terminal device that takes the longest time to perform one round of local training in each cluster as the longest latency of that cluster. The expression is:

[0097]

[0098] Among them, l k,max This represents the longest latency required for local training among all IoT terminal devices in the k-th cluster;

[0099] Step S403: Adjust the number of rounds of local training performed by each IoT terminal device in each cluster according to the ratio of the latency required for each IoT terminal device to perform one round of local training to the longest latency corresponding to its cluster. The calculation formula is as follows:

[0100]

[0101] Among them, e z This indicates the number of training rounds for the IoT terminal device z locally. This represents the local training round of the IoT terminal device in the k-th cluster that requires the longest latency to perform local training.

[0102] In steps S401-S403, for IoT terminal devices in a cluster, the longer the latency required to perform one round of local training, the fewer rounds of local training are needed. Ultimately, this maintains a roughly equal timeframe for each IoT terminal device to complete local training within one large training round, ensuring that the computing resources of all devices are fully utilized. It's important to clarify that a large round refers to the process where each IoT terminal device in the cluster downloads the global model, performs one global training task, one personalized training task, and returns a globally updated model to complete the aggregation. Local training refers to the process of locally training and updating both the global and personalized models.

[0103] On the other hand, the present invention also provides a clustered federated multi-task learning device for the Internet of Things, including a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the above method.

[0104] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0105] The present invention will now be described with reference to specific embodiments:

[0106] This embodiment proposes a clustered federated multi-task learning method for the Internet of Things (IoT), primarily based on the idea of ​​personalized federated learning. Personalized federated learning is a solution to the data heterogeneity problem built upon federated learning. While enjoying the advantages of knowledge sharing, personalized federated learning considers the specificity of each terminal, establishing a personalized model suitable for each terminal. Personalized federated learning methods mainly include federated multi-task learning, federated transfer learning, and federated meta-learning. Among them, multi-task learning obtains additional valuable information by mining the relationships between tasks. In federated learning, building models for different terminals can be regarded as multiple tasks. Therefore, multi-task learning can be naturally applied to federated learning to fit independent but related models for multiple terminals. However, in the case of data heterogeneity, data sharing and the personalization of terminal models are often mutually exclusive. For a certain terminal, the knowledge obtained from other terminals may be meaningless in the process of personalization. Therefore, this embodiment proposes a clustered federated multi-task learning scheme, which divides terminals with similar data distributions into clusters. Terminals within the same cluster undergo collaborative multi-task training due to their similarity. By learning some common low-level abstract features through the hidden layers at the bottom of the shared model network, data sharing within the cluster is achieved.

[0107] The system architecture constructed in this embodiment is as follows: Figure 1 As shown, the traditional federated learning training process mainly includes local model training on the terminal, uploading terminal parameters, and macro base station execution of model aggregation and parameter distribution. Unlike traditional federated learning tasks that uniformly perform collaborative training on all terminals, this embodiment clusters the terminals, linking highly correlated terminals together. Traditional federated learning tasks are executed within each cluster, and the global model within the cluster provides a reference for personalized federated learning on the terminals within that cluster. Terminals complete the training of personalized models through soft parameter sharing in multi-task learning. The scheme proposed in this embodiment balances data sharing within clusters and model personalization, improving model performance in areas such as accuracy. Furthermore, because the tasks within the same cluster are highly correlated, the scheme proposed in this embodiment also has an advantage in convergence speed compared to traditional federated learning schemes.

[0108] An IoT network consists of IoT terminal devices, micro base stations, macro base stations, and corresponding mobile edge computing (MEC) servers. The MEC servers within macro base stations possess powerful computing and communication resources. Micro base stations act as relays between IoT terminal devices and macro base stations. Each micro base station's MEC server has a certain computing capacity and covers several IoT terminal devices through connected base stations. The set of IoT terminal devices is denoted by Z, and D is the set of all such devices. z ={x z ,y zLet} represent the local dataset of terminal z, where z∈Z.

[0109] The clustered federated multi-task learning method for the Internet of Things shown in this embodiment is as follows: Figure 5 As shown, the steps are as follows:

[0110] 1.1 The macro base station clusters IoT terminal devices, and the IoT terminal devices initialize personalized models locally.

[0111] 1.2 The IoT terminal device downloads the global model of its current round from the macro base station and generates a copy locally.

[0112] 1.3 The IoT terminal device performs a global model training task on the local dataset, completes the global model copy update, and sends the copy to the macro base station.

[0113] 1.4 IoT terminal devices perform personalized model training tasks on local datasets to complete personalized model updates.

[0114] 1.5 At the macro base station, aggregate the global model copies of IoT terminal devices and update the global model.

[0115] 1.6 If the preset convergence condition is not met, proceed to step 1.2; otherwise, end the process.

[0116] In IoT scenarios, IoT terminal devices often perform similar tasks but their data distributions differ. During the clustering phase, the macro base station clusters the IoT terminal devices based on their data distribution. First, the macro base station sends requests to all IoT terminal devices to query their data distribution. Upon receiving the requests, the IoT terminal devices reply to the macro base station with their own data distribution. Based on the data distribution of all terminals, the macro base station uses the k-means algorithm to cluster the IoT terminal devices. IoT terminal devices with similar data distributions are grouped into one cluster. The clustering steps include:

[0117] 2.1 Initialize the centroid randomly according to the number of clusters.

[0118] 2.2 Calculate the distance from the terminal data distribution to all centroids, select the centroid with the closest distance, and assign the terminal to the cluster to which the centroid belongs.

[0119] 2.3 Adjust the position of the center of mass.

[0120] 2.4 Repeat steps 2.1 to 2.3 until the clustering of each IoT terminal device no longer changes.

[0121] The k-means algorithm requires a pre-defined number of clusters, but in this embodiment, the number of clusters for the IoT terminal devices is unknown beforehand. Therefore, the elbow method is used to determine the number of clusters. SSE (Self-Sensitive Segmentation) is used as a criterion for evaluating the quality of the cluster number setting. As the number of clusters increases, the sample division becomes more refined, and the SSE decreases. When the number of clusters is less than the optimal number, the SSE decreases rapidly with increasing cluster number. When the number of clusters is greater than the optimal number, the rate of SSE decrease slows down. Based on this, the optimal number of clusters can be determined.

[0122] After clustering, the cluster set is denoted by K. Federated multi-task learning is performed within each cluster, and the model trained on IoT terminal devices is considered to be a neural network. The hidden layers of the neural network are considered to play a role in feature extraction. Each model has multiple hidden layers. The bottom hidden layers learn some low-level features, and the shared bottom hidden layers within the cluster can learn some common low-level abstract features. A portion of higher-level hidden layers are reserved for each terminal's personalized model, so that the terminal's personalized model can learn unique, higher-level abstract features.

[0123] In traditional federated learning schemes, IoT terminal devices have the same number of local training rounds. However, in the IoT, due to differences in data volume and computing power among terminals, the time for each local training round varies. If the number of local training rounds is the same, the time for one round of global training depends on the client with the longest local training time. IoT terminal devices with shorter local training times remain in a waiting state after uploading their locally updated parameters, resulting in a waste of computing resources. To address this issue, this embodiment adjusts the number of local training rounds for terminals based on clustering results. The specific adjustment method can be found in steps S401-S403.

[0124] This embodiment achieves the sharing of the bottom hidden layer through soft parameter sharing. Soft parameter sharing does not require the parameters of the bottom hidden layer of different terminal models to be exactly the same, but rather encourages parameter similarity.

[0125] Reference Figure 6 Let the parameters of the personalized model trained on the IoT terminal device z in the nth round be w. z ,w z,i Indicates w z The parameters of the i-th layer, and the global model parameters of cluster k after the n-th round of global aggregation are: express The parameters of layer i, and the IoT terminal device z producing a copy of the global model of its respective cluster in round n. The set of hidden layers participating in soft parameter sharing is s. The local training performed on z will be split into two parts: a task for global model training.g and tasks used for personalized model training p Define the task to be executed on z. g The loss function at that time is The specific form of the loss function depends on the task. Taking a multi-class classification task as an example, the loss function is... Use the cross-entropy loss function. Execute the task on z. g The gradient at time is right The update is represented as:

[0126]

[0127]

[0128] Where, ε z Indicates the execution of task g The step size during each round of federated multi-task learning. g It can be executed multiple times locally. The data is sent from the terminal to the macro base station for this round of aggregation in the global model.

[0129] Execute task on z p The loss function at that time is F z,P (w z D z ), w z The update is represented as:

[0130]

[0131]

[0132] Where, η z Indicates the execution of task p The step size of time, This means executing the task on z. p The gradient over time. Similarly, during each round of federated multi-task learning, the task... p This process can be executed multiple times. To further improve the efficiency of federated multi-task learning, since there is a significant communication latency during parameter upload, the task can be started while parameters are being uploaded. p The execution. Here, λ z,i To control the parameters of similarity between the personalized model and the global model, λ is adjusted. z,i The degree of personalization in the model can be controlled. λ z,i When the value is large, on the shared hidden layer set s, the parameters of the terminal personalized model are forced to approach the parameters of the global model, resulting in a high degree of data sharing. z,iWhen the size is small, there are fewer constraints on the terminal personalization model, and the personalization level of the model is higher.

[0133] After collecting the global model update results of all terminals within cluster k in this round of federated multi-task learning, the macro base station aggregates the results and completes the global model update. The calculation formula is as follows:

[0134]

[0135] in, p represents the parameters of the global model for the k-th cluster in the (n+1)th round. z This represents the weight of the IoT terminal device z during aggregation. This represents the parameters of the global update model for IoT terminal device z in the nth round.

[0136] The effects of the present invention will be described below with reference to specific embodiments:

[0137] This embodiment conducted experiments on the MNIST and CIFAR10 datasets, comparing the experimental results of the PCFML algorithm (clustered federated multi-task learning algorithm) of this invention with the FedAvg algorithm (federated learning algorithm).

[0138] Both the MNIST and CIFAR-10 datasets contain 10 categories, and similar processing is applied to both datasets. The training set is divided into equal-length subsets based on the number of terminals, with each subset having a "main class." The "main class" is the label with the largest proportion in the subset, and is considered to represent the category to which a terminal belongs. A parameter `ratio` is set, which adjusts the proportion of the "main class" in the subsets; a larger parameter results in a more extreme data distribution. The test data is divided into 10 subsets, each with a "main class" at a ratio. A terminal's training and test sets share the same "main class."

[0139] A subset of the dataset was randomly assigned to the terminal as its local dataset. The terminal was trained using the local training set and tested on the test set. LeNet was used as the training model, with a model structure of 2 convolutional layers and 3 fully connected layers, and the Adam optimizer was used. The average accuracy of the terminal on the local test set was used as the primary evaluation metric.

[0140] Experimental results on the MNIST dataset are as follows: Figure 7 and 8 As shown, when the ratio is set to 0.7, the relationship between the number of training epochs and the average accuracy of the terminal on the test set is as follows: Figure 7As shown, the accuracy of both FedAvg and PCFML increases and then stabilizes with each training epoch. After convergence, PCFML shows an accuracy improvement of about 3% compared to FedAvg. It was also observed that PCFML converges faster.

[0141] When the ratio is set to 0.9, the data distribution on the terminal becomes more extreme. The relationship between the number of training epochs and the average accuracy of the terminal on the test set is as follows: Figure 8 As shown, the accuracy of FedAvg decreased significantly, while the accuracy of PCFML remained at a relatively high level. When both FedAvg and PCFML converged, PCFML's accuracy improved by approximately 24% compared to FedAvg, while maintaining its advantage of faster convergence. Meanwhile, when the data distribution was more extreme, the average accuracy of FedAvg showed significant fluctuations.

[0142] Experimental results on the CIFAR-10 dataset are as follows: Figure 9 and 10 As shown, when the dataset is CIFAR-10 and the ratio is set to 0.7, the relationship between the number of training epochs and the average accuracy of the terminal on the test set is as follows. Figure 9 As shown, FedAvg has a lower accuracy rate because the LeNet model has a simpler structure, and CIFAR-10 is more challenging for image recognition than MNIST. Overfitting occurred during training with PCFML, which was addressed using early stopping. Compared to FedAvg, PCFML still offers significant improvements in accuracy and convergence speed.

[0143] When the ratio is set to 0.5, the relationship between the number of training epochs and the average accuracy of the terminal on the test set is as follows: Figure 10 As shown, the accuracy of FedAvg improves as the extreme nature of the data distribution decreases, and the difference in average accuracy between FedAvg and PCFML after convergence narrows. Based on experiments on the MNIST dataset, it is believed that PCFML has a greater advantage when the data distribution of terminals within different clusters varies significantly.

[0144] Therefore, this embodiment provides a clustered federated multi-task learning method and apparatus for the Internet of Things (IoT), focusing on the personalization of terminal models to mitigate the adverse effects of data heterogeneity. Terminals are clustered based on their data characteristics, and federated multi-task learning is performed among terminals with higher similarity, eliminating the negative impact of weight divergence during aggregation when terminals with large data distribution differences are involved. The number of local training rounds for IoT terminal devices is adjusted according to the clustering situation. IoT terminal devices with strong computing power or small data volumes perform multiple local training rounds, avoiding idle waiting for some terminals and improving the utilization of computing resources. This drives the similarity of some hidden layer parameters in the personalized models of different terminals within the same cluster. It allows the personalized models of IoT terminal devices to balance data sharing within the cluster with model personalization, and the degree of model personalization can be adjusted. Simultaneously, the training of the global model and the training of the personalized model are decoupled, allowing personalized model training and parameter uploading and aggregation to be performed simultaneously, reducing the waste of computing resources.

[0145] In summary, the clustered federated multi-task learning method and apparatus for the Internet of Things (IoT) of this invention clusters IoT terminal devices to make the data distribution within the same cluster more similar, and executes a federated multi-task learning algorithm within each cluster. Global training tasks and personalized training tasks are performed on each IoT terminal device. While achieving data sharing within the cluster, it fully utilizes the local data of the IoT terminal devices for training on personalized tasks, efficiently utilizing the local data of each IoT terminal device and improving the training effect.

[0146] Furthermore, during the local training process of each IoT terminal device, the local training rounds are adjusted according to computing power to make full use of the computing resources of each IoT terminal device and improve the model training efficiency.

[0147] Furthermore, using a global model to apply regularization constraints to personalized training tasks can effectively prevent overfitting, control the degree of personalization, and improve model quality.

[0148] Corresponding to the above method, the present invention also provides a clustered federated multi-task learning method apparatus for the Internet of Things. The apparatus includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the apparatus implements the steps of the method as described above.

[0149] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0150] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0151] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0152] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0153] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A clustered federated multi-task learning method for the Internet of Things, characterized in that, The method runs on an IoT terminal device, which is connected to a macro base station via a micro base station relay. The IoT terminal device is divided into a predetermined number of clusters with the goal of minimizing data distribution differences. The method is used to perform federated multi-task training, and each training round includes: Download the global model of the current cluster to which the current IoT terminal device belongs in this round from the macro base station; The global model is trained using local data from current IoT terminal devices. The global training task uses a first loss function to update the parameters, resulting in a globally updated model. The local data of the current IoT terminal device is used to perform personalized training on the personalized model in this round. The personalized training uses a second loss function to update the parameters and obtain an updated personalized model. The second loss function combines the parameters of the global model with the first loss function to apply regularization constraints to the personalized training task. The global update model is sent to the macro base station and aggregated with the models obtained by other IoT terminal devices in the current cluster through global training tasks to update the global model of the current cluster. The method employs the k-means algorithm to cluster the IoT terminal devices. The IoT terminal devices are divided into a preset number of clusters with the goal of minimizing data distribution differences. The process includes: randomly initializing the preset number of IoT terminal devices as centroids to establish corresponding clusters; calculating the distribution distance between the local data of each remaining IoT terminal device and the local data of each centroid, assigning them to the cluster corresponding to the centroid with the closest distribution distance, and calculating the distribution mean of the local data of each IoT terminal device within each cluster to update the centroids, until all IoT terminal devices are classified. Alternatively, the method employs the elbow method to determine the preset number of clusters, wherein the IoT terminal devices are divided into a preset number of clusters with the goal of minimizing data distribution differences, including: randomly selecting an IoT terminal device as a cluster center; calculating the distribution distance between the local data of the remaining IoT terminal devices and the existing cluster centers one by one, and selecting the IoT terminal device with the largest distribution distance as a new cluster center; iterating until the preset number of cluster centers are selected.

2. The clustered federated multi-task learning method for the Internet of Things according to claim 1, characterized in that, The method also adjusts the local training rounds based on the computing power of each IoT terminal device in the cluster, including: The latency required for each IoT terminal device to perform one round of local training is calculated using the following formula: in, This represents the latency required for IoT terminal device z in the k-th cluster to perform one round of local training. This represents the amount of local data on the IoT terminal device z. This represents the computing power of the IoT terminal device z; The latency of the IoT terminal device that takes the longest time to perform one round of local training in each cluster is taken as the longest latency of that cluster. The expression is: in, This represents the longest latency required for local training among all IoT terminal devices in the k-th cluster; The number of rounds of local training for each IoT terminal device in each cluster is adjusted based on the ratio of the latency required for each IoT terminal device to perform one round of local training to the longest latency corresponding to its cluster. The calculation formula is as follows: in, This indicates the number of training rounds for the IoT terminal device z locally. This represents the local training round of the IoT terminal device in the k-th cluster that requires the longest latency to perform local training.

3. The clustered federated multi-task learning method for the Internet of Things according to claim 1, characterized in that, The second loss function, based on the first loss function and incorporating the parameters of the global model, applies regularization constraints to the personalized training task. The expression for the second loss function is: in, This represents the second loss function. This represents the first loss function. Let represent the parameters of the personalized model of IoT terminal device z in the nth round. This represents the local parameters of the IoT terminal device z in the nth round. express The parameters of the i-th layer, This represents the parameters of the i-th layer of the global model used by IoT terminal device z in the n-th round. This indicates the control parameters.

4. The clustered federated multi-task learning method for the Internet of Things according to claim 3, characterized in that, The global update model is sent to the macro base station and aggregated with the models obtained by other IoT terminal devices in the current cluster through global training tasks to update the global model of the current cluster. The calculation formula is as follows: in, This represents the parameters of the global model for the k-th cluster in the (n+1)th round. This represents the weight of the IoT terminal device z during aggregation. This represents the parameters of the global update model for IoT terminal device z in the nth round.

5. The clustered federated multi-task learning method for the Internet of Things according to claim 4, characterized in that, The method further includes: When the preset convergence condition is met, the federated multi-task training is stopped; the preset convergence condition is reaching the preset number of training rounds.

6. A clustered federated multi-task learning device for the Internet of Things, comprising a processor and a memory, characterized in that, The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images