Task granularity model aggregation method in edge side federated continuous learning
By employing a task-granular model aggregation method and utilizing task memory palace and knowledge distillation techniques, the problem of model aggregation for heterogeneous task sequences in federated learning is solved, thereby improving the model accuracy of edge devices and reducing resource overhead.
Patent Information
- Application Number
- CN202411504309.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2025-11-25
- Estimated Expiration
- 2044-10-25
AI Technical Summary
Existing federated learning algorithms suffer from negative transfer during model aggregation and high communication overhead on edge devices, especially in heterogeneous task sequence scenarios, leading to decreased model accuracy and wasted resources.
A task-granular model aggregation method is adopted, which uses a task memory palace data structure and four functional modules (parameter classification, knowledge extraction, similar model selection, model aggregation, and task knowledge integration) to identify and aggregate similar task knowledge. Optimal transmission and knowledge distillation techniques are used to reduce communication and computational overhead.
It improves the model's accuracy in the current task, reduces the computational and communication overhead of edge devices, and adapts to the edge environment of heterogeneous task sequences.
Smart Images
Figure CN119578584B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method for task-granularity model aggregation in edge-side federated continuous learning. Background Technology
[0002] Today, with hundreds of millions of Internet of Things (IoT) devices generating trillions of bytes of data at the network edge, opportunities have emerged for deploying artificial intelligence on edge devices. Specifically, the functionality of edge devices has expanded from simple photography to include facial recognition, text translation, and intelligent recommendations, encompassing applications such as image classification, natural language processing, graph recognition, and multimodal computing. To address these diverse needs, various neural network models (or simply models) can be deployed and trained at these edge devices. Considering that a single edge device's limited training samples cannot effectively handle local tasks, federated learning can be used to allow multiple edge devices to jointly train models. Through federated learning, while ensuring the privacy and security of local data on the edge clients (or simply clients), models on each edge device can leverage their strengths and compensate for their weaknesses to better handle local tasks.
[0003] However, in real-world applications, each edge device faces different tasks with varying data distributions and labels, and these tasks are constantly evolving. For example, one edge device might be performing a series of animal image classification tasks, while another device's model is continuously learning to classify different types of objects like cars and plants; the tasks differ significantly. This means that when edge devices interact (model aggregation), they will face completely inconsistent tasks.
[0004] Existing federated learning algorithms are primarily based on model-level aggregation mechanisms. Specifically, after each edge device sends its model to the server, the server aggregates these models directly into a global model using a specific method. However, when faced with inconsistent task sequences from edge devices, this aggregation strategy presents two challenges. First, due to the significant differences in tasks across edge devices, direct model aggregation may lead to "negative transfer," meaning that the accuracy of some edge device models after aggregation is actually lower than before aggregation. Second, as tasks continuously change, each aggregation process requires not only the transmission of a large number of model parameters but also the additional transmission of historical task information, resulting in substantial communication overhead. In edge computing environments, this significant communication overhead severely impacts system efficiency and performance. Existing federated learning aggregation strategies and their inherent problems include:
[0005] Federated Continuous Learning: Federated continuous learning methods evolved from general federated learning methods. They are used to address incremental categories or tasks on various clients (edge devices) to avoid catastrophic forgetting. These methods consider building a global knowledge base or using regularization parameters to constrain the learning of each task on the client, preventing the client model from excessively shifting towards new tasks and forgetting previously learned tasks. These methods can balance the task sequences on various clients well, but they all assume that the tasks on the clients being aggregated are consistent each time. However, in real-world edge scenarios, the tasks on each client are different. Using federated continuous learning methods will cause them to be affected in each round of aggregation, leading to a decrease in the accuracy of the client models. At the same time, in order to obtain task information from other clients, these methods need to transmit and compute client task parameters, which will cause additional communication and computational overhead, thus significantly increasing training time and communication burden.
[0006] Federated learning based on heterogeneous models: These methods aim to aggregate models with different architectures uploaded from different clients. To address the challenges posed by inconsistent model architectures, they employ knowledge distillation techniques for training. These methods mainly fall into two categories: one involves building additional small models with consistent architectures on the client side, which then perform aggregation by transmitting these architecture-consistent models, followed by knowledge transfer through knowledge distillation; the other maintains a shared dataset on the server, and clients aggregate by transmitting inference results from this shared dataset. However, these methods still model within similar task scenarios, as all client models are trained for the same task. Therefore, in real-world scenarios involving the aggregation of heterogeneous task sequences, a significant drop in accuracy may occur.
[0007] Cluster-based federated learning: This type of method divides participating clients into different clusters / groups, where clients within each cluster / group have similar data. It utilizes traditional clustering methods (such as KNN, K-means) for clustering and uses methods like cosine similarity to evaluate the task similarity among clients. This approach assumes that the client tasks are clustered during each aggregation, which is difficult to guarantee in scenarios involving heterogeneous task sequences. A client's current aggregation task might be similar to a task already learned by another client. This results in dissimilar clients being clustered, leading to a decrease in accuracy after aggregation.
[0008] The key to solving the above problems lies in considering the task differences on edge devices and being able to aggregate models at the task granularity. That is, it's crucial to identify the task differences between various edge devices and aggregate model parts with high task similarity. To achieve this, tasks for each edge device need to be stored and managed appropriately so that matching can be performed based on task similarity during aggregation, thereby avoiding interference between different tasks and improving the model's accuracy for the current task. Simultaneously, to work in resource-constrained edge device scenarios, key knowledge for each task needs to be extracted to enable rapid searching of similar task knowledge while reducing communication overhead. Based on this, this invention provides a task-granularity model aggregation method in edge-side federated continuous learning. Summary of the Invention
[0009] This invention proposes a task-granularity model aggregation method in edge-side federated continuous learning, which includes the following data structure for organizing and managing tasks from various edge devices:
[0010] The task memory palace has a data structure that is deployed on a server. It stores the knowledge of all client tasks and organizes this knowledge in a top-down manner using RTree. In the task memory palace, each leaf node stores the task knowledge or an index of a set of similar task knowledge. By building a task memory palace, new task knowledge can be added quickly, and edge device (client) models can be supported to quickly search for similar tasks.
[0011] The proposed method for task-granularity model aggregation in edge-side federated continuous learning utilizes the following functional modules to aggregate similar tasks:
[0012] The parameter classification module receives multiple sets of model parameters from the edge side, and transmits the merged parameters to each edge device participating in the training through an internal aggregation algorithm.
[0013] The knowledge extraction module identifies and extracts the most relevant parameter information for the task from the trained model as knowledge for the task. After each task switch, the knowledge extraction module extracts and stores the task knowledge.
[0014] The similarity model selection module can search the task memory palace for task knowledge similar to each distillation model participating in the aggregation.
[0015] The model aggregation module is used for model aggregation. It rearranges the models by using the optimal transportation method to ensure that the parameters aggregated at each position have similar functions, thereby maximizing the acquisition of useful information and improving the accuracy of the aggregated model.
[0016] The task knowledge integration module is used to calculate and update the gradients of the distilled model and the locally trained model. It uses knowledge distillation to transfer information from other clients to the local model.
[0017] In view of the defects or deficiencies of existing technologies in this field, the present invention proposes a task adaptation and federated learning system for heterogeneous task sequences at the edge, comprising the following steps:
[0018] Step 1: The knowledge extraction module extracts each client's local model into a compact knowledge, forming a knowledge distillation model, and sends it to the server.
[0019] Step 2: The server uses the heterogeneous model selection module to search the task memory palace. The task memory palace includes task knowledge storage, searching for similar task knowledge storage, adding new task knowledge, and finding a portion of the most similar task knowledge for each client.
[0020] Step 3: The model aggregation module uses optimal transmission to sequentially aggregate the distillation model of each client and the most similar task knowledge of its selected parts, ensuring that the parameters aggregated at each position have similar functions, so as to maximize the acquisition of useful information and improve the accuracy of the distillation model.
[0021] Step 4: The server returns the updated distillation models from each client to the client. Each client will evaluate the distillation models before and after aggregation and select the distillation model with higher accuracy to pass to the task knowledge integration module. The task knowledge integration module uses knowledge distillation to transmit task information from other clients to the local training model in order to improve the accuracy of the model.
[0022] Step 5: The knowledge extraction module of each client retains 5%-15% of the model parameters with the largest weights according to the current model, and sets the remaining parameters to 0, adjusts the model, and stores the adjusted model parameters in the knowledge storage module as the task knowledge of the current task.
[0023] Furthermore, in step 2, the server searches for the task memory palace in the heterogeneous model selection module, and the specific steps are as follows:
[0024] Step 2.1: Randomly select a portion of the common samples from the server, and each distillation model uses the selected samples to perform one inference.
[0025] Step 2.2: Each distillation model sequentially searches for similar task knowledge from the task memory palace;
[0026] Step 2.3: After each distillation model finds similar task knowledge according to Step 2.2, the similarity between each distillation model and the selected task knowledge is compared, and then a portion of the most similar task knowledge is selected.
[0027] Furthermore, in step 2, the task memory palace includes task knowledge storage, searching for similar task knowledge storage, and adding new task knowledge, specifically as follows:
[0028] Compact task knowledge storage: The task memory palace uses model pruning techniques (such as compressed column storage) to convert the model weights corresponding to each task into their knowledge. The knowledge corresponds to the portion of the weights with the highest values, in order to retain the most important information and reduce memory usage.
[0029] Quick and accurate search for similar task knowledge: The task memory palace uses R-tree to organize similar task knowledge into the same node and organizes all task knowledge hierarchically. Even if the number of clients and tasks is large, similar task knowledge can be searched quickly.
[0030] Accurately and quickly add new task knowledge: The Task Memory Palace supports adding / deleting tree nodes to update task knowledge. Each time new task knowledge is added, the following steps are included:
[0031] First, task knowledge is transformed into a low-dimensional task index. Then, given that RTree is more efficient in low-dimensional space, dimensionality reduction techniques are used for further compression. Finally, the RTree is searched for the most similar related knowledge to the currently added task. During the search, multi-head cross-entropy loss is used to evaluate the relationship between two task knowledges. If the similarity between two task knowledges exceeds a threshold, the two task knowledges are merged into the same node; otherwise, a new node is inserted into the RTree.
[0032] Furthermore, in step 3, the model aggregation module uses optimal transfer to sequentially aggregate the distillation model for each client and the most similar task knowledge selected by it, specifically including:
[0033] Step 3.1: Using the optimal transportation method, the selected task knowledge will be rearranged and adjusted according to its corresponding distillation model.
[0034] Step 3.2: Use the average aggregation method to aggregate the distillation model and each of its selected similar task knowledge adjusted in step 3.1.
[0035] Furthermore, in step 5, the model adjustment includes:
[0036] Step 5.1: Randomly select a portion of data samples from the current task dataset, and the model performs an inference based on these data samples;
[0037] Step 5.2: Calculate the loss using data labels and inference results;
[0038] Step 5.3: Calculate the gradient based on the loss and use the gradient to update the parameters that have not been set to 0;
[0039] Step 5.4: Determine whether the loss has reached the loss threshold or whether the number of adjustments has reached the upper limit. If the conditions are not met, return to step 5.2; otherwise, proceed to step 5.3.
[0040] Furthermore, in step 2.3, the similarity calculation and comparison formula between the distillation model and the task knowledge obtained from the task memory palace is as follows:
[0041]
[0042] in, Let W be the distillation model for client i. j For the distillation model, task knowledge is selected from the task memory palace, L MCE For multi-head cross-entropy loss calculation; S a S w The calculated similarity value. Each knowledge distillation model needs to be compared with the corresponding similar task knowledge, and 3-5 of the most similar task knowledge are selected.
[0043] Furthermore, the optimal transportation method used in step 3.1 rearranges the task knowledge, and each rearrangement of task knowledge is calculated as follows:
[0044]
[0045] in, Let β be the parameter of the l-th layer of the distillation model for client i. (l) The generated random matrix has the same size as the current layer of the model. `diag()` performs matrix diagonalization. T (l) The optimal transfer matrix is calculated based on the current task knowledge and its corresponding distillation model.
[0046] Furthermore, in step 4, the gradient calculation for the knowledge distillation model is as follows:
[0047]
[0048] Among them, L CE () represents the cross-entropy loss, L KL () represents the KL divergence loss during distillation. For the local model of client i, These are the parameters for the pre-polymerization distillation model. These are the parameters for the distillation model after polymerization. X i ,Y iThe gradient calculation for updating the local parameters is as follows, using the local training samples of the current client:
[0049]
[0050] Where, λ kd ,λ kd′ These are the evaluation parameters for the distillation model before and after polymerization. γ is used to determine λ. kd ,λ kd′ Which is better? If λ kd >λ kd′ Then γ = 1, otherwise γ = 0.
[0051] Compared with the prior art in this field, the superior effects of the present invention are as follows:
[0052] 1. The task-granularity model aggregation method in edge-side federated continuous learning described in this invention addresses the problem of large differences in tasks among various edge devices in real edge environments. Against this background, this invention starts from the perspective of task similarity and ensures that similar tasks are aggregated through key data structures and four modules, thereby improving the accuracy of local model training.
[0053] 2. The task-granularity model aggregation method in edge-side federated continuous learning described in this invention transforms the local model of each client and all its learning tasks into a compact network through weight position encoding. It uses distillation networks to share and exchange knowledge with other clients in global aggregation, and then transfers the learned knowledge to the local model through knowledge distillation. When the client finishes learning a new task (i.e., the last round of training for the current task), it further extracts the knowledge of the task into a compressed network and sends the compressed network with a smaller number of parameters to the task knowledge palace in the server. Through the distillation and model compression encoding used above, the computational and communication overhead during edge task training and interaction can be significantly reduced. Attached Figure Description
[0054] Figure 1 This is a schematic diagram of the overall process of the method described in this invention;
[0055] Figure 2 This is a flowchart illustrating step 1 of the method described in this invention;
[0056] Figure 3 This is a flowchart illustrating step 2 of the method described in this invention;
[0057] Figure 4 This is a flowchart illustrating step 3 of the method described in this invention;
[0058] Figure 5 This is a flowchart illustrating step 4 of the method described in this invention;
[0059] Figure 6 This is a flowchart of step 5 of the method described in this invention;
[0060] Figure 7 (a)- Figure 7 (f) is a schematic diagram comparing the time and accuracy of the method described in this invention and other methods on an edge device;
[0061] Figure 8 This is a schematic diagram comparing the time of the method described in this invention with other methods at different bandwidth sizes;
[0062] Figure 9 (a)- Figure 9 (b) is a comparison chart of the accuracy of the method described in this invention and other methods in large-scale scenarios. Detailed Implementation
[0063] To better understand the above-mentioned objectives, features and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.
[0064] Example
[0065] like Figure 1-7 As shown, the task-granularity model aggregation method in edge-side federated continuous learning includes:
[0066] Step 1: The server transmits the parameters to the client.
[0067] Step 1.1: Select the weights of the client-uploaded parameters based on the number of clients participating in the training and the set selection ratio;
[0068] Step 1.2: The selected client extracts information from the currently trained local model into a distilled model using the knowledge distillation method, and uploads the distilled model to the server. Step 1 ends.
[0069] Step 2: The server uses the heterogeneous model selection module to search the task memory palace and finds the most similar task knowledge for each client.
[0070] Step 2.1: Randomly select a portion of common samples from the server. Each distillation model uses the selected samples to perform one inference. After inference, the corresponding model vector is generated. Step 2.1 ends.
[0071] Step 2.2: Select similar task knowledge from the task memory palace based on the computational vector of the distillation model:
[0072] Step 2.2.1: Set a hyperparameter to represent the minimum similarity of task knowledge that the current distillation model needs to find;
[0073] Step 2.2.2: Search the task memory palace according to the hyperparameters. Use RTree to traverse from top to bottom to find the most similar task knowledge group. Step 2.2.2 ends.
[0074] Step 2.3: After each distillation model finds similar task knowledge according to Step 2.2, a similarity comparison is performed between each distillation model and the selected task knowledge.
[0075] Step 2.3: Select the most similar task knowledge based on the similarity comparison in Step 2.3;
[0076] Step 3, aggregation of knowledge about the distillation model and its selected similar tasks:
[0077] Step 3.1: Reorder the similar task knowledge selected for each distillation model using the optimal transportation method:
[0078] Step 3.1.1: Calculate the optimal transport matrix for each similar task knowledge and distillation model in sequence;
[0079] Step 3.1.2: Use the optimal transportation matrix to rearrange the selected task knowledge to form new parameters. Step 3.1.2 ends here.
[0080] Step 3.2: Use the average aggregation method to aggregate the distillation model and each of its selected similar task knowledge adjusted in Step 3.1;
[0081] Step 4: Transfer the information from the polymerized distillation model to the local model:
[0082] Step 4.1: Evaluate the distillation models before and after polymerization and generate corresponding evaluation scores;
[0083] Step 4.2: The local model and the distillation model calculate the loss using samples;
[0084] Step 4.3: Calculate the gradients of the distillation model and the local model using the evaluation scores from Step 4.1, and update the parameters of the local model.
[0085] Step 4.4: Check if the maximum number of training rounds has been reached. If it has, return to step 4.2; otherwise, end.
[0086] Step 5: The knowledge extraction module extracts the task knowledge and sends it to the task memory palace.
[0087] Step 5.1: Select the part with the largest weights based on the current model parameters, and set the remaining parameters to 0;
[0088] Step 5.2, Model fine-tuning:
[0089] Step 5.2.1: Randomly select a portion of data samples from the current task dataset, and the model performs an inference based on these data samples;
[0090] Step 5.2.2: Calculate the loss using data labels and inference results;
[0091] Step 5.2.3: Calculate the gradient based on the loss and use the gradient to update the parameters that have not been set to 0;
[0092] Step 5.2.4: Determine whether the loss has reached the loss threshold or whether the number of fine-tuning attempts has reached the upper limit. If the conditions are not met, return to step 5.2.2; otherwise, proceed to step 5.3.
[0093] Step 5.3: Send the fine-tuned model parameters to the task memory palace.
[0094] Specifically, in the above embodiments, a total of four platform architectures were selected: two Jetson TX2s with 256-core NVIDIA Pascal GPUs and 8GB of memory; two Jetson Nanos with NVIDIA Maxwell architecture, 128 NVIDIA CUDA cores and 4GB of memory; one Jetson Xavier NX with a 384-core NVIDIA Volta GPU, 48 Tensor cores and 16GB of memory; one Jetson AGX with a 512-core Volta GPU and 32GB of memory; and four Raspberry Pis with 4GB of memory. All Jetson and Raspberry Pi platforms were running Ubuntu 18.04.5LTS and supported PyTorch 1.9.0 (Python 3.6.9) as target edge devices, deploying various typical models on each edge device. The datasets considered include three image classification (computer vision) datasets: Cifar100, MiniImageNet, and TinyImageNet; text classification (natural language processing): Online-Shopping, ASC, and DSC datasets; graph node recognition (graph neural network learning): MiniGC and Reddit datasets; and image-text matching (multimodal): Cifar100-Text dataset. For Cifar100, MiniImageNet, Online-Shopping, DSC, and MiniGC datasets, each dataset was split into 100 private task sequences distributed across 10 clients (10 random, inconsistent tasks per client). For the ASC dataset, it was split into 190 task sequences randomly distributed across 10 clients (19 random, inconsistent tasks per client). For the Reddit dataset, it was split into 80 tasks randomly distributed across 10 clients (8 random, inconsistent tasks per client). Finally, for TinyImageNet, it was split into 200 task sequences distributed across 10 clients (20 random, inconsistent tasks per client).
[0095] In the above embodiments, different deep network models were used to train each dataset. For the image classification datasets (Cifar100 and MiniImageNet), seven models were randomly assigned to the client: Densenet, Six-CNN, Ten-CNN, TinyPiT, WideResnet, Resnet18, and MobilenetV2. For the text classification datasets (ASC and DSC), seven models were randomly assigned to the client: Adapter-Bert, Bert, TextCNN-1, TextCNN-2, RNN, LSTM, and MoELSTM. For the graph node recognition dataset (MiniGC), seven models were randomly assigned to the client: GCN-2, GCN-3, GCN-4, GCN-5, GCN-6, GAT-3, and GAT-4. Finally, for the image-text matching dataset (Cifar100-Text), three models were randomly assigned to the client: CLIP-ViT-B / 3, CLIP-RN50, and CLIP-RN101.
[0096] In step 1 of the above embodiment, when training the network on edge devices, the server selects 40% of the clients to participate in training in each round, which is 10 * 0.4 = 8 edge devices for aggregation. The training for one task is performed in a total of 10 rounds. After 10 rounds, the model for each edge device is tested, for example, including: accuracy, loss, and memory usage. Figure 7 As shown.
[0097] In step 2.1 of the above embodiment, 20% of the server samples are selected for calculation each time. In step 2.2.1, the similarity hyperparameter is set to 0.6. In step 2.3, the top 4 similar task knowledge selected from the task memory palace for each distillation model are selected.
[0098] In step 3 of the above embodiment, the task knowledge of similar tasks for each distillation model is first rearranged using optimal transportation, and then step 3.2 averages and aggregates the task knowledge of each distillation model and its four corresponding rearranged tasks.
[0099] In step 4 of the above embodiment, the aggregated task information needs to be transferred to the local model. In step 4.1, 20% of the local samples are selected to evaluate the distillation model before and after aggregation. Clients that did not participate in aggregation directly use the current distillation model. Step 4.2 calculates the gradient between the distillation model and the local model using the evaluation scores. The calculated gradient is as follows:
[0100]
[0101] Among them, LCE () represents the cross-entropy loss, L KL () represents the KL divergence loss during distillation. For the local model of client i, These are the parameters for the pre-polymerization distillation model. X represents the parameters of the distillation model after polymerization. i ,Y i Using the local training samples of the current client, the final gradient of the local model is calculated:
[0102]
[0103] Use g' i Update the local model.
[0104] In step 5 of the above embodiment, after a task is trained, the client extracts the relevant task parameters as task knowledge. In step 5.1, the model selects 10% of the maximum weight parameters for saving and sets the remaining parameters to 0. In step 5.2, 20% of the samples are selected to fine-tune these selected parameters. The maximum number of fine-tunings is set to 10, and the error threshold is set to one percent of the previous error value. Then, the trained parameters are sent to the task memory palace on the server.
[0105] To verify the superior technical effect of the method described in the above embodiments, tests were conducted on multiple datasets across multiple applications, such as... Figure 7 As shown, where Figure 7 (a)- Figure 7 (b) indicates testing on a computer vision dataset; Figure 7 (c)- Figure 7 (d) indicates testing on a natural language processing dataset; Figure 7 (e) indicates testing on graph data; Figure 7 (f) indicates testing on a multimodal dataset. Experimental results show that the method described in this invention performs excellently in federated learning and task-adaptive scenarios. The method improves accuracy by 40.62% compared to federated continuous learning algorithms, by 35.67% for cluster-based federated learning algorithms, and by 24.74% for heterogeneous network-based federated learning algorithms, while maintaining the lowest training time. Furthermore, in practice, the method performs well under bandwidth limitations ranging from 50KB to 10MB. With a bandwidth of 1MB, communication time accounts for only 10% of training time, reducing communication time by 83.19% compared to the latest algorithms in the prior art. Figure 8 As shown.
[0106] The method described in this invention, through testing under extreme conditions, primarily including a large number of clients and a large number of tasks, still maintains the highest accuracy and shorter training time. Firstly, for large-scale clients, 100 clients were selected for testing. Figure 9 As shown in (a), even with a large number of clients, the method described in this invention still maintains the highest accuracy, achieving a 46.1% improvement in accuracy compared to some of the latest methods in the prior art. Furthermore, for multi-tasking scenarios, such as... Figure 9 As shown in (b), a total of 500 heterogeneous task sequences were included across 10 clients, which improved the accuracy by 37.8% compared to the latest methods in the prior art.
[0107] This invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope. All such changes and modifications fall within the scope of the invention as defined by the appended claims.
Claims
1. A method for task-granularity model aggregation in edge-side federated continuous learning, comprising the following steps: Step 1: The knowledge extraction module extracts each client's local model into a compact knowledge, forming a knowledge distillation model, and sends it to the server. Step 2: The server uses the heterogeneous model selection module to search the task memory palace. The task memory palace includes task knowledge storage, searching for similar task knowledge storage, and adding new task knowledge. It finds a portion of the most similar task knowledge for each client. Specifically, the task memory palace uses model pruning technology to convert the model weights corresponding to each task into its knowledge. The knowledge corresponds to the portion of weights with the highest values to retain the most important information and reduce memory usage. Quick and accurate search for similar task knowledge: The task memory palace uses R-tree to organize similar task knowledge into the same node and organizes all task knowledge hierarchically. Even if the number of clients and tasks is large, similar task knowledge can be searched quickly. Accurately and quickly add new task knowledge: The Task Memory Palace supports adding / deleting tree nodes to update task knowledge. For each new task knowledge added: First, task knowledge is transformed into a low-dimensional task index. Next, given that R-trees are more efficient in low-dimensional spaces, dimensionality reduction techniques are used for further compression. Finally, the R-tree searches for the most similar related knowledge to the currently added task. During the search, multi-head cross-entropy loss is used to evaluate the relationship between two pieces of task knowledge. If the similarity between two pieces of task knowledge exceeds a threshold, the two pieces of task knowledge are merged into the same node; otherwise, a new node is inserted into the R-tree. Step 3: The model aggregation module uses optimal transmission to sequentially aggregate the distillation model of each client and the most similar task knowledge of its selected parts, ensuring that the parameters aggregated at each position have similar functions, so as to maximize the acquisition of useful information and improve the accuracy of the distillation model. Step 4: The server returns the updated distillation model to each client. Each client evaluates the distillation model before and after aggregation and selects the more accurate distillation model to be passed to the task knowledge integration module. The task knowledge integration module uses knowledge distillation to transmit the task information of other clients to the local training model in order to improve the accuracy of the model. Step 5: The knowledge extraction module of each client retains 5%-15% of the model parameters with the largest weights according to the current model, and sets the remaining parameters to 0, adjusts the model, and stores the adjusted model parameters in the knowledge storage module as the task knowledge of the current task.
2. In the task-granularity model aggregation method for edge-side federated continuous learning as described in claim 1, in step 2, the server searches for the task memory palace in the heterogeneous model selection module, and the specific steps are as follows: Step 2.1: Randomly select a portion of the common samples from the server, and each distillation model uses the selected samples to perform one inference. Step 2.2: Each distillation model sequentially searches for similar task knowledge from the task memory palace; Step 2.3: After each distillation model finds similar task knowledge according to Step 2.2, the similarity between each distillation model and the selected task knowledge is compared, and then a portion of the most similar task knowledge is selected.
3. In the task-granularity model aggregation method for edge-side federated continuous learning as described in claim 1, step 3 involves the model aggregation module using optimal transfer to sequentially aggregate the distillation model of each client and its selected most similar task knowledge, specifically including: Step 3.1: Using the optimal transportation method, the selected task knowledge will be rearranged and adjusted according to its corresponding distillation model. Step 3.2: Use the average aggregation method to aggregate the distillation model and each of its selected similar task knowledge adjusted in step 3.
1.
4. In the task-granularity model aggregation method for edge-side federated continuous learning according to claim 1, step 5, wherein model adjustment includes: Step 5.1: Randomly select a portion of data samples from the current task dataset, and the model performs an inference based on these data samples; Step 5.2: Calculate the loss using data labels and inference results; Step 5.3: Calculate the gradient based on the loss and use the gradient to update the parameters that have not been set to 0; Step 5.4: Determine whether the loss has reached the loss threshold or whether the number of adjustments has reached the upper limit. If the conditions are not met, return to step 5.2; otherwise, proceed to step 5.
3.
5. In the task-granularity model aggregation method for edge-side federated continuous learning according to claim 2, step 2.3, the similarity calculation and comparison formula between the distillation model and the task knowledge obtained from the task memory palace is as follows: ， in, Represented as client Distillation model, Task knowledge selected from the task memory palace for the distillation model. Calculation of multi-head cross-entropy loss; , To calculate the similarity value, each knowledge distillation model needs to perform similarity calculations with the corresponding similar task knowledge and select 3-5 of the most similar task knowledge.
6. In the task granularity model aggregation method for edge-side federated continuous learning according to claim 3, step 3.1 uses the optimal transportation method to rearrange the task knowledge, and each task knowledge rearrangement is calculated as follows: ， in, For the client The distillation model of the first Layer parameters, The generated random matrix has the same size as the current layer of the model. For the matrix diagonalization performed, The optimal transfer matrix is calculated based on the current task knowledge and its corresponding distillation model.
7. In the task-granularity model aggregation method for edge-side federated continuous learning according to claim 1, step 4, the model gradient calculation for knowledge distillation is as follows: ， in, This is expressed as cross-entropy loss. The KL divergence loss during distillation, For the client The local model, These are the parameters for the pre-polymerization distillation model. These are the parameters for the distillation model after polymerization. , The gradient calculation for updating the local parameters is as follows, using the local training samples of the current client: ， in, The parameters are for evaluating the distillation model before and after polymerization. It is a judgment Who is better, if So ,otherwise .