Federal continuous learning method for ubiquitous embedded terminal
By initializing single-class models on the server side and dynamically allocating them, the client performs comparison learning and knowledge distillation, the problems of poor generalization ability and data drift under extreme data heterogeneity are solved, and better model generalization and robustness are achieved.
Patent Information
- Application Number
- CN202510569066.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-30
- Publication Date
- 2025-08-15
AI Technical Summary
Under extreme data heterogeneity conditions, overfitting local data in personalized models leads to poor generalization capabilities and data drift problems during testing.
The server initializes each single-class model and dynamically allocates it to the client according to the training progress. The client minimizes the distance between similar samples and maximizes the distance between different samples through comparative learning. The server integrates the single-class model into a generalized model through knowledge distillation technology. The client detects data drift during the inference stage and selects a suitable model for inference.
Effectively respond to data heterogeneity problems, improve the generalization ability of the model on global data, enhance the robustness of data distribution changes, and reduce the training delay caused by the model's convergence time imbalance.
Smart Images
Figure CN120494130A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of federated learning methods for embedded terminals, and in particular to a federated continuous learning method and system for ubiquitous embedded terminals. Background Art
[0002] In recent years, with the rapid development of mobile internet and IoT technologies, federated learning, a distributed machine learning paradigm, has gradually become an important technology for resolving the contradiction between data privacy protection and efficient model training. Federated learning enables high-performance global model training without sharing raw data through collaborative training across multiple clients (such as mobile devices and edge devices), thereby fully leveraging the value of distributed data while protecting data privacy. However, in actual mobile application scenarios, federated learning faces many challenges, the most prominent of which are data heterogeneity and data drift during testing.
[0003] Although research has made some progress in addressing data heterogeneity between clients, existing methods mostly focus on static data distribution and ignore the dynamic changes in data distribution within clients. Under conditions of extreme data heterogeneity, personalized models often overfit local data, resulting in insufficient generalization capabilities on global data. Furthermore, existing methods also have certain limitations when dealing with data drift during testing. For example, some methods enhance the generalization capabilities of personalized models by introducing a global model during federated training, but these methods may not work effectively when the data distribution is highly heterogeneous. Summary of the Invention
[0004] The main purpose of this application is to provide a federated continuous learning method and system for ubiquitous embedded terminals, aiming to solve the technical problems that under extreme data heterogeneity conditions, personalized models often overfit local data, resulting in poor generalization ability on global data, and data drift when responding to tests.
[0005] To achieve the above-mentioned objectives, the present application provides a federated continuous learning method for ubiquitous embedded terminals, including: the server initializes each single-class model according to the number of classification task categories, each single-class model corresponds to a specific category, and is dynamically assigned to the client according to the training progress of each single-class model; the client locally trains the assigned single-class model based on local data, and during the training process, the distance between samples of the same class is minimized and the distance between samples of different classes is maximized through comparative learning, and the single-class model assigned to the client is trained to obtain the trained single-class model; the server aggregates the parameters of the trained single-class model, and integrates the knowledge of each single-class model into a generalized model through knowledge distillation technology; the client detects the feature distribution of the input data in real time during the inference stage. If data drift is determined to occur based on the feature distribution, the generalized model is used for inference, otherwise the trained single-class model is used for inference.
[0006] Optionally, the dynamic allocation to the client according to the training progress of each single-class model includes: the server monitors the training progress of each single-class model in real time, and preferentially allocates the client to the single-class model with the slowest training progress to minimize the imbalance of convergence time.
[0007] Optionally, the client performs local training on the assigned single-class model based on local data. During the training process, the distance between samples of the same class is minimized and the distance between samples of different classes is maximized through contrastive learning, and the single-class model assigned by the client is trained to obtain a trained single-class model, including: the client divides the target category samples to obtain target category samples and non-category samples; randomly enhances the target category samples to obtain enhanced target category samples; calculates the contrast loss of the target category samples and the enhanced target category samples to obtain a distance minimization item between samples of the same class; calculates the cross entropy between the enhanced target category samples and the non-category samples to obtain a distance maximization item between samples of different classes; obtains a contrast loss function based on the sum of the distance minimization item between samples of the same class and the distance maximization item between samples of different classes; and locally trains the single-class model assigned by the client based on the contrast loss function training to obtain a trained single-class model.
[0008] Optionally, the knowledge of each single-class model is integrated into a generalized model through knowledge distillation technology, including: the server uses a pure noise data set to calculate the mean and variance of the output of each trained single-class model; normalizes the prediction probability based on the mean and variance to obtain a normalized prediction probability; uses the normalized prediction probability as a soft label, and trains the trained generalization model of each single-class model based on the soft label and the parameters of each single-class model to obtain a generalized model with determined parameters.
[0009] Optionally, the normalized predicted probability is used as a soft label, and a training generalization model of each single-class model is trained based on the soft label and the parameters of each single-class model to obtain a generalization model with determined parameters, including: the server obtains the predicted probability distribution of the generalization model for the input sample; based on the minimized KL divergence of the soft label and the predicted probability distribution, the generalization model is trained to obtain a generalization model with determined parameters.
[0010] Optionally, the client detects the feature distribution of the input data in real time during the inference phase. If data drift is determined to have occurred based on the feature distribution, a generalized model is used for inference, including: the client obtains the feature distribution of the input data; calculates the output logits entropy value based on the feature distribution of the input data; if the logits entropy value exceeds a preset threshold, it is determined that data drift has occurred in the feature distribution, and the generalized model is used for inference.
[0011] Optionally, the server aggregates the trained parameters of the single-class model, including: performing weighted averaging on the trained model parameters uploaded by each client according to the classification task category to obtain the aggregated trained parameters of the single-class model.
[0012] Optionally, the knowledge distillation of the generalization model is performed simultaneously with the training of the single-class model.
[0013] Optionally, the random data enhancement includes at least one operation of cropping, rotation and noise addition.
[0014] In order to achieve the above-mentioned purpose, the present application also provides a federated continuous learning system for an embedded terminal, comprising: a client and a server; wherein the server is used to initialize each single-class model according to the number of classification task categories, each single-class model corresponds to a specific category, and is dynamically assigned to the client according to the training progress of each single-class model; the client is used to perform local training on the assigned single-class model based on local data. During the training process, the distance between samples of the same class is minimized and the distance between samples of different classes is maximized through comparative learning, and the single-class model assigned by the client is trained to obtain the trained single-class model; the server is also used to aggregate the parameters of the trained single-class model, and integrate the knowledge of each single-class model into a generalized model through knowledge distillation technology; the client is also used to detect the feature distribution of the input data in real time during the inference stage. If data drift is determined to occur based on the feature distribution, the generalized model is used for inference, otherwise the trained single-class model is used for inference.
[0015] The embodiment of the present application proposes a federated continuous learning method and system for ubiquitous embedded terminals. The server initializes each single-class model according to the number of classification task categories. Each single-class model corresponds to a specific category and is dynamically assigned to the client according to the training progress of each single-class model. The client performs local training on the assigned single-class model based on local data. During the training process, the distance between samples of the same class is minimized and the distance between samples of different classes is maximized through comparative learning to train the single-class model assigned to the client and obtain the trained single-class model. The server aggregates the parameters of the trained single-class model and integrates the knowledge of each single-class model into a generalized model through knowledge distillation technology. The client detects the feature distribution of the input data in real time during the inference stage. If it is determined that a new feature has occurred based on the feature distribution, If there is data drift, the generalized model is used for inference, otherwise the trained single-class model is used for inference. By decomposing model training into multiple category-centric single-class model training tasks, category-specific bias caused by data distribution differences is avoided, thereby effectively addressing the data heterogeneity problem between clients; knowledge distillation technology is used to aggregate the knowledge of multiple single-class models into a generalized model, thereby improving the model's generalization ability on global data, thereby addressing the data drift problem during testing; in addition, through dynamic training scheduling strategies, all single-class models are ensured to converge synchronously, reducing training delays caused by imbalanced model convergence time; in the inference stage, by detecting data drift and dynamically selecting appropriate models for prediction, the model's robustness to data distribution changes is further enhanced. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 A flowchart of an embodiment of a federated continuous learning method for ubiquitous embedded terminals provided by this application;
[0017] Figure 2 A schematic diagram comparing traditional clustering and category clustering in a schematic diagram of an embodiment of a federated continuous learning method for ubiquitous embedded terminals provided in this application;
[0018] Figure 3 A schematic diagram of knowledge distillation from a cluster model to a generalized model is provided for an embodiment of a federated continuous learning method for ubiquitous embedded terminals of this application.
[0019] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0020] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.
[0021] Reference Figure 1 and Figure 3 , Figure 1The flowchart of the federated continuous learning method for ubiquitous embedded terminals provided in the embodiment of the present application is as follows: Figure 3 A schematic diagram of knowledge distillation from a cluster model to a generalized model is provided for an embodiment of a federated continuous learning method for ubiquitous embedded terminals of this application. The federated continuous learning method for ubiquitous embedded terminals can be executed by a processor of a server or terminal. The federated continuous learning method for ubiquitous embedded terminals may include:
[0022] S10. The server initializes each single-class model according to the number of classification task categories. Each single-class model corresponds to a specific category and is dynamically allocated to the client according to the training progress of each single-class model.
[0023] In one embodiment of the present application, the dynamic allocation to the client according to the training progress of each single-class model includes:
[0024] The server monitors the training progress of each single-class model in real time and preferentially assigns the client to the single-class model with the slowest training progress to minimize the imbalance in convergence time.
[0025] See also Figure 2 , Figure 2 A schematic diagram comparing traditional clustering and category clustering of the principle diagram provided in an embodiment of the federated continuous learning method for ubiquitous embedded terminals of this application; Figure 2 Different from the traditional clustering method, in the actual implementation process of the method of this application, the server first initializes the single-class model, and initializes N single-class models {w1,w2,...,w N Each model corresponds to a specific category, and the training progress of each single-category model is monitored in real time (with the number of completed iterations en as an indicator). The client is assigned to the model with the slowest progress first to minimize the convergence time imbalance index.
[0026] S20. The client performs local training on the assigned single-class model based on local data. During the training process, the client trains the assigned single-class model by minimizing the distance between samples of the same class and maximizing the distance between samples of different classes through comparative learning, thereby obtaining a trained single-class model.
[0027] In an embodiment of the present application, step S20 may include the following execution process:
[0028] S201: The client classifies target category samples into target category samples and non-category samples.
[0029] S202, randomly enhance the target category samples to obtain enhanced target category samples;
[0030] The random data enhancement includes at least one operation of cropping, rotation, and noise addition.
[0031] S203, calculating the contrast loss of the target category sample and the enhanced target category sample to obtain the distance minimization term between the same type of samples;
[0032] S204, calculating the cross entropy between the enhanced target category samples and non-category samples to obtain a distance maximization term between samples of different categories;
[0033] S205, obtaining a contrast loss function based on the sum of the distance minimization term between samples of the same class and the distance maximization term between samples of different classes;
[0034] S206 , performing local training on the single-class model assigned to the client based on contrastive loss function training to obtain a trained single-class model.
[0035] In the actual execution process, the client data is divided and enhanced, and the client m receives the assigned model w n , based on the local dataset D m Divide target category samples ξ + and non-target class samples ξ - , for + Perform random enhancement (cropping, rotation, noise addition) generation Define the total loss function L = L con +L classify , where L con For contrast loss, minimize the distance between enhanced representations of similar samples: L classify For classification loss, feature classification is constrained by cross entropy: L classify =∑CrossEntropy(w classifier (w(ξ)))
[0036] S30: The server aggregates the parameters of the trained single-class models and integrates the knowledge of each single-class model into a generalized model through knowledge distillation technology;
[0037] The parameters of the single-class model after server aggregation training may include:
[0038] The trained model parameters uploaded by each client according to the classification task category are weighted averaged to obtain the parameters of the aggregated trained single-class model.
[0039] In an embodiment of the present application, step S30 may include the following execution process:
[0040] S301. Integrate the knowledge of each single-class model into a generalized model through knowledge distillation technology, including:
[0041] S302. The server calculates the mean and variance of the output of each trained single-class model using the pure noise data set;
[0042] S303, normalizing the predicted probability based on the mean and variance to obtain a normalized predicted probability;
[0043] S304: Using the normalized prediction probability as a soft label, and training a generalized model of each single-class model based on the soft label and the parameters of each single-class model to obtain a generalized model with determined parameters.
[0044] The normalized prediction probability is used as a soft label, and the training generalization model of each single-class model is trained based on the soft label and the parameters of each single-class model. The parameter-determined generalization model may include:
[0045] The server obtains the predicted probability distribution of the generalization model for the input sample;
[0046] Based on the minimized KL divergence between the soft labels and the predicted probability distribution, the generalization model is trained to obtain a generalization model with determined parameters.
[0047] In the specific execution process, the client uploads the updated single-class model to the server, and the server aggregates the parameters of each model by category, and trains the generalization model at the same time, using the pure noise data set to calculate the mean μ of the output of each single-class model. n and variance σ n , normalize the predicted probability: Using the normalized probabilities as soft labels, the generalization model Wg is trained to minimize the KL divergence: L(W g ,W 1...N )=KL(P i ,f(ξ i ,W g )).
[0048] Formula L(W g , W 1…N )=KL(P,f(ξ i , W g )) indicates that the goal of training the generalization model Wg is to minimize the KL divergence (Kullback-Leibler divergence). Among them, Wg is the parameter of the generalization model to be trained, W 1…N represents the parameter set of each single-class model before; P is the normalized probability (soft label) calculated previously; f(ξ i , W g ) represents the generalization model Wg for the input sample ξ i The predicted probability distribution of i , W g)) and train the generalization model by minimizing this difference so that its prediction results are as close as possible to the normalized probability (soft label) obtained based on the single-class model, so as to achieve the generalization model training goal.
[0049] It is worth noting that the knowledge distillation of the generalized model can be performed simultaneously with the training of the single-class model, and the knowledge distillation is completed after all the single-class models converge.
[0050] S40. The client detects the feature distribution of the input data in real time during the inference phase. If data drift is determined to have occurred based on the feature distribution, the generalized model is used for inference; otherwise, the trained single-class model is used for inference.
[0051] Specifically, during the inference phase, the client detects the feature distribution of the input data in real time. If data drift is determined based on the feature distribution, the client uses the generalized model for inference, including:
[0052] The client obtains the characteristic distribution of input data;
[0053] Calculate the output logits entropy value based on the characteristic distribution of the input data;
[0054] If the logits entropy value exceeds the preset threshold, it is determined that data drift has occurred in the feature distribution, and the generalization model is used for inference.
[0055] Based on the above method embodiments, the present application further provides a federated continuous learning system for embedded terminals, including:
[0056] Client and server;
[0057] The server is used to initialize each single-class model according to the number of classification task categories. Each single-class model corresponds to a specific category and is dynamically allocated to the client based on the training progress of each single-class model.
[0058] The client is used to perform local training on the assigned single-class model based on local data. During the training process, the client minimizes the distance between samples of the same class and maximizes the distance between samples of different classes through comparative learning, thereby training the assigned single-class model to obtain the trained single-class model.
[0059] The server is also used to aggregate the parameters of the trained single-class models and integrate the knowledge of each single-class model into a generalized model through knowledge distillation technology;
[0060] The client is also used to detect the feature distribution of input data in real time during the inference phase. If data drift is determined to have occurred based on the feature distribution, the generalized model is used for inference; otherwise, the trained single-class model is used for inference.
[0061] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A federated continuous learning method for ubiquitous embedded terminals, characterized by: include: The server initializes each single-class model according to the number of classification task categories. Each single-class model corresponds to a specific category and is dynamically allocated to the client based on the training progress of each single-class model. The client performs local training on the assigned single-class model based on local data. During the training process, the client minimizes the distance between samples of the same class and maximizes the distance between samples of different classes through comparative learning, thus training the assigned single-class model to obtain the trained single-class model. The server aggregates the parameters of the trained single-class models and integrates the knowledge of each single-class model into a generalized model through knowledge distillation technology; The client detects the feature distribution of the input data in real time during the inference phase. If data drift is determined to have occurred based on the feature distribution, the generalized model is used for inference; otherwise, the trained single-class model is used for inference.
2. The federated continuous learning method for ubiquitous embedded terminals according to claim 1, characterized in that: The dynamic allocation to the client according to the training progress of each single-class model includes: The server monitors the training progress of each single-class model in real time and preferentially assigns the client to the single-class model with the slowest training progress to minimize the imbalance in convergence time.
3. The federated continuous learning method for ubiquitous embedded terminals according to claim 1, characterized in that: The client performs local training on the assigned single-class model based on local data. During the training process, the distance between samples of the same class is minimized and the distance between samples of different classes is maximized through comparative learning. The trained single-class model is obtained, including: The client classifies the target category samples into target category samples and non-category samples; Randomly enhance the target category samples to obtain enhanced target category samples; Calculate the contrast loss of the target category samples and the enhanced target category samples to obtain the distance minimization term between similar samples; Calculate the cross entropy between the enhanced target category samples and non-category samples to obtain the distance maximization term between samples of different categories; The contrast loss function is obtained based on the sum of the distance minimization term between samples of the same type and the distance maximization term between samples of different types; Based on the contrastive loss function training, the single-class model assigned by the client is locally trained to obtain the trained single-class model.
4. The federated continuous learning method for ubiquitous embedded terminals according to claim 1, characterized in that: The knowledge distillation technology is used to integrate the knowledge of each single-class model into a generalized model, including: The server uses a pure noise dataset to calculate the mean and variance of the output of each trained single-class model; Normalize the predicted probability based on the mean and variance to obtain the normalized predicted probability; The normalized prediction probability is used as a soft label, and the training generalization model of each single-class model is trained based on the soft label and the parameters of each single-class model to obtain a generalization model with determined parameters.
5. The federated continuous learning method for ubiquitous embedded terminals according to claim 4, characterized in that: The normalized predicted probability is used as a soft label, and a training generalization model of each single-class model is trained based on the soft label and the parameters of each single-class model to obtain a generalization model with determined parameters, including: The server obtains the predicted probability distribution of the generalization model for the input sample; Based on the minimized KL divergence between the soft labels and the predicted probability distribution, the generalization model is trained to obtain a generalization model with determined parameters.
6. The federated continuous learning method for ubiquitous embedded terminals according to claim 1, characterized in that: The client detects the feature distribution of the input data in real time during the inference phase. If data drift is determined to have occurred based on the feature distribution, the client uses the generalized model for inference, including: The client obtains the characteristic distribution of input data; Calculate the output logits entropy value based on the characteristic distribution of the input data; If the logits entropy value exceeds the preset threshold, it is determined that data drift has occurred in the feature distribution, and the generalization model is used for inference.
7. The federated continuous learning method for ubiquitous embedded terminals according to claim 1, characterized in that: The server aggregates the parameters of the trained single-class model, including: The trained model parameters uploaded by each client according to the classification task category are weighted averaged to obtain the parameters of the aggregated trained single-class model.
8. The federated continuous learning method for ubiquitous embedded terminals according to claim 1, characterized in that: The knowledge distillation of the generalized model is performed simultaneously with the training of the single-class model.
9. The federated continuous learning method for ubiquitous embedded terminals according to claim 3, characterized in that: The random enhancement of target category samples includes: At least one of a cropping operation, a rotation operation, and a noise addition operation is performed on the target category sample.
10. A federated continuous learning system for embedded terminals, characterized in that: include: Client and server; The server is used to initialize each single-class model according to the number of classification task categories. Each single-class model corresponds to a specific category and is dynamically allocated to the client based on the training progress of each single-class model. The client is used to perform local training on the assigned single-class model based on local data. During the training process, the client minimizes the distance between samples of the same class and maximizes the distance between samples of different classes through comparative learning, thereby training the assigned single-class model to obtain the trained single-class model. The server is also used to aggregate the parameters of the trained single-class models and integrate the knowledge of each single-class model into a generalized model through knowledge distillation technology; The client is also used to detect the feature distribution of input data in real time during the inference phase. If data drift is determined to have occurred based on the feature distribution, the generalized model is used for inference; otherwise, the trained single-class model is used for inference.