Federal continuous learning method based on multi-scale expert collaboration

Abstract

The invention belongs to the technical field of computers, and relates to a federal continuous learning method based on multi-scale expert collaboration. The method comprises the following steps: 1) each client constructs a multi-adapter enhancement architecture based on a pre-training basic model, and adapter parameter fine tuning is carried out in a first task stage; 2) performing prototype initialization on each client by using the semantic features of the old task and the sample features of the new task, and fusing historical knowledge into new category representation learning; 3) each client divides the pre-training model into a plurality of feature learning expert modules, extracts feature representations of different levels, and collaboratively optimizes the classification performance of each expert through multi-scale feature learning loss; 4) each client adopts an uncertainty weighted multi-teacher self-distillation mechanism to dynamically fuse shallow texture features and deep semantic knowledge to a final expert, and 5) the server integrates adapter parameters and multi-scale category prototypes of each client through a weighted aggregation mechanism to form a global knowledge base and distributes the global knowledge base to all clients.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and in particular relates to a federated continuous learning method based on multi-scale expert collaboration. Background Technology

[0002] In traditional centralized training models, all data must be centralized on a server for unified modeling and parameter optimization. While this approach ensures globally optimal model performance under a unified data distribution, centralized storage and training pose serious privacy risks and data security vulnerabilities in privacy-sensitive and data compliance-critical fields such as healthcare, finance, education, government affairs, and the Internet of Things. Furthermore, the significant differences in data scale, type, and quality among different institutions, users, or devices make centralized training ill-suited for the personalized modeling needs of distributed, heterogeneous data environments.

[0003] To address the challenges posed by data privacy protection and distributed heterogeneity, federated learning, as a distributed machine learning paradigm, has emerged. Federated learning allows multiple clients to collaboratively optimize a global model through local training and parameter aggregation without sharing the original data. This mechanism effectively protects user privacy and meets data security and compliance requirements. Traditional federated learning methods typically assume that clients have identical tasks, consistent data distribution, and that the dataset remains static during training. However, in real-world applications, the environments, task types, and data sources of different terminal devices or clients often differ significantly, and data distribution evolves over time. This dynamic change necessitates continuous learning of new tasks, categories, or modalities, leading to problems such as concept drift, task increments, and data heterogeneity, making it difficult to achieve long-term stable performance through a single centralized training session.

[0004] To address this issue, federated continuous learning combines continuous learning techniques to enable models to continuously learn new tasks without forgetting old knowledge while protecting data privacy. Existing federated continuous learning methods mainly include the following categories: (1) parameter preservation strategies based on regularization constraints, which limit the model's deviation from key parameters during training on new tasks by adding stability constraints to the optimization objective, thereby preserving knowledge of old tasks; (2) memory replay-based methods, which cache some old task samples or their feature representations and perform joint optimization during training on new tasks to mitigate catastrophic forgetting; (3) knowledge distillation-based methods, which use historical models as teacher networks to pass knowledge to new models, enabling them to maintain stable performance on old tasks while learning new tasks.

[0005] While the aforementioned methods alleviate the forgetting problem in federated continuous learning to some extent, they still have several limitations. On the one hand, these methods often rely on additional data caching or auxiliary models, leading to potential privacy risks and additional communication and storage overhead. On the other hand, they struggle to balance local feature preservation and global semantic generalization when facing multimodal tasks or complex scenarios with significant differences in multi-scale features. Furthermore, the inconsistency in feature spaces across different clients makes it difficult for the globally aggregated model to adapt to the personalized task requirements of each client, thus limiting the effectiveness of federated continuous learning in real-world multi-source heterogeneous environments. Summary of the Invention

[0006] To address the aforementioned problems, the present invention aims to provide a federated continuous learning method based on multi-scale expert collaboration.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A federated continuous learning method based on multi-scale expert collaboration includes the following steps:

[0009] 1) Each client performs personalized model initialization locally based on the pre-trained model and inserts lightweight adapter modules into each layer of the model to achieve rapid adaptation to new tasks through small-scale parameter updates;

[0010] 2) During local training, the semantic information of the old task and the data features of the new task are combined, and a multimodal feature fusion mechanism is used to generate new category prototypes in order to achieve knowledge transfer and retention of old task memory.

[0011] 3) Divide the pre-trained model into multiple feature learning modules, with each module regarded as an independent expert. Different experts are responsible for the extraction and representation of features at different scales, thereby achieving joint learning of low-level visual features and high-level features.

[0012] 4) Each expert performs multi-teacher dynamic self-distillation within the client, with shallow experts acting as teachers and deep experts acting as students, and cross-layer knowledge transfer based on a multi-teacher self-distillation mechanism with dynamic uncertainty weighting.

[0013] 5) After completing local training, each client uploads the adapter parameters and the category prototypes generated by the experts to the server. Without accessing the client's original data, the server aggregates and updates the uploaded parameters to form a global model, thereby realizing cross-client knowledge sharing and continuous optimization.

[0014] A further optimization of this technical solution involves step 1), where each client initializes a personalized model locally based on a pre-trained model and inserts lightweight adapter modules into each layer of the model. The method for achieving rapid adaptation to new tasks through small-scale parameter updates is as follows:

[0015] 1.1) Insert a lightweight adapter module into the Transformer layer of the client pre-trained model. This module consists of a dimensionality reduction layer, a nonlinear activation function layer, and an dimensionality increase layer, and is used to achieve rapid adjustment of task-specific features while keeping the backbone network parameters frozen.

[0016] 1.2) Each client trains only the adjustable parameters in the lightweight adapter module locally, using a mini-batch local dataset for finite-step gradient updates. This allows the model to quickly adapt to new tasks without disrupting the pre-trained feature space, and training is performed only on the first task. The training process is as follows: , For the original pre-trained model, For pre-trained models with adapters, For the client The dataset for the first task, For the client Adapter parameters, The loss function;

[0017] 1.3) The characteristic output of the adapter is ,in and These are the input and output characteristic representations after passing through an adapter, respectively. It is a non-linear activation function. , These are the dimension reduction and dimension increase transformation matrices, respectively.

[0018] A further optimization of this technical solution is that, in step 2), the method of generating new category prototypes by combining the semantic information of the old task with the data features of the new task during local training, using a multimodal feature fusion mechanism to achieve knowledge transfer and retention of old task memory, is as follows:

[0019] 2.1) Extract visual feature vectors from new task image samples using a pre-trained visual encoder and standardize them to ensure consistency in the feature space;

[0020] 2.2) Calculate the similarity between the semantic embeddings of the old task and the visual features of the new task. Select the most relevant old task category based on the similarity distribution, and generate a new category prototype using a cross-attention mechanism. The prototype initialization formula is as follows: , For category The initial prototype, For categories obtained through multimodal initialization prototype, ,in and Semantic features and class prototypes of the old task most similar to the current class, respectively. For cross attention, As a query vector For key vectors, As a value vector;

[0021] 2.3) The online update formula for the prototype is: , in To process the first Category after each sample prototype, To process the first Category after each sample prototype, For category The Feature representation of a sample For sample indexing, from arrive (category In the mission (Based on the number of samples in the model), as more samples are seen, the prototype gradually converges to the center of the categorical features.

[0022] A further optimization of this technical solution, specifically step 3), refers to:

[0023] 3.1) Divide the client-side pre-trained model into multiple feature extraction units according to network layers. Each unit is defined as an expert, and different experts are responsible for learning feature representations at different scales.

[0024] 3.2) Each expert calculates the classification loss function locally. The total loss for multi-expert feature learning, among which For the number of experts, For the first training samples, For the first The true labels of each training sample For the first One expert on the sample The predicted probability, The dataset for the current task;

[0025] 3.3) Through multi-scale feature learning, multi-level experts among clients can make decisions on the same sample from different perspectives, thereby capturing the diversity and complexity of the sample more comprehensively. As data distribution continues to evolve, clients become more flexible in adapting to new tasks.

[0026] A further optimization of this technical solution, specifically step 4), refers to:

[0027] 4.1) Use shallow experts as teacher models and deep experts as student models;

[0028] 4.2) Distillation loss is defined by calculating the difference between the teacher and student output distributions using relative entropy (Kullback-Leibler divergence). ;

[0029] 4.3) Calculate the uncertainty weight based on the entropy values ​​output by each expert. And normalize the alignment. ,in It is a small positive constant used to prevent the denominator from becoming zero and to avoid the weight from becoming infinitely large. For the first The predicted output of a sample on a certain expert model For the first The predicted output of each sample on the final expert model For normalization function, Indicates the first The weighting of each expert, Represents the weight coefficients of a certain expert model;

[0030] 4.4) The final loss function is This is to achieve dynamically weighted cross-layer knowledge fusion.

[0031] In a further optimization of this technical solution, in step 5), the method for the server to aggregate the prototypes and parameters uploaded by the client is as follows:

[0032] 5.1) In each round of communication during the first round of task training, each client uploads its locally trained adapter to the server for aggregation, enabling the model to acquire more comprehensive knowledge. The formula for the server to aggregate adapter parameters is as follows: A weighted average is used, with weights based on the number of samples each client performs on the first task. The server will distribute the adapter parameters to the clients in the first round of tasks and then freeze them; where For the first Global adapter parameters for round-robin communication For the client In the Local adapter parameters for round-robin communication, For the client Number of samples on the first task Total number of clients;

[0033] 5.2) In each round of the task, the server performs a weighted average of the client prototypes based on the number of client samples: ,in For the first After round communication category The global prototype For the client In category The number of samples on For the client No. Experts in the category The prototype on the surface, Total number of clients For the number of experts;

[0034] 5.3) Server aggregation yields a global prototype It is then distributed to each client for new task initialization.

[0035] Unlike existing technologies, the above technical solution has the following beneficial effects:

[0036] 1) The client's original data is processed and trained entirely locally, without the need to transmit any original sample data to the server or other clients. Knowledge sharing is achieved only by uploading lightweight adapter parameters and category prototypes, which fundamentally protects user data privacy and security.

[0037] 2) This invention introduces a multi-scale expert structure, dividing the pre-trained model into multiple feature learning modules. Experts at different levels are responsible for feature extraction and representation at different semantic levels, and cross-level knowledge fusion is achieved through collaborative optimization among experts. This hierarchical feature learning mechanism enables the model to understand input data from multiple granularities, thereby obtaining a more comprehensive and robust feature representation.

[0038] 3) The use of multimodal prototype initialization enables the model to retain historical knowledge while continuously learning new tasks, which significantly alleviates the catastrophic forgetting problem in federated continuous learning and achieves a good balance between stability and plasticity.

[0039] 4) In the pre-trained model of each client, only a small number of lightweight adapter modules are inserted into specific layers for training, while the backbone network parameters are frozen, thereby significantly reducing the number of parameters that need to be updated and transmitted. This approach not only reduces local computation and communication overhead but also avoids overfitting and knowledge forgetting problems caused by full model fine-tuning, preserves the model's generalization ability, and improves training efficiency and robustness in a federated learning environment. Attached Figure Description

[0040] Figure 1 This is a flowchart of a federated continuous learning method based on multi-scale expert collaboration. Detailed Implementation

[0041] To explain in detail the technical content, structural features, objectives, and effects of the technical solution, the following description is provided in conjunction with specific implementation examples and accompanying drawings.

[0042] like Figure 1 The diagram shows a flowchart of a federated continuous learning method based on multi-scale expert collaboration. This detection method includes the following steps performed sequentially:

[0043] 1) Each client performs personalized model initialization locally based on the pre-trained model and inserts lightweight adapter modules into each layer of the model to achieve rapid adaptation to new tasks through small-scale parameter updates;

[0044] 1.1) Insert a lightweight adapter module into the Transformer layer of the client pre-trained model. This module consists of a dimensionality reduction layer, a nonlinear activation function layer, and an dimensionality increase layer, and is used to achieve rapid adjustment of task-specific features while keeping the backbone network parameters frozen.

[0045] 1.2) Each client trains only the adjustable parameters in the lightweight adapter module locally, using a mini-batch local dataset for finite-step gradient updates. This allows the model to quickly adapt to new tasks without disrupting the pre-trained feature space, and training is performed only on the first task. The training process is as follows: , For the original pre-trained model, For pre-trained models with adapters, For the client The dataset for the first task, For the client Adapter parameters, The loss function;

[0046] 1.3) The characteristic output of the lightweight adapter is ,in and These are the input and output feature representations after passing through a lightweight adapter, respectively. It is a non-linear activation function. , These are the dimension reduction and dimension increase transformation matrices, respectively.

[0047] 2) During local training, the semantic information of the old task and the data features of the new task are combined, and a multimodal feature fusion mechanism is used to generate new category prototypes in order to achieve knowledge transfer and retention of old task memory.

[0048] 2.1) Extract visual feature vectors from new task image samples using a pre-trained visual encoder and standardize them to ensure consistency in the feature space;

[0049] 2.2) Calculate the similarity between the semantic embeddings of the old task and the visual features of the new task. Select the most relevant old task category based on the similarity distribution, and generate a new category prototype using a cross-attention mechanism. The prototype initialization formula is as follows: , For category The initial prototype, For categories obtained through multimodal initialization prototype, ,in and These are the semantic features and class prototype of the old task most similar to the current class, respectively. For cross attention, As a query vector For key vectors, As a value vector; ,in For similarity calculation function, The feature representation extracted from the new category samples by a pre-trained model with an adapter. For the text encoder of the pre-trained model, Given a set of old task labels, calculate the similarity of the semantic features of new category samples with old task labels to find the old task knowledge most relevant to the new category;

[0050] 2.3) The online update formula for the prototype is: , in To process the first Category after each sample prototype, To process the first Category after each sample prototype, For category The Feature representation of a sample For sample indexing, from arrive ( For category In the mission (Based on the number of samples in the model), as more samples are seen, the prototype gradually converges to the center of the categorical features.

[0051] 3) Divide the pre-trained model into multiple feature learning modules, with each module regarded as an independent expert. Different experts are responsible for the extraction and representation of features at different scales, thereby achieving joint learning of low-level visual features and high-level features.

[0052] 3.1) Divide the client-side pre-trained model into multiple feature extraction units according to network layers. Each unit is defined as an expert, and different experts are responsible for learning feature representations at different scales.

[0053] 3.2) Each expert calculates the classification loss function locally. The total loss for multi-expert feature learning, among which For the number of experts, For the first training samples, For the first The true labels of each training sample For the first One expert on the sample The predicted probability, The dataset for the current task;

[0054] 3.3) Through multi-scale feature learning, multi-level experts among clients can make decisions on the same sample from different perspectives, thereby capturing the diversity and complexity of the sample more comprehensively. As data distribution continues to evolve, clients become more flexible in adapting to new tasks.

[0055] 4) Each expert performs multi-teacher dynamic self-distillation within the client, with shallow experts acting as teachers and deep experts acting as students, and cross-layer knowledge transfer based on a multi-teacher self-distillation mechanism with dynamic uncertainty weighting.

[0056] 4.1) Use shallow experts as teacher models and deep experts as student models;

[0057] 4.2) Distillation loss is defined by calculating the difference between the teacher and student output distributions using relative entropy (Kullback-Leibler divergence). ;

[0058] 4.3) Calculate the uncertainty weight based on the entropy values ​​output by each expert. And normalize the alignment. ,in It is a small positive constant used to prevent the denominator from becoming zero and to avoid the weight from becoming infinitely large. For the first The predicted output of a sample on a certain expert model For the first The predicted output of each sample on the final expert model For normalization function, Indicates the first The weighting of each expert, Represents the weight coefficients of a certain expert model;

[0059] 4.4) The final loss function is This is to achieve dynamically weighted cross-layer knowledge fusion.

[0060] 5) After completing local training, each client uploads the lightweight adapter parameters and the expert-generated category prototypes to the server. Without accessing the client's original data, the server aggregates and updates the uploaded parameters to form a global model, thereby enabling cross-client knowledge sharing and continuous optimization.

[0061] 5.1) In each round of communication during the first round of task training, each client uploads its locally trained adapter to the server for aggregation, enabling the model to acquire more comprehensive knowledge. The formula for the server to aggregate adapter parameters is as follows: A weighted average is used, with weights based on the number of samples each client performs on the first task. The server will distribute the adapter parameters to the clients in the first round of tasks and then freeze them; where For the first Global lightweight adapter parameters for round communication, For the client In the Parameters of the local lightweight adapter for round communication. For the client Number of samples on the first task Total number of clients;

[0062] 5.2) In each round of the task, the server performs a weighted average of the client prototypes based on the number of client samples: ,in For the first After round communication category The global prototype For the client In category The number of samples on For the client No. Experts in the category The prototype on the surface, Total number of clients For the number of experts;

[0063] 5.3) Server aggregation yields a global prototype It is then distributed to each client for new task initialization.

[0064] Although the above embodiments have been described, those skilled in the art, once they understand the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the above descriptions are merely embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structural or procedural transformations made using the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A federated continuous learning method based on multi-scale expert collaboration, characterized in that, Includes the following steps: 1) Each client performs personalized model initialization locally based on the pre-trained model and inserts lightweight adapter modules into each layer of the model to achieve rapid adaptation to new tasks through small-scale parameter updates; 2) During local training, new category prototypes are generated by combining the semantic information of the old task with the data features of the new task using a multimodal feature fusion mechanism; 3) Divide the pre-trained model into multiple feature learning modules, with each module regarded as an independent expert. Different experts are responsible for the extraction and representation of features at different scales, thereby achieving joint learning of low-level visual features and high-level features. 4) Each expert performs multi-teacher dynamic self-distillation within the client, with shallow experts acting as teachers and deep experts acting as students, and cross-layer knowledge transfer based on a multi-teacher self-distillation mechanism with dynamic uncertainty weighting. 5) After completing local training, each client uploads the adapter parameters and the category prototypes generated by the experts to the server. Without accessing the client's original data, the server aggregates and updates the uploaded parameters to form a global model, thereby realizing cross-client knowledge sharing and continuous optimization.

2. The federated continuous learning method based on multi-scale expert collaboration as described in claim 1, characterized in that, In step 1), each client initializes a personalized model locally based on the pre-trained model and inserts lightweight adapter modules into each layer of the model. The method to achieve rapid adaptation to new tasks through small-scale parameter updates is as follows: 1.1) Insert a lightweight adapter module into the converter model layer of the client pre-trained model. This module includes a dimensionality reduction layer, a non-linear activation function layer, and a dimensionality increase layer. 1.2) Each client trains only the adjustable parameters in the lightweight adapter module locally, and performs finite-step gradient updates using a small batch of local datasets, so that the model can quickly adapt to new tasks without destroying the pre-trained feature space. Furthermore, training is only performed on the first task, and then frozen to prevent catastrophic forgetting. 1.3) The characteristic output of the lightweight adapter is ,in and These are the input and output feature representations after passing through a lightweight adapter, respectively. It is a non-linear activation function. , These are the dimension reduction and dimension increase transformation matrices, respectively.

3. The federated continuous learning method based on multi-scale expert collaboration as described in claim 2, characterized in that, The training process for the lightweight adapter module on the first task is as follows: , For the original pre-trained model, For pre-trained models with adapters, For the client The dataset for the first task, For the client Adapter parameters, The loss function is used to train the adapter parameters only on the first task. Then freeze the model to ensure it adapts quickly to downstream tasks while preserving the generalization ability of the pre-trained model.

4. The federated continuous learning method based on multi-scale expert collaboration as described in claim 1, characterized in that, In step 2), the process of generating a new category prototype by combining the semantic information of the old task with the data features of the new task during local training includes the following steps: 2.1) Extract visual feature vectors from new task image samples using a pre-trained visual encoder and standardize them to ensure consistency in the feature space; 2.2) Calculate the similarity between the semantic embeddings of the old task and the visual features of the new task. Select the most relevant old task category based on the semantic similarity distribution, and generate a new category prototype using a cross-attention mechanism. The prototype initialization formula is as follows: , For category The initial prototype, For categories obtained through multimodal initialization prototype; 2.3) The online update formula for the prototype is: , in To process the first Category after each sample prototype, To process the first Category after each sample prototype, For category The Feature representation of a sample For sample indexing, from arrive , For category In the mission The number of samples in the prototype gradually converges to the center of the category feature.

5. The federated continuous learning method based on multi-scale expert collaboration as described in claim 4, characterized in that, In section 2.2, selecting the most relevant old task category based on semantic similarity distribution specifically refers to: selecting the most similar label semantic features and merging them with the sample features of the new class through a cross-attention mechanism to initialize the new class prototype, thereby incorporating prior implicit biases. ,in and Semantic features and class prototypes of the old task most similar to the current class, respectively. For cross attention, As a query vector For key vectors, As a value vector.

6. The federated continuous learning method based on multi-scale expert collaboration as described in claim 5, characterized in that, The semantic similarity calculation formula is as follows: ,in For similarity calculation function, The feature representation extracted from the new category samples by a pre-trained model with an adapter. For the text encoder of the pre-trained model, This is a collection of old task tags.

7. The federated continuous learning method based on multi-scale expert collaboration as described in claim 1, characterized in that, Step 3) refers to: 3.1) Divide the client-side pre-trained model into multiple feature extraction units according to network layers. Each unit is defined as an expert, and different experts are responsible for learning feature representations at different scales. 3.2) Each expert calculates the classification loss function locally. The total loss for multi-expert feature learning, among which For the number of experts, For the first training samples, For the first The true labels of each training sample For the first One expert on the sample The predicted probability, The dataset for the current task; 3.3) Through multi-scale feature learning, multi-level experts among customers can make decisions on the same sample from different perspectives.

8. The federated continuous learning method based on multi-scale expert collaboration as described in claim 1, characterized in that, The specific implementation of the multi-teacher dynamic self-distillation mechanism in step 4) includes: 4.1) Use shallow experts as teacher models and deep experts as student models; 4.2) Distillation loss is defined by calculating the difference between the teacher's and student's output distributions using relative entropy. ; 4.3) Calculate the uncertainty weight based on the entropy values ​​output by each expert. And normalize the alignment. ,in It is a small positive constant used to prevent the denominator from becoming zero and to avoid the weight from becoming infinitely large. For the first The predicted output of a sample on a certain expert model For the first The predicted output of each sample on the final expert model For normalization function, Indicates the first The weighting of each expert, Represents the weight coefficients of a certain expert model; 4.4) The final loss function is .

9. The federated continuous learning method based on multi-scale expert collaboration as described in claim 1, characterized in that, The method for the server to aggregate the prototypes and parameters uploaded by the client in step 5) is as follows: 5.1) In each round of communication during the first round of task training, each client uploads its locally trained adapter to the server for aggregation, enabling the model to acquire more comprehensive knowledge. The formula for the server to aggregate adapter parameters is as follows: A weighted average is used, with weights based on the number of samples each client performs on the first task. The server will distribute the adapter parameters to the clients in the first round of tasks and then freeze them; where For the first Global adapter parameters for round-robin communication For the client In the Local adapter parameters for round-robin communication, For the client Number of samples on the first task Total number of clients; 5.2) In each round of the task, the server performs a weighted average of the client prototypes based on the number of client samples: ,in For the first After round communication category The global prototype For the client In category The number of samples on For the client No. Experts in the category The prototype on the surface, Total number of clients For the number of experts; 5.3) Server aggregation yields a global prototype It is then distributed to each client for new task initialization.

Images