Personalized federal incremental learning method based on instance-level prompt generation

Abstract

The invention discloses a personalized federal incremental learning method based on instance-level prompt generation, which realizes finer-grained feature capture of data by deploying a dynamic adjustable prompt generation module for a basic network of each client. The core of the module is to mine hidden attributes in data, and the attributes not only can significantly improve the generalization ability of the model for new classes, but also can realize effective migration of base class knowledge to the new classes. The hidden attribute features learned from the base class can be naturally generalized into unseen classes, so that the adaptability of the model to unknown distribution is enhanced while historical knowledge is kept. According to the method, the bottleneck of the static prompt pool in the aspects of expandability and adaptability is broken through, and a more flexible solution is provided for personalized federal continuous learning.

Description

Technical Field

[0001] The present invention relates to the fields of pattern recognition, machine learning, as well as federated learning and incremental learning technologies, and particularly relates to a personalized federated incremental learning method based on instance-level prompt generation. Background Art

[0002] With the rapid development of artificial intelligence technology, federated learning and continual learning have gradually become research hotspots. Federated learning allows multiple clients to collaboratively train a global model while protecting data privacy, and continual learning aims to enable the model to adapt to new tasks without forgetting old knowledge. However, in practical applications, the data distributions of clients often have a high degree of heterogeneity, that is, the data of different clients may come from different tasks, categories, or domains. Therefore, how to achieve efficient personalized federated continual learning in such a dynamic and non-independent and identically distributed environment has become an extremely challenging problem.

[0003] Existing personalized federated continual learning methods usually adopt an architecture based on a fixed-size prompt pool, which is designed to store and reuse task- or category-related knowledge to alleviate the catastrophic forgetting problem. However, this static design has two key limitations. One is the lack of scalability: in real scenarios, the data distributions of clients may change dynamically over time, and new tasks or categories may emerge continuously. The fixed-size prompt pool cannot flexibly adapt to this dynamic change, resulting in a decline in the model's performance when facing new tasks. For example, when a certain client encounters a new category not covered in the training phase, the predefined prompts may not provide effective guidance, thus affecting the generalization ability of the model. The other is limited adaptability: static prompt pools usually rely on a coarse-grained division of tasks or categories while ignoring the fine-grained differences in data distributions. For example, different instances within the same task may have significantly different features, but the fixed prompt pool cannot be dynamically adjusted to adapt to this change. In addition, the globally shared prompt pool may not be able to fully capture the personalized needs of clients, resulting in poor performance of the model on local data.

[0004] These limitations not only restrict the adaptability of the model but may also hinder its long-term deployment in an open environment. For example, in medical federated learning, the patient data of different hospitals may evolve over time, such as the emergence of new diseases or the update of diagnosis and treatment methods. It is difficult for a static prompt pool to be dynamically adjusted to adapt to this evolution, thus affecting the diagnostic accuracy of the model. Similarly, in the intelligent Internet of Things scenario, the data distributions of devices may be dynamically adjusted due to environmental changes or user behavior changes. The rigid structure of the fixed prompt pool may cause the model to be unable to effectively learn new knowledge or quickly adapt to the new environment. Summary of the Invention

[0005] In view of the problem of insufficient adaptability of the existing fixed prompt pool in a dynamic data environment, the present invention proposes a personalized federated learning framework based on adaptive prompt generation. Different from the rigid architecture of the traditional fixed prompt pool, by deploying a dynamically adjustable prompt generation module for the basic network of each client, finer-grained feature capture of data is achieved. The core of this module lies in mining the hidden attributes in the data, which can not only significantly improve the generalization ability of the model to new categories, but also realize the effective transfer of base class knowledge to new categories. Specifically, the hidden attribute features learned from the base class can be naturally generalized to unseen categories, thereby enhancing the model's adaptability to unknown distributions while maintaining historical knowledge. This design breaks through the bottleneck of static prompt pools in terms of scalability and adaptability, providing a more flexible solution for personalized federated continuous learning.

[0006] The technical solution of the present invention is a personalized federated incremental learning method based on instance-level prompt generation. The implementation steps of this method are as follows:

[0007] Step 1: Local clients train an image classification model;

[0008] S11 Input: Image data of each client's current task.

[0009] S12 Model training: Each client uses Vision Transformer (ViT) as the basic network framework. A lightweight adaptive prompt generator is embedded in each Encoding Layer of ViT. This module can dynamically generate corresponding personalized instance-level prompts according to the instance features of the input data. During the training process, the system constructs a loss function through the supervision signal of the classification head, and jointly optimizes the parameters of the adaptive prompt generator and the classification head in an end-to-end manner, while keeping the parameters of the ViT backbone network frozen to ensure the stability of the basic features.

[0010] S13 Output: The trained local client image classification model

[0011] Step 2: The server side performs global classification head aggregation

[0012] S21 Parameter interaction: Each client only uploads the trained classification head parameters to the server for average aggregation. The parameters of the dynamic prompt generation module are retained locally on each client as personalized parameters.

[0013] S22 Classification head parameter reset: The aggregated global classification head model is securely distributed to each client as the initial model for the next round of local training.

[0014] Compared with the prior art, the present invention proposes a personalized federated learning framework based on dynamic prompt generation. By deploying a dynamically adjustable adaptive prompt generation module for the base network of each client, the rigid architecture of the traditional fixed prompt pool is broken, and finer-grained feature capture of data is achieved. The core of this module lies in mining the hidden attributes in the data, which can significantly improve the generalization ability of the model to new categories.

[0015] Vision Transformer (ViT)

[0016] As a unified base network architecture, all clients perform personalized learning based on this backbone network. The model first divides the input image into several image patches and converts them into a sequence of d-dimensional embedding vectors through the patch embedding layer. At the same time, a learnable [cls] classification token is introduced as a global feature aggregation node, which is connected to the embedding vector sequence to form e0 ∈ R H×W×C . e (n+1)×d . e l represents the overall input sequence:

[0017] e l = [cls; e0, e1.......e n ∈ R (n+1)×d

[0018] In the feature extraction stage, multiple layers of Transformer encoders process the input sequence in turn. Each layer of encoder contains a multi-head self-attention module (MHSA) and a multi-layer perceptron (MLP), and residual connection and layer normalization (LN) techniques are used to ensure the stability of gradient propagation.

[0019] e l ' = LN(MHSA(e l , e l , e l ))

[0020] e l ” = MLP(e l ')

[0021] e l+1 = LN(e l ' + e l ”)

[0022] This architecture design based on global self-attention can effectively model the long-range dependencies between image patches, thereby achieving accurate understanding of the overall semantics of the image.

[0023] Adaptive Prompt Generation Module

[0024] The adaptive prompt generation module dynamically generates adaptive prompts related to the input instance through a feed-forward network and linear transformation. Based on the transposed input, this module uses layer normalization (LN), multi-layer perceptron (MLP), and linear transformation layer (LT) to extract instance-specific information and generate dynamic prompts in real time, thereby guiding the model to adjust the feature representation between the current task and future tasks. Specifically, we first normalize the input features and transpose their dimensions from (n + 1)×d to d×(n + 1), and then input them into the MLP for processing. The MLP converts the (n + 1)-dimensional input into a p-dimensional output to generate instance-level prompts, which fuse the global context information from each feature channel. Such prompts serve as adaptive guidance, enabling the model to retain key task-related knowledge and effectively adapt to different client-specific dynamic data.

[0025] To enhance the personalization and adaptability of the prompts across different tasks, the prompt generation module introduces a feature dimension transformation framework (LT). LT uses affine transformation to adjust the generated prompts, where two types of parameters are used: the scaling factor γ e ∈R d and the offset factor β e =R d , both of which are generated by the task-specific embedding e, and their calculation form is as follows:

[0026] [γ e ,β e =ψ(e)

[0027] On this basis, the prompt generation module further designs a way to combine task information to enhance the effect of prompt generation. Specifically, for each sample x∈D t , first calculate its similarity with the task key vector k t through the following matching function:

[0028] l matching (x,k t )=S C (fb(fp(x))[cls],k t )

[0029] where, S C represents the cosine similarity function, and fb(fp(.))[cls] represents the [cls] token representation of the last layer Transformer in the frozen pre-trained model. The model selects the task most relevant to the input sample argmax t through the matching degree with the key vectors k t∈{1,...T} of multiple tasks. Finally, the selected task key vector k tIt will be used as input together with the generated sample-level prompts to guide the model to make more effective predictions. This mechanism integrates instance-aware and task-driven prompt information, which not only improves the adaptability of the model but also effectively alleviates the forgetting problem in continuous task learning.

[0030] Different from traditional methods that rely on a large-scale static prompt pool, this method adopts a lightweight grouped optimization strategy to avoid excessive memory overhead while ensuring efficient knowledge encoding. In addition, this module can control feature transfer at a fine-grained level and enhance the model's adaptability in the presence of significant domain differences. Given the embedding sequence E ∈ R (n+1)×d , this module directly generates an adaptive prompt P based on the relationship of the embedding vectors a , and the calculation process is as follows:

[0031] P a = LT(MLP(LN(E) T )); ψ(e)) T

[0032] = (γ e MLP(LN(E) T ) + β e ) T ∈ R p×d ψ is a linear layer used to predict two sets of affine transformation parameters [γ, β], where e is the conditional input embedding of ψ. This design keeps the prompt-related parameters optimized locally on the client side, which not only ensures efficient adaptation but also meets the privacy protection requirements of federated learning, and has scalability for practical applications.

[0033] Classification loss function

[0034] At the final stage of ViT (Vision Transformer), the classifier is introduced as the final component of the model. It consists of a single-layer feed-forward neural network and takes [cls] from the last Transformer layer as input. [cls] is used to summarize the global information of the input image, and the classifier maps it to the label space to complete the final prediction:

[0035]

[0036] where represents the predicted class probability distribution.

[0037] The adaptive prompt generation module is updated by minimizing the following total loss function:

[0038]

[0039] where (x, y) belongs to the dataset D of task t t , where C i:tDenote the t-th task of client i. l in the loss function ce represents cross-entropy loss, l matching is the matching loss term of the adaptive prompt generation module. λ is the balance coefficient between the two losses. In this design, it is 0.1. During the training process, the parameters of the ViT backbone network are kept frozen, and only the parameters φ of the prompt generation module 1:L and the parameter f in the classifier c are updated.

[0040] In this personalized federated learning framework, the training process adopts an iterative parameter aggregation module. After each training cycle, each participating client will upload the classification head parameters H in its local model l to the central server. The server aggregates these distributed classification head parameters through a weighted average algorithm to generate a globally shared classification head H g :

[0041]

[0042] where l ∈ {1, 2.....M} indicates that there are M clients. At the beginning of the next round of training, this aggregated and optimized global classification head will be distributed to each client as the initialization parameter to guide the subsequent training process of its local model. This module not only preserves the privacy of client data but also realizes knowledge transfer through parameter sharing, effectively improving the generalization performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is the specific implementation flowchart of the present invention.

[0044] Figure 2 is the average accuracy on the local test set of the aggregated global model (CIFAR-100, each client contains 5 class incremental tasks). It is divided into synchronous tasks and asynchronous tasks.

[0045] Figure 3 is the experimental result in the federated incremental learning scenario on the CIFAR100 dataset. The final performance is measured by temporal knowledge retention and spatial knowledge retention on synchronous and asynchronous tasks respectively:

[0046]

[0047] where represents that after the r-th round of training, client A iThe test accuracy of the local model on the 0th task, represents the test accuracy of the local model of client A at the initial round on the 0th task. i The test accuracy of the local model on the 0th task.

[0048]

[0049] where represents the accuracy of the global model on the local task T of client A i after the r-th round of training, i r and represents the accuracy of the local model on its current local task T i r after the r-th round of training.

[0050] (a) is Asynchronous KRs; (b) is Asynchronous KRt; (c) is Synchronous KRs; (d) is Synchronous KRt Detailed implementation method

[0051] CIFAR-100 is a benchmark dataset commonly used in continual learning research, which contains a total of 60,000 color images of 32×32 pixels, covering 100 categories. This dataset is often used to evaluate the performance of models in class-incremental tasks.

[0052] In the synchronous (Synchronous Task) setting, all clients follow the same task order, but the class sample distributions of each client are different. This common configuration of federated class-incremental learning simulates the data heterogeneity by introducing the Dirichlet distribution prior. In our experiment, the parameter of the Dirichlet distribution is set to 1. The dataset is divided into five tasks, each task contains 20 non-overlapping categories, and the categories between tasks are also completely non-overlapping. In each task, the class samples are randomly divided into several non-overlapping subsets, and the number of subsets is the same as the number of clients.

[0053] In the asynchronous (Asynchronous Task) setting, some categories are globally visible to all clients, while others are only visible to specific clients, which reflects the typical non-independent and identically distributed (Non-IID) characteristics in static federated learning. Each client is assigned 15 private categories that are only accessible to that client, and there are also 25 categories shared among all clients. Therefore, each client has data from 40 categories in total, and these categories are randomly divided into five tasks, each task contains 8 categories.

[0054] On the CIFAR100 dataset, as Figure 2 Our method has achieved a significant performance improvement, reaching accuracies of 93.13% and 87.16% in synchronous and asynchronous settings respectively, which are 2.57% and 3.7% higher than the 90.56% and 83.46% of the previous state-of-the-art FedMGP respectively. This effectively improves the recognition accuracy.

[0055] In Figure 3 Figures (a) and (c), whether in asynchronous or synchronous settings, our method performs excellently in maintaining spatial knowledge, with almost no forgetting compared to other methods, fully demonstrating its effectiveness in alleviating spatio-temporal catastrophic forgetting.

[0056] In Figures (b) and (d), when evaluating the ability to maintain temporal knowledge, although our method shows a downward trend compared to FedMGP, with a 13.2% decrease in the asynchronous setting and a 6% decrease in the synchronous setting. It is worth noting that we still maintain competitive performance overall compared to other methods.

Claims

1. A personalized federated incremental learning method based on instance-level prompt generation, characterized in that The implementation steps of this method are as follows: Step 1: The local client trains an image classification model; S11 Input: The image data of each client's current task; S12 Model training: Each client uses ViT as the basic network framework; in each Encoding Layer of ViT, a lightweight adaptive prompt generator is embedded to dynamically generate corresponding personalized instance-level prompts according to the instance features of the input data; during the training process, the system constructs a loss function through the supervision signal of the classification head, and jointly optimizes the parameters of the adaptive prompt generator and the classification head in an end-to-end manner, while keeping the parameters of the ViT backbone network frozen to ensure the stability of the basic features; S13 Output: The locally trained image classification model of the client; Step 2: The server side performs global classification head aggregation; S21 Parameter interaction: Each client only uploads the trained classification head parameters to the server for average aggregation; the parameters of the dynamic prompt generation module are retained locally on each client as personalized parameters; S22 Classification head parameter reset: The aggregated global classification head model will be securely distributed to each client as the initial model for the next round of local training.

2. The personalized federated incremental learning method generated based on instance-level prompts according to claim 1, wherein, ViT serves as a unified basic network architecture, and all clients perform personalized learning based on this backbone network; first, for an input image x ∈ R H×W×C The patch embedding layer divides the input image into several image patches and converts them into a sequence of d-dimensional embedding vectors. At the same time, a learnable [cls] classification token is introduced as a global feature aggregation node, which is concatenated with the embedding vector sequence to obtain e0 ∈ R (n +1)×d ; e l represents the overall input sequence: e l = [cls; e0, e1.......e n ∈ R (n+1)×d In the feature extraction stage, multiple layers of Transformer encoders process the input sequence in turn, where each layer of encoder contains a multi-head self-attention module MHSA and a multi-layer perceptron MLP, and residual connection and layer normalization LN techniques are used to ensure the stability of gradient propagation; e l ' = LN(MHSA(e l , e l , e l )) e l ” = MLP(e l ') e l+1 = LN(e l '+e l ”) The architecture design based on global self-attention can effectively model the long-range dependence relationship between image patches, so as to achieve accurate understanding of the overall semantics of the image.

3. The personalized federated incremental learning method generated based on instance-level prompts according to claim 1, wherein The adaptive prompt generator dynamically generates adaptive prompts related to the input instance through a feed-forward network and a linear transformation; This module extracts instance-specific information based on the transposed input, uses layer normalization LN, multi-layer perceptron MLP and linear transformation layer LT to generate dynamic prompts in real time, so as to guide the model to adjust the feature representation between the current task and future tasks; first, the input features are normalized, and their dimensions are transposed from (n + 1)×d to d×(n + 1), and then input into the MLP for processing. The MLP converts the (n + 1)-dimensional input into a p-dimensional output to generate instance-level prompts, and these prompts fuse the global context information from each feature channel; Such prompts serve as adaptive guidance, enabling the model to retain key task-related knowledge and effectively adapt to different client-specific dynamic data.

4. The personalized federated incremental learning method generated based on instance-level prompts according to claim 1, wherein To enhance the personalization and adaptability of prompts across different tasks, the prompt generation module introduces the feature dimension transformation framework LT. LT uses affine transformation to adjust the generated prompts, with two types of parameters: the scaling factor γ e ∈R d and the offset factor β e =R d , both of which are generated by the task-specific embedding e, and their calculation forms are as follows: [γ e ,β e = ψ(e) The prompt generation module designs a way to combine task information to enhance the effect of prompt generation; for each sample x∈D t , first calculate its similarity with the task key vector k t using the following matching function: l matching (x, k t ) = S C (fb(fp(x))[cls], k t ) Among them, S C represents the cosine similarity function, and fb(fp(.))[cls] represents the [cls] token representation of the last layer Transformer in the frozen pre-trained model; the model selects the task argmax that is most relevant to the input sample by matching with the key vectors k t of multiple tasks t∈{1,...T} Finally, the selected task key vector k t will be used as the input together with the generated sample-level prompt to guide the model to make predictions more effectively.

5. The personalized federated incremental learning method generated based on instance-level prompts according to claim 1, characterized in that, Adopt a lightweight grouping optimization strategy to avoid excessive memory overhead while ensuring efficient knowledge encoding; given the embedding sequence E ∈ R (n+1)×d , directly generate an adaptive prompt P based on the relationship of the embedding vectors a , and the calculation process is as follows: P a = LT(MLP(LN(E) T );ψ(e)) T = (γ e MLP(LN(E) T ) + β e ) T ∈R p×d ψ is a linear layer used to predict two sets of affine transformation parameters [γ,β], where e is the conditional input embedding of ψ; this design keeps the prompt-related parameters optimized locally on the client, which not only ensures efficient adaptation but also meets the privacy protection requirements of federated learning, and has practical application scalability.

6. The personalized federated incremental learning method generated based on instance-level prompts according to claim 1, characterized in that, In the final stage of ViT, a classifier is introduced as the final component of the model; it consists of a single-layer feed-forward neural network and takes [cls] from the last Transformer layer as the input; [cls] is used to summarize the global information of the input image, and the classifier maps it to the label space to complete the final prediction: Among them represents the predicted class probability distribution.

7. The personalized federated incremental learning method based on instance-level prompts according to claim 1, characterized in that, The adaptive prompt generator is updated by minimizing the following total loss function: where (x, y) belongs to the dataset D of task t t , where C i:t represents the t-th task of client i; l in the loss function ce represents the cross-entropy loss, l matching is the matching loss term of the adaptive prompt generator; λ is the balance coefficient between the two losses; in this design, it is 0.

1. During the training process, the parameters of the ViT backbone network are kept frozen, and only the parameters φ of the prompt generation module are updated 1:L and the parameters of f c in the classifier.

8. The personalized federated incremental learning method generated based on instance-level prompts according to claim 1, wherein In the personalized federated learning framework, the training process adopts an iterative parameter aggregation module; after each training cycle is completed, each participating client will upload the classification head parameter H in its local model l to the central server; the server side aggregates these distributed classification head parameters through a weighted average algorithm to generate a globally shared classification head H g : where l ∈ {1, 2.....M} represents that there are M clients; at the beginning of the next round of training, this globally aggregated and optimized classification head will be distributed to each client as the initialization parameter to guide the subsequent training process of its local model; It not only preserves the privacy of client data but also realizes knowledge transfer through parameter sharing.

Images