A prompt-based federated class-incremental learning method and system

Abstract

The application provides a prompt-based federated class incremental learning method and system, and relates to the field of federated class incremental learning. For each incremental learning task in a task sequence, the specific steps of federated class incremental learning are as follows: a global server distributes the latest global model to selected clients; the clients optimize the prompt vector in the local training process based on the local private training sample and the prompt pool by using a dynamic prompt selection mechanism based on key-value pairs, so as to guide the model to output more distinctive features; the clients upload the optimized prompt pool and model parameters to the global server; the global server globally fuses the prompt pool and the model parameters, and the fused prompt pool and model parameters constitute a new global model; and the application effectively solves the data heterogeneity problem caused by the continuous arrival of new class data and the dynamic addition of new clients in the federated class incremental problem, and solves the catastrophic forgetting problem of old knowledge.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of federated class incremental learning, and in particular to a prompt-based federated class incremental learning method and system. BACKGROUND

[0002] In today's society, people's concern for data security is growing, and collaborative model training with privacy guarantees has become a hot trend. Federated learning, as a new distributed machine learning paradigm, maintains a global model by communicating with distributed clients without accessing their local data, thus providing opportunities for multiple users to collaborate in training while protecting the data privacy of participants, thus attracting great interest from academia and industry.

[0003] Although federated learning has been successfully applied to a large number of research fields such as smartphones and medical diagnosis, it still faces key challenges in practical deployment. Most of the current proposed federated learning models are based on idealized assumptions, i.e., the entire federated learning framework is trained in a static application scenario, i.e., the data classes learned over time are fixed. However, in real-world applications, the data of clients evolves due to changes in the environment, trends, or new interests, and new class data from local clients arrives continuously in the form of a stream. In addition, some new clients receive new class data that other local clients have not seen before, and these new clients may also join the federated training irregularly in practical applications. Under the traditional federated learning setting, the dynamic arrival of such data and the dynamic changes of clients will cause the model to continuously receive new class information, leading to the gradual coverage or forgetting of previously acquired knowledge, resulting in a serious "catastrophic forgetting" problem, which greatly affects the overall performance and effectiveness of the model, bringing many challenges to the training of the model. SUMMARY

[0004] To solve the above problems, the present application proposes a prompt-based federated class incremental learning method and system, which effectively solves the data heterogeneity problem caused by the continuous arrival of new class data and the dynamic joining of new clients in federated class incremental learning, and solves the problem of catastrophic forgetting of old knowledge.

[0005] According to some embodiments, the present application adopts the following technical solutions:

[0006] A prompt-based federated class incremental learning method, for each incremental learning task in the stream task sequence, the specific steps of federated class incremental learning are as follows:

[0007] The global server distributes the latest global model to selected clients;

[0008] The client optimizes the prompt vector in the local training process by using a dynamic prompt selection mechanism based on key-value pairs, based on the local private training samples and the prompt pool, to guide the model to output more discriminative features; after the local training is completed, the client uploads the optimized prompt pool and the model parameters to the global server;

[0009] The global server globally fuses the prompt pool and the model parameters, and the fused prompt pool and model parameters constitute a new global model for the next round of global training until all global training rounds are completed.

[0010] According to some embodiments, the application adopts the technical scheme as follows:

[0011] A prompt-based federated class-incremental learning system, for each incremental learning task in a task sequence, performs federated class-incremental learning, comprising:

[0012] The global distribution module is configured to: the global server distributes the latest global model to selected several clients;

[0013] The local training module is configured to: the client optimizes the prompt vector in the local training process by using a dynamic prompt selection mechanism based on key-value pairs, based on the local private training samples and the prompt pool, to guide the model to output more discriminative features; after the local training is completed, the client uploads the optimized prompt pool and the model parameters to the global server;

[0014] The global fusion module is configured to: the global server globally fuses the prompt pool and the model parameters, and the fused prompt pool and model parameters constitute a new global model for the next round of global training until all global training rounds are completed.

[0015] According to some embodiments, the application adopts the technical scheme as follows:

[0016] A computer program product comprising a computer program, which, when executed by a processor, implements the prompt-based federated class-incremental learning method.

[0017] According to some embodiments, the application adopts the technical scheme as follows:

[0018] A non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implements the prompt-based federated class-incremental learning method.

[0019] According to some embodiments, the application adopts the technical scheme as follows:

[0020] An electronic device comprises a processor, a memory and a computer program; wherein the processor is connected with the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the prompt-based federated class incremental learning method.

[0021] Compared with the prior art, the beneficial effects of the present application are:

[0022] 1. The present application uses prompts to optimize the model, and uses prompts to represent the local knowledge learned by the client; the parameters for training and optimization are only the learnable prompts; the model with a large number of frozen parameters is transmitted in the federated communication, and the lightweight parameters are transmitted, so that the communication overhead and the calculation and storage burden of the client are greatly reduced.

[0023] 2. The present application constructs a prompt pool to store prompts, and adopts a knowledge distillation strategy to alleviate the catastrophic forgetting problem; knowledge is transmitted through the output of the prompt and the global model, so that the real data from the old task is avoided to be stored and replayed, the stability of the model is enhanced, and the data privacy protection principle is met.

[0024] 3. The present application uses complete image information and fine-grained image block information, and considers multi-level feature learning of global and local, so that the model can better distinguish the key local details of new and old categories, and the plasticity of the model is improved.

[0025] 4. The present application adopts the design of a dynamic prompt pool, aggregates the prompts from different clients in the server through federated aggregation, forms a more diversified global prompt pool, alleviates the data heterogeneity problem caused by non-independent and identically distributed under the premise of ensuring data privacy, and makes the model have better generalization ability and stronger adaptability to new tasks in the federated class incremental learning scene. BRIEF DESCRIPTION OF DRAWINGS

[0026] The drawings accompanying the specification of the present application serve to provide a further understanding of the present application, and the schematic embodiments of the present application and the description thereof serve to explain the present application, and do not constitute an improper limitation on the present application.

[0027] Figure 1 The method flowchart of example 1.

[0028] Figure 2 The structure diagram of the classification model of example 1. DETAILED DESCRIPTION

[0029] The present application will be further described below in combination with the drawings and examples.

[0030] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0031] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0032] Embodiment 1

[0033] The embodiment provides a prompt-based federated class incremental learning method. For each incremental learning task in a task sequence, the specific steps of federated class incremental learning are as follows:

[0034] Step S1: The global server distributes the latest global model to selected clients.

[0035] Step S2: The client optimizes the prompt vector in the local training process based on the local private training sample and the prompt pool, and guides the model to output more discriminative features by using a dynamic prompt selection mechanism based on key-value pairs. After the local training is completed, the client uploads the optimized prompt pool and model parameters to the global server.

[0036] Step S3: The global server globally fuses the prompt pool and the model parameters, and the fused prompt pool and model parameters constitute a new global model, which is used for global training in the next round until all global training rounds are completed.

[0037] The specific implementation process is described as follows:

[0038] Federated learning focuses on training a decentralized global model by aggregating the network parameters of different local models, thus solving the problem of privacy data isolation. FedAvg proposes an average weighting strategy, which receives the model parameters uploaded by the client on the server side, then simply averages the parameter values and returns them to each client. FedProx uses a new framework to solve the data heterogeneity of aggregated local models, adding a proximal term on the server side to improve the performance difference caused by system heterogeneity between different clients, helping the local model to approximate the global model. LDP-FL designs a local differential privacy mechanism that acts on the transmission process of federated learning parameters, and also proposes and designs a performance loss constraint mechanism suitable for federated learning, which reduces performance loss by optimizing the constraint range of the loss function. However, the federated class-incremental problem allows models on different clients to learn different numbers of new classes in each task and classify all classes observed so far. Existing federated learning methods cannot continuously learn new classes in a stream form, and due to the limited memory of local clients to store old classes, they will forget old classes when receiving enough data, resulting in a significant decline in model performance.

[0039] Class-incremental learning focuses on continuously learning and recognizing new classes in the real world while solving the forgetting problem of old classes, thus addressing the challenge of changing data classes in dynamic environments. Based on this feature, the setting of class-incremental learning is more consistent with the scenario of continuously generating new data in the real world. Existing methods such as iCaRL build a memory buffer on the local client, and under the cardinality constraint, select samples by approximating the average value of the training data. When a new task arrives, the model is trained jointly with old task samples and new task data. DDE considers extracting the causal relationship between training samples with class imbalance problems to alleviate the class imbalance distribution problem between new and old classes. In addition, DGR proposes a dual collaboration model, including a memory generator for synthesizing old classes and a task solver for solving the forgetting problem, to generate pseudo data to strengthen the model's memory of old knowledge. However, to solve the problem of federated class-incremental learning, existing class-incremental learning methods require privacy knowledge about when and where to collect new classes, which violates the requirement of privacy protection and is difficult to implement in real-world applications.

[0040] The key to federated class-incremental learning is to overcome the catastrophic forgetting problem under the condition of heterogeneous data distribution between different clients. For the problem of federated class-incremental learning, this embodiment proposes a prompt-based federated class-incremental learning method. Specifically, the entire model framework consists of a global server and multiple clients, and the order of tasks containing new data arrives at the client. All clients do not share any original data between each other and between the client and the global server. Each client trains a local model using the received data and uploads the model parameters to the global server to aggregate the global model.

[0041] This embodiment addresses federated incremental learning scenarios with several consecutive incremental tasks, proposing a prompt-based federated incremental learning method. First, a prompt pool is constructed on the client side, employing a key-value pair-based dynamic prompt selection mechanism to optimize the prompt vector during training, guiding the model to output more discriminative features. Second, knowledge distillation is performed using the global model as the teacher model and the local model as the student model to address the forgetting problem caused by the non-independent and identically distributed nature of data across clients. Finally, fine-grained block-level features are utilized to enable the model to better focus on key details of the data, thereby improving the model's fine-grained recognition capability.

[0042] The following is a detailed explanation in three parts:

[0043] The first part introduces the problem definition of federated incremental learning and related preparatory knowledge.

[0044] (1.1) The structure of the classification model for federated incremental learning is as follows: Figure 2 As shown, the classification model includes an input module, a multimodal model CLIP, and a similarity module. The input module assembles categories into text based on the text template "a photo of [class name]". For example, if the category is "dog", the assembled text is "a photo of dog", forming image-text pairs. The similarity of the image-text pairs is used to determine the image category. The similarity is calculated by calculating the cosine similarity between image features and text features. Therefore, the multimodal model CLIP uses a visual encoder and a text encoder to encode the image x and text t respectively, obtaining the image features. and text features Finally, the similarity module calculates the similarity between image features and text features based on cosine similarity, and selects the most similar category as the image classification.

[0045] (1.2): For federated incremental learning problems, there exists a sequence of streaming tasks. , Represents the number of tasks. The training set is divided into... Non-overlapping subsets , No. Task Corresponding to the training subset ,in, Depend on For the sample and its unique heat-coded label The composition here is that the sample is an image, and the one-hot encoded label is the category corresponding to the image. Indicates by The first new category consists of The label space of the nth task has no overlapping label spaces between any two learning tasks. The tasks contained A new category compared to the previous ones What I learned from each task Each of the old categories is different.

[0046] Use the global server express, A local client uses This indicates that in each round of global training, several clients are randomly selected from the local clients to participate in the training. When the [number]th [round] is [number], [the number] clients are selected from the local clients to participate in the training. Select the first incremental learning task. Local clients During training, the global server will distribute the latest global model to it. (in This indicates the number of rounds of global training, totaling [number] rounds. (Global training round).

[0047] Local client In the current task training subset Randomly select 60% of the categories as your own private training dataset. and in Optimization Thus, its first Local updates in incremental tasks .

[0048] The global server collects all updated models from selected clients and aggregates them into a global model for the next global round. Repeat the above process to obtain the latest global model. Distribute to clients for local training.

[0049] In this embodiment, all The local clients are divided into three categories, namely Specifically, The client-side model in the model is trained on previous tasks and retains old knowledge, but it cannot receive new categories for the current task; The client-side model has participated in the training of old tasks and can also receive new categories for the current task; The client model in the code has no knowledge of the old categories and only receives the new category for the current task.

[0050] When each incremental task arrives, it is randomly determined. , and Due to the setting of the federated incremental learning problem, new clients will be irregularly added to the federated training , 、 and are also dynamically changing.

[0051] (1.3): Pre-trained models have strong generalization ability and representation learning ability, which enable them to perform well in diversified downstream tasks. One of the typical representatives is the CLIP (Contrastive Language-Image Pre-training) model, which consists of an image encoder and a text encoder. The input is an image and its corresponding text, and the output is an image feature and a text feature. CLIP learns to align the semantic representations of images and texts in a unified embedding space by jointly training a visual encoder (such as Vision Transformer) and a text encoder (such as Transformer Encoder).

[0052] In this embodiment, ViT (Vision Transformer) is used as the visual encoder to encode the input image into a fixed-dimensional image feature vector. Specifically, given an input image and a pre-trained visual encoder ViT, ViT is represented as , where is the input embedding layer, and is a number of self-attention layers.

[0053] In existing ViT, is usually implemented by a convolutional layer with a kernel size and stride equal to the image block size, thus implicitly completing image segmentation and feature extraction. Specifically, first, the input image is uniformly divided into image blocks of fixed size , and then each image block is flattened to obtain a feature embedding , which forms a feature embedding sequence .

[0054] After introducing a learnable classification token [CLS] token into the sequence, it is input into a number of self-attention layers , which utilize the [CLS] token to represent the global image feature , providing rich representation ability for cross-modal alignment and prompt guidance.

[0055] The second part introduces the local client training process, including prompt pool construction, knowledge distillation, and image block-level feature learning, and introduces the model-related loss functions.

[0056] In the local client training stage, based on the pre-training CLIP framework, combined with prompt pool, knowledge distillation and image block-level fine-grained feature learning, a local training strategy suitable for non-independent and identically distributed data environment is designed. The goal of this stage is to make full use of the private data of the client, guide through dynamic prompts and refine local features, so that the model learns discriminative global and local representations, while avoiding catastrophic forgetting.

[0057] (2.1): Prompt-based parameter tuning strategy

[0058] In order to make the pre-trained model better adapt to diversified downstream tasks, a prompt-based parameter tuning strategy is adopted 。

[0059] Specifically, the backbone parameters of the CLIP model are frozen, and a set of learnable prompt vectors are introduced , a prompt pool is selected Prompt pool subset , inserted into the input sequence of several self-attention layers of the visual encoder ViT , so as to guide the model to pay attention to the features related to the current task. For the input image , , the input sequence of

[0060]

[0061] Among them, represents the prompt vector, represents the feature embedding of the image block, represents the spliced sequence. By optimizing and updating the prompt vector, the pre-trained model is adapted to the downstream task in a lightweight manner, while ensuring that its original strong generalization ability is not updated destructively.

[0062] In the federated class incremental learning environment, the data distribution of different clients is usually heterogeneous, and each client may only have data of a part of the class. In order to improve the adaptability of the model, the prompt pool is designed and constructed The prompt pool consists of several groups of key-value pairs:

[0063]

[0064] Among them, represents the learnable prompt key, which is used to represent the semantic index of the prompt; represents the prompt value, that is, the prompt vector.

[0065] It is expected that the input image itself can determine which prompts to select by matching the query key, selecting the prompt closest to the input itself, and thus achieving dynamic selection of prompts. In order to match the input image to the corresponding prompt in the prompt pool, a query function is introduced The input image is mapped to the same dimension as the prompt key, and the embedding layer of ViT is used as the query function, thereby obtaining the query feature . The cosine distance is used as the standard to measure the relevance between the query feature and the prompt key, given the input image , the similarity is calculated as follows:

[0066]

[0067] wherein, represents the cosine similarity between two vectors, and the top-m prompt keys most relevant to the input image in the prompt pool are retrieved according to the similarity, i.e.:

[0068]

[0069] wherein, represents a subset of all prompt keys in the prompt pool, containing the top-m prompt keys most relevant to the image selected from the prompt pool.

[0070] The selected prompt vector is spliced with the image embedding, and a classification label is introduced to obtain the input sequence of several self-attention layers :

[0071]

[0072] The spliced sequence is input into several self-attention layers , and the output sequence is:

[0073]

[0074] wherein, is the global image feature extracted, represented by the [CLS] token, used for global image representation and classification; represents the prompt after the self-attention layer ; represents the feature vector at the image block level; represents the sequential splicing operation.

[0075] ​​Through dynamic selection prompts, the model can adaptively adjust the introduction of guidance information for different tasks and data distributions, and improve the feature learning ability of the local client.

[0076] (2.2): symmetric contrastive loss function

[0077] In order to effectively utilize the CLIP cross-modal feature learning to learn the cross-modal semantic association between images and texts, the embodiment first introduces a basic symmetric contrastive loss function for complete pictures , which maximizes the similarity between matched image-text pairs and minimizes the similarity between unmatched image-text pairs.

[0078] Specifically, focusing on each client, for a given training subset with a batch size of , the class name to which the image belongs is obtained through the label in the sample, the class name is filled into the text template "a photo of [class name]", and the text description corresponding to the image is constructed, obtaining image-text pairs , the image and the text are input into the visual encoder and the text encoder respectively, and the image feature vector and the text feature vector after the concatenation prompt are extracted. L2 normalization is performed on all image features and text features to obtain normalized features and , and the cosine similarity between image features and text features in the batch is calculated to obtain an output matrix with a dimension of :

[0079]

[0080] wherein, denotes the cosine similarity between two vectors, is a learnable temperature parameter.

[0081] Using the standard cross-entropy loss, the following image-to-text loss and text-to-image loss are constructed:

[0082]

[0083]

[0084] Finally, the contrastive loss As follows:

[0085]

[0086] (2.3): Fine-grained feature learning at image block level

[0087] ViT exhibits strong global feature modeling capability by dividing the input image into fixed-size image blocks and then modeling the relationship between image blocks through self-attention (i.e., several self-attention layers ). However, for subtle local differences between different categories (such as texture structure, edge contour, etc.), using only the [CLS] token may mask key local features, resulting in insufficient model discrimination.

[0088] To overcome this limitation, the embodiment proposes fine-grained feature learning at the image block level, which applies semantic constraints to each image block feature embedding output by the embedding layer of ViT, guiding the model to explicitly focus on the alignment of local regions and text semantics, thereby improving the model's fine-grained discrimination ability.

[0089] According to the workflow of the visual encoder ViT, for any input image in a batch , after processing by ViT, a set of image block-level feature vectors corresponding to each local region of the image is naturally obtained , i.e., a fine-grained visual description of the image, which can be expressed as follows:

[0090]

[0091] where denotes the feature vector of the th image block of the th image, denotes the total number of image blocks, and for a batch containing images, a feature tensor with a dimension of is obtained.

[0092] To establish the connection between the local details of the image and the text semantics, the cosine similarity between the feature of each image block and the features of all text descriptions within the batch is calculated. Let the text features corresponding to the text descriptions in the batch be , where . After L2 normalization of all feature vectors, the similarity is calculated as follows:

[0093]

[0094] where Cosine similarity is used to represent the cosine similarity between two vectors. Indicates the first The first image Image patch features obtained by normalizing the feature vectors of each image patch. Indicates the first Text features after normalization of individual text features.

[0095] The similarity matrix is ​​obtained through the above process. The elements in the matrix represent the similarity between the image and the text, which are used to characterize the matching scores between the corresponding images and text.

[0096] In order to select more representative image patches from a high-dimensional similarity matrix, this embodiment designs a top-K aggregation method to balance the disadvantages of max pooling being sensitive to noise and average pooling being easily affected by background information.

[0097] Specifically, for any image-text pair Firstly, from its The highest matching score among the image patches is selected. indivual:

[0098]

[0099] in, This indicates selecting from largest to smallest. indivual.

[0100] Then, through this The final fine-grained matching score for the image-text pair is obtained by averaging the similarity scores. :

[0101]

[0102] For all images in the batch, after temperature factor... After adjustment, a two-dimensional image patch-level image-text matching score matrix is ​​finally obtained. .

[0103] Based on image-text matching score matrix Constructing image-to-text loss at the image patch level and text-to-image loss :

[0104]

[0105]

[0106] in, for The Middle Line 1 column corresponding elements.

[0107] Finally, an independent contrastive loss is constructed for the fine-grained matching score matrix , which optimizes the image block-level fine-grained feature learning and drives the model to pay attention to and utilize the local discriminative information of the image, which is expressed by the formula as:

[0108]

[0109] (2.4): Knowledge distillation

[0110] Since the client can only access the local data of the current task and cannot obtain the data of the old task, the continuous arrival of new classes will gradually aggravate the forgetting of the model for the old knowledge. In order to alleviate the problem of catastrophic forgetting, the embodiment transfers the knowledge learned by the model in the previous task to the new task through knowledge distillation.

[0111] Specifically, in each round of global training, the global server distributes the global model to the local clients participating in the training as an old model, and after the client receives the old model, it uses the old model to predict the current training data and uses the output as a soft label to guide the training of the student model (i.e. the local model that the client needs to train).

[0112] Traditional knowledge distillation methods are usually performed at the final output probability distribution layer. Considering that CLIP is a multi-modal model, its output layer has fused visual and language information, which is not conducive to the independent alignment of image features. Therefore, the embodiment introduces knowledge distillation at the feature level to align the output representations of the global model and the local model in the visual space, helping the local model to obtain old knowledge from the global model.

[0113] The global model is , and the local model of the client is , and the input image is encoded by the ViT visual encoder, and the feature representations of the global model and the client model in the visual space are and respectively. In order to make the local model consistent with the global model at the visual representation level, define the feature layer distillation loss as:

[0114]

[0115] where is the local data set owned by the client at the th task.

[0116] The third part introduces global model aggregation, which realizes aggregation based on prompt keys to reduce the influence of data heterogeneity on model performance.

[0117] In federated incremental learning, due to the heterogeneity of client data distribution and the use of prompt pool mechanism, the distribution of prompt vectors optimized by each client in the semantic space may be quite different. The traditional aggregation algorithm based on model weight average may cause the prompt information to be blurred, making it difficult to effectively integrate the local knowledge from each client in the global model.

[0118] To solve the above problems, the embodiment proposes a clustering aggregation mechanism based on prompt keys, which enables the global model to integrate the prompt information from each client under heterogeneous data distribution in a more structured manner. Specifically:

[0119] (3.1) In the aggregation phase of each round of federated learning, the server first collects the local updated model parameters from the client participating in this round of training.

[0120] Since the relevant knowledge learned by the local model is stored in the prompt pool during the local training process, the prompt keys related to the prompt pool in the parameters are focused on for the client . Let the prompt pool uploaded by the client after local training be:

[0121]

[0122] Integrate the prompt key set and the prompt set from the client:

[0123]

[0124]

[0125] (3.2): To facilitate subsequent clustering operations based on prompt keys, the prompt key set is L2 normalized to obtain .

[0126] All normalized prompt keys are regarded as points in the feature space, and the K-Means algorithm is used to cluster the normalized prompt key set into clusters, denoted as , where is the global prompt pool size, represents the K-Means algorithm aims to find the K centroids that can minimize the within-cluster sum of squares , which can be expressed as:

[0127]

[0128] where, is the cluster .

[0129] By clustering, each prompt key and its corresponding prompt vector is assigned a cluster label . The prompt values and prompt keys within each cluster are similar in function, and by averaging within each cluster, the global prompt value and prompt key are obtained.

[0130] For the th cluster, find the index set of all prompt values and prompt keys belonging to this cluster , then the new global prompt value and prompt key of this cluster are:

[0131]

[0132]

[0133] where, and are the original prompt values and prompt keys.

[0134] Through the intra-cluster averaging operation, it can be ensured that similar prompts in the prompt key semantic space are aggregated together, aligning different client prompt pools, generating a more robust and better generalization ability global prompt pool, and finally the global prompt pool is updated to:

[0135]

[0136] (3.3): For the client model, other trainable parameters in addition to the prompt pool , still use the standard federated averaging algorithm for aggregation, which can be expressed as:

[0137]

[0138] Through this hybrid aggregation strategy, both specialized prompt knowledge and the stability of other parts of the model can be preserved, and the final global model is composed of the aggregated prompt pool-related parameters and other parameters.

[0139] A specific example is provided, including:​

[0140] The first step: at the beginning of each incremental learning task, the global server selects a part of the clients from all the clients to participate in the current training, which may only master the old class knowledge, master both the old and new class knowledge, or only access the new class knowledge.

[0141] The second step: the server distributes the latest global model to these clients as the base model of their local training; the clients receive the global model, generate a query feature for each image combined with their local data, use the query feature to match in the prompt pool, and dynamically select a set of prompt vectors most relevant to the current image, and the selected prompt vectors and the features of the image are input into the visual encoder.

[0142] The third step: using the image and text features, the contrast loss is calculated to narrow the distance between the overall image feature and its corresponding text description through contrast learning; for the image block level feature, the Top-K aggregation method is used to select the most representative K image blocks, and the fine-grained image block contrast loss is calculated to align the image features of the local area with the text semantics.

[0143] The fourth step: the visual feature output of the global model is used as a soft label during local model training, and the distillation loss is constructed by minimizing the L2 norm distance between the local model visual feature and the global model visual feature.

[0144] The fifth step: after the local training is completed, the client uploads the optimized prompt pool and a small amount of other parameters to the global server for global model aggregation.

[0145] The sixth step: after the server receives all the parameters uploaded by the clients, the prompt key-based aggregation method is used to perform K-Means clustering on all prompt keys, and the cluster average is performed on each cluster after clustering to obtain new prompt keys and prompt values; for other trainable parameters, standard federated average aggregation is performed. The updated prompt pool and other related parameters constitute a new global model, which is used for the next round of global training.

[0146] The seventh step: repeat the second step to the sixth step until all global training rounds are completed.

[0147] Embodiment 2

[0148] In an embodiment of the present application, a prompt-based federated class incremental learning system is provided, which performs federated class incremental learning for each incremental learning task in a task sequence, comprising:

[0149] The global distribution module is configured to: the global server distributes the latest global model to a plurality of selected clients;

[0150] The local training module is configured to: based on the local private training sample and the prompt pool, the client adopts a dynamic prompt selection mechanism based on a key-value pair to optimize the prompt vector in the local training process, and guide the model to output more discriminative features; after the local training is completed, the client uploads the optimized prompt pool and the model parameter to the global server;

[0151] The global fusion module is configured to: the global server globally fuses the prompt pool and the model parameter, the fused prompt pool and the model parameter constitute a new global model, which is used for global training in the next round until all global training rounds are completed.

[0152] Embodiment 3

[0153] In an embodiment of the present application, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the prompt-based federated class incremental learning method.

[0154] Embodiment 4

[0155] In an embodiment of the present application, a non-transitory computer readable storage medium is provided, which is used to store computer instructions, and the computer instructions, when executed by a processor, implement the prompt-based federated class incremental learning method.

[0156] Embodiment 5

[0157] In an embodiment of the present application, an electronic device is provided, comprising a processor, a memory and a computer program; wherein the processor is connected with the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the prompt-based federated class incremental learning method.

[0158] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks

[0159] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer implemented process, so that the instructions executed on the computer or other programmable data processing devices provide a process for implementing the functions specified in the flowchart Figure 1 one flow or multiple flows and / or the functions specified in the block Figure 1 one flow or multiple flows and / or the functions specified in the block

[0160] The above describes the specific embodiments of the present application in conjunction with the drawings, but is not a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the scope of protection of the present application.

Claims

1. A prompt-based federated incremental learning method, characterized in that, The specific steps for federated incremental learning for each incremental learning task in the streaming task sequence are as follows: The global server distributes the latest global model to a selected number of clients; The client uses a key-value pair-based dynamic suggestion selection mechanism based on local private training samples and suggestion pools to optimize suggestion vectors during local training and guide the model to output more discriminative features. After local training is complete, the client uploads the optimized cue pool and model parameters to the global server; The global server performs global fusion of the cue pool and model parameters. The fused cue pool and model parameters constitute a new global model, which is used for the next round of global training, until all global training rounds are completed. The prompt pool consists of several sets of key-value pairs: in, A learnable suggestion key, used to represent the semantic index of the suggestion; Represents the prompt vector; A key-value pair-based dynamic suggestion selection mechanism is adopted to optimize the suggestion vector during local training, specifically as follows: The cue pool consists of cue keys and corresponding cue vectors. The query features of the input image are calculated through a query function. The cosine distance is used as a standard to measure the similarity between the query features and the cue keys. Based on the similarity, a preset number of cue vectors that are most relevant to the input image are selected from the cue pool. The selected cue vectors are embedded and concatenated with the image, and a classification label is introduced as the input sequence of the final visual encoder. The loss functions for local training include contrastive loss based on image and text features, fine-grained image patch contrastive loss, and distillation loss. Contrastive loss based on image and text features narrows the distance between the overall image features and their corresponding text descriptions through contrastive learning. Fine-grained image patch contrastive loss aligns the image features of local regions with the text semantics. Distillation loss uses the image features of the global model as soft labels and minimizes the L2 norm distance between the image features of the local model and the global model, thus maintaining consistency between the local model and the global model at the visual representation layer. The global fusion of the prompt pool employs a clustering and aggregation mechanism based on prompt keys, enabling the global model to fuse prompt information from various clients across heterogeneous data distributions in a more structured manner. Specifically: Treating suggestion keys as points in space, the K-Means algorithm is used to aggregate suggestions with similar semantic spaces by minimizing the sum of squares within each cluster. This aligns suggestion pools across different clients, generating a more robust global suggestion pool with better generalization capabilities. For the ... Given a cluster, find the set of indices of all hint values ​​and hint keys belonging to that cluster. Then the new global hint value for that cluster and prompt key for: in, and This refers to the original prompt value and prompt key.

2. The prompt-based federated incremental learning method as described in claim 1, characterized in that, In a streaming task sequence, each task has its own training subset, which consists of training samples and their one-hot encoded labels. The one-hot encoded labels are used to represent the category of the samples, and there is no overlapping label space between any two tasks.

3. The prompt-based federated incremental learning method as described in claim 1, characterized in that, In federated incremental learning, the model to be trained is a classification model, which consists of an input module, a multimodal model CLIP, and a similarity module. The input module assembles categories into text, forming image-text pairs; The multimodal model CLIP uses a visual encoder and a text encoder to encode images and text respectively, obtaining image features and text features. The similarity module calculates the similarity between image features and text features, and selects the most similar category as the image classification.

4. A prompt-based federated incremental learning system, characterized in that, For each incremental learning task in the streaming task sequence, federated incremental learning is performed, including: The global distribution module is configured so that the global server distributes the latest global model to a number of selected clients. The local training module is configured as follows: the client uses a key-value pair-based dynamic suggestion selection mechanism based on local private training samples and suggestion pool to optimize suggestion vectors during local training, guiding the model to output more discriminative features; after local training is completed, the client uploads the optimized suggestion pool and model parameters to the global server. The global fusion module is configured as follows: the global server performs global fusion of the cue pool and model parameters, and the fused cue pool and model parameters constitute a new global model for the next round of global training, until all global training rounds are completed. The prompt pool consists of several sets of key-value pairs: in, A learnable suggestion key, used to represent the semantic index of the suggestion; Represents the prompt vector; A key-value pair-based dynamic suggestion selection mechanism is adopted to optimize the suggestion vector during local training, specifically as follows: The cue pool consists of cue keys and corresponding cue vectors. The query features of the input image are calculated through a query function. The cosine distance is used as a standard to measure the similarity between the query features and the cue keys. Based on the similarity, a preset number of cue vectors that are most relevant to the input image are selected from the cue pool. The selected cue vectors are embedded and concatenated with the image, and a classification label is introduced as the input sequence of the final visual encoder. The loss functions for local training include contrastive loss based on image and text features, fine-grained image patch contrastive loss, and distillation loss. Contrastive loss based on image and text features narrows the distance between the overall image features and their corresponding text descriptions through contrastive learning. Fine-grained image patch contrastive loss aligns the image features of local regions with the text semantics. Distillation loss uses the image features of the global model as soft labels and minimizes the L2 norm distance between the image features of the local model and the global model, thus maintaining consistency between the local model and the global model at the visual representation layer. The global fusion of the prompt pool employs a clustering and aggregation mechanism based on prompt keys, enabling the global model to fuse prompt information from various clients across heterogeneous data distributions in a more structured manner. Specifically: Treating suggestion keys as points in space, the K-Means algorithm is used to aggregate suggestions with similar semantic spaces by minimizing the sum of squares within each cluster. This aligns suggestion pools across different clients, generating a more robust global suggestion pool with better generalization capabilities. For the ... Given a cluster, find the set of indices of all hint values ​​and hint keys belonging to that cluster. Then the new global hint value for that cluster and prompt key for: in, and This refers to the original prompt value and prompt key.

5. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements a prompt-based federated incremental learning method according to any one of claims 1-3.

6. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement a prompt-based federated incremental learning method as described in any one of claims 1-3.

7. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to perform a prompt-based federated incremental learning method as described in any one of claims 1-3.

Images