Federated continuous learning image classification method based on diffusion driven data replay
By using the diffusion model reverse engineering category in the continuous learning of the federated class and performing comparative learning, the problem of the model forgetting historical knowledge when learning new tasks is solved, and the efficient image classification performance improvement under privacy protection is achieved.
Patent Information
- Application Number
- CN202411191850.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-28
AI Technical Summary
There is a catastrophic forgetting problem in the continuous learning of the federated class. The model loses the knowledge it has previously acquired when learning new tasks. In a privacy-sensitive environment, existing policies such as experience playback are limited by data privacy protection and are difficult to apply directly.
Using a data playback method based on diffusion model, the pre-trained conditional diffusion model reversely engineers each category, generates new input conditions, reduces computing resources and time consumption, and improves the domain generalization ability of generated data through comparative learning.
Without revealing user data privacy, the performance of image classification models is significantly improved, avoiding forgetting problems, and not significantly increasing training costs.
Smart Images

Figure CN118710996B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of deep learning, federated learning, continuous learning and computer vision, and in particular to a federated continuous learning image classification method based on diffusion-driven data playback. Background Art
[0002] Federated learning is an emerging machine learning paradigm that provides a way to perform federated learning across decentralized devices while ensuring data privacy. Its ability to preserve privacy makes it indispensable in fields such as healthcare and finance that are highly sensitive to data security. However, the practical application of federated learning has also encountered challenges, such as the introduction of new data categories by clients over time and the variability of participants in the federation. This situation has led to the development of federated class continual learning, a new concept that requires the model to incorporate new class information into the model during federated training without compromising previously acquired knowledge.
[0003] The main challenge facing federated continuous learning is the problem of catastrophic forgetting, where the model loses previously acquired knowledge when learning a new task. While this problem has been explored to some extent in traditional continuous learning, privacy protection in federated continuous learning introduces unique limitations that may prevent the direct application of existing strategies. For example, experience replay, the main approach in continuous learning to mitigate forgetting by retaining and replaying data from previous tasks, faces significant obstacles in the federated learning setting. Specifically, in privacy-sensitive environments such as healthcare, users may not be allowed to store historical data for a long time. In addition, the departure of a federated participant will result in the loss of its stored data, further complicating data management and continuity.
[0004] Therefore, how to implement federated continuous learning algorithms while ensuring the privacy and security of user data remains an urgent problem to be solved. Summary of the invention
[0005] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention discloses a method for federated continuous learning image classification based on diffusion-driven data replay. The method can allow the image classification model to learn on multiple different tasks without forgetting the knowledge of historical tasks without leaking the privacy of user data. In order to solve the difficulty of catastrophic forgetting in federated continuous learning, the present method proposes a new data replay method based on a diffusion model. We reverse engineer each category by using a pre-trained conditional diffusion model, and search for the corresponding input conditions of each category in the input space of the model, thereby significantly reducing computing resources and time consumption while ensuring effective generation. In addition, the present method improves the domain generalization ability of the classifier for generated data and real data through contrastive learning, and indirectly improves the representation ability of generated data for real data. Compared with existing methods, the present method significantly improves the performance of image classification models while not significantly increasing training cost consumption.
[0006] The object of the present invention is to achieve the following technical solution: a federated class continuous learning image classification method based on diffusion driven data playback, the method comprising: multiple clients and a server for task sequence The tasks on the above are continuously learned, where Indicates tasks, , is the total number of tasks; the data set of each task is scattered among various clients, Indicates Task No. The client dataset is: , Represents the total number of clients; datasets for different tasks do not have overlapping image categories, and when learning a new task, the real data of the old task cannot be accessed.
[0007] The learning of a new task comprises the following steps:
[0008] Step 201, initialization phase; the server sends the pre-trained conditional latent diffusion model and image classification model to each client respectively; if the new task is the first task, the server will initialize the image classification model and send the initialized image classification model to each client, otherwise, the server sends the image classification model that has learned the previous task to each client;
[0009] Step 202, category reverse encoding stage: multiple clients jointly reverse encode all categories of the new task, encode each category of the new task and obtain a category embedding vector;
[0010] Step 203, image generation phase; the server generates data of each category using the category embedding vector of the new task; if the new task is not the first task, the server also generates data of each category of the historical task using the category embedding vector obtained from the historical task; finally, the server sends all generated data to the client;
[0011] Step 204, image classification model training phase; each client uses its own real data and the generated data obtained from the server to jointly train the image classification model.
[0012] The category reverse encoding stage: multiple clients jointly reverse encode all categories of the new task, and encode each category of the new task into a category embedding vector, including the following steps:
[0013] Step 2201, the server randomly initializes all categories of the new task into a category embedding vector, and then sends the category embedding vector to each client;
[0014] In step 2202, each client performs reverse encoding on each new task category data of the local data set, and inputs the encoding and the category embedding vector received from the server into the pre-trained conditional latent diffusion model, and finally searches and optimizes a new category embedding vector from the conditional input space of the pre-trained conditional latent diffusion model. The expression of the optimization objective function of the reverse encoding is:
[0015]
[0016] in, represents the new category embedding vector, Indicates Task No. The first The category embedding vector of the class, Indicates Task No. A client dataset, No. Task No. The first All data of the class, represents the image dimension reduction encoder of the pre-trained conditional latent diffusion model, is a preset text. represents the text encoder of the pre-trained conditional latent diffusion model, represents the current diffusion step number of the pre-trained conditional latent diffusion model, represents the hyperparameters of the pre-trained conditional latent diffusion model, represents the pre-trained conditional latent diffusion model, is the noise sampled from a standard normal distribution, represents the standard normal distribution, Express The latent vector obtained after the image dimensionality reduction in represents mathematical expectation;
[0017] Each client optimizes the category embedding vector received from the server using the optimized objective function of the inverse coding and after reaching a preset number of steps, sends the optimized category embedding vector to the server;
[0018] Step 20203, the server receives the category embedding vectors sent by each client and aggregates them. The expression is:
[0019]
[0020] in, Represents the first Task No. All category embedding vectors of classes, Indicates Task No. The first The category embedding vector of the class, Indicates the total number of clients;
[0021] If the preset number of rounds is reached, go to step 20205, otherwise, go to step 20204;
[0022] Step 2204: the server sends the aggregated category embedding vector to the client, and then goes to step 20202.
[0023] Step 20205, the server stores the aggregated category embedding vectors of each class and uses them for image generation;
[0024] The image generation stage; the server generates data of each category using the category embedding vector of the new task; if the new task is not the first task, the server also generates data of each category of the historical task using the category embedding vector obtained from the historical task; the server sends all the generated data to the client, including the following steps:
[0025] The server uses the aggregated category embedding vector as a condition and inputs it into the pre-trained conditional latent diffusion model to obtain the generated data for the new task. ,in, represents the generated image for the new task, The label of the generated image for the new task; the server takes the category embedding vector of the old task as a condition and inputs it into the pre-trained conditional latent diffusion model to obtain the generated data of the old task ,in, represents the generated image of the old task, The label of the generated image of the old task; the server and Sent to each client.
[0026] The image classification model training phase: each client uses its own real data and the generated data obtained from the server to jointly train the image classification model, including the following steps:
[0027] Step 20401: Each client receives the image classification model and the generated data of the new task sent by the server. And the generated data of the old task ;
[0028] Step 20402, each client calculates the new task classification loss function, which is expressed as:
[0029]
[0030] in, represents the new task classification loss function, represents the generated image for the new task, denotes the labels of the generated images for the new task, represents the real image of the new task, The labels of the real images representing the new task, for the Task No. Clients, yes The images in yes The tags in represents the sampled image, represents the sampled image label, represents the image classification model, Represents the probability vector obtained by the image classification model to predict the sampled image, Indicates the calculation of the cross entropy loss function;
[0031] Step 20403, each client calculates the supervised contrastive learning loss function, which is expressed as:
[0032]
[0033] in, is the supervised contrastive learning loss function, Represents an image classification model The feature extraction module Indicates the use For the sampled The image features obtained after feature extraction of images are Indicates the use For the sampled The image features obtained after feature extraction of images are Indicates The image feature set of samples of the same category, express The samples in Indicates the calculation of similarity. Indicates that a two-layer multi-layer perceptron is used to map the features. is a preset hyperparameter;
[0034] The feature extraction module of the image classification model is an image classification model The model obtained after removing the last fully connected layer;
[0035] If the new task is the first task, go to step 20406, otherwise, go to step 20404;
[0036] Step 20404, each client calculates the historical task classification loss function, which is expressed as:
[0037]
[0038] in, represents the historical task classification loss function, represents the generated image of the old task, Labels representing the generated images for the old task;
[0039] Step 20405, each client calculates the knowledge distillation loss function, which is expressed as:
[0040]
[0041] in, represents the knowledge distillation loss function, represents the generated image of the old task, Indicates the calculation of KL divergence, Indicates the image classification model obtained after the previous task training;
[0042] Step 20406, calculate the joint loss function; if the new task is the first task, the expression of the joint loss function is:
[0043]
[0044] in, represents the joint loss function, is a preset hyperparameter;
[0045] Otherwise, the expression of the joint loss function is:
[0046]
[0047] in, and is a preset hyperparameter;
[0048] Step 20407: Each client optimizes the image classification model using the joint loss function. If the preset number of optimization steps is reached, each client sends the optimized image classification model to the server.
[0049] In step 20408, the server receives the image classification models sent by each client, and averages the image classification models to achieve aggregation. If the preset number of rounds is reached, the training is terminated and the aggregated image classification model is output. Otherwise, the aggregated image classification model is continued to be sent to each client, and the process goes to step 20401 to continue optimization.
[0050] Compared with the existing method, the beneficial effects of the present invention are:
[0051] By using a pre-trained conditional diffusion model to reverse engineer each category, the corresponding input conditions of each category are searched in the model's input space, thereby significantly reducing computing resources and time consumption while ensuring effective generation. In addition, this method improves the domain generalization ability of the classifier for generated data and real data through contrastive learning, and indirectly improves the representation ability of generated data for real data. Compared with existing methods, this method significantly improves the performance of image classification models without significantly increasing training cost consumption. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 A schematic diagram of a flow chart of an embodiment of the present invention is shown.
[0053] Figure 2 A schematic diagram of the process flow of the category reverse encoding stage in an embodiment of the present invention is shown.
[0054] Figure 3 A schematic diagram of the image classification model training phase process in an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0055] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0056] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0057] In this example, we assume that we want to develop a medical image classification system. Due to the privacy of medical images, various institutions cannot share data. Due to the increasing types of medical images, we require the model to continuously learn new task knowledge while not forgetting the knowledge of old tasks. Therefore, the key technology of the medical image classification system lies in federated class continuous learning. We use the federated class continuous learning image classification method based on diffusion-driven data playback proposed in this paper to solve this key technology.
[0058] Therefore, a federated continuous learning image classification method based on diffusion driven data playback includes:
[0059] Multiple clients and one server for task sequence The tasks on the above are continuously learned, where Indicates tasks, , is the total number of tasks; the data set of each task is scattered among various clients, Indicates Task No. The client dataset is: , Represents the total number of clients; datasets for different tasks do not have overlapping image categories, and when learning a new task, the real data of the old task cannot be accessed.
[0060] like Figure 1 As shown, the learning of a new task includes the following steps:
[0061] Step 201, initialization phase; the server sends the pre-trained conditional latent diffusion model and image classification model to each client respectively; if the new task is the first task, the server will initialize the image classification model and send the initialized image classification model to each client; otherwise, the server sends the image classification model that has learned the previous task to each client;
[0062] Among them, the pre-trained conditional latent diffusion model is an auxiliary model required in the learning process and is used for data generation, while the image classification model is the target model to be learned in the federated continuous learning image classification method and is used to classify all learned data categories.
[0063] Step 202, category reverse encoding stage: multiple clients jointly reverse encode all categories of the new task, encode each category of the new task, and obtain a category embedding vector;
[0064] The combination refers to that a plurality of clients generate their own category embedding vectors respectively and send them to the server, and the server aggregates the received category embedding vectors and generates category data;
[0065] Step 203, image generation phase; the server generates data of each category using the category embedding vector; if the new task is not the first task, the server also generates data of each category of the historical task using the category embedding vector obtained from the historical task; finally, the server sends all generated data to the client;
[0066] It should be noted that, since the data generated by the server using the category embedding vector is different from the real data, it will not cause the leakage of data privacy, thereby playing a role in effectively protecting data privacy;
[0067] Step 204, image classification model training phase; each client uses its own real data and generated data obtained from the server to jointly train the image classification model.
[0068] Among them, the "joint training" here means that each client uses its own real data and the generated data received from the server to train its own image classification model, and then sends the trained image classification model to the server, which aggregates the image classification models. If the preset number of aggregations or training rounds is not reached, the image classification model continues to be jointly trained until the image classification model converges.
[0069] Specifically, the pre-trained conditional latent diffusion model is a generation model based on the diffusion process, which has important research value. It shows excellent performance in tasks such as image generation, and the generated samples are of high quality and strong diversity. Through conditional input, the pre-trained conditional latent diffusion model can generate samples that meet specific conditions, which is of great significance for application scenarios that require precise control of generation results, such as image editing and style transfer. The pre-trained conditional latent diffusion model not only reduces the computational complexity by diffusing in a low-dimensional latent space, but also maintains efficient generation capabilities, and is suitable for multimodal generation tasks, such as image generation of subtitles and text generation of images. The structure of the model includes an encoder, a conditional network, a diffusion network and a decoder. The encoder maps high-dimensional data to a low-dimensional latent space, improves computational efficiency and extracts latent representations; the conditional network processes conditional inputs and combines conditional inputs with latent representations to guide the diffusion process; the diffusion network gradually converts noise into samples that meet the conditions in the latent space to ensure the diversity and authenticity of the samples; the decoder decodes the diffused latent representations into high-dimensional data to generate the final samples.
[0070] Furthermore, the specific workflow of the pre-trained conditional latent diffusion model is to convert the input data into latent representation through the encoder, the conditional input is processed by the conditional network and combined with the latent representation, the diffusion network performs a diffusion process in the latent space to generate a new latent representation, and finally decodes it through the decoder to generate high-quality samples that meet the conditions. This model demonstrates superior performance in application scenarios that require high quality, diversity and controllable generation.
[0071] Specifically, Figure 2 As shown, the category reverse encoding stage in step 202; multiple clients jointly reverse encode all categories of the new task, encode each category of the new task, and obtain a category embedding vector, including the following steps:
[0072] Step 2201: the server randomly initializes all categories of the new task into a category embedding vector, and then sends the category embedding vector to each client;
[0073] Step 2202, each client performs reverse encoding on each new task category data of the local data set, and inputs the encoding and the category embedding vector received from the server into the pre-trained conditional latent diffusion model, and finally searches and optimizes the conditional input space of the pre-trained conditional latent diffusion model to obtain a new category embedding vector;
[0074] Wherein, the conditional input space represents the space formed by all possible conditional inputs of the pre-trained conditional latent diffusion model;
[0075] Furthermore, the inverse coding and optimized expression is:
[0076]
[0077] in, represents the new category embedding vector, Indicates Task No. The first The category embedding vector of the class, Indicates Task No. A client dataset, No. Task No. The first All data of the class, represents the image dimension reduction encoder of the pre-trained conditional latent diffusion model, is a preset text. represents the text encoder of the pre-trained conditional latent diffusion model, represents the current diffusion step number of the pre-trained conditional latent diffusion model, Representation and The hyperparameters of the pre-trained conditional latent diffusion model, represents the pre-trained conditional latent diffusion model, is the noise sampled from a standard normal distribution, represents the standard normal distribution, Express The latent vector obtained after the image dimensionality reduction in represents mathematical expectation;
[0078] Furthermore, after each client optimizes the category embedding vector using the optimized objective function of the inverse encoding and reaches a preset number of steps or times, the optimized category embedding vector is sent to the server, and the server aggregates the optimized category embedding vectors;
[0079] Specifically, the optimization method is diverse, and the optimization method can use SGD, Adam, and AdamW optimization algorithms. Taking the Adam optimization algorithm as an example, the Adam (Adaptive Moment Estimation) optimization algorithm is an optimization algorithm for training deep learning models. It combines the advantages of momentum and RMSProp algorithms, can achieve adaptive learning rate adjustment, and improve convergence speed and stability. The core idea of Adam is to calculate the first-order moment estimation and second-order moment estimation of the gradient, and use these estimates to update the parameters.
[0080] The expression of Adam optimization algorithm is as follows:
[0081] 1. Initialization parameters:
[0082] - : Initial parameters
[0083] - and : Initialize first-order and second-order moment estimates
[0084] - and : Attenuation coefficient, usually 0.9 and 0.999
[0085] - : A small constant to prevent division by zero, usually valued at
[0086] - : Time step
[0087] 2. At each time step Do the following:
[0088] -
[0089] - Calculate the current gradient
[0090] - Update the first moment estimate:
[0091]
[0092] - Update the second moment estimate:
[0093]
[0094] - Compute the bias correction for the first moment estimate:
[0095]
[0096] - Compute the bias correction for the second moment estimate:
[0097]
[0098] - Update parameters:
[0099]
[0100] in, is the learning rate.
[0101] The Adam optimization algorithm corrects the deviation of the first-order and second-order moment estimates of the gradient, so that the learning rate can be dynamically adjusted during the training process, thereby improving the efficiency and stability of model training.
[0102] Step 20203: The server receives the category embedding vectors sent by each client and aggregates them. The expression is:
[0103]
[0104] in, Represents the first Task No. The category embedding vector of the class, Indicates Task No. The first The category embedding vector of the class, Indicates the total number of clients;
[0105] If the preset number of rounds is reached, go to step 20205, otherwise, go to step 20204;
[0106] Step 20204: the server sends the aggregated category embedding vector to the client, and then goes to step 20202.
[0107] Step 20205, the server stores the aggregated category embedding vectors of each class and uses them for image generation;
[0108] Specifically, the image generation stage; the server generates data of each category using the category embedding vector; if the new task is not the first task, the server also generates data of each category of the historical task using the category embedding vector obtained from the historical task; finally, the server sends all the generated data to the client, which includes the following steps:
[0109] The server takes the category embedding vector of the new task as a condition and inputs it into the pre-trained conditional latent diffusion model to obtain the generated data of the new task. ,in, represents the generated image for the new task, The server uses the category embedding vector of the old task as a condition and inputs it into the pre-trained conditional latent diffusion model to obtain the generated data of the old task. ,in, represents the generated image of the old task, The label of the generated image of the old task; the server and Send to each client;
[0110] Specifically, Figure 3 As shown, the image classification model training phase; each client uses its own real data and the generated data obtained from the server to jointly train the image classification model, including the following steps:
[0111] Step 20401: Each client receives the image classification model and the generated data of the new task sent by the server. And the generated data of the old task ;
[0112] Step 20402, each client calculates the new task classification loss function, which is expressed as:
[0113]
[0114] in, represents the new task classification loss function, represents the generated image for the new task, denotes the labels of the generated images for the new task, represents the real image of the new task, The labels of the real images representing the new task, for the Task No. Clients, yes The images in yes The tags in represents the sampled image, represents the sampled image label, represents the image classification model, Represents the probability vector obtained by the image classification model to predict the sampled image, Indicates the calculation of the cross entropy loss function;
[0115] Step 20403, each client calculates the supervised contrastive learning loss function, which is expressed as:
[0116]
[0117] in, is the supervised contrastive learning loss function, Represents an image classification model The feature extraction module Indicates the use For the sampled The image features obtained after feature extraction of images are Indicates the use For the sampled The image features obtained after feature extraction of images are Indicates The image feature set of samples of the same category, express The samples in Indicates the calculation of similarity. Indicates that a two-layer multi-layer perceptron is used to map the features. is a preset hyperparameter, represents transpose;
[0118] Among them, the feature extraction module of the image classification model is the image classification model The model obtained after removing the last fully connected layer;
[0119] If the new task is the first task, go to step 20406, otherwise, go to step 20404;
[0120] Step 20404, each client calculates the historical task classification loss function, which is expressed as:
[0121]
[0122] in, represents the historical task classification loss function, represents the generated image of the old task, Labels representing the generated images for the old task;
[0123] Step 20405, each client calculates the knowledge distillation loss function, which is expressed as:
[0124]
[0125] in, represents the knowledge distillation loss function, represents the generated image of the old task, Indicates the calculation of KL divergence, Indicates the image classification model obtained after the previous task training is completed;
[0126] Step 20406, calculate the joint loss function; if the new task is the first task, the expression of the joint loss function is:
[0127]
[0128] in, represents the joint loss function, is a preset hyperparameter;
[0129] Otherwise, the expression of the joint loss function is:
[0130]
[0131] in, and is a preset hyperparameter;
[0132] Step 20407: Each client optimizes the image classification model using a joint loss function. If a preset number of optimization steps is reached, each client sends the optimized image classification model to the server.
[0133] In step 20408, the server receives the image classification models sent by each client, and averages the image classification models to achieve aggregation. If the preset number of aggregation times or rounds is reached, the training is terminated and the aggregated image classification model is output. Otherwise, the aggregated image classification model continues to be sent to each client, and the process goes to step 20401 to continue optimization.
[0134] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
Claims
1. A federated continuous learning image classification method based on diffusion-driven data playback, characterized in that: The method comprises: a plurality of clients and a server for task sequence The tasks on the above are continuously learned, where Indicates tasks, , is the total number of tasks; the data set of each task is scattered in various clients; the data sets of different tasks do not have overlapping image categories, and when learning a new task, the real data of the old task cannot be accessed, and the server includes a pre-trained conditional latent diffusion model and an image classification model; The learning of a new task comprises the following steps: Step 201, initialization phase; the server sends the pre-trained conditional latent diffusion model and the image classification model to each client respectively; if the new task is the first task, the server will initialize the image classification model and send the initialized image classification model to each client, otherwise, the server sends the image classification model that has learned the previous task to each client; Step 202, category reverse encoding stage: multiple clients jointly perform reverse encoding on all categories of the new task, encode each category of the new task and obtain a category embedding vector; Step 203, image generation phase; the server generates data of each category using the category embedding vector; if the new task is not the first task, the server also generates data of each category of the historical task using the category embedding vector obtained from the historical task; finally, the server sends all generated data to the client; Step 204, image classification model training phase; each client uses its own real data and the generated data obtained from the server to jointly train the image classification model.
2. The method for federated continuous learning image classification based on diffusion-driven data playback according to claim 1 is characterized in that: The category reverse coding stage includes the following steps: Step 2201: the server randomly initializes all categories of the new task into a category embedding vector, and then sends the category embedding vector to each client; In step 2202, each client performs reverse encoding on each new task category data of the local data set, and inputs the encoding and the category embedding vector received from the server into the pre-trained conditional latent diffusion model, and finally searches and optimizes the conditional input space of the pre-trained conditional latent diffusion model to obtain a new category embedding vector; the reverse encoding and optimization expression is: ; in, represents the new category embedding vector, Indicates Task No. The first The category embedding vector of the class, Indicates Task No. A client dataset, Indicates Task No. The first All data of the class, represents the image dimension reduction encoder of the pre-trained conditional latent diffusion model, is a preset text. represents the text encoder of the pre-trained conditional latent diffusion model, Represents the current diffusion step number of the pre-trained conditional diffusion model, Representation and The hyperparameters of the pre-trained conditional latent diffusion model, represents the pre-trained conditional latent diffusion model, is the noise sampled from a standard normal distribution, represents the standard normal distribution, Express The latent vector obtained after the image dimensionality reduction in represents mathematical expectation; Each client uses the optimized objective function of the inverse encoding to optimize the category embedding vector received from the server and after reaching a preset number of steps, sends the optimized category embedding vector to the server; Step 20203: The server receives the category embedding vectors sent by each client and aggregates them. The expression is: ; in, Represents the first Task No. All the category embedding vectors of the classes, Indicates Task No. The first The category embedding vector of the class, Indicates the total number of clients; If the preset number of rounds is reached, go to step 20205, otherwise, go to step 20204; Step 20204: the server sends the aggregated category embedding vector to the client, and then goes to step 20202. In step 20205, the server stores the category embedding vectors of each aggregated class and uses them for image generation.
3. The method for federated continuous learning image classification based on diffusion-driven data playback according to claim 2 is characterized in that: The image generation stage comprises the following steps: The server uses the aggregated category embedding vector as a condition and inputs it into the pre-trained conditional latent diffusion model to obtain the generated data for the new task. ,in, represents the generated image for the new task, The server uses the category embedding vector of the old task as a condition and inputs it into the pre-trained conditional latent diffusion model to obtain the generated data of the old task. ,in, represents the generated image of the old task, The label of the generated image of the old task; the server and Sent to each client.
4. The method for federated continuous learning image classification based on diffusion-driven data playback according to claim 3 is characterized in that: The image classification model training phase includes the following steps: Step 20401: Each client receives the image classification model and the generated data of the new task sent by the server. And the generated data of the old task ; Step 20402, each client calculates the new task classification loss function, which is expressed as: ; in, represents the new task classification loss function, represents the generated image for the new task, denotes the labels of the generated images for the new task, represents the real image of the new task, The labels of the real images representing the new task; Task No. Clients, yes The images in yes The tags in represents the sampled image, represents the sampled image label, represents the image classification model, Represents the probability vector obtained by the image classification model to predict the sampled image, Indicates the calculation of the cross entropy loss function; Step 20403, each client calculates the supervised contrastive learning loss function, which is expressed as: ; in, is the supervised contrastive learning loss function, Represents an image classification model The feature extraction module Indicates the use For the sampled The image features obtained after feature extraction of images are Indicates the use For the sampled The image features obtained after feature extraction of images are Indicates The image feature set of samples of the same category, express The samples in Indicates the calculation of similarity. Indicates that a two-layer multi-layer perceptron is used to map the features. is a preset hyperparameter, represents transpose; If the new task is the first task, go to step 20406, otherwise, go to step 20404; Step 20404, each client calculates the historical task classification loss function, which is expressed as: ; in, represents the historical task classification loss function, represents the generated image of the old task, Labels representing the generated images for the old task; Step 20405, each client calculates the knowledge distillation loss function, which is expressed as: ; in, represents the knowledge distillation loss function, represents the generated image of the old task, Indicates the calculation of KL divergence, Indicates the image classification model obtained after the previous task training is completed; Step 20406, calculate the joint loss function; if the new task is the first task, the expression of the joint loss function is: ; in, represents the joint loss function, is a preset hyperparameter; Otherwise, the expression of the joint loss function is: ; in, and is a preset hyperparameter; Step 20407: Each client optimizes the image classification model using a joint loss function. If a preset number of optimization steps is reached, each client sends the optimized image classification model to the server. In step 20408, the server receives the image classification models sent by each client, and averages the image classification models to achieve aggregation. If the preset number of aggregation rounds is reached, the training is terminated and the aggregated image classification model is output. Otherwise, the aggregated image classification model is continued to be sent to each client, and the process goes to step 20401 to continue optimization.
Citation Information
Patent Citations
Continuous learning method for federated learning
CN113657607A
Image classification method based on federal knowledge distillation and ensemble learning
CN117523291A
