Training method of federated continual learning image classification model based on frequency domain prototype

Abstract

This invention proposes a training method for a federated continuous learning image classification model based on frequency domain prototypes. The implementation steps are as follows: constructing a federated continuous learning scenario; initializing federated continuous learning parameters; each client iteratively training its local model; and a central server performing federated aggregation of model parameters uploaded by all clients. In the process of each client generating mixed pseudo-samples based on the frequency domain prototype, the computations required for calculating the frequency domain prototype and generating the mixed pseudo-samples are linear transformations, avoiding the high computational complexity of existing technologies due to the use of diffusion model inference, and effectively reducing the computational overhead of the client. In the process of generating mixed pseudo-samples, this invention generates mixed pseudo-samples using the phase spectrum of the original image and the mixed amplitude spectrum based on the frequency domain prototype. Since the phase spectrum provides structural constraints mathematically, it avoids the randomness of diffusion model generation, improving the stability of the classification accuracy of the trained model.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and distributed machine learning technology, and specifically relates to a training method for a federated continuous learning image classification model based on frequency domain prototypes, which can be used in multi-center dynamic data scenarios such as security monitoring, autonomous driving and medical imaging. Background Technology

[0002] Federated learning, as a privacy-preserving distributed machine learning paradigm, allows a massive number of clients to collaboratively train a global model without sharing the original data. However, in real-world applications, the distribution of client data often evolves dynamically over time, with a series of local tasks arriving over time; this scenario is known as federated continuous learning. Image classification is the most common and representative application of federated continuous learning. Image classification models in federated continuous learning need to continuously utilize data from newly added categories for multiple rounds of collaborative training. During this process, the model parameters are constantly updated, which can easily lead to the gradual degradation of learned old categories or historical discriminative knowledge, resulting in the catastrophic forgetting problem of image classification models.

[0003] To address the catastrophic forgetting problem in image classification models, Jinglin Liang et al., in their 2024 paper "Diffusion-Driven DataReplay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning" published at the European Conference on Computer Vision, proposed a federated continuous learning image classification model training method based on a diffusion model. This method requires the client to use a frozen pre-trained diffusion model to continuously backpropagate images to obtain class embedding vectors representing old classes, and then use synthetic data generated from these class embedding vectors for replay. Contrastive learning is also employed to align the feature distributions of the synthetic data with those of the real data. However, this method introduces diffusion model inference into the client during the acquisition of class embedding vectors, significantly increasing the client's computational overhead. Furthermore, due to the probabilistic randomness in the diffusion model generation process, it is prone to generating illogical synthetic data, interfering with model training and affecting the stability of the trained model's classification accuracy. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and propose a training method for a federated continuous learning image classification model based on frequency domain prototypes, which is used to solve the technical problems of low training efficiency and low classification accuracy stability in the existing technology.

[0005] To achieve the above objectives, the technical solution adopted by the present invention includes the following steps:

[0006] (1) Constructing a federated continuous learning scenario:

[0007] The construction includes a central server and The number of continuous learning tasks per client is In a federated continuous learning scenario, the central server sends the image classification network as a global model to each client; each client builds a system including... Target categories Cropped natural images and category labels and the local training sample set for each learning task based on the hybrid pseudo-samples generated from the frequency domain prototype;

[0008] (2) Initialize federated continuous learning parameters:

[0009] The initial communication rounds between the central server and each client for continuous learning are as follows: The maximum number of communication rounds is and order ;

[0010] (3) Each client iteratively trains the local model:

[0011] Each client uses the global model sent by the central server as its local model and performs training on each learning task. The local training sample set is used to iteratively train the model, and then the trained model parameters are uploaded to the central server.

[0012] (4) The central server performs federated aggregation of model parameters uploaded by all clients:

[0013] Central server The model parameters uploaded by each client are federated and aggregated, and then a judgment is made. Is it true? If so, obtain the first... Learning tasks The parameters are The image classification model is used until all learning tasks are completed, resulting in parameters as follows: The image classification model, otherwise let And perform step (3).

[0014] Compared with the prior art, the present invention has the following advantages:

[0015] 1. In the process of generating hybrid pseudo-samples based on the frequency domain prototype by each client in this invention, the Fourier transform required to calculate the frequency domain prototype and the inverse Fourier transform required to generate the hybrid pseudo-samples are both linear transformations. This avoids the high computational complexity caused by the use of diffusion model inference to obtain class embedding vectors in the prior art, and effectively reduces the computational overhead of the client.

[0016] 2. In the process of generating mixed pseudo-samples based on the frequency domain prototype for each client in this invention, mixed pseudo-samples are generated by using the phase spectrum of the original image and the mixed amplitude spectrum based on the frequency domain prototype. Since the phase spectrum carries the structural information of the image, it provides a hard structural constraint in mathematics, ensuring that the generated pseudo-samples always maintain a high degree of consistency with the real samples in terms of visual structure. This avoids the problem of probabilistic randomness in the generation process of the diffusion model and improves the stability of the classification accuracy of the training model. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the implementation of the present invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0019] Reference Figure 1 The present invention includes the following steps:

[0020] Step 1) Construct a federated continuous learning scenario:

[0021] The construction includes a central server and The number of continuous learning tasks per client is In the context of federated continuous learning, and in the learning of the first When performing a task, previous access is unavailable. The raw training data for each task. The central server sends the image classification network as a global model to each client.

[0022] In this embodiment, the widely recognized CIFAR-100 dataset in the field of computer vision is used. This dataset contains 60,000 color natural images in 100 categories. Each image has a physical resolution of 32×32 pixels and 3 channels. The images in these 100 categories are divided as follows: each task contains images of 10 unique categories, and the remaining 50 categories are randomly assigned to 5 sequence tasks to simulate a typical class incremental continuous learning scenario.

[0023] Non-independent and identically distributed data partitioning was employed: a Dirichlet distribution (α=0.5) was used to assign data for each task to different clients to simulate real-world data heterogeneity. Each client's data consisted of... Target categories Cropped natural images and category labels and the local training sample set for each learning task based on the hybrid pseudo-samples generated from the frequency domain prototype;

[0024] The global model adopts the standard ResNet-18 network structure, including cascaded feature extraction and classification modules, wherein:

[0025] The feature extraction module includes eight cascaded residual blocks, which are divided into four groups (two blocks per group). The number of output channels in each group are 64, 128, 256, and 512, respectively. Each residual block includes two stacked convolutional layers that perform nonlinear transformations on the input feature map through convolution operations, two normalization layers, and two activation function layers. The input and output of each residual block are connected in a skip connection.

[0026] The classification module consists of a stacked global average pooling layer and a fully connected layer.

[0027] The method for obtaining the hybrid pseudo-samples is as follows:

[0028] (1a) Section Each client processes each cropped natural image. Perform a Fast Fourier Transform (FFT) and analyze the amplitude spectrum obtained from the FFT. Radial integral compression yields a one-dimensional radial vector. .

[0029] The formula for calculating the Fast Fourier Transform is as follows:

[0030] ;

[0031] in, Represents absolute value. The frequency coordinates are respectively , Real images Frequency domain representation;

[0032] radial vector The calculation formula is:

[0033] ;

[0034] ;

[0035] ;

[0036] in, For the first One element, , amplitude spectrum The center coordinates, Indicating amplitude spectrum to The Euclidean distance, after rounding, equals The set of coordinates, Represents a set The number of pixels in Represents the frequency domain coordinates after rounding down. arrive The Euclidean distance.

[0037] Then according to each natural image The categories are used to classify and group the radial vectors obtained from compression, resulting in... The reason for classifying and grouping radial vectors of each category is that different categories may have drastically different style features in the frequency domain. By constructing a frequency domain prototype of class conditions, we can capture the unique style distribution of each category more precisely.

[0038] Calculate the next Learning tasks Each category The mean of the set of radial vectors With variance Forming a frequency domain prototype ,in Indicates the first The set of categories contained in the local training sample set of each client. hour It is an empty value because when At that time, the concept of old tasks did not exist.

[0039] mean and variance The calculation formulas are as follows:

[0040] ;

[0041] ;

[0042] in, For the first Learning tasks Category The total number of natural images, For the first in this category The radial vector of a cropped natural image.

[0043] (1b) A client based on Calculate the first Learning tasks Amplitude spectrum and through by and The mixed amplitude spectrum and each natural image phase spectrum Dynamic calculation Hybrid pseudo-samples :

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] ;

[0049] in, Indicates by , The one-dimensional vector obtained by sampling from the constructed Gaussian distribution, Indicates amplitude spectrum The center coordinates, This indicates rounding down. For Using a preset threshold as the center A binary matrix representing the radius, where the value is 1 if the frequency domain coordinates fall within this radius, and 0 otherwise. This represents the argument operation for complex numbers. This represents the inverse fast Fourier transform.

[0050] Amplitude spectrum This describes the energy distribution of an image at different frequencies. For natural images, low-frequency components typically correspond to overall illumination, hue, and contrast, while high-frequency components correspond to texture details. These energy distributions change most significantly when the environment of a federated continuous learning task changes (e.g., from "day" to "night"). Therefore, by storing frequency domain prototypes of older tasks... This invention actually preserves knowledge of the old task environment.

[0051] The process of generating hybrid pseudo-samples on the client side can be completed through linear transformation alone, which greatly reduces the computational overhead. In addition, the hybrid pseudo-samples in this embodiment are dynamically generated with the input of natural images during iterative training, and they are deleted immediately after use, avoiding the shortcomings of existing technologies that occupy local storage space on the client side.

[0052] The client utilizes the mixed amplitude spectrum and each natural image phase spectrum Dynamic calculation Hybrid pseudo-samples , A defined spatial structure constraint is imposed on the synthesis of pseudo-samples, ensuring that pseudo-samples can produce output responses that are highly consistent with real samples in the local model feature space.

[0053] Step 2) Initialize federated continuous learning parameters:

[0054] The initial communication rounds between the central server and each client for continuous learning are as follows: The maximum number of communication rounds is and order ;

[0055] Step 3) Each client iteratively trains the local model:

[0056] Each client uses the global model sent by the central server as its local model and performs training on each learning task. The local training sample set is used to iteratively train the model, and then the trained model parameters are uploaded to the central server.

[0057] Each client iteratively trains the local model, which involves the following steps:

[0058] (3a) When During initialization For the local model in the current learning task Weight parameters before training begins; initialization of local iteration count. The maximum number of local iterations is , No. The weight parameters of the local model in the next iteration are and order ;

[0059] (3b) The feature extraction module extracts features from each natural image and the mixed pseudo-samples, and the classification module performs classification prediction on the extracted natural image features and the mixed pseudo-samples to obtain the classification prediction value for each natural image. And the classification prediction value of each mixed pseudo-sample. ;

[0060] The feature extraction module extracts features from each natural image and each mixed pseudo-sample separately. The steps are as follows:

[0061] The feature extraction module consists of eight residual blocks that extract features layer by layer from each natural image and the mixed pseudo-sample. The input for each layer is the element-wise sum of the input and output of the previous residual block. The first two residual blocks output a 64-channel feature map, the next two output a 128-channel feature map, the following two output a 256-channel feature map, and the last two output a 512-channel feature map. Each residual block in the feature extraction module contains two convolutional layers that extract features from the input feature map through convolution operations. A normalization layer and an activation function layer perform non-linear transformations on the features. The input of the residual block is added to the non-linearly transformed features to obtain the feature map for each natural image and the mixed pseudo-sample.

[0062] The classification prediction process involves classifying the extracted feature maps. First, the spatial dimension of the final feature map is reduced to 1×1 by downsampling through the global average pooling layer of the classification module, generating a column vector with a dimension of 512. Then, it is input into the fully connected layer, which outputs a one-dimensional vector with a length of 100 to obtain the prediction probability of each category. The one with the highest probability is the final classification result for each natural image and the mixed pseudo-sample.

[0063] (3c) Employ the cross-entropy loss function to predict the classification value of each natural image. With category labels Calculate the cross-entropy loss value of the current local model. ;like Then through For weight parameters Perform gradient descent updates; if Simultaneously, a consistency loss function is employed, using the classification prediction value of each mixed pseudo-sample. and the learning task of storage Local model weight parameters before training begins Classification prediction values Calculate the consistency loss value of the current local model. ; Based on the calculation of mixed pseudo-samples, the current local model is constrained. When faced with mixed pseudo-samples, its output must be consistent with the old model. Maintaining consistency allows the current local model to review the classification prediction boundaries of the old model using mixed pseudo-samples without touching any original images from the old task.

[0064] Then, the stochastic gradient descent method is used, through... and The calculated loss value of the current local model For weight parameters Update the model to obtain the local model parameters for this local iteration. ;

[0065] The loss value The calculation formula is:

[0066] ;

[0067] ;

[0068] ;

[0069] in, Represents the balance coefficient. This represents the Euclidean norm.

[0070] The weight parameters The update is performed using the following formula:

[0071] ;

[0072] in, It's the learning rate. For gradient operators, yes about The gradient.

[0073] (3d) judgment Is it true? If so, obtain the first... Communication No. Weight parameters of a local model trained on a client Otherwise, let Then proceed with step (3b).

[0074] Step 4) The central server performs federated aggregation of model parameters uploaded by all clients:

[0075] Central server The model parameters uploaded by each client are federated and aggregated, and then a judgment is made. Is it true? If so, obtain the first... Learning tasks The parameters are The image classification model is used until all learning tasks are completed, resulting in parameters as follows: The image classification model, otherwise let And perform step (3).

[0076] The aforementioned federated aggregation has the following aggregation formula:

[0077] .

Claims

1. A training method for a federated continuous learning image classification model based on frequency domain prototypes, characterized in that, Includes the following steps: (1) Constructing a federated continuous learning scenario: The construction includes a central server and The number of continuous learning tasks per client is In a federated continuous learning scenario, the central server sends the image classification network as a global model to each client; each client builds a system including... Target categories Cropped natural images and category labels and the local training sample set for each learning task based on the hybrid pseudo-samples generated from the frequency domain prototype; (2) Initialize federated continuous learning parameters: The initial communication rounds between the central server and each client for continuous learning are as follows: The maximum number of communication rounds is and order ; (3) Each client iteratively trains the local model: Each client uses the global model sent by the central server as its local model and performs training on each learning task. The local training sample set is used to iteratively train the model, and then the trained model parameters are uploaded to the central server. (4) The central server performs federated aggregation of model parameters uploaded by all clients: Central server The model parameters uploaded by each client are federated and aggregated, and then a judgment is made. Is it true? If so, obtain the first... Learning tasks The parameters are The image classification model is used until all learning tasks are completed, resulting in parameters as follows: The image classification model, otherwise let And perform step (3).

2. The method according to claim 1, characterized in that, The global model described in step (1) includes a cascaded feature extraction module and a classification module, wherein: The feature extraction module includes multiple cascaded residual blocks. Each residual block includes at least two stacked convolutional layers, a normalization layer, and an activation function layer, and the input and output of each residual block are connected in a skip connection. The classification module includes stacked global average pooling layers and fully connected layers.

3. The method according to claim 2, characterized in that, The method for obtaining the mixed pseudo-samples mentioned in step (1) is as follows: (1a) Section Each client processes each cropped natural image. Perform a Fast Fourier Transform (FFT) and analyze the amplitude spectrum obtained from the FFT. Radial integral compression is performed, then applied to each natural image. The categories are used to classify and group the radial vectors obtained from compression, resulting in... The set of radial vectors for the nth category, then calculate the nth... Learning tasks Each category The mean of the set of radial vectors With variance Forming a frequency domain prototype ,in Indicates the first The set of categories contained in the local training sample set of each client. hour It is a null value; (1b) A client based on Calculate the first Learning tasks Amplitude spectrum and through by and The mixed amplitude spectrum and each natural image phase spectrum calculate Hybrid pseudo-samples .

4. The method according to claim 3, characterized in that, The mean mentioned in step (1a) and variance The calculation formulas are as follows: ； ； in, For the first Learning tasks Category The total number of natural images, For the first in this category The radial vector of a cropped natural image.

5. The method according to claim 3, characterized in that, The mixed pseudo-samples described in step (1b) The calculation formula is: (1b) A client based on Calculate the first Learning tasks Amplitude spectrum and through by and The mixed amplitude spectrum and each natural image phase spectrum calculate Hybrid pseudo-samples : ； ； ； ； ； in, Indicates by , The one-dimensional vector obtained by sampling from the constructed Gaussian distribution, Indicates amplitude spectrum The center coordinates, This indicates rounding down. For Using a preset threshold as the center A binary matrix with radius, This represents the argument operation for complex numbers. This is the inverse fast Fourier transform.

6. The method according to claim 4, characterized in that, The steps for each client to iteratively train the local model as described in step (3) are as follows: (3a) When During initialization For the local model in the first Weight parameters for each task before training begins; initialization of the local iteration count. The maximum number of local iterations is , No. The weight parameters of the local model in the next iteration are and order ; (3b) The feature extraction module extracts features from each natural image and the mixed pseudo-samples, and the classification module performs classification prediction on the extracted natural image features and the mixed pseudo-samples to obtain the classification prediction value for each natural image. And the classification prediction value of each mixed pseudo-sample. ; (3c) Employ the cross-entropy loss function to predict the classification value of each natural image. With category labels Calculate the cross-entropy loss value of the current local model. ;like Simultaneously, a consistency loss function is employed, using the classification prediction value of each mixed pseudo-sample. and the Local model weight parameters before the start of training for each learning task Classification prediction values Calculate the consistency loss value of the current local model. Then, the stochastic gradient descent method is used to... and The calculated loss value of the current local model For weight parameters Update the model to obtain the local model parameters for this local iteration. ; (3d) judgment Is it true? If so, obtain the first... Communication No. Weight parameters of a local model trained on a client Otherwise, let Then proceed with step (3b).

7. The method according to claim 5, characterized in that, The feature extraction module described in step (3b) extracts features from each natural image and the mixed pseudo-sample separately. The steps are as follows: The feature extraction module uses multiple residual blocks to extract features from each natural image and mixed pseudo-sample layer by layer. The parameters used for each layer extraction are the result of element-wise summation of the input and output of the previous residual block.

8. The method according to claim 5, characterized in that, The loss value described in step (3c) The calculation formula is: ； ； ； in, Represents the balance coefficient. This represents the Euclidean norm.

9. The method according to claim 5, characterized in that, The weighting parameters described in step (3c) The update is performed using the following formula: ； in, It's the learning rate. For gradient operators, yes about The gradient.

10. The method according to claim 1, characterized in that, The federated aggregation described in step (4) has the following aggregation formula: 。

Images