Federated Learning Method for Long-Tail Image Classification Combining Rotation Self-Supervised and CLIP-Guided Approach

Abstract

This invention relates to a federated learning method for long-tail image classification that combines rotation self-supervised learning and CLIP guidance, belonging to the field of long-tail federated learning. First, this invention uses rotation self-supervised learning to guide the training of the local client, reducing the impact of long-tail data on the model's feature extractor and enabling the model to better learn image features. Second, this invention utilizes pre-trained CLIP to guide the rotated image, transferring the rich knowledge from CLIP to the client model, further improving the performance of the model's feature extractor. Finally, the server generates a set of class-balanced federated features guided by CLIP, which are used to retrain the classification layer of the aggregated model to improve the accuracy of the global model. This invention, by utilizing rotation self-supervised learning and CLIP to guide model training, effectively improves the ability of the client model's feature extractor to extract features from images, mitigates the impact of long-tail data, and improves the recognition accuracy of the final global model.

Description

Technical Field

[0001] This invention discloses a federated learning method for long-tail image classification that combines rotational self-supervision and CLIP guidance, belonging to the field of long-tail federated learning. Background Technology

[0002] Federated learning is a distributed machine learning method that allows multiple clients to collaboratively train a high-performance shared model without transferring their private datasets. However, in the real world, data distribution often exhibits a long-tail distribution, with the head class having a large amount of data and the tail class having only a small amount. Such data is detrimental to model training. When federated learning trains a model on long-tailed data, it can cause the model to favor the head class data while ignoring the tail class data, resulting in poor classification ability for the tail class data.

[0003] Currently, many methods have been proposed to address the problem of non-independent and identically distributed (IID) data in federated learning. Methods such as Scaffold (Scaffold: Stochastic controlled averaging for federated learning[C], International conference on machine learning.PMLR,2020:5132–5143.) and FedProx (Federated optimization in heterogeneous networks[J].Proceedings of Machine learning and systems,2020,2:429–450.) improve the client-side process, aligning the optimization direction of the client model with that of the global model. Methods such as CCVR (No fear of heterogeneity: Classifier calibration for federated learning with non-iID data[J].Advances in Neural Information Processing Systems,2021,34:5972–5984.) improve the performance of the global model by making improvements to the server. However, these methods are not suitable for cases where the global data has a long-tailed distribution. In such cases, the model ignores tail-class data during training, leading to a decrease in the accuracy of the global model. In centralized long-tail learning methods, data resampling and decoupling have proven effective. However, federated learning is a distributed machine learning approach, and centralized training methods are not suitable for federated learning scenarios. CLIP2FL (CLIP-Guided Federated Learning on Heterogeneity and Long-Tailed Data[C], Proceedings of the AAAI Conference on Artificial Intelligence.2024,38(13):14955–14963.) uses CLIP (Contrastive Language-Image Pre-training) to guide the generation of federated features and client training to obtain a high-precision global model. However, it only focuses on retraining the model classifier and does not balance the model's feature extractor. As a result, the model's feature extractor is still affected by long-tail data. At the same time, it does not give full play to the guiding role of CLIP.

[0004] In summary, addressing the model bias caused by the long-tail distribution of data in federated learning remains an important issue. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a method for long-tail federated learning image classification for global image data with a long-tail distribution, so as to alleviate the bias of federated learning models trained under long-tail distribution on the majority and minority class data in the long-tail data, and ultimately improve the accuracy of the global model.

[0006] The technical solution of the present invention is as follows: Figure 1 As shown, this paper presents a long-tail federated learning method for cases where global data has a long-tail distribution. On the client side, rotational self-supervised learning and CLIP are used to guide the training of the local client model. The gradient of the classification layer of the local data under the global model is obtained using the global model distributed by the server, which is then used by the server to generate federated features. On the server side, the server first aggregates the models uploaded by the clients into a global model on an average basis. Then, it generates federated features by using the classification layer gradients collected from the local clients and CLIP. Finally, the classification layer of the aggregated global model is retrained using the generated federated features.

[0007] The specific steps are as follows:

[0008] Step 1: The client uses the global model distributed by the server to calculate the gradient of the classification layer on the local dataset, and then uses the currently received global model as the current local client model. The client trains the current local client model using the local dataset and uses rotation self-supervision and CLIP to guide the training of the current local client model.

[0009] Step 2: The server aggregates the client models trained in Step 1 into the global model for the next round, and uses the classifier gradients collected from the client to generate federated features. During the generation of federated features, CLIP is used to guide the generation of federated features.

[0010] Step 3: The server copies the aggregated global model obtained in Step 2, retrains the classification layer of the copied global model using federated features, and finally distributes the aggregated global model obtained in Step 2 to the client.

[0011] Step 4: Repeat Step 1 to Step 3 until the maximum number of communications is reached to complete the training.

[0012] Specifically, Step 1 is as follows:

[0013] S1.1 For each client k and each category c, based on its local dataset Use the currently issued global model w t Calculate the classifier gradient for the c-th class.

[0014]

[0015] in, For the client in the t-th communication round, L ce Let cross-entropy be the loss function. For the d-dimensional true features generated for the c-th class of the k-th client, y i For sample labels;

[0016] S1.2 For client k, the currently received global model w t As a local model Its local dataset For each sample x in the dataset, rotate sample x by {0°, 90°, 180°, 270°} respectively, and the rotated labels are {1, 2, 3, 4}. The client additionally creates a four-classifier v, and then feeds the rotated samples and labels into the client's feature extractor f. k Then, the feature extractor f k The output is input into a four-class classifier v to construct a rotationally self-supervised learning model. Thus, the losses from supervised learning and rotationally self-supervised learning during model training are expressed as:

[0017] L sup+self =L ce sup (x, y) + ε·L ce self (x r y r );

[0018] Among them, L ce sup (x, y) represents the supervised training loss during normal training on the client side, where y is the label of sample x, and L... ce self Let x be the loss for rotational self-supervised learning, ε be a hyperparameter controlling the effect of rotational self-supervised learning, and x be the loss. r For the rotated sample, y r The labels corresponding to the rotated samples;

[0019] S1.3 For each client k in the training, set its local dataset... Each sample x and the rotated x r The samples are input into CLIP, and the logit vectors output by CLIP are obtained, represented as follows: and Using knowledge distillation on the local client model Provide guidance, and the loss is expressed as:

[0020]

[0021] in, and Sample x and rotated x output for the local client model r The dataset outputs a logit vector, where KL is the KL divergence, and β and δ are hyperparameters that control the degree of CLIP guidance.

[0022] S1.4 The total loss for training the local client model is:

[0023] L local =L sup+self +L KL total .

[0024] Specifically, Step 2 is as follows:

[0025] S2.1 The server receives the local model trained by the client and further aggregates the client model using the following formula to form the global model for the next round:

[0026]

[0027] Among them, D k Let w represent the local dataset of the k-th client. t+1 For the aggregated global model, φ t This refers to the set of clients selected by the server to participate in training in the previous round. Upload the trained local model to the client;

[0028] S2.2, The average true gradient collected by the server from the client. The true gradient average is calculated using the following formula:

[0029]

[0030] in, Let be the gradient of class c among the clients selected in round t;

[0031] S2.3 The server first randomly generates m federated features that conform to the dimensions of the real features. Then use a classifier. For randomly generated federated features Generate the corresponding class gradient And optimize randomly generated federated features using gradient matching loss function and CLIP-guided optimization. Make the generated federated features The formula is more consistent with the characteristics of real data and is as follows:

[0032]

[0033]

[0034]

[0035] L all =L g +η·L p ;

[0036] in, For the i-th randomly generated federated feature in the c-th class of the t-th round, L g For gradient matching loss function, Federal characteristics The corresponding class gradient value in the j-th row, L represents the value of the class gradient in the j-th row corresponding to the true average gradient. p To compare the loss functions, z c，i For the i-th randomly generated federated feature in class c, z j For federal features of category i, C is the total number of categories, and p C For randomly generated federated features The in-class prototype after multiplying with the semantic information in CLIP, where τ is temperature and η is the federated feature randomly generated under CLIP guidance. The generated hyperparameters, L all For the total loss, y i For sample labels.

[0037] Specifically, Step 3 is as follows:

[0038] S3.1 After copying the aggregated global model, retrain the classifier of the copied global model. First, freeze the parameters of the feature extractor of the copied global model, and then retrain the classifier γ of the copied global model. t Then, the aggregated global model is sent to the client;

[0039]

[0040] Where, γ t+1 For the trained and replicated global model classifier, For the federated features completed in round t, To retrain the learning rate of the model's classification layer, For γ t The gradient of L ce Let y be the cross-entropy loss function. i For sample labels.

[0041] The beneficial effects of this invention are:

[0042] 1. Existing long-tail federated learning methods lack feature extractor balancing, resulting in models trained on long-tailed data distributions that favor head classes while performing poorly on tail classes, making it difficult for the model to classify tail-class data. This invention proposes a federated learning method for long-tail image classification that combines rotational self-supervision and CLIP guidance. By using rotational self-supervision and CLIP guidance for model training, the imbalance of feature extractors during training is alleviated.

[0043] 2. To address the issue of long-tailed distribution in global data, this invention employs rotation-based self-supervised learning during client-side training to encourage the model to learn high-quality image features, thereby mitigating the impact of the long-tailed distribution. Simultaneously, it leverages the rich knowledge of CLIP to guide both normal training images and rotated images, transferring CLIP's knowledge to the local client-side model and further enhancing its performance. This method mitigates the impact of long-tailed data distribution on the model, thereby improving the accuracy of global model classification. Attached Figure Description

[0044] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0045] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0046] Example 1: As Figure 1 As shown, this implementation example provides 20 users K = {k: k = 1, 2, ...}. The server and client communicate a total of 200 times during the entire training process. Before each communication, the server randomly selects 8 clients as the clients for training the model. The client trains the model locally for 15 iterations, with each batch consisting of 32 iterations.

[0047] Step 1: The client uses the global model distributed by the server to calculate the gradient of the classification layer on the local dataset, and then uses the currently received global model as the current local client model. The client trains the current local client model using the local dataset and uses rotation self-supervision and CLIP to guide the training of the current local client model.

[0048] S1.1 For each client k and each category c, based on its local dataset Use the currently issued global model w t Calculate the classifier gradient for the c-th class.

[0049]

[0050] in, For the client in the t-th communication round, L ce Let cross-entropy be the loss function. For the d-dimensional true features generated for the c-th class of the k-th client, y i For sample labels;

[0051] S1.2 For client k, the currently received global model w t As a local model Its local dataset For each sample x in the dataset, rotate sample x by {0°, 90°, 180°, 270°} respectively, and the rotated labels are {1, 2, 3, 4}. The client additionally creates a four-classifier v, and then feeds the rotated samples and labels into the client's feature extractor f. k Then, the feature extractor f k The output is input into a four-class classifier v to construct a rotationally self-supervised learning model. Thus, the losses from supervised learning and rotationally self-supervised learning during model training are expressed as:

[0052] L sup+self =L ce sup (x, y) + ε·L ce self (x r y r );

[0053] Among them, L cesup (x, y) represents the supervised training loss during normal training on the client side, where y is the label of sample x, and L... ce self Let x be the loss for rotational self-supervised learning, ε be a hyperparameter controlling the effect of rotational self-supervised learning, and x be the loss. r For the rotated sample, y r The labels corresponding to the rotated samples;

[0054] For each client k during training, its local dataset is... Each sample x and the rotated x r The samples are input into CLIP, and the logit vectors output by CLIP are obtained, represented as follows: and Using knowledge distillation on the local client model Provide guidance, and the loss is expressed as:

[0055]

[0056] in, and Sample x and rotated x output for the local client model rThe dataset outputs a logit vector, where KL is the KL divergence, and β and δ are hyperparameters that control the degree of CLIP guidance.

[0057] S1.4 The total loss for training the local client model is:

[0058] L local =L sup+self +L KL total ;

[0059] Step 2: The server aggregates the client models trained in Step 1 into the global model for the next round, and uses the classifier gradients collected from the client to generate federated features. During the generation of federated features, CLIP is used to guide the generation of federated features.

[0060] S2.1 The server receives the local model trained by the client and further aggregates the client model using the following formula to form the global model for the next round:

[0061]

[0062] Among them, D k Let w represent the local dataset of the k-th client. t+1 For the aggregated global model, φ t This refers to the set of clients selected by the server to participate in training in the previous round. Upload the trained local model to the client;

[0063] S2.2, The average true gradient collected by the server from the client. The true gradient average is calculated using the following formula:

[0064]

[0065] in, Let be the gradient of class c among the clients selected in round t.

[0066] S2.3 The server first randomly generates m federated features that conform to the dimensions of the real features. Then use a classifier. For randomly generated federated features Generate the corresponding class gradient And optimize randomly generated federated features using gradient matching loss function and CLIP-guided optimization. Make the generated federated features The formula is more consistent with the characteristics of real data and is as follows:

[0067]

[0068]

[0069]

[0070] L all =L g +η·L p ;

[0071] in, For the i-th randomly generated federated feature in the c-th class of the t-th round, L g For gradient matching loss function, Federal characteristics The corresponding class gradient value in the j-th row, L represents the value of the class gradient in the j-th row corresponding to the true average gradient. p To compare the loss functions, z c，i For the i-th randomly generated federated feature in class c, z j For federal features of category i, C is the total number of categories, and p C For randomly generated federated features The in-class prototype after multiplying with the semantic information in CLIP, where τ is temperature and η is the federated feature randomly generated under CLIP guidance. The generated hyperparameters, L all For the total loss, y i For sample labels.

[0072] Step 3: The server copies the aggregated global model obtained in Step 2, retrains the classification layer of the copied global model using federated features, and finally distributes the aggregated global model obtained in Step 2 to the client.

[0073] S3.1 After copying the aggregated global model, retrain the classifier of the copied global model. First, freeze the parameters of the feature extractor of the copied global model, and then retrain the classifier γ of the copied global model. t Then, the aggregated global model is sent to the client;

[0074]

[0075] Where, γ t+1 For the trained and replicated global model classifier, Let θ represent the federated features optimized in round t, and θ be the learning rate of the retrained classification layer of the model. For γ t The gradient of L ce Let y be the cross-entropy loss function. i For sample labels.

[0076] Step 4: Repeat Step 1 to Step 3 until 200 communications are reached to complete the training.

[0077] Finally, the performance of this invention was evaluated through experimental comparison. The Python version used in the experiment was 3.10.12, the PyTorch deep learning framework version was 2.0.1, and the hardware environment used in the experiment was an NVIDIA GeForce RTX 4090 graphics card.

[0078] This experiment uses the CIFAR-10-LT and CIFAR-100-LT datasets, which are constructed based on CIFAR-10 / CIFAR-100 respectively. The long-tail distribution and heterogeneity of the data are simulated by adjusting the imbalance factor IF and the Dirichlet distribution parameter α.

[0079] The model was trained using ResNet-8, with SGD optimizer and a learning rate of 0.1.

[0080] To demonstrate the effectiveness of this invention in training on long-tailed data, experiments were conducted comparing it with other federated learning methods on the CIFAR-10-LT and CIFAR-100-LT datasets, as shown in Table 1. Table 1 shows that this invention exhibits superior performance on long-tailed distributions. This is mainly due to the rotational self-supervised learning and CLIP guidance, which mitigates the model's bias towards different data classes caused by the long-tailed distribution. Compared to CLIP2FL, the accuracy of this invention is improved by 2.35% to 4.72%.

[0081] Table 1

[0082]

[0083]

[0084] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A federated learning method for long-tail image classification combining rotational self-supervision and CLIP guidance, characterized in that, The specific steps are as follows: Step 1: The client uses the global model distributed by the server to calculate the gradient of the classification layer on the local dataset, then uses the currently received global model as the current local client model, trains the current local client model using the local dataset, and uses rotation self-supervision and CLIP to guide the training of the current local client model. Step 2: The server aggregates the client models trained in Step 1 into the global model for the next round, and uses the classifier gradients collected from the client to generate federated features. During the generation of federated features, CLIP is used to guide the generation of federated features. Step 3: The server copies the aggregated global model obtained in Step 2, retrains the classification layer of the copied global model using federated features, and finally distributes the aggregated global model obtained in Step 2 to the client. Step 4: Repeat Step 1 to Step 3 until the maximum number of communications is reached to complete the training; Step 1 specifically involves: S1.1, For each client and each category Based on its local dataset Use the currently issued global model Calculate the first Classifier gradients for each category ; ； in, For the first Each communication round of the client, Let cross-entropy be the loss function. For the first The first client's The class generates True characteristics, For sample labels; S1.2, For the client The currently received global model As a local model Its local dataset Each sample , sample Rotate separately The rotated label is The client additionally creates a four-classifier. Then, the rotated samples and labels are fed into the client's feature extractor. Then the feature extractor The output is input to a four-class classifier. In this way, we construct rotational self-supervised learning. Thus, the losses from supervised learning and rotational self-supervised learning in model training are expressed as follows: ； in, The loss is the supervised training loss used for normal training on the client side. y For the sample x The tag, For the loss of self-supervised learning by rotation, To control the hyperparameters affected by rotational self-supervised learning, The rotated sample The labels corresponding to the rotated samples; S1.3 For each client during training , its local dataset Each sample and after rotation The samples are input into CLIP, and the CLIP outputs are obtained respectively. Vector, represented as and Using knowledge distillation on the local client model Provide guidance, and the loss is expressed as: ； in, and Samples output for local client model and after rotation Dataset output vector, KL is KL divergence, and Hyperparameters for controlling the degree of CLIP guidance; S1.4 The total loss for training the local client model is: 。 2. The federated learning method for long-tail image classification combining rotational self-supervision and CLIP guidance as described in claim 1, characterized in that, Step 2 specifically involves: S2.1 The server receives the local model trained by the client and further aggregates the client model using the following formula to form the global model for the next round: ； in, Indicates the first Local datasets for each client, For the aggregated global model, This refers to the set of clients selected by the server to participate in training in the previous round. Upload the trained local model to the client; S2.2, The average true gradient collected by the server from the client. The true gradient average is calculated using the following formula: ； in, For the first The first client in the round-robin selection Gradient of a class; S2.3, The server first generates randomly. A federated feature that matches the dimensions of real features Then use a classifier For randomly generated federated features Generate the corresponding class gradient And it uses gradient matching loss function and CLIP-guided optimization to optimize randomly generated federated features. This enables the generation of federal features. The formula is more consistent with the characteristics of real data and is as follows: ； ; ; ； in, For the first Round The first in the class A randomly generated federal feature, For gradient matching loss function, Federal characteristics The corresponding category gradient The value of the row, The class gradient corresponding to the true average gradient. The value of the row, To compare loss functions, For the first The first in the class A randomly generated federal feature, For category not for federal characteristics, The total number of categories, For randomly generated federated features The in-class prototype after multiplying with the semantic information in CLIP. For temperature, CLIP guides the random generation of federated features The generated hyperparameters, For the total loss, For sample labels.

3. The federated learning method for long-tail image classification combining rotational self-supervision and CLIP guidance as described in claim 1, characterized in that, Step 3 specifically involves: S3.1 After copying the aggregated global model, retrain the classifier of the copied global model. First, freeze the parameters of the feature extractor of the copied global model, and then retrain the classifier of the copied global model. Then, the aggregated global model is sent to the client; ； in, For the trained and replicated global model classifier, For the first The federated features completed in the first round of optimization To retrain the learning rate of the model's classification layer, for gradient, Let cross-entropy be the loss function. For sample labels.

Images