Federal learning method for long-tail image classification in combination with rotation self-supervision and CLIP guidance

Abstract

The invention relates to a federated learning method for long-tail image classification in combination with rotation self-supervision and CLIP guidance, and belongs to the field of long-tail federated learning. Firstly, training of a local client is guided by using rotation self-supervised learning, the influence of long-tail data on a model feature extractor is reduced, and a model better learns the features of an image. And secondly, the pre-trained CLIP is utilized to guide the rotated image, rich knowledge in the CLIP is transferred to a client model, and the performance of a model feature extractor is further improved. And finally, the server generates a group of class-balanced federal features guided by the CLIP, and the federal features are used for retraining the classification layer of the aggregated model so as to improve the precision of the global model. According to the invention, by utilizing rotation self-supervision and CLIP guidance model training, the capability of extracting the features in the image by a client model feature extractor is effectively improved, the influence caused by long-tail data is relieved, and the recognition accuracy of a final global model is improved.

Description

Technical Field

[0001] The invention discloses a federated learning method for long-tail image classification combining rotation self-supervision and CLIP guidance, belonging to the field of long-tail federated learning. Background Art

[0002] Federated learning is a distributed machine learning method that allows multiple clients to collaboratively train a powerful shared model without transmitting their own private data sets. However, in the real world, data distribution usually presents a long-tail distribution, with the head class having a large amount of data and the tail class having only a small amount of data. Such data is not conducive to model training. When federated learning trains a model based on long-tail distributed data, it will cause the model to favor the head class data and ignore the tail class data, making the model's ability to classify the tail class data poor.

[0003] At present, many methods have been proposed to solve the problem of non-IID in federated learning. Methods such as Scaffold (Scaffold: Stochastic controlled averaging for federated learning [C], International conference on machine learning. PMLR, 2020: 5132–5143.) and FedProx (Federated optimization in heterogeneous networks [J]. Proceedings of Machine learning and systems, 2020, 2: 429–450.) make improvements to the client process so that the optimization direction of the client model is consistent with the optimization direction of the global model. Methods such as CCVR (No fear of heterogeneity: Classifier calibration for federated learning with non-iid data [J]. Advances in Neural Information Processing Systems, 2021, 34: 5972–5984.) improve the performance of the global model by making improvements on the server. However, the above methods are not applicable to the case where the global data is distributed in a long tail. In the case of a long tail distribution, the model will ignore the tail class data during training, and such methods will lead to a decrease in the accuracy of the global model. In centralized long-tail learning methods, methods such as data resampling and decoupling have been proven to be effective, but federated learning is distributed machine learning, and centralized training methods are not applicable to federated learning scenarios. CLIP2FL (CLIP-Guided Federated Learning on Heterogeneity and Long-Tailed Data[C], Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(13): 14955–14963.) uses CLIP (Contrastive Language-Image Pre-training) to guide the generation of federated features and guide client training to obtain a high-precision global model, but it only focuses on retraining the model classifier and does not balance the model's feature extractor, resulting in the model's feature extractor still being affected by long-tail data. At the same time, it does not give full play to the guiding role of CLIP.

[0004] In summary, solving the model deviation caused by the long-tail distribution of data in federated learning remains an important issue. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method for long-tail federated learning image classification for the case where global image data has a long-tail distribution, so as to alleviate the bias of the federated learning model trained under the long-tail distribution towards the majority class data and minority class data in the long-tail data, and ultimately improve the accuracy of the global model.

[0006] The technical solution of the present invention is: Figure 1 As shown in the figure, a long-tail federated learning method for the case of long-tail distribution of global data is presented. On the client side, rotational self-supervision and CLIP are used to guide the training of the local client model, and the global model sent by the server is used to obtain the gradient of the classification layer of the local data under the global model, which is used for the subsequent server to generate federated features; on the server side, the server first averages and aggregates the models uploaded by the client into a global model, and then uses the classification layer gradients collected from the local client and CLIP to guide the server to generate federated features, and then uses the generated federated features to retrain the classification layer of the aggregated global model.

[0007] The specific steps are:

[0008] Step 1: The client uses the global model sent by the server to calculate the gradient of the classification layer on the local dataset, then uses the currently received global model as the current local client model, trains the current local client model using the local dataset, and uses rotational self-supervision and CLIP to guide the training of the current local client model;

[0009] Step 2: The server aggregates the client models trained in Step 1 to become the global model for the next round, and uses the classifier gradients collected from the clients to generate federated features. In the process of generating federated features, CLIP is used to guide the generation of federated features.

[0010] Step 3: The server copies the aggregated global model obtained in Step 2, and retrains the classification layer of the copied global model using the federated features. Finally, the aggregated global model obtained in Step 2 is distributed to the client.

[0011] Step 4: Repeat Step 1 to Step 3 until the maximum number of communications is reached and the training is completed.

[0012] Specifically, the step 1 is as follows:

[0013] S1.1. For each client k and each category c, based on its local dataset Use the currently issued global model w t Calculate the gradient of the classifier for the cth category

[0014]

[0015] in, is the client in the tth communication round, L ce is the cross entropy loss function, The d-dimensional true features generated for the c-th class of the k-th client, y i is the sample label;

[0016] S1.2. For client k, the currently received global model w t As a local model Its local dataset For each sample x in , rotate the sample x by {0°, 90°, 180°, 270°} respectively, and the labels after rotation are {1, 2, 3, 4}; the client creates an additional four-category classifier v, and then sends the rotated samples and labels to the client's feature extractor f k Then the feature extractor f k The output of is input into the four-category classifier v to construct rotation self-supervised learning. So far, the loss of supervised learning and rotation self-supervision in model training is expressed as:

[0017] L sup+self =L ce sup (x, y)+ε·L ce self (x r ,y r );

[0018] Among them, L ce sup (x, y) is the loss of supervised training of normal training on the client side, y is the label of sample x, L ce self is the loss of rotation self-supervised learning, ε is the hyperparameter that controls the impact of rotation self-supervised learning, and x r is the rotated sample, y r is the label corresponding to the rotated sample;

[0019] S1.3. For each client k in training, its local dataset Each sample x and the rotated x r The samples are input into CLIP, and the logit vectors output by CLIP are obtained, expressed as and Using knowledge distillation to improve local client models For guidance, the loss is expressed as:

[0020]

[0021] in, and Sample x and rotated x output by the local client model r The logit vector output by the dataset, KL is the KL divergence, β and δ are hyperparameters that control the degree of CLIP guidance;

[0022] S1.4. The total loss of local client model training is:

[0023] L local =L sup+self +L KL total .

[0024] Specifically, the step Step2 is as follows:

[0025] S2.1. The server receives the local model trained by the client and further aggregates the client model through the following formula to form the next round of global model:

[0026]

[0027] Among them, D k represents the local dataset of the kth client, w t+1 is the global model after aggregation, φ t is the set of clients selected by the server to participate in training in the last round. Upload the trained local model to the client;

[0028] S2.2. The server averages the true gradients collected from the client And the true gradient average is calculated by the following formula:

[0029]

[0030] in, is the gradient of the cth class of clients selected in the tth round;

[0031] S2.3. The server first randomly generates m federated features that conform to the real feature dimensions. Then use the classifier is a randomly generated federated feature Generate corresponding category gradients And use the gradient matching loss function and CLIP guidance to optimize the randomly generated federated features Enables the generated federation feature It is more in line with the characteristics of real data. The formula is as follows:

[0032]

[0033]

[0034]

[0035] L all =L g +η·L p ;

[0036] in, is the randomly generated federated feature in the cth category in the tth round, L g is the gradient matching loss function, Federal characteristics The value of the jth row of the corresponding category gradient, is the value of the class gradient in the jth row corresponding to the true gradient average, L p is the contrast loss function, z c，i is the randomly generated federated feature in the cth class, z j is the federated feature whose category is not i, C is the total number of categories, p C is a randomly generated federated feature The in-class prototype after multiplication with the semantic information in CLIP, τ is the temperature, and η is the federated feature randomly generated by CLIP The generated hyperparameters, L all is the total loss, y i is the sample label.

[0037] Specifically, the step Step 3 is as follows:

[0038] S3.1. After copying the aggregated global model, retrain the model classifier of the copied global model. First, freeze the parameters of the feature extractor of the copied global model, and retrain the classifier γ of the copied global model. t , and then send the aggregated global model to the client;

[0039]

[0040] Among them, γ t+1 is the trained copied global model classifier, is the federated feature optimized in the tth round, is the learning rate for retraining the classification layer of the model, For γ t The gradient of L ce is the cross entropy loss function, y i is the sample label.

[0041] The beneficial effects of the present invention are:

[0042] 1. Since the existing long-tail federated learning does not balance the feature extractor of the model, the model trained in the case of long-tail data distribution is biased towards the head class, and the classification performance for the tail class is poor, and the model is difficult to classify the tail class data. The present invention proposes a federated learning method for long-tail image classification combined with rotational self-supervision and CLIP guidance. Through rotational self-supervision and CLIP guidance model training, the imbalance of the feature extractor of the model during training is alleviated.

[0043] 2. In view of the situation that the global data is distributed in a long tail, the present invention uses rotation self-supervised learning in the client training process to prompt the model to learn high-quality image features, thereby alleviating the impact of the long tail distribution; at the same time, the rich knowledge of CLIP is used to guide the normal training images and the rotated images, and the rich knowledge of CLIP is transferred to the model of the local client, further enhancing the performance of the local client model. This method alleviates the impact of the long tail distribution data on the model, thereby improving the accuracy of global model classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0046] Example 1: Figure 1 As shown, this implementation example gives 20 users K = {k: k = 1, 2, ...}. The server and the client communicate 200 times in total during the entire training process. Before each communication, the server randomly selects 8 clients as clients for training the model. The client local training model iterates 15 times, with each batch being 32.

[0047] Step 1: The client uses the global model sent by the server to calculate the gradient of the classification layer on the local dataset, then uses the currently received global model as the current local client model, trains the current local client model using the local dataset, and uses rotational self-supervision and CLIP to guide the training of the current local client model;

[0048] S1.1. For each client k and each category c, based on its local dataset Use the currently issued global model w t Calculate the gradient of the classifier for the cth category

[0049]

[0050] in, is the client in the tth communication round, L ce is the cross entropy loss function, The d-dimensional true features generated for the c-th class of the k-th client, y i is the sample label;

[0051] S1.2. For client k, the currently received global model w t As a local model Its local dataset For each sample x in , rotate the sample x by {0°, 90°, 180°, 270°} respectively, and the labels after rotation are {1, 2, 3, 4}; the client creates an additional four-category classifier v, and then sends the rotated samples and labels to the client's feature extractor f k Then the feature extractor f k The output of is input into the four-category classifier v to construct rotation self-supervised learning. So far, the loss of supervised learning and rotation self-supervision in model training is expressed as:

[0052] L sup+self =L ce sup (x, y)+ε·L ce self (x r ,y r );

[0053] Among them, L cesup (x, y) is the loss of supervised training of normal training on the client side, y is the label of sample x, L ce self is the loss of rotation self-supervised learning, ε is the hyperparameter that controls the impact of rotation self-supervised learning, and x r is the rotated sample, y r is the label corresponding to the rotated sample;

[0054] For each client k in training, its local dataset Each sample x and the rotated x r The samples are input into CLIP, and the logit vectors output by CLIP are obtained, expressed as and Using knowledge distillation to improve local client models For guidance, the loss is expressed as:

[0055]

[0056] in, and Sample x and rotated x output by the local client model rThe logit vector output by the dataset, KL is the KL divergence, β and δ are hyperparameters that control the degree of CLIP guidance;

[0057] S1.4. The total loss of local client model training is:

[0058] L local =L sup+self +L KL total ;

[0059] Step 2: The server aggregates the client models trained in Step 1 to become the global model for the next round, and uses the classifier gradients collected from the clients to generate federated features. In the process of generating federated features, CLIP is used to guide the generation of federated features.

[0060] S2.1. The server receives the local model trained by the client and further aggregates the client model through the following formula to form the next round of global model:

[0061]

[0062] Among them, D k represents the local dataset of the kth client, w t+1 is the global model after aggregation, φ t is the set of clients selected by the server to participate in training in the last round. Upload the trained local model to the client;

[0063] S2.2. The server averages the true gradients collected from the client And the true gradient average is calculated by the following formula:

[0064]

[0065] in, is the gradient of the cth class of clients selected in round t.

[0066] S2.3. The server first randomly generates m federated features that conform to the real feature dimensions. Then use the classifier is a randomly generated federated feature Generate corresponding category gradients And use the gradient matching loss function and CLIP guidance to optimize the randomly generated federated features Enables the generated federation feature It is more in line with the characteristics of real data. The formula is as follows:

[0067]

[0068]

[0069]

[0070] L all =L g +η·L p ;

[0071] in, is the randomly generated federated feature in the cth category in the tth round, L g is the gradient matching loss function, Federal characteristics The value of the corresponding category gradient row j, is the value of the class gradient in the jth row corresponding to the true gradient average, L p is the contrast loss function, z c，i is the randomly generated federated feature in the cth class, z j is the federated feature whose category is not i, C is the total number of categories, p C is a randomly generated federated feature The in-class prototype after multiplication with the semantic information in CLIP, τ is the temperature, and η is the federated feature randomly generated by CLIP The generated hyperparameters, L all is the total loss, y i is the sample label.

[0072] Step 3: The server copies the aggregated global model obtained in Step 2, and retrains the classification layer of the copied global model using the federated features. Finally, the aggregated global model obtained in Step 2 is distributed to the client.

[0073] S3.1. After copying the aggregated global model, retrain the model classifier of the copied global model. First, freeze the parameters of the feature extractor of the copied global model, and retrain the classifier γ of the copied global model. t , and then send the aggregated global model to the client;

[0074]

[0075] Among them, γ t+1 is the trained copied global model classifier, is the federated feature optimized in the tth round, θ is the learning rate of the classification layer of the retrained model, For γ t The gradient of L ce is the cross entropy loss function, y i is the sample label.

[0076] Step 4: Repeat Step 1 to Step 3 until 200 communications are reached and the training is completed.

[0077] Finally, the performance of the present invention is evaluated through experimental comparison. The version of Python used in the experiment is 3.10.12, the version of the Pytorch deep learning framework is 2.0.1, and the hardware environment used in the experiment is NVIDIA GeForce RTX 4090 graphics card.

[0078] This experiment uses the CIFAR-10-LT and CIFAR-100-LT datasets, which are built based on CIFAR-10 / CIFAR-100 respectively. The long-tail distribution and heterogeneity of the data are simulated by adjusting the imbalance factor IF and the Dirichlet distribution parameter α.

[0079] The model uses ResNet-8 as the training model, the optimizer uses the SGD optimizer, and the learning rate is 0.1.

[0080] In order to prove the effectiveness of the present invention in training under long-tail data, the present invention is compared with other federated learning methods on CIFAR-10-LT and CIFAR-100-LT datasets, and Table 1 is obtained. As can be seen from Table 1, the present invention shows superior performance under long-tail distribution, which is mainly due to the guidance of rotational self-supervised learning and CLIP, which alleviates the bias of the model caused by long-tail distribution on different data classes. Compared with CLIP2FL, the accuracy of the present invention method is improved by 2.35% to 4.72%.

[0081] Table 1

[0082]

[0083]

[0084] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A federated learning method for long-tail image classification combining rotational self-supervision and CLIP guidance, characterized in that, The specific steps are: Step 1: The client uses the global model sent by the server to calculate the gradient of the classification layer on the local dataset, then uses the currently received global model as the current local client model, trains the current local client model using the local dataset, and uses rotational self-supervision and CLIP to guide the training of the current local client model; Step 2: The server aggregates the client models trained in Step 1 to become the global model for the next round, and uses the classifier gradients collected from the clients to generate federated features. In the process of generating federated features, CLIP is used to guide the generation of federated features. Step 3: The server copies the aggregated global model obtained in Step 2, and retrains the classification layer of the copied global model using the federated features. Finally, the aggregated global model obtained in Step 2 is distributed to the client. Step 4: Repeat Step 1 to Step 3 until the maximum number of communications is reached and the training is completed.

2. The federated learning method for long-tail image classification combining rotational self-supervision and CLIP guidance according to claim 1, characterized in that: The step 1 is specifically as follows: S1.

1. For each client k and each category c, based on its local dataset Use the currently issued global model w t Calculate the gradient of the classifier for the cth category in, is the client in the tth communication round, L ce is the cross entropy loss function, The d-dimensional true features generated for the c-th class of the k-th client, y i is the sample label; S1.

2. For client k, the currently received global model w t As a local model Its local dataset For each sample x in , rotate the sample x to {0°, 90°, 180°, 270°} respectively, and the rotated labels are {1, 2, 3, 4}; the client creates an additional four-category classifier v, and then sends the rotated samples and labels to the client's feature extractor f k Then the feature extractor f k The output of is input into the four-category classifier v to construct rotation self-supervised learning. So far, the loss of supervised learning and rotation self-supervision in model training is expressed as: L sup+self =L ce sup (x,y)+ε·L ce self (y r ,y r ); Among them, L ce sup (x,y) is the loss of supervised training of normal training on the client side, y is the label of sample x, L ce self is the loss of rotation self-supervised learning, ε is the hyperparameter that controls the impact of rotation self-supervised learning, and x r is the rotated sample, y r is the label corresponding to the rotated sample; S1.

3. For each client k in training, its local dataset Each sample x and the rotated x r The samples are input into CLIP, and the logit vectors output by CLIP are obtained, expressed as and Using knowledge distillation to improve local client models For guidance, the loss is expressed as: in, and Sample x and rotated x output by the local client model r The logit vector output by the dataset, KL is the KL divergence, β and δ are hyperparameters that control the degree of CLIP guidance; S1.

4. The total loss of local client model training is: L local =L sup+self +L KL total .

3. The federated learning method for long-tail image classification combining rotational self-supervision and CLIP guidance according to claim 1, characterized in that: The step 2 is specifically as follows: S2.

1. The server receives the local model trained by the client and further aggregates the client model through the following formula to form the next round of global model: Among them, D k represents the local dataset of the kth client, w t+1 is the global model after aggregation, φ t is the set of clients selected by the server to participate in training in the last round. Upload the trained local model to the client; S2.

2. The server averages the true gradients collected from the client And the true gradient average is calculated by the following formula: in, is the gradient of the cth class of clients selected in the tth round; S2.

3. The server first randomly generates m federated features that conform to the real feature dimensions. Then use the classifier is a randomly generated federated feature Generate corresponding category gradients And use the gradient matching loss function and CLIP guidance to optimize the randomly generated federated features Enables the generated federation feature It is more in line with the characteristics of real data. The formula is as follows: L all =L g +η·L p ; in, is the randomly generated federated feature in the cth category in the tth round, L g is the gradient matching loss function, Federal characteristics The value of the jth row of the corresponding category gradient, is the value of the class gradient in the jth row corresponding to the true gradient average, L p is the contrast loss function, z c,i is the randomly generated federated feature in the cth class, z j is the federated feature whose category is not i, C is the total number of categories, p C is a randomly generated federated feature The in-class prototype after multiplication with the semantic information in CLIP, τ is the temperature, and η is the federated feature randomly generated by CLIP The generated hyperparameters, L all is the total loss, y i is the sample label.

4. The federated learning method for long-tail image classification combining rotational self-supervision and CLIP guidance according to claim 1, characterized in that: The step Step 3 is specifically as follows: S3.

1. After copying the aggregated global model, retrain the model classifier of the copied global model. First, freeze the parameters of the feature extractor of the copied global model, and retrain the classifier γ of the copied global model. t , and then send the aggregated global model to the client; Among them, γ t+1 is the trained copied global model classifier, is the federated feature optimized in the tth round, θ is the learning rate of the classification layer of the retrained model, For γ t The gradient of L ce is the cross entropy loss function, y i is the sample label.

Images