Federated course learning image classification method oriented to class imbalance data
By introducing a two-layer course weighting strategy in federated learning, the class imbalance problem is solved, the classification accuracy of minority classes and the generalization ability of the model are improved, and it is suitable for scenarios with high precision and high privacy requirements such as medical image classification.
Patent Information
- Application Number
- CN202510618369.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-14
- Publication Date
- 2025-09-16
AI Technical Summary
Under the federated learning framework, image classification models find it difficult to effectively identify minority class data when faced with category-imbalanced data, resulting in low classification accuracy and a lack of careful consideration of the difficulty of training samples and the balance at the category level.
A two-layer course weighting strategy is adopted. The sample-level difficulty score is calculated by recording historical prediction results in an adaptive window, and the category-level difficulty score is calculated by combining the expected calibration error of the federated categories. After fusion, the data is rearranged, and the local model training is performed by combining the cross entropy loss function and the mean square error loss function. Finally, knowledge sharing is performed through the global prototype.
It significantly improves the classification accuracy of minority classes and the generalization ability of the overall model, and is particularly suitable for scenarios such as medical image classification that have high requirements for classification accuracy and data privacy.
Smart Images
Figure CN120655964A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a federated curriculum learning image classification method for category imbalanced data, belonging to the technical field of image recognition. Background Art
[0002] With the rapid development of artificial intelligence and deep learning, image classification technology has been widely used in fields such as medical diagnosis, autonomous driving, and security monitoring. In particular, it has become a crucial tool for assisting doctors in decision-making in medical image analysis. However, in real-world applications, image data often suffers from severe class imbalance, where the amount of data for some classes far exceeds that for others. This class imbalance causes classification models to perform poorly when processing minority class data, impacting both classification accuracy and the model's generalization ability. Therefore, effectively addressing data imbalance and improving classification performance has long been a key focus of image classification research.
[0003] Traditional deep learning methods rely on large-scale labeled datasets for training, but in the case of class imbalance, a single dataset often fails to fully capture the characteristics of minority classes. To this end, researchers have proposed techniques such as data augmentation, undersampling, and oversampling to balance the number of samples in each category. Although these methods have improved model performance to a certain extent, they still face problems such as data distortion, overfitting, and low training efficiency. In particular, they are often limited in their effectiveness in extracting minority class features. In addition, most existing methods assume that data is trained in a centralized environment, with all data aggregated and processed on a central server. This centralized training not only increases data privacy and security risks, but also wastes computing resources due to the large amount of data, reducing training efficiency.
[0004] Compared to traditional methods, federated learning, an emerging distributed machine learning technology, avoids the risk of data leakage by keeping data locally and improves model training efficiency through collaborative learning among multiple clients. Federated learning can fully utilize distributed data while protecting user privacy, thus showing broad application prospects in scenarios involving large amounts of distributed data. However, within the federated learning framework, differences in data distribution, particularly class imbalance, make it difficult for traditional methods to effectively classify minority class data, significantly reducing model accuracy. This problem is particularly prominent in fields such as healthcare and security, where the ability to identify minority classes (such as rare diseases and abnormal events) cannot meet practical needs.
[0005] Currently, research on federated learning primarily focuses on addressing data heterogeneity and differences in client distribution, while research on class imbalance remains insufficient. Due to privacy protection requirements in federated learning, traditional methods for addressing class imbalance (such as oversampling and undersampling) are difficult to apply directly, resulting in significant challenges for federated learning in addressing class imbalance. While strategies based on loss function weighting and client selection have been attempted to balance the impact by adjusting class weights, these methods still have significant limitations in improving minority class classification performance. More importantly, existing methods for addressing class imbalance in federated learning often overlook the fine-grained control of the difficulty of training samples and the consideration of overall balance at the class level. There is a lack of an effective federated curriculum learning mechanism, particularly one that can combine a two-layer weighting strategy, that is, one that simultaneously considers the difficulty and importance at both the sample level and the class level to guide model training. Summary of the Invention
[0006] The present invention aims to solve the problems in the prior art of classifying class-imbalanced images under the federated learning framework, such as the difficulty of the model in effectively identifying minority class data, low classification accuracy, and lack of careful consideration of the difficulty of training samples and the balance at the class level. A federated curriculum learning image classification method for class-imbalanced data is proposed.
[0007] The technical solution adopted by the present invention to solve the above problems is: the present invention comprises the following steps:
[0008] Step 1: Build a federated learning system, where the federated communication system includes a server and M clients participating in training;
[0009] Step 2: Initialize the global prototype on the server The client initializes the local model w i (t-1) Parameters;
[0010] Step 3: The server sends the initialized global prototype to the client for the current round of communication;
[0011] Step 4: In each round of communication, each client i uses its local dataset and local models Data rearrangement was performed using a two-tier course weighting strategy;
[0012] Step 5: Each client i uses the rearranged local data to train the local model according to the received global prototype and update the local model parameters w i and local prototype P i ;
[0013] Step 6: The client will update the local prototype P iUpload to the server;
[0014] Step 7: The server clusters the received local prototype P i And update the global prototype used in the next round of communication;
[0015] Step 8: Repeat steps 2 to 7 until the preset number of training rounds is reached or the convergence condition is met;
[0016] Step 9: The server finishes the federated training and sends the final global prototype to all clients.
[0017] Furthermore, the global prototype in step 2 is the feature vector representation of each category, which is used for knowledge sharing in the federated learning process, where K is the number of categories, is the global prototype of the kth category, d is the feature dimension, and the local model Includes a feature extractor f for extracting image features i (·) and the classifier h for classification prediction i (·).
[0018] Furthermore, step 4 specifically includes:
[0019] Step 4.1: For client i’s local dataset Each sample x i,j , based on the local model in the historical communication round and the current communication round The prediction results are obtained by using an adaptive window W(t) to record the historical prediction results, and the sample-level difficulty score of each sample is obtained based on the historical prediction results. Among them, the window size is dynamically adjusted with the training round t;
[0020] Step 4.2: Calculate the federated class expected calibration error for each data class k and use the federated class expected calibration error as the class-level difficulty score
[0021] Step 4.3: Score the sample-level difficulty and category-level difficulty scores Fusion is performed to obtain the comprehensive difficulty score of each sample The adjustment parameter α is introduced to balance the comprehensive difficulty score of the current communication round and the historical difficulty score to obtain the final difficulty score of the corresponding sample
[0022] Step 4.4: The final difficulty score of the corresponding sample Normalized to course probability According to the course probability Rearrange local training data;
[0023] The expression of the adaptive window W(t) is:
[0024] W(t)=W min +(W max -W min )·log(t+1) / log(T+1), (1);
[0025] In formula (1), W min is the minimum window size, W max is the maximum value of the window size, T is the total number of communications, and t is the current communication round;
[0026] Sample-level difficulty score The calculation formula is:
[0027]
[0028] In formula (2), is the local model pair sample x in the tth round of communication i,j The prediction result, y i,j For sample x i,j The true label, is an indicator function, which is 1 when the prediction is wrong and 0 otherwise;
[0029] Sample comprehensive difficulty score The calculation formula is:
[0030]
[0031] Final difficulty score of the sample The calculation formula is:
[0032]
[0033] Course Probability The calculation formula is:
[0034]
[0035] Furthermore, step 4.2 specifically includes:
[0036] Step 4.2.1: Divide the predicted probability into S intervals,
[0037] Step 4.2.2: Calculate the set of samples with label k whose predicted probability falls within the sth interval in, is the sample set with the true label k on client i, is the size of its sample set;
[0038] Step 4.2.3: Calculate the average confidence level within each interval and average accuracy And calculate the average confidence and average accuracy The weighted average result of the federated category calibration error is obtained, that is, the category level difficulty score
[0039] Average confidence The calculation formula is:
[0040]
[0041] Average accuracy The calculation formula is:
[0042]
[0043] Category-level difficulty score The calculation formula is:
[0044]
[0045] Furthermore, in step 5, the cross entropy loss function and the mean square error loss function are used as the total loss function L during the local model training process;
[0046] The expression of the total loss function L is:
[0047]
[0048] In formula (9), log(·) is a logarithmic operation with the natural constant e as the base. After the j-th sample of the i-th client is input into the local model, the output is the probability of belonging to the k-th class. is the local prototype of the i-th client category k in the t-th round, and the k-th category training image is input to the feature extractor f i The eigenvectors obtained in (·) are averaged;
[0049] The calculation formula is:
[0050]
[0051] Furthermore, step 7 specifically includes:
[0052] The server cluster receives all local prototypes P from the client. i Then, the federated learning aggregation algorithm is used to aggregate the local prototypes P from different clients. i Perform a weighted draw to obtain the global prototype used in the next round of communication
[0053] The global prototype used in the next round of communication The calculation formula is:
[0054]
[0055] In formula (11), is the set of all clients containing category k.
[0056] The beneficial effects of the present invention are:
[0057] By introducing a two-layer weighted federated curriculum learning strategy, this paper can more effectively address the class imbalance problem in federated learning, improving the classification accuracy of minority classes and the generalization ability of the overall model. For example, on the Fed-ISIC2019 dataset, this method achieved significant performance improvements compared to existing methods. By combining global prototypes for knowledge sharing, the model's performance is further improved, making it particularly suitable for scenarios with high requirements for classification accuracy and data privacy, such as medical image classification. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] Figure 1 A flowchart of a federated curriculum learning image classification method for class-imbalanced data provided by the present invention;
[0059] Figure 2 Schematic diagram of the local model training framework provided by the present invention;
[0060] Figure 3 This is a comparison chart of the confusion matrices of the proposed method and the existing federated learning method for classification on the Fed-ISIC2019 dataset. Figure 3 In the figure, (a) is a schematic diagram of the confusion matrix of the existing federated learning method for classification on the Fed-ISIC2019 dataset, and (b) is a schematic diagram of the confusion matrix of the method proposed in the present invention for classification on the Fed-ISIC2019 dataset. DETAILED DESCRIPTION
[0061] Specific implementation method 1: Combination Figure 1 and Figure 2 This embodiment is described as follows. Figure 1 As shown, the steps of the federated curriculum learning image classification method for class imbalanced data described in this embodiment include:
[0062] S1: Initialize the global prototype and client model;
[0063] S101: This embodiment first constructs a federated learning system, which includes a server and M clients participating in training. Each client has its own local class imbalanced image dataset.
[0064] S102: Randomly initialize the global prototype on the server side The global prototype is the feature vector representation of each category, which is used for knowledge sharing in the federated learning process, where K is the number of categories. is the global prototype of the kth category, d is the feature dimension; and the initial global prototype is sent to each client, and the client randomly initializes the local model parameters The local model of client i includes a feature extractor f for extracting image features i (·) and the classifier h for classification prediction i (·) The local models of each client can be heterogeneous. For example, different clients can use convolutional neural networks with different numbers of layers as feature extractors, or use different types of classifiers (such as fully connected layers, support vector machines, etc.).
[0065] S2: The client sends the global prototype to the client;
[0066] S3: Each client uses a two-tier course strategy to rearrange local data based on its local dataset;
[0067] S301: The client calculates the sample level difficulty;
[0068] For each sample x of client i i,j , based on the prediction results of the local model in the historical communication rounds and the current communication round, the misclassification error of each sample is calculated as the sample-level difficulty score To maintain high flexibility in the early stages of training and avoid over-reliance on outdated information in the later stages of training, this implementation uses an adaptive window size W(t) to record historical prediction results. The window size is dynamically adjusted with the number of training rounds t, for example, using a logarithmic growth function.
[0069] The expression of the adaptive window W(t) is:
[0070] W(t)=W min +(W max -W min )·log(t+1) / log(T+1), (1);
[0071] In formula (1), W min is the minimum window size, W maxis the maximum value of the window size, T is the total number of communications, and t is the current communication round;
[0072] Sample-level difficulty score The calculation formula is:
[0073]
[0074] In formula (2), is the local model pair sample x in the tth round of communication i,j The prediction result, y i,j For sample x i,j The true label, is an indicator function, which is 1 when the prediction is wrong and 0 otherwise;
[0075] S302: Client extreme category level difficulty;
[0076] Although the sample-level difficulty helps prioritize misclassified samples, it cannot directly address the class imbalance problem. To this end, for each class k, this implementation proposes the federated class expected calibration error As a category-level difficulty score, it is used to reflect the learning difficulty and calibration of each category. The calculation steps are as follows:
[0077] S30201: Divide the predicted probability into S intervals,
[0078] S30202: Calculate the set of samples with label k whose predicted probability falls within the sth interval in, is the sample set with the true label k on client i, is the size of its sample set;
[0079] S30203: Calculate the average confidence level within each interval and average accuracy And calculate the average confidence and average accuracy The weighted average result of the federated category calibration error is obtained, that is, the category level difficulty score
[0080] Average confidence The calculation formula is:
[0081]
[0082] Average accuracy The calculation formula is:
[0083]
[0084] Category-level difficulty score The calculation formula is:
[0085]
[0086] S303: The client integrates the sample-level and category-level difficulty scores and weights them based on the historical difficulty scores.
[0087] S30301: Score the sample level difficulty and category-level difficulty scores Fusion is performed to obtain the comprehensive difficulty score of each sample
[0088]
[0089] S30302: Introduce the adjustment parameter α to balance the difficulty score of the current round with the historical difficulty score to obtain the final difficulty score of the sample
[0090]
[0091] S304: Generate course weights and rearrange local data;
[0092] The final difficulty score Normalized to course probability Map the difficulty score to the range [0,1], based on the course probability Rearrange local training data;
[0093] Course Probability The calculation formula is:
[0094]
[0095] By rearranging data, samples with higher difficulty have higher priority during local training. For example, data can be arranged in descending order of course probability, making samples with higher difficulty easier to be learned by the model in each local training cycle.
[0096] S4: The client uses the rearranged local data to train the local model based on the received global prototype;
[0097] The local training process is as follows Figure 2 As shown, each client i uses the rearranged local data to train the local model and update the local model parameters w according to the received global prototype i and local prototype P i The local training uses the cross entropy loss function and the mean square error loss function as the total loss function L:
[0098]
[0099] In formula (9), log(·) is a logarithmic operation with the natural constant e as the base. After the j-th sample of the i-th client is input into the local model, the output is the probability of belonging to the k-th class. is the local prototype of the i-th client category k in the t-th round, and the k-th category training image is input to the feature extractor f i The eigenvectors obtained in (·) are averaged;
[0100] The calculation formula is:
[0101]
[0102] S5: The client uploads the updated local prototype to the server;
[0103] S6: The server aggregates the received local prototypes and updates the new round of global prototypes;
[0104] After the server receives the local prototypes from all clients, this implementation adopts a federated learning aggregation algorithm (such as FedProto) to integrate the local information from different clients and update the global prototype. For each category k, the new global It can be obtained by taking a weighted average of the local prototypes uploaded by all clients:
[0105]
[0106] In formula (11), is the set of all clients containing category k.
[0107] S7: Repeat S2-S6 until the preset number of training rounds is reached or the convergence condition is met;
[0108] S8: The server ends the federated training and sends the final global prototype to all clients.
[0109] In summary, the present invention can more effectively solve the category imbalance problem in federated learning by introducing a two-layer weighted federated curriculum learning strategy, improve the classification accuracy of minority classes and the generalization ability of the overall model. At the same time, the present invention has achieved significant performance improvement compared with existing methods. By combining global prototypes for knowledge sharing, the performance of the model is further improved. It is particularly suitable for scenarios with high requirements on classification accuracy and data privacy, such as medical image classification.
[0110] Specific implementation method 2: Combination Figure 3To illustrate this embodiment, the following verification experiment was designed to verify the technical effect and effectiveness of the proposed method for federated curriculum learning image classification for class-imbalanced data:
[0111] The medical image dataset used in practical applications in this embodiment is the dermatoscope dataset Fed-ISIC2019, which includes eight types of skin lesions and a total of 23,247 images. It is divided into six clients, and the number of images in each client is {12413, 3954, 3363, 2259, 819, 439}, and the image size is 200×200×3.
[0112] This implementation uses FedAvg, FedProx, FedBN, CCVR, FedProto, PRR-Imbalance, FPL, and FedConcat as benchmark methods, and uses BACC (balanced accuracy) as the classification metric. In the experiment, the local model uses ResNet18 as the feature extractor, followed by a fully connected layer as the classifier. The optimizer uses SGD, and the learning rate is set to 0.01. The confusion matrix of the proposed method for classification on the Fed-ISIC2019 dataset is shown below: Figure 3 As shown in (b), the BACC (%) values of the proposed method and the existing 8 benchmark methods are compared in Table 1. The confusion matrix of the existing federated learning method for classification on the Fed-ISIC2019 dataset is shown in Figure 3 (a) shows that according to Figure 3 (a) Figure 3 (b) and Table 1 show that the proposed method improves by 12.00%, 13.41%, 5.23%, 10.05%, 3.51%, 9.34%, 5.76% and 5.98% compared with FedAvg, FedProx, FedBN, CCVR, FedProto, PRR-Imbalance, FPL and FedConcat, respectively.
[0113] Table 1
[0114]
[0115] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the present profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical content disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement of the above embodiments made according to the technical essence of the present invention, within the spirit and principles of the present invention, without departing from the content of the technical solution of the present invention, shall still fall within the scope of protection of the technical solution of the present invention.
Claims
1. A federated curriculum learning image classification method for class imbalanced data, characterized by: The steps of the federated curriculum learning image classification method for class imbalanced data include: Step 1: Build a federated learning system, where the federated communication system includes a server and M clients participating in training; Step 2: Initialize the global prototype on the server Client initializes local model Parameters; Step 3: The server sends the initialized global prototype to the client for the current round of communication; Step 4: In each round of communication, each client i uses its local dataset and local models Data rearrangement was performed using a two-tier course weighting strategy; Step 5: Each client i uses the rearranged local data to train the local model according to the received global prototype and update the local model parameters w i and local prototype P i ; Step 6: The client will update the local prototype P i Upload to the server; Step 7: The server clusters the received local prototype P i And update the global prototype used in the next round of communication; Step 8: Repeat steps 2 to 7 until the preset number of training rounds is reached or the convergence condition is met; Step 9: The server finishes the federated training and sends the final global prototype to all clients.
2. The method for image classification using federated curriculum learning for class-imbalanced data according to claim 1, wherein: Global prototype in step 2 is the feature vector representation of each category, which is used for knowledge sharing in the federated learning process, where K is the number of categories, is the global prototype of the kth category, d is the feature dimension, and the local model Includes a feature extractor f for extracting image features i (·) and the classifier h for classification prediction i (·).
3. The method for image classification using federated curriculum learning for class-imbalanced data according to claim 1, wherein: Step 4 specifically includes: Step 4.1: For client i’s local dataset Each sample x i,j , based on the local model in the historical communication round and the current communication round The prediction results are obtained by using an adaptive window W(t) to record the historical prediction results, and the sample-level difficulty score of each sample is obtained based on the historical prediction results. Among them, the window size is dynamically adjusted with the training round t; Step 4.2: Calculate the federated class expected calibration error for each data class k and use the federated class expected calibration error as the class-level difficulty score Step 4.3: Score the sample-level difficulty and category-level difficulty scores Fusion is performed to obtain the comprehensive difficulty score of each sample The adjustment parameter α is introduced to balance the comprehensive difficulty score of the current communication round and the historical difficulty score to obtain the final difficulty score of the corresponding sample Step 4.4: The final difficulty score of the corresponding sample Normalized to course probability According to the course probability Rearrange local training data; The expression of the adaptive window W(t) is: W(t)=W min +(W max -W min )·log(t+1) / log(T+1), (1); In formula (1), W min is the minimum window size, W max is the maximum value of the window size, T is the total number of communications, and t is the current communication round; Sample-level difficulty score The calculation formula is: In formula (2), is the local model pair sample x in the tth round of communication i,j The prediction result, y i,j For sample x i,j The true label, is an indicator function, which is 1 when the prediction is wrong and 0 otherwise; Sample comprehensive difficulty score The calculation formula is: Final difficulty score of the sample The calculation formula is: Course Probability The calculation formula is:
4. The method for image classification using federated curriculum learning for class-imbalanced data according to claim 3, wherein: Step 4.2 specifically includes: Step 4.2.1: Divide the predicted probability into S intervals, Step 4.2.2: Calculate the set of samples with label k whose predicted probability falls within the sth interval in, is the sample set with the true label k on client i, is the size of its sample set; Step 4.2.3: Calculate the average confidence level within each interval and average accuracy And calculate the average confidence and average accuracy The weighted average result of the federated category calibration error is obtained, that is, the category level difficulty score Average confidence The calculation formula is: Average accuracy The calculation formula is: Category-level difficulty score The calculation formula is:
5. The method for image classification using federated curriculum learning for class imbalanced data according to claim 1, wherein: In step 5, the cross entropy loss function and the mean square error loss function are used as the total loss function L during the local model training process; The expression of the total loss function L is: In formula (9), log(·) is a logarithmic operation with the natural constant e as the base. After the jth sample of the i-th client is input into the local model, the output probability belongs to the k-th category, is the local prototype of the i-th client category k in the t-th round, and the k-th category training image is input to the feature extractor f i The eigenvectors obtained in (·) are averaged; The calculation formula is:
6. The method for image classification using federated curriculum learning for class imbalanced data according to claim 1, wherein: Step 7 specifically includes: The server cluster receives all local prototypes P from the client. i Then, the federated learning aggregation algorithm is used to aggregate the local prototypes P from different clients. i Perform a weighted draw to obtain the global prototype used in the next round of communication The global prototype used in the next round of communication The calculation formula is: In formula (11), is the set of all clients containing category k.