Federal image classification method oriented to class imbalance

Abstract

The invention relates to a federated image classification method, in particular to a federated image classification method oriented to class imbalance. The method aims at solving the problem that an existing federal learning method mainly pays attention to data heterogeneity and cannot reduce the influence of class imbalance on model performance. The method specifically comprises the steps that a server initializes a global Gaussian distribution prototype, and a client initializes local model parameters; issuing the global Gaussian distribution prototype to a client; the client generates intra-class variance information by using a Gaussian prototype based on a local data set, and performs local model training through a configured loss function; the client generates a balanced virtual feature set through random sampling; the client uploads the updated local prototype to the server; the server aggregates the local prototypes uploaded by the clients to generate a new global Gaussian distribution prototype for the next round of training; and judging whether a set training round or model convergence is reached or not. The invention belongs to the technical field of image classification.

Description

Technical Field

[0001] The invention relates to a federated image classification method and belongs to the technical field of image classification. Background Art

[0002] Image classification is an important task in the field of computer vision and is widely used in scenarios such as autonomous driving, smart security, and medical image analysis. In recent years, deep learning models have performed well in centralized data environments. However, in practical applications, data is usually distributed across different devices or institutions, making it difficult to train them centrally. To solve this problem, federated learning has become an important technical means.

[0003] Federated learning is a distributed learning paradigm that allows multiple parties to jointly train a global model without sharing data. It not only protects data privacy, but also uses decentralized data for effective training. Therefore, it has been widely used in fields with high privacy protection requirements such as finance and medicine. Through federated learning, each party only needs to share local model updates without transmitting actual data, which effectively reduces the risk of data leakage.

[0004] Although federated learning improves model training efficiency while protecting data privacy, it still faces many challenges. One of the most notable problems is class imbalance. Class imbalance refers to the fact that the number of samples in some categories in the dataset is significantly less than that in other categories. This phenomenon is particularly common in the field of medical image classification. For example, in medical diagnosis tasks, cancer samples are usually far less than healthy samples, resulting in poor classification performance of the model on minority classes, thus affecting the accuracy and reliability of the diagnosis results.

[0005] Existing federated learning methods mainly focus on data heterogeneity, and there is relatively little research on the problem of class imbalance. Compared with centralized learning, the class imbalance problem in federated learning is more complicated, which may exist in the local data of the client and may also differ between clients. Due to the requirements of privacy protection, traditional class imbalance processing methods (such as oversampling and undersampling) are difficult to apply directly. Although some methods have tried to deal with class imbalance, the effect is limited, especially in improving the classification performance of a few categories. For example, some studies suggest uploading local class distribution information for client selection, but this violates the principle of privacy protection. Other methods try to use auxiliary balanced datasets for retraining on the server side, but such datasets are often difficult to obtain in practical applications.

[0006] Therefore, there is an urgent need to design an image classification method that can effectively deal with the problem of class imbalance in a federated learning environment while taking into account data privacy protection. This method should be able to improve the classification performance of minority classes without relying on external data, thereby solving the shortcomings of existing technologies and better meeting the needs of practical applications. Summary of the invention

[0007] In order to solve the problem that the existing federated learning methods mainly focus on data heterogeneity and cannot reduce the impact of category imbalance on model performance, the present invention further proposes a federated image classification method for category imbalance.

[0008] The technical solution adopted by the present invention to solve the above-mentioned problem is: the steps of the present invention include:

[0009] Step 1: The server initializes the global Gaussian distribution prototype, and the client initializes the local model parameters;

[0010] Step 2: Send the global Gaussian distribution prototype to the client;

[0011] Step 3: Based on the local data set, the client generates intra-class variance information using the Gaussian prototype, performs local model training using the configured loss function, and generates a local prototype.

[0012] Step 4: The client generates a balanced virtual feature set through random sampling and retrains the classifier by combining the global and local prototypes;

[0013] Step 5: The client uploads the updated local prototype to the server;

[0014] Step 6: The server aggregates the local prototypes uploaded by each client to generate a new global Gaussian distribution prototype for the next round of training;

[0015] Step 7: Determine whether the set training rounds or model convergence has been reached. If so, the training ends. If not, repeat steps 2 to 6.

[0016] Furthermore, the local Gaussian prototype P i,k :N(μ i,k ,Σ i,k ) is obtained by calculating the mean and covariance of each class of samples to capture the intra-class variance information and enhance the robustness of the classification model:

[0017]

[0018] In formulas (1) and (2), Represents the jth input sample of the kth class of the i-th client The feature vector of .

[0019] Furthermore, category-aware Gaussian prototype learning is introduced on the client side, and its loss function is as follows:

[0020]

[0021] Furthermore, the virtual feature set is generated by fusing the global and local Gaussian prototypes through random sampling, ensuring that the fused features have both global information and can retain local features;

[0022]

[0023] In formula (4), represents a local virtual feature, represents the global virtual feature, and β is the fusion coefficient, which is used to balance the global and local features.

[0024] Furthermore, the client classifier is retrained using the fused features to ensure a balanced class distribution to reduce the bias of the classifier.

[0025] Furthermore, when the server aggregates the uploaded local prototypes, it generates a new global Gaussian prototype using the global mean and covariance of each type of samples.

[0026]

[0027] In formulas (5) and (6), Represents the total number of samples of the kth category of all clients.

[0028] The beneficial effects of the present invention are: 1) Reducing the impact of category imbalance: By introducing a prototype based on Gaussian distribution and a classifier retraining mechanism, the present invention effectively reduces the classification bias caused by category imbalance, especially improves the classification accuracy of minority classes, and solves the problem that traditional methods are prone to bias and misclassification when processing category imbalance data. 2) Improving the adaptability of federated learning: Through the generation and balancing of virtual feature sets, the present invention improves the adaptability of the model to heterogeneous data while maintaining data privacy, and enhances the robustness and generalization ability of the federated learning system. 3) Enhancing the classification performance of minority classes: By using Gaussian prototypes and intra-class variance information, the present invention improves the discrimination ability of the classifier when there are fewer minority class samples. 4) Reducing the burden of data transmission: The present invention only uploads the updated local prototype, which reduces the data transmission bandwidth requirements, reduces the risk of data leakage, and meets the privacy protection requirements of federated learning. 5) Broad application prospects: The present invention can adapt to various category imbalance problems, such as medical image classification, financial risk analysis, etc., and has broad application value and market potential. These beneficial effects not only improve the performance of federated learning models in class imbalance scenarios, but also show significant advantages in protecting data privacy and improving computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart of the present invention;

[0030] Figure 2It is a schematic diagram of the global model training framework of the present invention;

[0031] Figure 3 This is a comparison chart of whether the classifier of the method proposed in the present invention is retrained. DETAILED DESCRIPTION

[0032] Specific implementation method 1: Figures 1 to 3 As shown, a federated image classification method for class imbalance is provided, and the specific steps include:

[0033] Step 1: The server initializes the global Gaussian distribution prototype, and the client initializes the local model parameters;

[0034] Step 2: Send the global Gaussian distribution prototype to the client;

[0035] Step 3: Based on the local data set, the client generates intra-class variance information using the Gaussian prototype, performs local model training using the configured loss function, and generates a local prototype.

[0036] Step 4: The client generates a balanced virtual feature set through random sampling and retrains the classifier by combining the global and local prototypes;

[0037] Step 5: The client uploads the updated local prototype to the server;

[0038] Step 6: The server aggregates the local prototypes uploaded by each client to generate a new global Gaussian distribution prototype for the next round of training;

[0039] Step 7: Determine whether the set training rounds or model convergence has been reached. If so, the training ends. If not, repeat steps 2 to 6.

[0040] Specific implementation method 2: Figures 1 to 3 As shown, the local Gaussian prototype P i,k :N(μ i,k ,∑ i,k ) is obtained by calculating the mean and covariance of each class of samples to capture the intra-class variance information and enhance the robustness of the classification model:

[0041]

[0042] In formulas (1) and (2), Represents the jth input sample of the kth class of the i-th client The feature vector of .

[0043] Specific implementation method three: Figures 1 to 3 As shown in the figure, category-aware Gaussian prototype learning is introduced on the client side, and its loss function is as follows:

[0044]

[0045] Specific implementation method four: Figures 1 to 3 As shown, the virtual feature set is generated by fusing the global and local Gaussian prototypes through random sampling, ensuring that the fused features have both global information and can retain local features;

[0046]

[0047] In formula (4), represents a local virtual feature, represents the global virtual feature, and β is the fusion coefficient, which is used to balance the global and local features.

[0048] Specific implementation method five: Figures 1 to 3 As shown, the client classifier is retrained using the fused features to ensure a balanced class distribution to reduce the bias of the classifier.

[0049] Specific implementation method six: Figures 1 to 3 As shown, when the server aggregates the uploaded local prototypes, it generates a new global Gaussian prototype using the global mean and covariance of each type of samples.

[0050]

[0051]

[0052] In formulas (5) and (6), Represents the total number of samples of the kth category of all clients.

[0053] Example

[0054] A federated image classification method for class imbalance includes the following steps:

[0055] Step 1: Initialize the global Gaussian prototype on the server The client initializes local model parameters The local models of each client can be heterogeneous;

[0056] Step 2: The server converts the global Gaussian prototype Send to each client;

[0057] Step 3: The client performs local training based on the global Gaussian prototype sent down, especially for the processing of minority class samples.

[0058] In the tth communication, the client uses local private data D i , and extract the feature vector of each sample through a feature extraction network (such as ResNet18). Specifically, for the jth input sample of the i-th client Its eigenvector is where f i (·) is the feature extractor of the i-th client. Since the features generated by deep neural networks can be approximated by a mixed Gaussian distribution, in order to better model the intra-class features, we assume that the features of each category k follow a Gaussian distribution P i,k :N(μ i,k ,∑ i,k ), where for class k, μ i,k is the mean of the local Gaussian prototype, ∑ i,k is the covariance matrix. The calculation process is as follows:

[0059]

[0060] Among them, N i,k =|D i,k | represents the number of samples of category k in the i-th client.

[0061] In order to learn a discriminative representation, it is necessary not only to ensure that the prototype boundaries are clear, but also to align between clients to preserve semantic information. Therefore, category-aware Gaussian prototype learning is introduced on the client side, and its loss function is as follows:

[0062]

[0063] Among them, L CGPL represents the loss function of the class-aware Gaussian prototype learning of the i-th client. For k'∈K of the i-th client, and k'≠k. s(·,·) is the similarity metric function used to measure the distance between the input sample and the k-th class mean of the global prototype

[0064]

[0065] Here, λ represents the temperature coefficient hyperparameter, which is set to 0.1 in this experiment.

[0066] Therefore, the local objective function of the i-th client is

[0067] L i =L CE +λL CGPL (11) Among them, L CE is the cross entropy loss function, and λ is a hyperparameter used to balance the impact of category-aware Gaussian prototype learning.

[0068] Step 4: The client uploads the updated local Gaussian prototype to the server.

[0069] Step 5: The server performs Gaussian fusion on the uploaded local Gaussian prototypes, generates a new Gaussian distribution for each category, and updates the global Gaussian prototype. The calculation process is as follows:

[0070]

[0071] in, Represents the total number of samples of the kth class of all clients. The global covariance calculation process of the kth class is as follows:

[0072]

[0073] Step 6: Feature fusion and trainer training.

[0074] Determine whether the set training rounds or global model convergence has been reached; if not, repeat steps 1 to 5; if reached, use the local Gaussian prototype and the global Gaussian prototype to perform random sampling on the local client to generate a virtual feature set and M represents the number of virtual features sampled for each category. This virtual feature set is used to balance the number of samples in each category, especially to increase the number of samples in the minority class. In order to make the virtual samples have both global information and local features, the client performs weighted fusion on the virtual features.

[0075]

[0076] in, represents a local virtual feature, represents the global virtual feature, and β is the fusion coefficient, which is used to balance the global and local features. Since the classifier is the main cause of model deviation, in order to further alleviate the problem of category imbalance, the client fixes the feature extractor and retrains the classifier using the generated fusion features.

[0077] In order to illustrate the effectiveness of the present invention, the following experimental demonstration is carried out. First, the medical image dataset used in the experiment is introduced: the dermoscopy dataset Fed-ISIC2019. The dataset contains a total of 23,247 images, divided into six clients, including eight types of skin lesions. The image size is 200×200×3. We use FedAvg, FedProx, FedBN, FedProto, PRR-Imbalance, FedProc, FPL and FedIIC as benchmark methods. We use BACC (balanced accuracy) as the classification indicator;

[0078] like Figure 3As shown in Table 1, the class-imbalanced federated image classification method proposed in the present invention can obtain higher classification accuracy, which is improved by 15.49%, 16.90%, 8.72%, 7.00%, 12.83%, 19.84%, 9.25% and 12.89% respectively compared with FedAvg, FedProx, FedBN, FedProto, PRR-Imbalance, FedProc, FPL and FedIIC.

[0079] Table 1 Comparison of BACC (%) values ​​of the proposed method and eight benchmark methods

[0080] FedAvg FedProx FedBN FedProto PRR-Imbalance FedProc FPL FedIIC Proposed method 51.57 50.16 58.34 60.06 54.23 47.22 57.81 54.17 67.06

[0081] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. A federated image classification method for class imbalance, characterized in that: The specific steps include: Step 1: The server initializes the global Gaussian distribution prototype, and the client initializes the local model parameters; Step 2: Send the global Gaussian distribution prototype to the client; Step 3: Based on the local data set, the client generates intra-class variance information using the Gaussian prototype, performs local model training using the configured loss function, and generates a local prototype. Step 4: The client generates a balanced virtual feature set through random sampling and retrains the classifier by combining the global and local prototypes; Step 5: The client uploads the updated local prototype to the server; Step 6: The server aggregates the local prototypes uploaded by each client to generate a new global Gaussian distribution prototype for the next round of training; Step 7: Determine whether the set training rounds or model convergence has been reached. If so, the training ends. If not, repeat steps 2 to 6.

2. The method for class imbalance federated image classification according to claim 1, characterized in that: Local Gaussian prototype P i,k :N(μ i,k ,Σ i,k ) is obtained by calculating the mean and covariance of each class of samples to capture the intra-class variance information and enhance the robustness of the classification model: In formulas (1) and (2), Represents the jth input sample of the kth class of the i-th client The feature vector of .

3. The method for class imbalance federated image classification according to claim 1, characterized in that: Category-aware Gaussian prototype learning is introduced on the client side, and its loss function is as follows:

4. The method for class imbalance federated image classification according to claim 1, characterized in that: The virtual feature set is generated by fusing the global and local Gaussian prototypes through random sampling, ensuring that the fused features have both global information and can retain local features; In formula (4), represents a local virtual feature, represents the global virtual feature, and β is the fusion coefficient, which is used to balance the global and local features.

5. The method for class imbalance federated image classification according to claim 1, characterized in that: The client classifier is retrained using the fused features to ensure a balanced class distribution to reduce the bias of the classifier.

6. The method for class imbalance federated image classification according to claim 1, characterized in that: When the server aggregates the uploaded local prototypes, it generates a new global Gaussian prototype using the global mean and covariance of each type of samples. In formulas (5) and (6), Represents the total number of samples of the kth category of all clients.

Images