Class imbalance oriented federated learning image classification method and system

Abstract

The application discloses a class imbalance-oriented federated learning image classification method and system, multiple clients jointly train a global model according to a standard federated learning method; each client divides local training data into head classes and tail classes, calculates corresponding confusion classes of the tail classes, extracts features of the tail classes and the corresponding confusion classes in the local training data based on a class activation map, and obtains tail class-specific features and confusion class-common features; each client fuses the tail class-specific features and the corresponding confusion class-common features to obtain augmented samples of the tail classes, so as to enhance the local training data; each client fine-tunes the global model by using the enhanced local training data, and uploads the global model to a server to further update the global model. In addition, the application designs a new loss function TailDistillation Loss, which can reduce the influence of global class imbalance.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and in particular to a federated learning image classification method and system for class imbalance. Background Technology

[0002] The statements in this section merely refer to the background art related to this invention and do not necessarily constitute prior art.

[0003] With the rapid development of next-generation information technologies such as big data, cloud computing, and artificial intelligence, modern society has placed new demands on privacy protection and information security. Currently, machine learning, especially deep learning, has achieved tremendous success in fields such as computer vision and natural language processing, all of which are built upon massive amounts of data. However, in many application areas, data often exists in a distributed form, and due to legal, regulatory, and copyright requirements, effective data circulation is difficult, forcing people to confront the problem of data silos that are difficult to bridge. Against this backdrop, federated learning has emerged as a key technology for breaking down data silos, preventing privacy leaks, and better protecting data privacy while sharing data value more efficiently.

[0004] Federated learning, as an emerging distributed machine learning paradigm, utilizes decentralized data from multiple clients to jointly train a shared global model under the coordination of a central server. However, a major practical challenge of federated learning is the data heterogeneity caused by the non-independent and identically distributed (i.i.d.) of the local data on each client, which can significantly impact the performance of the global model. Furthermore, the union of all client datasets involved in training often exhibits a long-tailed distribution rather than a balanced distribution. This can lead to the model performing well on classes with a large number of samples (head classes) but poorly on classes with a small number of samples (tail classes). Although many methods have been proposed to mitigate the negative impact of non-independent and identically distributed data, existing solutions often perform poorly on tail classes due to a lack of consideration for the long-tailed distribution. Therefore, researching data heterogeneity and the long-tailed distribution problem in federated learning is of great significance.

[0005] To address data heterogeneity in federated learning, methods can be broadly categorized into client-side and server-side approaches. Client-side methods stabilize local training by adjusting the bias between the local and global models in the parameter space, while server-side methods improve model aggregation performance by refining aggregation strategies. While these methods alleviate data heterogeneity in federated learning to some extent, most rely on the assumption of a balanced global data distribution. However, in the real world, global data distributions are typically imbalanced, often exhibiting a long-tailed distribution with significant class imbalance. Currently, only a few studies have addressed the decline in model accuracy in federated learning due to class imbalance (not specific to long-tailed distributions). Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a federated learning image classification method and system for class imbalance, which aims to improve model performance and image recognition accuracy in federated learning while satisfying privacy protection and data security.

[0007] On the one hand, it provides a federated learning image classification method for class imbalance;

[0008] Federated learning image classification methods for class imbalance include:

[0009] The server sends the current global model parameters to the client. The client uses the received model parameters and local training data to update the model and uploads the updated local model to the server. The server aggregates all the local models uploaded by the clients to obtain the new global model parameters. The server then conducts the next round of communication based on the new global model parameters until the global model converges.

[0010] Each client divides the local training data into head class and tail class, calculates the confusion class corresponding to the tail class, and extracts features from the tail class and its corresponding confusion class in the local training data based on the class activation graph to obtain tail class-specific features and confusion class-general features.

[0011] Each client fuses the tail-class-specific features with the corresponding obfuscated class-general features to obtain fused features, which are then used as augmented samples for the tail class. The augmented samples for the tail class are then added to the local training data to obtain enhanced local training data.

[0012] Each client uses enhanced local training data to retrain the global model, uploads the retrained model to the server, aggregates the models uploaded by all clients, further updates the global model, and proceeds to the next round of communication until the global model converges again.

[0013] Each client uses the finally converged global model to perform image classification on the image to be recognized.

[0014] On the other hand, it provides a federated learning image classification system for class imbalance.

[0015] A federated learning image classification system for class imbalance includes: a server and a client;

[0016] The server sends the current global model parameters to the client. The client uses the received model parameters and local training data to update the model and uploads the updated local model to the server. The server aggregates all the local models uploaded by the clients to obtain the new global model parameters. The server then conducts the next round of communication based on the new global model parameters until the global model converges.

[0017] Each client divides the local training data into head class and tail class, calculates the confusion class corresponding to the tail class, and extracts features from the tail class and its corresponding confusion class in the local training data based on the class activation graph to obtain tail class-specific features and confusion class-general features.

[0018] Each client fuses the tail-class-specific features with the corresponding obfuscated class-general features to obtain fused features, which are then used as augmented samples for the tail class. The augmented samples for the tail class are then added to the local training data to obtain enhanced local training data.

[0019] Each client uses enhanced local training data to retrain the global model, uploads the retrained model to the server, aggregates the models uploaded by all clients, further updates the global model, and proceeds to the next round of communication until the global model converges again.

[0020] Each client uses the finally converged global model to perform image classification on the image to be recognized.

[0021] One of the above technical solutions has the following advantages or beneficial effects:

[0022] This invention investigates the problems of data heterogeneity and severe class imbalance (long-tail distribution) in federated learning, and proposes a novel long-tail federated learning framework. It enhances the feature space of client-side data through a unique method of general compensation and designs a novel loss function that allows the client-side local model to further retain knowledge about the tail classes from the global model during training, thus mitigating the classifier's bias towards the head classes and addressing the class imbalance problem in both the global and local aspects of federated learning. This invention improves the performance of the client-side local model, thereby enhancing the performance of the final aggregated global model, outperforming current federated learning methods designed for heterogeneous and imbalanced data. Attached Figure Description

[0023] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0024] Figure 1 This is a flowchart of the method in Example 1;

[0025] Figure 2 This is a client data distribution diagram for Example 1;

[0026] Figure 3 This is a diagram illustrating the client-side augmented sample generation process in Example 1.

[0027] Figure 4 This is a diagram of the ResNet-8 network structure in Example 1. Detailed Implementation

[0028] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0029] Example 1

[0030] This embodiment provides a federated learning image classification method for class imbalance.

[0031] like Figure 1 As shown, federated learning image classification methods for class imbalance include:

[0032] S101: The server sends the current global model parameters to the client. The client uses the received model parameters and local training data to update the model and uploads the updated local model to the server. The server aggregates all the local models uploaded by the clients to obtain the new global model parameters. The server then conducts the next round of communication based on the new global model parameters until the global model converges.

[0033] S102: Each client divides the local training data into head class and tail class, calculates the confusion class corresponding to the tail class, and extracts features from the tail class and its corresponding confusion class in the local training data based on the class activation graph to obtain tail class-specific features and confusion class-general features.

[0034] S103: Each client fuses the tail-class-specific features with the corresponding confusion-class general features to obtain fused features, and uses the fused features as augmented samples for the tail class; the augmented samples for the tail class are then added to the local training data to obtain enhanced local training data.

[0035] S104: Each client uses enhanced local training data to retrain the global model trained in S101, uploads the retrained model to the server, aggregates the models uploaded by all clients, further updates the global model, and proceeds to the next round of communication until the global model converges again.

[0036] S105: Each client uses the finally converged global model to perform image classification on the image to be identified.

[0037] Further, in S101: the server sends the current global model parameters to the client, the client uses the received model parameters and local training data to update the model, and uploads the updated local model to the server. The server aggregates all the local models uploaded by clients to obtain new global model parameters. The server then conducts the next round of communication based on the new global model parameters until the global model converges, specifically including:

[0038] The server communicates with several clients, and both the server and clients have the same image classification model, which is implemented using ResNet-8. The ResNet-8 network structure diagram is shown below. Figure 4 As shown;

[0039] The initial model parameters of the image classification model on the server are set values;

[0040] The server sends the initial model parameters to all clients;

[0041] Each client receives the initial model parameters sent by the server and sets the initial model parameters into the image classification model. Each client trains its own image classification model using local training data to obtain the trained image classification model. Each client then uploads the parameters of the trained image classification model to the server.

[0042] The server receives model parameters uploaded by all clients, performs a weighted sum of the uploaded model parameters based on the amount of training data from each client, obtains new global model parameters, and distributes the new global model parameters to all clients for the next round of communication, until the server's model parameters converge.

[0043] In this embodiment of the invention, a ResNet-8 network architecture containing a single fully connected layer was selected for training. ResNet-8 is a neural network φ with parameters w = {u, v}. w It consists of feature extractor f u The classifier and two other parts h v The model consists of a classifier that takes input from a global average pooling layer. The parameters of the local model for client k are represented as w. k .

[0044] Multiple benchmark datasets are adapted to the long-tail federated learning setup, a federated learning environment is set up and the model is initialized, the client model is updated and uploaded to the server, and the models uploaded by the client are aggregated on the server.

[0045] Adapt the benchmark datasets CIFAR-10 and CIFAR-100 to a long-tail federated learning setting.

[0046] The CIFAR-10 dataset consists of 60,000 32x32 color images across 10 classes, with 6,000 images per class. There are 50,000 training images and 10,000 test images. The dataset is divided into five training batches and one test batch, each containing 10,000 images. The test batch contains 1,000 randomly selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. In between, each training batch contains 5,000 images from each class.

[0047] The CIFAR-100 dataset contains 100 classes, each with 600 images. Each class has 500 training images and 100 test images. The 100 classes in CIFAR-100 are divided into 20 superclasses. Each image has a "fine" label (of its class) and a "coarse" label (of its superclass).

[0048] The two datasets were reshaped into long-tail versions, CIFAR-10-LT and CIAFR-100-LT, with different imbalance factors (IF = 100, 50, 10) to suit long-tail federated learning settings. The imbalance factor IF is the ratio of the number of samples in the largest class to the number of samples in the smallest class across all clients. A Dirichlet distribution was used to partition the non-IID data among different clients. The heterogeneity α on CIFAR-10-LT and CIAFR-100-LT was set to 0.5, representing their local data. The client data distribution is shown in the figure below. Figure 2 As shown.

[0049] To compute class activation maps in subsequent steps, this embodiment of the invention uses the ResNet-8 deep convolutional neural network architecture, which contains a single fully connected layer, as the base model for CIFAR-10 / 100-LT. In this model, the classifier takes input from a global average pooling layer. ResNet-8 is a neural network φ with parameters w = {u, v}. w It consists of feature extractor f u The classifier and two other parts h v Composition. The parameters of the local model of client k are represented as w. k By default, this embodiment of the invention runs a total of 200 rounds of global communication, with a total of 20 clients, and an active user ratio C = 40% in each round. For each real sample in the tail class of each client, this embodiment of the invention selects N. a =3. In this embodiment of the invention, the local training batch size is set to 128, the learning rate is 0.001, and the optimizer is SGD.

[0050] The server randomly selects a subset S from K clients. t Participate in the t-th round of communication and send the global model w to the selected client. t (In the first round of communication, the global model is randomly initialized by the server.) S t Each client in the process receives the global model w sent by the server. t Then, it will use its local data. The model is updated using stochastic gradient descent (SGD) with a learning rate of η for k = 1, ..., K:

[0051]

[0052] After the local update, the client will update the obtained local model. The data is uploaded to the server, where the server aggregates the local model parameters from each client using a weighted average to obtain the global model for the (t+1)th round of communication.

[0053]

[0054] Furthermore, since models tend to favor the head class (with sufficient samples) and neglect the tail class when data exhibits a long-tailed distribution, this embodiment of the invention designs a Tail Distillation Loss based on the original loss function. This allows the client-side local model to further retain knowledge about the tail class from the global model during training, thereby reducing the model's bias towards the head class. Specifically, this embodiment of the invention considers the CrossEntropy Loss function, widely used in multi-class classifiers, as the basic term, and defines the tail distillation loss function, i.e., the total loss function, as follows:

[0055] L TD = (1-λ)*CE(gt,σ(z) s ;T=1))+λ*CE(σ(z′ t ;T=τ),σ(z′ s ;T=τ)) (3)

[0056] Where gt is the true label of the local training data, and z s It is the prediction score of the local model on the local training data, z′ t It is the prediction score of the global model on the local tail class training data, z′ s σ is the prediction score of the local model on the local tail class training data, CE is the cross-entropy loss function, σ is the softmax function parameterized by temperature T, and λ is the tradeoff hyperparameter, which takes values ​​in the range [0,1].

[0057] Further, in S102: each client divides its local training data into a head class and a tail class, specifically including:

[0058] Local training data includes: images with known classification labels;

[0059] Construct a Cartesian coordinate system. The horizontal axis represents the known classification labels, which are arranged from left to right in descending order of the number of images. The vertical axis represents the number of images corresponding to each label, with the vertical axis being zero at the end closest to the origin of the coordinate system. Plot the distribution curve of the number of images in the Cartesian coordinate system.

[0060] The classification label with the fewest images is selected as the tail class label, and the remaining classification labels are all head class labels. The local training data corresponding to the head class labels is the head class training data, and the local training data corresponding to the tail class labels is the tail class training data.

[0061] It should be understood that in the distribution curve of the number of images, the number of images corresponding to the tail category label is the smallest, which is similar to "a tail" in the distribution curve diagram.

[0062] Furthermore, the calculation of the obfuscation class corresponding to the tail class specifically includes:

[0063] In the tail class training data labeled a1 in the p-th client, there are a total of M images;

[0064] Take the q-th image from the tail class training data labeled a1, input it into the image classification model of the p-th client, and output the predicted sub-labels of the q-th image as a1, a2, ..., a n The probability value;

[0065] Similarly, the M images in the tail class training data labeled a1 are sequentially input into the image classification model of the p-th client, and the probability values ​​of the M images being predicted by the sub-label are output.

[0066] Calculate the sub-labels of M images as a r The average probability values ​​are sorted from highest to lowest, and the top Q image sub-labels with the largest average values ​​are selected as the confusion classes corresponding to class a1. Here, r ranges from 2 to n. Q is a set value, for example, Q = 2.

[0067] For example, the calculation of the obfuscation class corresponding to the tail class by each client specifically includes:

[0068] The first client contains 400 images in the training data for the cat tail category;

[0069] The first image in the training data labeled as cat is input into the image classification model of the first client. The output probability values ​​of the first image being predicted as cat, fox, ..., dog are 0.6, 0.2, ..., 0.12, respectively.

[0070] The second image in the training data labeled as cat is input into the image classification model of the first client. The output probability values ​​of the second image being predicted as cat, fox, ..., dog are 0.55, 0.23, ..., 0.14, respectively.

[0071] Similarly, the 400th image in the training data labeled as cat is input into the image classification model of the first client, and the output probability values ​​of the 400th image being predicted as cat, fox, ..., dog are 0.62, 0.16, ..., 0.10 respectively.

[0072] Calculate the average probability values ​​of the sub-labels of 400 images as fox, ..., dog, sort the average values ​​in descending order, and select the top two image labels with the largest average values ​​as the confusion class corresponding to cat.

[0073] Furthermore, the step of extracting features from the tail class and its corresponding confusion class in the local training data based on the class activation graph to obtain tail class-specific features and confusion class-general features specifically includes:

[0074] The tail class training data is input into the image classification model of the current client, the class activation mapping is calculated, and the tail class-specific features are separated according to the set threshold.

[0075] The confused training data is input into the image classification model of the current client, the class activation map is calculated, and the general features of the confused class are separated according to the set threshold.

[0076] It should be understood that, firstly, it is necessary to determine how to select the class from which to extract common features, given a tail class. This embodiment of the invention considers classes relatively close to a given tail class in the feature space to be more likely to be confused with it, which has a significant impact on recovering the distribution of the tail class. Specifically, each client uses the global model obtained in S101 to predict the classification scores of all other classes for each training sample in the tail class, and finds its confusion class C by sorting the average classification scores of all other classes for the samples in the tail class. f .

[0077] Furthermore, the step of extracting features from the tail class and its corresponding confusion class in the local training data based on the class activation graph to obtain tail class-specific features and confusion class-general features specifically includes:

[0078] First, calculate the score y of class c. c (Before softmax) The feature map A of the l-th channel relative to the output of the last convolutional layer in the image classification model. l gradient

[0079] Then, global average pooling is performed on the gradient along the width and height dimensions (indexed as i and j, respectively) to obtain the neuron weights.

[0080]

[0081] Here, Z represents i*j;

[0082] Finally, based on neuron weights For feature activation mapping A l We perform a weighted combination followed by an activation function operation to obtain the class activation map M of class c. c :

[0083]

[0084] Where BeLU represents the activation function;

[0085] Class activation map M c The values ​​are normalized to the range of 0 to 1. Therefore, given a threshold of 0 < ξ < 1, the class activation map M is... c Decomposed into and The feature vector is divided into two parts: general features for confusion classes and tail-specific features.

[0086] Common features of obfuscation

[0087]

[0088] Tail-like characteristics

[0089]

[0090] Where ⊙ is the Hadamard product between two tensors, and sgn is the sign function, sgn(x) = 1 when x ≥ 0 and sgn(x) = 0 when x < 0.

[0091] The beneficial effect of the above technical solution is that it can distinguish between class-general features that are not significantly different between categories and class-specific features that are significantly different between categories.

[0092] Further, S103: Each client fuses the tail-class-specific features with the corresponding confusion-class general features to obtain fused features, and uses the fused features as augmented samples for the tail class; the augmented samples for the tail class are then added to the local training data to obtain enhanced local training data, specifically including:

[0093] Each client fuses the tail-class-specific features with the corresponding obfuscated class-general features in a linear manner to obtain the fused features.

[0094] The tail class label is used as the label corresponding to the fusion feature, and the fusion feature and the label corresponding to the fusion feature are used as augmented samples of the tail class.

[0095] Augmented samples of the tail class are added to the local training data to obtain enhanced local training data.

[0096] Furthermore, the linear method can be: weighted summation.

[0097] It should be understood that class-specific features are separated from a given tail class image, and class-general features are separated from the corresponding confused classes, using class activation maps. To suppress noise and potential bias, the two are fused linearly in the feature space, and the fusion process is guided by randomly generated combination ratios, thereby generating augmented samples for a given tail class to recover its inherent data distribution, such as... Figure 3As shown. For each real sample in the tail class, a total of N are generated. a An augmented sample.

[0098] Augmented samples can be generated at any stage of the network; however, compared to shallow features, deep features have a larger receptive field and stronger semantic information, which greatly helps in the feature fusion of tail classes and their confusion classes. This embodiment of the invention chooses to use features prior to the last average pooling layer to help generate augmented samples to improve classification performance. Since the average pooling layer accumulates features from all spatial locations, the spatial distribution of class-general and class-specific features becomes irrelevant in the augmented samples. Therefore, when combining features, only the ratio of the two classes of features needs to be given.

[0099] Furthermore, S104: Each client uses enhanced local training data to retrain the global model trained in S101, uploads the retrained model to the server, aggregates the models uploaded by all clients, further updates the global model, and proceeds to the next round of communication until the global model converges again, specifically including:

[0100] Each client uses enhanced local training data to train its own image classification model;

[0101] Each client uploads the trained model parameters to the server;

[0102] The server performs a weighted sum of the model parameters of all clients based on the amount of training data from each client, and obtains new global model parameters. The server then distributes the new global model parameters to all clients and initiates the next round of communication until the server's model parameters no longer change, at which point training stops.

[0103] The server will send the final model parameters to all clients.

[0104] Each client uses local data, including the original samples and the augmented samples of the generated tail classes, to fine-tune the global model trained in S101. The fine-tuned model is then uploaded to the server for aggregation to further update the global model, and the next round of communication is performed until the global model converges again.

[0105] Example 2

[0106] This embodiment provides a federated learning image classification system for class imbalance.

[0107] A federated learning image classification system for class imbalance includes: a server and a client;

[0108] The server sends the current global model parameters to the client. The client uses the received model parameters and local training data to update the model and uploads the updated local model to the server. The server aggregates all the local models uploaded by the clients to obtain the new global model parameters. The server then conducts the next round of communication based on the new global model parameters until the global model converges.

[0109] Each client divides the local training data into head class and tail class, calculates the confusion class corresponding to the tail class, and extracts features from the tail class and its corresponding confusion class in the local training data based on the class activation graph to obtain tail class-specific features and confusion class-general features.

[0110] Each client fuses the tail-class-specific features with the corresponding obfuscated class-general features to obtain fused features, which are then used as augmented samples for the tail class. The augmented samples for the tail class are then added to the local training data to obtain enhanced local training data.

[0111] Each client uses enhanced local training data to retrain the global model, uploads the retrained model to the server, aggregates the models uploaded by all clients, further updates the global model, and proceeds to the next round of communication until the global model converges again.

[0112] Each client uses the finally converged global model to perform image classification on the image to be recognized.

[0113] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A federated learning image classification method for class imbalance, characterized in that, The server sends the current global model parameters to the client, the client uses the received model parameters and the local training data to update the model, and uploads the updated local model to the server, the server aggregates all the local models uploaded by the clients to obtain new global model parameters, and the server performs the next round of communication based on the new global model parameters until the global model converges. Each client divides the local training data into head classes and tail classes, calculates the corresponding confusion classes of the tail classes, and extracts features of the tail classes and their corresponding confusion classes in the local training data based on the class activation map to obtain tail class-specific features and confusion class-common features, specifically including: Each client fuses the tail class-specific features with the corresponding confusion class-common features to obtain fusion features, and uses the fusion features as augmented samples of the tail classes; the augmented samples of the tail classes are supplemented into the local training data to obtain enhanced local training data; First, a class score is calculated with respect to a gradient of a feature map of an nth channel of an output of a last convolutional layer in an image classification model Then, global average pooling is performed on the gradient in the width and height dimensions to obtain the neuron weight : wherein The meaning of ; Finally, based on the neuron weights , the feature activation maps are weighted combined and followed by an activation function operation to obtain class activation maps :​ wherein, represents an activation function; normalizing the values of the class activation map to the range 0 to 1, thus, given a threshold the class activation map is decomposed into and two parts, separating the feature vector into a confusable class general feature and a tail class specific feature; Confusing class-wide features : tail-specific features : wherein is the Hadamard product between two tensors, sgn is the sign function, when , , , ; Each client re-trains the global model obtained by training using the enhanced local training data, uploads the re-trained model to the server, the server aggregates all the models uploaded by the clients to further update the global model, and performs the next round of communication until the global model converges again. Each client uses the finally converged global model to perform image classification on the to-be-recognized image. The server sends the current global model parameters to the client, the client uses the received model parameters and the local training data to update the model, and uploads the updated local model to the server, the server aggregates all the local models uploaded by the clients to obtain new global model parameters, and the server performs the next round of communication based on the new global model parameters until the global model converges, specifically including:

2. The class imbalance oriented federated learning image classification method of claim 1, wherein, The server communicates with a plurality of clients respectively, and the same image classification model is arranged in the server and the clients; The initial model parameters of the image classification model of the server are set values; The server sends the initial model parameters to all the clients; Each client receives the initial model parameters sent by the server and sets the initial model parameters to the image classification model, each client trains the image classification model of itself using the local training data to obtain a trained image classification model, and each client uploads the parameters of the trained image classification model to the server; The server receives the model parameters uploaded by all the clients, performs weighted summation on the uploaded model parameters of each client according to the number of training data of each client to obtain new global model parameters, and distributes the new global model parameters to all the clients for the next round of communication until the model parameters of the server converge. Each client divides the local training data into head classes and tail classes, specifically including:

3. The class imbalance oriented federated learning image classification method of claim 1, wherein, The local training data includes images with known classification labels. ​ A rectangular coordinate system is constructed, the horizontal coordinate is a known classification label, and the classification labels of the horizontal coordinate are arranged from left to right in the order of decreasing image quantity; the vertical coordinate is the number of images corresponding to each label, and the vertical coordinate is zero near the origin of the coordinate system; in the rectangular coordinate system, a distribution curve of the number of images is drawn; The classification label with the least number of images is selected as the tail classification label, and the remaining classification labels are all head classification labels; the local training data corresponding to the head classification label is the head class training data, and the local training data corresponding to the tail classification label is the tail class training data.

4. The class imbalance oriented federated learning image classification method of claim 1, wherein, The tail class corresponding to the confusion class is calculated, specifically including: The total number of images in the tail class training data labeled in the first client is . ​ The tail class training data labeled with the first image, is input into the image classification model of the first client, and the first image is output, which is predicted to be labeled with 、 、……、 probability value; And so on, labeling as In the tail class training data Each image is sequentially input into the first... In the image classification model for each client, the output The probability value of an image being predicted to have a sub-label; calculate Image sub-labels are The average of the probability values ​​is sorted from highest to lowest, and the values ​​with the largest averages are selected. Image sub-labels, as The obfuscation class corresponding to the class, where, The range of values ​​for is 2. ~ .

5. The class imbalance oriented federated learning image classification method of claim 1, wherein, The tail class and its corresponding confusion class in the local training data are extracted based on the class activation map to obtain tail class specific features and confusion class general features, specifically including: The tail class training data is input into the image classification model of the current client to calculate the class activation mapping, and the tail class specific features are separated according to the set threshold; The confusion class training data is input into the image classification model of the current client to calculate the class activation mapping, and the confusion class general features are separated according to the set threshold.

6. The class imbalance oriented federated learning image classification method of claim 1, wherein, Each client fuses the tail class specific features with the corresponding confusion class general features to obtain fusion features, and takes the fusion features as augmented samples of the tail class; The augmented samples of the tail class are supplemented into the local training data to obtain enhanced local training data, specifically including: Each client fuses the tail class specific features with the corresponding confusion class general features in a linear manner to obtain fusion features; The tail class label is taken as the label corresponding to the fusion features, and the fusion features and the label corresponding to the fusion features are taken as augmented samples of the tail class; The augmented samples of the tail class are supplemented into the local training data to obtain enhanced local training data.

7. The class imbalance oriented federated learning image classification method of claim 1, wherein, Each client uses the enhanced local training data to retrain the global model obtained by training, uploads the retrained model to the server, the server aggregates all the model parameters uploaded by the clients, further updates the global model, and performs the next round of communication until the global model converges again, specifically including: Each client uses the enhanced local training data to train its own image classification model; Each client uploads the trained model parameters to the server; The server weights and sums all the model parameters of the clients according to the number of training data of each client to obtain new global model parameters, and the server distributes the new global model parameters to all the clients and performs the next round of communication until the model parameters of the server no longer change and the training stops; The server distributes the final model parameters to all the clients.

8. The class imbalance oriented federated learning image classification method of claim 1, wherein, The total loss function of the model in the training process is defined as: where, is the true label of the local training data, is the prediction score of the local model on the local training data, is the prediction score of the global model on the local tail class training data, is the prediction score of the local model on the local tail class training data, is the cross-entropy loss function, is the softmax function parameterized by temperature , is the trade-off hyperparameter, which takes values in the range .

9. A federated learning image classification system oriented to class imbalance, characterized in that, including: The server and the client; The server sends the current global model parameters to the client, the client updates the model using the received model parameters and the local training data, and uploads the updated local model to the server, the server aggregates all the local models uploaded by the clients to obtain new global model parameters, and the server performs the next round of communication based on the new global model parameters until the global model converges; Each client divides the local training data into head classes and tail classes, calculates the confusion classes corresponding to the tail classes, performs feature extraction on the tail classes and the corresponding confusion classes in the local training data based on the class activation map, and obtains tail-specific features and confusion-class general features, specifically including: First, computational classes fractions Compared to the output of the last convolutional layer in the image classification model, the first... Feature map of each channel gradient ; Then, global average pooling is performed on the gradient in the width and height dimensions to obtain the neuron weight : wherein the meaning is ; Finally, based on neuron weights , for feature activation mapping Perform a weighted combination, followed by an activation function operation, to obtain the class. Class activation mapping : wherein, denotes an activation function; normalizing the values of the class activation map to the range 0 to 1, thus, given a threshold the class activation map is decomposed into and two parts, separating the feature vector into a confusion class general feature and a tail class specific feature; Confusing class-wide features : tail-specific feature : wherein is the Hadamard product between two tensors, sgn is the sign function, when , , , ; Each client fuses the tail-specific features with the corresponding confusion-class general features to obtain fused features, and uses the fused features as augmented samples of the tail classes; the augmented samples of the tail classes are supplemented to the local training data to obtain enhanced local training data; Each client uses the enhanced local training data to retrain the global model obtained by training, uploads the retrained model to the server, the server aggregates the models uploaded by all clients, further updates the global model, and performs the next round of communication until the global model converges again; Each client uses the finally converged global model to perform image classification on the to-be-recognized image.

Images