Data class adaptive balancing federated learning method fusing dual contrast loss

Abstract

The application discloses a data class adaptability balancing federated learning method fusing double contrast loss and belongs to the technical field of federated learning. In the data preprocessing stage, the application solves the problem of class distribution imbalance in the training sample through oversampling operation, guarantees the learning effect of the model on various class samples, thereby reducing the deviation degree of the model, and optimizes the sample features learned by the model in the feature space by designing two loss functions with contrast properties, i.e., intra-class compactness loss and supervised contrast loss, solves the overfitting problem possibly caused by oversampling, and thereby improves the classification performance and generalization ability of the model.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, specifically a data category adaptive balancing federated learning method that incorporates dual contrastive loss. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, data privacy and security issues have received increasing attention, especially in the medical field, where the arbitrary collection and sharing of data is no longer feasible. To address this challenge, federated learning, as a paradigm of edge intelligence, has gradually gained widespread attention. Federated learning enables parties to jointly train machine learning models without sharing the original data, breaking down the limitations of data silos while ensuring privacy protection and data security. However, in practical applications, datasets from different clients are often not independent and identically distributed, which brings new challenges.

[0003] To address the challenges of data heterogeneity, a range of methods have been developed to improve the performance of federated learning. Some methods approach the global optimum by optimizing local training or regularizing client-side training, but these methods may lead to privacy breaches or increased communication and computational overhead. Other methods take a server-centric approach, such as FedDisco, which considers dataset differences and sizes, and FedLC, which selects beneficial models for aggregation through label clustering. However, these methods may have limited effectiveness under severe data heterogeneity or may ignore potentially useful information, affecting the model's generalization ability.

[0004] To address the limitations of existing research methods and their inability to fundamentally resolve the issue of non-independent and identically distributed data among clients, some studies have begun to focus on incorporating data preprocessing techniques to further improve the performance of federated learning. For example, FRAug proposed using data augmentation in the low-dimensional feature space to enhance model performance. These optimization operations targeting data during the preprocessing stage are not only efficient but also avoid additional communication overhead. However, these approaches do not consider the potential overfitting problem that data augmentation can introduce, which may cause the model to overfit to noise in the training set, thus losing its generalization ability. Summary of the Invention

[0005] To address the problem of imbalanced client data categories and resulting model performance degradation caused by the diversity of client-generated data and different collection methods in federated learning, this invention provides a data category adaptive balancing federated learning method that incorporates dual contrastive loss.

[0006] This invention provides a class-adaptive balanced federated learning method that incorporates dual contrastive loss, applicable to federated learning frameworks, including a central server and a central server... There are 10 clients; each client has its own independent local original dataset; at the beginning of each round of federated learning communication, the central server sends the global model to all clients, and after the clients have finished training, it receives the local models from the clients and aggregates all the received client local models to form a new global model; the federated learning method includes the following steps:

[0007] The federated learning method includes the following steps:

[0008] Step 1, in the During the turn, the client After receiving the global model from the central server, the local raw dataset is oversampled.

[0009] Step 2, Client On the oversampled dataset, the client-side local training loss function is obtained by combining the classification loss function and the double contrast loss function, and the client-side local model is trained to obtain the updated local model.

[0010] Step 3: The client sends the updated local model to the central server, which then aggregates the data according to the client's data ratio to form a new model. Round global model.

[0011] Furthermore,

[0012] The specific steps of step 1 are as follows:

[0013] Step 1.1: Before each round of training, the client performs a sampling operation, first counting the number of samples of each class in the local original dataset, using the following formula:

[0014] ,

[0015] in, For indicator functions, Indicates client The categories contained in it are The number of samples, Indicates the first The local raw dataset of each client, express The samples in Indicates corresponding to The label. Step 1.2: Find the category with the most samples in the local original dataset, and obtain a target sample value based on the number of samples in that category:

[0016] ,

[0017] in, Indicates the target sample value. This represents the total number of sample categories in the original local dataset. It is a hyperparameter used to determine a reasonable standard to measure the local dataset, ensuring that the class distribution of the dataset is relatively consistent and reasonable.

[0018] Step 1.3, for each category Based on the number of samples in that category With target value The magnitude of the values ​​determines whether oversampling should be performed. The specific operation is as follows:

[0019] like If so, an oversampling operation is performed, and the number of oversampled samples is set to... The oversampled subset is represented as:

[0020] ,

[0021] in, For the client The above category is The sample set, To obtain from the sample set The oversampled subset obtained by random uniform sampling includes One sample;

[0022] Step 1.4: After oversampling, the oversampled subset obtained by randomly sampling from the local original dataset will be merged back into the local original dataset.

[0023] Furthermore, in step 2, the dual contrastive loss function consists of two loss functions: intra-class compactness loss and supervised contrastive loss.

[0024] The formula for the client-side local training loss function is as follows:

[0025] ,

[0026] in, Hyperparameters used to control the influence of intra-class compactness loss and supervised contrast loss; Represents cross-entropy loss, Intra-class compactness loss, This indicates that there is a supervised comparison of losses. Local training loss on the client side.

[0027] Furthermore, the intra-class compactness loss consists of an inter-class distance term and a class center distance term;

[0028] The formula for the inter-class distance term is:

[0029] ,

[0030] in, , representing the sample features extracted from the model, the sample tags , For the client Local model parameters; Indicates the distance between classes; This represents the samples extracted from the model. Features;

[0031] The formula for the class center distance term is:

[0032] ,

[0033] ,

[0034] in, Represents the class center distance; Represents class-centric features;

[0035] The formula for the intra-class compactness loss function is as follows:

[0036] ,

[0037] in, Intra-class compactness loss, This represents hyperparameters.

[0038] Furthermore, the supervised contrastive loss compares the features of samples within a batch. The contrast operation decreases the distance between positive sample pairs and increases the distance between negative sample pairs. The distance between positive sample pairs is defined as follows:

[0039] ,

[0040] in, As the decision boundary, This represents the distance between positive sample pairs;

[0041] For a batch and samples All samples with different labels, they are related to the sample The distance between the resulting negative sample pairs is defined as follows:

[0042] ,

[0043] in, Indicates the distance between negative sample pairs;

[0044] The supervised contrastive loss is formed by adding the distances between positive and negative sample pairs, as shown in the following formula:

[0045] ,

[0046] in, This indicates that there is a supervised comparison loss.

[0047] Furthermore, in step 2, based on the client's local training loss function, the client updates the local model by performing gradient descent as follows:

[0048] ,

[0049] in, It is the learning rate for model training. Indicates client The current local model, This represents the client-side local gradient.

[0050] The improvement of the present invention compared with the prior art is as follows:

[0051] 1) In the data preprocessing stage, the present invention performs oversampling operation, which increases the frequency of occurrence of minority class samples in training, balances the class distribution of training samples, ensures that each class can receive sufficient attention and fair treatment in training, thereby improving the generalization ability of the model.

[0052] 2) This invention designs two contrastive loss functions: intra-class compactness loss and supervised contrastive loss. These ensure the compactness of features among samples of the same class while increasing the differences between samples of different classes. These two contrastive loss functions can solve the overfitting problem that may be caused by oversampling of duplicate samples and promote the model to learn more effective feature representations in the latent feature space.

[0053] 3) The method proposed in this invention can effectively solve the performance degradation problem when client data is not independently and identically distributed. Compared with other advanced algorithms, the method of this invention shows a more significant advantage in performance, without requiring additional communication overhead, and without increasing the risk of data privacy leakage. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of a data category adaptive balanced federated learning method that incorporates dual contrastive loss, designed according to the present invention.

[0055] Figure 2 This is a schematic diagram of the algorithm framework for a data category adaptive balance federated learning method that incorporates dual contrastive loss, as designed in this invention. Detailed Implementation

[0056] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the embodiments described.

[0057] This invention can reduce the model bias when the class distribution in the client data training samples is unbalanced, ensure that each class receives sufficient attention and fair treatment during training, improve the model's generalization ability, solve the overfitting problem, and improve the model's classification performance and generalization ability.

[0058] This invention discloses a data category adaptive balanced federated learning method that incorporates dual contrastive loss, and its network model is as follows: Figure 1 As shown, the model consists of a client layer and a central server layer, wherein:

[0059] Client layer: This layer consists of The system consists of several clients, each with its own independent local dataset, and the data between clients is not shared. Due to the diversity of how client data is generated or collected, the local datasets of each client exhibit an unbalanced label distribution. Furthermore, due to this diversity, the data distribution among clients exhibits a non-independent and identically distributed characteristic.

[0060] Central Server Layer: The central server layer consists of a central server responsible for communicating with all clients. The central server has two main functions: at the beginning of each round of federated learning communication, it sends the global model to all clients; after the clients have finished training, it receives the local models from the clients and aggregates all the received client local models to form a new global model.

[0061] This invention provides a framework for a class-adaptive balanced federated learning method that incorporates dual contrastive loss, such as... Figure 2 As shown, it includes the following steps:

[0062] Step 1, in the During the turn, the client After receiving the global model from the central server, the local original dataset is oversampled to bring the originally severely imbalanced distribution of sample classes into a more balanced state.

[0063] The specific steps of step 1 are as follows:

[0064] Step 1.1: The client's local original dataset is a non-independent, identically distributed dataset. This step is executed before each round of training on the client side. The client will count the number of samples of each class in the local original dataset. The statistical formula is as follows:

[0065] ,

[0066] in, For indicator functions, Indicates client The categories contained in it are The number of samples, Indicates the first The local raw dataset of each client, express The samples in Indicates corresponding to The tag.

[0067] Step 1.2: Next, find the category with the largest number of samples in the local original dataset, and obtain a target sample value based on the number of samples in that category:

[0068] ,

[0069] in, Indicates the target sample value. This represents the total number of sample categories in the original local dataset. It is a hyperparameter used to determine a reasonable standard to measure the local dataset, ensuring that the class distribution of the dataset is relatively consistent and reasonable.

[0070] Step 1.3, for each category Based on the number of samples in that category With target value The magnitude of the values ​​determines whether oversampling should be performed. The specific operation is as follows:

[0071] like This indicates that the number of samples in the current category is too small, and oversampling is required. The number of samples to be oversampled should be set to... An oversampled subset can be represented as:

[0072] ,

[0073] in, For the client The above category is The sample set, To obtain from the sample set The oversampled subset obtained by random uniform sampling includes One sample.

[0074] Step 1.4: After oversampling, the oversampled subset obtained by randomly sampling from the local original dataset is merged back into the local original dataset. This merging operation not only ensures that the oversampled samples are used for local model training, but also increases the diversity of the samples. This diversity helps the local model to better capture the features and distribution of the data during training.

[0075] Step 2: After the data preprocessing in Step 1 above, the client... In addition to using the classification loss function, a dual contrastive loss function will be combined to train the local model on the oversampled dataset, resulting in an updated local model.

[0076] The dual contrastive loss function consists of two loss functions: intra-class compactness loss and supervised contrastive loss. Intra-class compactness loss is mainly used to optimize the distribution of samples within the same class in the feature space, promote the model's learning ability for samples of the same class, and improve the model's generalization ability. The goal of supervised contrastive loss is to maximize the similarity between samples of the same class and minimize the similarity between samples of different classes, so that the model can better distinguish samples of different classes in the feature space.

[0077] 1) The intra-class compactness loss consists of two parts: the inter-class distance term and the class center distance term.

[0078] The first term of the intra-class compactness loss is the inter-class distance term, which is mainly used to reduce the distance between low-dimensional features obtained by model projection of samples of the same class. Its formula can be defined as:

[0079] ,

[0080] in, , representing the sample features extracted from the model, the sample tags , For the client Local model parameters; Indicates the distance between classes; This represents the samples extracted from the model. Features;

[0081] This term reflects the average distance between features of samples within the same category. Minimizing this term allows the model to focus more on the distribution features of samples within a category, rather than simply memorizing duplicate samples. This improves the model's generalization ability, enhances its ability to distinguish between categories, and reduces overfitting to duplicate samples.

[0082] The second term in the intra-class compactness loss is the class center distance term. Similar to the idea of ​​clustering, this term first calculates the class center feature of each class based on the category. The calculation method is as follows:

[0083] ,

[0084] in, This represents the class center feature;

[0085] The class center distance loss is calculated as the distance from a sample feature to the class center of that class, i.e.:

[0086] ,

[0087] in, Represents the class center distance;

[0088] Intra-class compactness loss is composed of inter-class distance and class center distance terms, expressed through hyperparameters. The modulated hyperparameters are in the following form:

[0089] ,

[0090] in, Intra-class compactness loss, This represents hyperparameters.

[0091] 2) Supervised contrastive loss compares the features of samples within a batch; the larger the batch, the better the effect. The main contrastive operation is to reduce the distance between features of positive sample pairs and increase the distance between features of negative sample pairs. The distance between positive sample pairs is defined as follows:

[0092] ,

[0093] in, As the decision boundary, This represents the distance between positive sample pairs;

[0094] For a batch and samples All samples with different labels, they are related to the sample The distance between the resulting negative sample pairs is defined as follows:

[0095] ,

[0096] in, Indicates the distance between negative sample pairs;

[0097] The supervised contrastive loss is formed by adding the distances between positive and negative sample pairs:

[0098]

[0099] in, This indicates that there is a supervised comparison loss.

[0100] 3) Combining the cross-entropy loss (classification loss function) with the intra-class compactness loss and supervised contrastive loss, we obtain the client-side local training loss function, as shown below:

[0101] ,

[0102] in, Hyperparameters used to control the influence of intra-class compactness loss and supervised contrast loss; Represents cross-entropy loss, Local training loss on the client side.

[0103] 4) Based on the obtained client-side local training loss function, the client updates the local model by performing gradient descent as follows:

[0104] ,

[0105] in, It is the learning rate for model training; Indicates client The current local model, This represents the client-side local gradient.

[0106] Step 3: After training is complete, the client sends the updated local model to the central server, which then aggregates the data according to the client's data ratio to form a new model. Round global model.

[0107] Based on the above, this invention presents a simulation experiment of a class-adaptive balanced federated learning method incorporating dual contrastive loss in a real-world scenario. The experimental dataset used is the CIFAR10 dataset, and the network model consists of three parts: a basic encoder, a projector, and an output layer. The basic encoder includes two convolutional layers, two max-pooling layers, and two fully connected layers with ReLU activation functions. The projector consists of a two-layer multilayer perceptron, and the output layer is a fully connected layer. The method of this invention is compared with three existing federated learning algorithms: FedAvg, FedProx, and MOON.

[0108] The dataset is distributed to each client using a label-imbalanced Dirichlet distribution method. When there are 10 clients, the concentration parameters under different Dirichlet distributions are... , , The learning test accuracy of all methods is shown in Table 1. The larger the concentration parameter value, the lower the data heterogeneity. It can be seen that as the heterogeneity decreases, the leading effect of the method of this invention becomes more significant. At that time, the accuracy of the present invention on CIFAR10 was more than 3.81% higher than other methods. At that time, the accuracy of the method of this invention on CIFAR10 was more than 9.26% higher than that of other methods. This is because in the case of mildly heterogeneous data, the local dataset of the client is likely to cover samples of all categories. In this case, the oversampling strategy can effectively adjust the imbalance distribution between categories, thereby ensuring that the model can fully learn the knowledge of each category.

[0109] Table 1. Final model accuracy results of various algorithms on the CIFAR10 dataset with different concentration parameters.

[0110]

[0111] Further investigation was conducted on the CIFAR10 dataset to examine the accuracy of all methods under varying numbers of clients, thus validating the scalability of the proposed method. The accuracy of all methods is shown in Table 2. As the number of clients increases, the accuracy of FedProx and MOON is close to that of FedAvg, indicating that simply optimizing the client training process has limited effect on improving model performance.

[0112] Table 2. Final model accuracy results for various algorithms on the CIFAR10 dataset with different numbers of clients.

[0113]

[0114] Among these four federated learning methods, the method of this invention consistently maintains the highest accuracy, highlighting the scalability and robustness of the present invention.

[0115] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention.

Claims

1. A class-adaptive balanced federated learning method incorporating dual contrastive loss, characterized in that, The method is applicable to federated learning frameworks, including central servers. There are 10 clients; each client has its own independent local raw dataset; at the beginning of each round of federated learning communication, the central server sends the global model to all clients, and after the clients have finished training, it receives the local models from the clients and aggregates all the received client local models to form a new global model. The local original dataset is an image dataset; The local model consists of three parts: a basic encoder, a projector, and an output layer. The basic encoder includes two convolutional layers, two max pooling layers, and two fully connected layers with ReLU activation functions. The projector is composed of a two-layer multilayer perceptron, and the output layer is a fully connected layer. The local model learns knowledge of each category from the image dataset. The federated learning method includes the following steps: Step 1, in the During the round, the client After receiving the global model from the central server, the local raw dataset is oversampled. Step 2, Client On the oversampled dataset, the client-side local training loss function is obtained by combining the classification loss function and the double contrast loss function, and the client-side local model is trained to obtain the updated local model. Step 3: The client sends the updated local model to the central server, which then aggregates the data according to the client's data ratio to form a new model. Round global model; In step 2, the dual contrast loss function consists of two loss functions: intra-class compactness loss and supervised contrast loss. The formula for the client-side local training loss function is as follows: ， in, Hyperparameters used to control the influence of intra-class compactness loss and supervised contrast loss; Represents cross-entropy loss, Indicates the intra-class compactness loss. This indicates that there is a supervised comparison of losses. Client-side local training loss; The intra-class compactness loss consists of an inter-class distance term and a class center distance term. The formula for the inter-class distance term is: ， in, , representing the sample features extracted from the model, the sample tags , For the client Local model parameters; Indicates the distance between classes; This represents the samples extracted from the model. Features; For the client The above category is The sample set; The formula for the class center distance term is: ， ， in, Represents the class center distance; Represents class-centric features; The formula for the intra-class compactness loss function is as follows: ， in, Intra-class compactness loss, Indicates hyperparameters; The supervised contrastive loss compares the features of samples within a batch. The contrast operation decreases the distance between positive sample pairs and increases the distance between negative sample pairs. The distance between positive sample pairs is defined as follows: ， in, As the decision boundary, This represents the distance between positive sample pairs; For a batch and samples All samples with different labels, they are related to the sample The distance between the resulting negative sample pairs is defined as follows: ， in, Indicates the distance between negative sample pairs; The supervised contrastive loss is formed by adding the distances between positive and negative sample pairs, as shown in the following formula: ， in, This indicates that there is a supervised comparison loss.

2. The class-adaptive balanced federated learning method incorporating dual contrastive loss according to claim 1, characterized in that, The specific steps of step 1 are as follows: Step 1.1: Before each round of training, the client performs a sampling operation, first counting the number of samples of each class in the local original dataset, using the following formula: ， in, For indicator functions, Indicates client The categories contained in it are The number of samples, Indicates the first The local raw dataset of each client, express The samples in Indicates corresponding to Tags; Step 1.2: Find the category with the largest number of samples in the local original dataset, and obtain a target sample value based on the number of samples in that category: ， in, Indicates the target sample value. This represents the total number of sample categories in the original local dataset. It's a hyperparameter; Step 1.3, for each category Based on the number of samples in that category With target value The magnitude of the values ​​determines whether oversampling should be performed. The specific operation is as follows: like If so, an oversampling operation is performed, and the number of oversampled samples is set to... The oversampled subset is represented as: ， in, For the client The above category is The sample set, To obtain from the sample set The oversampled subset obtained by random uniform sampling includes One sample; Step 1.4: After oversampling, the oversampled subset obtained by randomly sampling from the local original dataset will be merged back into the local original dataset.

3. The class-adaptive balanced federated learning method incorporating dual contrastive loss according to claim 1, characterized in that, In step 2, based on the client's local training loss function, the client updates the local model by performing the following gradient descent: , in, It is the learning rate for model training. Indicates client The current local model, This represents the client-side local gradient.

Images