A federated semi-supervised learning method based on class rebalancing

Abstract

The application discloses a kind of federal semi-supervised learning methods based on class rebalancing, which comprises: initializing global model, and assigning initialization model parameter to local model, to obtain initial local model;Based on adaptive sample selection sampling strategy, image samples are pretreated to obtain training samples;Non-independent and identically distributed division is carried out on training samples to obtain non-independent and identically distributed samples;Based on non-independent and identically distributed samples and local sampling rate adjustment strategy, initial local model is repeatedly trained to obtain trained local model;Three-stage parameter weighting is carried out on trained local model to obtain robust global model.By using the application, the problem of class imbalance in data non-independent and identically distributed can be effectively improved, and the accuracy of semi-supervised federated learning on long-tail data can be improved.The application can be widely applied to the field of federal semi-supervised learning.

Description

Technical Field

[0001] This invention relates to the field of federated semi-supervised learning technology, and more particularly to a federated semi-supervised learning method based on category rebalancing. Background Technology

[0002] Traditional machine learning methods typically require training the entire dataset on a centralized server. This process involves the transfer of large amounts of sensitive data, posing significant privacy risks. In recent years, Federated Learning (FL) has adopted a decentralized approach, distributing the training process across local devices. This enhances machine learning capabilities while protecting sensitive data, making it increasingly valuable in today's data segmentation field.

[0003] Existing federated learning (FL) methods focus on supervised learning settings, assuming that local private data is fully labeled. However, in real-world scenarios, it's unlikely that all data examples will have complex annotations. To address this issue, some research has introduced the concept of semi-supervised federated learning (FSSL), extending the application of semi-supervised learning methods to federated learning. Existing federated semi-supervised learning (FSSL) methods typically assume that training data follows independent and identically distributed (IID) conditions. However, in real-world scenarios, data often deviates from this assumption, exhibiting class imbalance, a common characteristic of non-IID data. Existing federated semi-supervised learning (FSSL) methods struggle to handle class imbalance in non-IID data. Summary of the Invention

[0004] To address the aforementioned technical problems, the objective of this invention is to provide a federated semi-supervised learning method based on class rebalancing, which can effectively improve the class imbalance problem in non-independent and identically distributed data and enhance the accuracy of semi-supervised federated learning on long-tail data.

[0005] The technical solution adopted in this invention is: a federated semi-supervised learning method based on class rebalancing, comprising the following steps:

[0006] Initialize the global model and assign the initial model parameters to the local model to obtain the initial local model;

[0007] Image samples are preprocessed based on an adaptive sample selection sampling strategy to obtain training samples;

[0008] The training samples are partitioned into non-independent and identically distributed samples to obtain non-independent and identically distributed samples.

[0009] The initial local model is repeatedly trained based on non-independent and identically distributed samples and a local sampling rate adjustment strategy to obtain the trained local model.

[0010] The trained local models are divided into groups, and the first parameter is weighted based on the amount of data in each group to obtain the standard model;

[0011] The local model is weighted by a second parameter based on the L2 norm of the gradient of the standard model and the gradient of the trained local model to obtain a sub-model.

[0012] By weighting the sub-models with the third parameter in the same proportion, a robust global model is obtained.

[0013] The step of preprocessing image samples based on the adaptive sample selection sampling strategy to obtain training samples specifically includes:

[0014] Randomly assign weight values ​​to image samples to obtain weight-optimized image samples;

[0015] Based on the weight values ​​of the image samples, the image samples after weight optimization are adaptively selected, and the image samples with high weights are selected to obtain the training samples.

[0016] This optimization step focuses on the samples most critical to improving model performance, avoiding excessive attention to noise or irrelevant samples. This improves the model's generalization ability and reduces computational overhead during training.

[0017] Furthermore, the step of partitioning the training samples into non-independent and identically distributed samples to obtain non-independent and identically distributed samples specifically includes:

[0018] Several class labels are set for the training samples;

[0019] For each category label, perform non-independent and identically distributed partitioning to obtain the category label distribution matrix;

[0020] Training samples are assigned to the initial local model based on the category label distribution matrix, resulting in non-independent and identically distributed samples.

[0021] Through this optimization step, the training samples of each category will be divided into different initial local models according to different proportions on the generated distribution, thereby simulating the non-independent and identically distributed data samples.

[0022] Furthermore, the step of repeatedly training the initial local model based on non-independent and identically distributed samples and a local sampling rate adjustment strategy to obtain a trained local model specifically includes:

[0023] The initial local model is locally trained based on the labeled dataset of non-independent and identically distributed samples to obtain the first local model;

[0024] Based on the first local model, pseudo-labels are generated for the unlabeled dataset in the non-independent and identically distributed samples to obtain the pseudo-labeled dataset.

[0025] To obtain the class sampling rate, a corresponding sampling rate is set for each class of sample dataset based on the amount of data for each class label.

[0026] Data within the pseudo-label dataset is selected based on the class sampling rate, and the label dataset is updated to obtain the updated label dataset.

[0027] The first local model is trained based on the updated label dataset to obtain the trained local model.

[0028] This optimized step involves repeatedly training the initial local model by updating the label dataset, which significantly reduces the prediction bias of the trained local model.

[0029] Furthermore, the first parameter is weighted, and its expression is as follows:

[0030]

[0031]

[0032] Among them, K i θ represents the amount of data in the i-th local model. avg Represents the standard model, K total θ represents the total amount of data. i This represents the local model after training.

[0033] Furthermore, the second parameter is weighted, and its expression is as follows:

[0034]

[0035]

[0036]

[0037] in, Let K denote the sub-model, β denote the hyperparameters, and K represent the sub-model. i θ represents the amount of data in the i-th local model. avg Representing the standard model, ||θ i -θ avg ||2 represents the 2-norm of the model gradient between the standard model and the trained local model. Let K represent the i-th model in the m-th round. total ω represents the total amount of data. i ω represents the dynamic scaling factor of the i-th model. j This represents the dynamic scaling factor of the j-th model. θ represents the dynamic scaling factor after normalization of the i-th model. i This represents the local model after training.

[0038] Furthermore, the weighted third parameter is expressed as follows:

[0039]

[0040] in, Representing the sub-model, This represents the new global model, and M represents the number of sub-models.

[0041] The beneficial effects of the method of this invention are as follows: This invention simulates the non-independent and identically distributed nature of sample data by dividing the training samples into non-independent and identically distributed groups; it alleviates the problem of decreased recall caused by class imbalance by using a local adaptive sampling rate adjustment strategy to rebalance the class imbalanced data in the non-independent and identically distributed group, thereby reducing the cost of manual data processing in the field of federated semi-supervised learning; it makes the global model more robust through three-stage parameter weighting; and it improves the accuracy of FSSL on long-tailed data, reduces prediction bias, improves model accuracy, enhances model generalization ability, and has relatively low computational cost and time complexity, all without modifying the structure of the original classification model or manually balancing the data categories. Attached Figure Description

[0042] Figure 1 This is a flowchart of the steps of a federated semi-supervised learning method based on category rebalancing according to the present invention.

[0043] Figure 2 This is a flowchart illustrating the workflow of constructing and training a local model using a federated semi-supervised learning method based on class rebalancing, as described in this invention.

[0044] Figure 3 This is a flowchart illustrating the workflow of constructing a global model using a federated semi-supervised learning method based on category rebalancing, as described in this invention. Detailed Implementation

[0045] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0046] Reference Figure 1 This invention provides a federated semi-supervised learning method based on class rebalancing, which includes the following steps:

[0047] S1. Initialize the global model and assign the initialization model parameters to the local model to obtain the initial local model;

[0048] Specifically, a global model is initialized, and the parameters of the initialized global model are passed to the local model of each client for initialization, resulting in an initialized model. The constructed local model contains feature extraction and feature processing units, which can extract and process features from the input sample images and use these features to classify the image samples. The model structure adopts a convolutional neural network structure, including but not limited to ResNet, GoogLeNet, and other convolutional neural network structures, to construct the local model.

[0049] S2. Preprocess the image samples based on the adaptive sample selection sampling strategy to obtain training samples;

[0050] Specifically, after obtaining image samples, the image samples are divided into a training sample set and a test sample set. Traditional random partitioning methods tend to lead to significant data drift between training and test samples, thus affecting the model's generalization ability. The image samples in the training sample set are used to optimize the parameters of the local model, while the images in the test sample set are used to evaluate the classification performance of the global model. Therefore, this invention employs a local adaptive sample selection sampling strategy to optimize the sample partitioning.

[0051] First, image samples are randomly assigned weight values ​​to obtain image samples with optimized weights. Typically, misclassified samples are given higher weights, while correctly classified samples are given lower weights, allowing the model to learn more about and correct these misclassified samples.

[0052] Next, based on the weight values ​​of the image samples, adaptive selection is performed on the weighted image samples to filter out high-weight image samples and obtain training samples. Possible adaptive selection strategies include sorting by weight value and selecting samples with higher weights as part of the training set; or setting a threshold and selecting only samples with weights exceeding the threshold as part of the training set; then, selection is performed based on gradient updates, that is, selecting samples according to their contribution to the model optimization process, and grouping the selected image samples together to form the training samples.

[0053] S3. Divide the training samples into non-independent and identically distributed samples to obtain non-independent and identically distributed samples.

[0054] Specifically, in order to obtain non-independent and identically distributed samples, the specific embodiments of the present invention divide the training samples into non-independent and identically distributed groups according to the label distribution of the training samples.

[0055] First, we set K class labels for the training samples and N initial local models. The training samples for each class label need to be distributed across different initial local models in different proportions. Therefore, let matrix X∈R K*N The category label distribution matrix has row vectors x k ∈R N This represents the probability distribution vector of class k on different initial local models (each dimension represents the proportion of samples of class k assigned to different initial local models). The probability distribution vector is obtained by sampling the class labels using a Dirichlet distribution. Based on the class label distribution matrix, training samples are assigned to the initial local models to achieve the effect that training samples for each class label need to be assigned to different initial local models in different proportions, thereby simulating the non-independent and identically distributed nature of the sample data.

[0056] S4. Based on the non-independent and identically distributed samples and the local sampling rate adjustment strategy, the initial local model is repeatedly trained to obtain the trained local model.

[0057] S4.1, Reference Figure 2 Using the labeled dataset X = {(x i ,y i ):i(1,...n)} (The labels here are not the class labels set in S3, but the annotation information carried by the sample itself) Local training is performed on the initial local model to obtain the first local model, and the formula of its training loss function is expressed as follows:

[0058]

[0059] Among them, L s Let y represent the loss function for training. i This represents the prediction result obtained through training. The label representing the sample.

[0060] S4.2, Using the first local model obtained from training, all unlabeled datasets u in the non-independent and identically distributed samples. m Generate pseudo tags The pseudo-labeled dataset U is obtained, and its expression is as follows:

[0061]

[0062] Where U represents the pseudo-label dataset, u m This represents an unlabeled dataset. denoted by pseudo-label, and M represents the number of unlabeled datasets.

[0063] Next, we define the local sampling rate adjustment strategy for each first local model and select pseudo-labels.

[0064] S4.3. Based on the amount of data for each label, set a corresponding sampling rate for each class of sample dataset to obtain the class sampling rate. In this specific embodiment of the invention, it is assumed that the number of categories is in the order N1≥N2≥N3......≥N n If the order is such that the sampling rate of class i is calculated, the expression for calculating the sampling rate is as follows:

[0065]

[0066] Where, μ i N represents the sampling rate of class i. i Let α represent the number of samples in class i, and let α represent the sampling ratio.

[0067] S4.4. Select data from the pseudo-label dataset based on the class sampling rate, and update the label dataset to obtain the updated label dataset, the expression of which is as follows:

[0068]

[0069] Where Y represents the updated label dataset, and X represents the label dataset. This represents the data within the selected pseudo-label dataset.

[0070] S4.5. Train the first local model based on the updated label dataset, repeating steps S4.1-S4.4 to obtain the trained local model, which greatly reduces the prediction bias of the trained local model.

[0071] The next step is to perform parameter weighting on the trained local model to obtain a robust global model. The process can be found in [reference needed]. Figure 3 .

[0072] S5. Divide the trained local model into groups and weight the first parameter based on the amount of data in each group to obtain the standard model;

[0073] Specifically, the first parameter is weighted, and its expression is as follows:

[0074]

[0075]

[0076] Among them, K i θ represents the amount of data in the i-th local model. avg Represents the standard model, K total θ represents the total amount of data. i This represents the local model after training.

[0077] S6. The local model is weighted by the second parameter based on the L2 norm of the gradient of the standard model and the gradient of the trained local model to obtain a sub-model.

[0078] Specifically, the second parameter is weighted, and its expression is as follows:

[0079]

[0080]

[0081]

[0082] in, Let K denote the sub-model, β denote the hyperparameters, and K represent the sub-model. i θ represents the amount of data in the i-th local model. avg Representing the standard model, ||θ i -θ avg ||2 represents the 2-norm of the model gradient between the standard model and the trained local model. Let K represent the i-th model in the m-th round. total ω represents the total amount of data. i ω represents the dynamic scaling factor of the i-th model. j This represents the dynamic scaling factor of the j-th model. θ represents the dynamic scaling factor after normalization of the i-th model. i This represents the local model after training.

[0083] S7. Weight the sub-models with the third parameter according to the same proportion to obtain a robust global model.

[0084] Specifically, the weighted third parameter is expressed as follows:

[0085]

[0086] in, Representing the sub-model, This represents the new global model, and M represents the number of sub-models.

[0087] After obtaining the new global model, the parameters of the new global model are used to initialize the local model on the client side. Steps S5-S7 are repeated until a robust global model is obtained for image classification.

[0088] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A federated semi-supervised learning method based on class rebalancing, characterized in that, Includes the following steps: Initialize the global model and assign the initial model parameters to the local model to obtain the initial local model; Image samples are preprocessed based on an adaptive sample selection sampling strategy to obtain training samples; The training samples are partitioned into non-independent and identically distributed samples to obtain non-independent and identically distributed samples. The initial local model is repeatedly trained based on non-independent and identically distributed samples and a local sampling rate adjustment strategy to obtain the trained local model. The trained local models are divided into groups, and the first parameter is weighted based on the amount of data in each group to obtain the standard model; The second parameter is used to weight the trained local model based on the L2 norm of the gradient of the standard model and the gradient of the trained local model to obtain a sub-model. By weighting the sub-models with the third parameter in the same proportion, a robust global model is obtained. The step of repeatedly training the initial local model based on non-independent and identically distributed samples and a local sampling rate adjustment strategy to obtain a trained local model specifically includes: The initial local model is locally trained based on the labeled dataset of non-independent and identically distributed samples to obtain the first local model; Based on the first local model, pseudo-labels are generated for the unlabeled dataset in the non-independent and identically distributed samples to obtain the pseudo-labeled dataset. To obtain the class sampling rate, a corresponding sampling rate is set for each class of sample dataset based on the amount of data for each class label. Data within the pseudo-label dataset is selected based on the class sampling rate, and the label dataset is updated to obtain the updated label dataset. The first local model is trained based on the updated label dataset to obtain the trained local model. The first parameter is weighted, and its expression is as follows: in, This represents the amount of data in the i-th local model. Represents the standard model, This represents the total amount of data. This represents the local model after training; The second parameter is weighted, and its expression is as follows: in, Representing the sub-model, Indicates hyperparameters, This represents the amount of data in the i-th local model. Represents the standard model, The α2 norm represents the model gradient between the standard model and the trained local model. This represents the i-th model in the m-th round. This represents the total amount of data. This represents the dynamic scaling factor of the i-th model. This represents the dynamic scaling factor of the j-th model. This represents the dynamic scaling factor after normalization of the i-th model. This represents the local model after training.

2. The federated semi-supervised learning method based on class rebalancing according to claim 1, characterized in that, The step of preprocessing image samples based on an adaptive sample selection sampling strategy to obtain training samples specifically includes: Randomly assign weight values ​​to image samples to obtain weight-optimized image samples; Based on the weight values ​​of the image samples, the image samples after weight optimization are adaptively selected, and the image samples with high weights are selected to obtain the training samples.

3. The federated semi-supervised learning method based on class rebalancing according to claim 1, characterized in that, The step of partitioning the training samples into non-independent and identically distributed samples to obtain non-independent and identically distributed samples specifically includes: Set several category labels for the training samples; For each category label, perform non-independent and identically distributed partitioning to obtain the category label distribution matrix; Training samples are assigned to the initial local model based on the category label distribution matrix, resulting in non-independent and identically distributed samples.

4. The federated semi-supervised learning method based on class rebalancing according to claim 1, characterized in that, The third weighted parameter is expressed as follows: in, Representing the sub-model, This represents the new global model. Indicates the number of sub-models.

Images