Random sampling consensus joint semi-supervised learning

Abstract

A method and system are provided for Random Sampling Consensus Joint (RSCFed) learning in a non-IID setting. The method includes: randomly sampling local clients; initializing the current global model by assigning it to the randomly sampled local clients at the start of a synchronization round; training the randomly sampled local clients locally; collecting local models from the randomly sampled local clients and performing distance-reweighted model aggregation (DMA) on the collected local models to obtain sub-consensus models; repeating the above steps multiple times to obtain a set of sub-consensus models; and aggregating a new model based on the sub-consensus models as the next global model.

Description

Background Technology

[0001] The core idea of ​​federated learning (FL) is to train machine learning models on independent datasets distributed across different locations or devices, thereby protecting local data privacy to some extent. In the past few years, FL has become an important research tool, and its application in fields such as medical image diagnosis [10,14,28], image classification

[16] , and object detection

[22] has attracted attention. Various FL methods have been proposed, such as FedAvg

[23] , SCAFFOLD

[12] , and MOON

[16] , and the initial results obtained are promising. However, the practical application of these methods is still limited because images need to be fully labeled on each local client.

[0002] Recently, joint semi-supervised learning (FSSL) [8,19,21,28], aimed at enhancing global models of FL using unlabeled images, has emerged as a new area of ​​research. One line of FSSL focuses on each client with both partially labeled and unlabeled images. For example, Jeong et al. [8] introduced the concept of client-to-client consistency loss, which improves the global model by encouraging consistent outputs from multiple clients. Another line of FSSL [21,28] makes the assumption that some local clients have fully labeled images and are represented as labeled clients, while other local clients have unlabeled images and are represented as unlabeled clients. However, these approaches are limited in two ways. First, they do not take into account non-independent and identically distributed (non-IID) data among local clients, which leads to a decrease in performance accuracy [9,15]. Second, some approaches

[21] share the correlation matrix among local clients, which can lead to information leakage.

[0003] One potential solution to this problem is to extend the application of existing FSSL methods (such as FedIRM

[21] and Fed-Consist

[28] ) to non-IID settings. However, this approach has failed to generalize these methods to non-IID settings. The FedIRM method shares the inter-class correlation matrix among clients, which fails to learn correctly due to the heterogeneous data among local clients, thus affecting model performance. The Fed-Consist method averages the model weights equally from labeled and unlabeled clients. However, as the number of unlabeled clients increases, model performance degrades significantly as the global model may be dominated by unlabeled clients. Another possible solution is to adjust the aggregate weights of labeled and unlabeled clients, where the weights of labeled clients are increased and the weights of unlabeled clients are decreased. However, preliminary results show only limited performance improvements. Summary of the Invention

[0004] There is still a need in this field for improved designs and techniques for methods and systems to perform random sampling consensus federated (RSCFed) learning in non-IID settings.

[0005] According to an embodiment of the present invention, a Random Sampled Consensus Joint (RSCFed) learning method is provided. The method includes: randomly sampling local clients; at the start of a synchronization round, assigning the current global model to the randomly sampled local clients for initialization; performing local training on the randomly sampled local clients; collecting local models from the randomly sampled local clients and performing Distance Reweighted Model Aggregation (DMA) on the collected local models to obtain sub-consensus models; repeating the above steps multiple times to obtain a set of sub-consensus models; and

[0006] A new model is aggregated based on the sub-consensus model to serve as the next global model. Local clients include labeled local clients with labeled local data and unlabeled local clients with unlabeled local data. Furthermore, initializing the current global model by assigning it to randomly sampled clients includes: initializing local models with the current global model to perform local training on the randomly sampled clients. Local training includes standard supervised training and unsupervised training on labeled and unlabeled local clients, respectively. The cross-entropy loss L, defined as the primary objective by the following equation, is utilized. CE The local training is performed on the labeled local client:

[0007]

[0008] in This involves predictions from randomly sampled local clients derived from the corresponding local models. The unlabeled local clients are trained locally using a mean-teacher-based consistency regularization framework, treating the student model as the local model. Furthermore, Distance-Reweighted Model Aggregation (DMA) is configured to dynamically adjust the weights of the collected models. During local training on the unlabeled local clients, after generating predictions from both the student and teacher models, a sharpening method is configured to increase the predictive heat of the teacher model. Once local training is complete, the student model is provided as the local model to the corresponding unlabeled local clients. Additionally, performing Distance-Reweighted Model Aggregation (DMA) includes: calculating the average model within each subset; scaling the weights for the local clients within each subset; and normalizing the model weights within the subset to the range [0,1].

[0009] In some embodiments of the present invention, a system for performing Random Sampled Consensus Joint (RSCFed) learning is provided. The system includes a joint server coupled to multiple local clients via a communication network, and the joint server is configured to: randomly sample multiple local clients; initialize the current global model by assigning it to the randomly sampled local clients at the start of a synchronization round; perform local training on the randomly sampled local clients; collect local models from the randomly sampled local clients; perform distance reweighted model aggregation (DMA) on the collected local models to obtain sub-consensus models; repeat the above steps multiple times to obtain a set of sub-consensus models; and aggregate a new model based on the sub-consensus models as the next global model. Local clients include labeled local clients with labeled local data and unlabeled local clients with unlabeled local data. Furthermore, initializing the current global model by assigning it to the randomly sampled clients includes: initializing local models with the current global model to perform local training on the randomly sampled clients. Local training includes standard supervised training and unsupervised training on labeled and unlabeled local clients, respectively. The system utilizes cross-entropy loss L, which is the primary objective defined by the following equation. CE The local training is performed on the labeled local client:

[0010]

[0011] in This involves predictions from randomly sampled local clients derived from the corresponding local models. Local training is performed on the unlabeled local clients using a mean-teacher-based consistency regularization framework, treating the student model as the local model. Furthermore, Distance-Reweighted Model Aggregation (DMA) is configured to dynamically adjust the weights of the collected models. During local training on the unlabeled local clients, after generating predictions from both the student and teacher models, a sharpening method is configured to increase the predictive heat of the teacher model. Once local training is complete, the student model is provided as the local model to the corresponding unlabeled local clients. Additionally, Distance-Reweighted Model Aggregation (DMA) is performed by: calculating the average model within each subset; scaling the weights for the local clients within each subset; and normalizing the model weights within the subset to the range [0,1]. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating the steps of the Random Sampling Consensus Joint (RSCFed) learning method according to an embodiment of the present invention.

[0013] Figure 2This is a schematic diagram outlining the RSCFed method according to an embodiment of the present invention, wherein labeled local clients and unlabeled local clients are respectively subjected to supervised cross-entropy loss. CE and mean-teacher-based consistency loss L MSE To optimize this, RSCFed uses distance-reweighted model aggregation (DMA) to perform multiple random subsamplings across all clients to increase the weights of clients closer to the sub-consensus model and vice versa, thereby suppressing the impact of the applied deviation of the local model on the global model.

[0014] Figure 3 This is a graph showing the experimental results comparing the accuracy of the RSCFed method with the accuracy of the conventional FedIRM method and the Fed-Consist method under the original settings and under the weighted settings according to embodiments of the present invention.

[0015] Figure 4 This is a schematic diagram comparing the RSCFed method of the present invention with the traditional Fed-Consist method. The RSCFed method extracts multiple sub-consensus models from the local model and updates the global model by aggregating the sub-consensus models according to the embodiments of the present invention, while the traditional Fed-Consist method performs standard model aggregation according to the FedAvg method. Detailed Implementation

[0016] Embodiments of the present invention relate to methods and systems for performing joint semi-supervised learning (FSSL), aimed at deriving a global model by jointly training fully labeled clients and fully unlabeled clients.

[0017] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. Unless the context clearly indicates otherwise, the singular forms "a," "an," and "the" are intended to include both the plural and singular forms as used herein. It will also be understood that, when used in this specification, the terms "comprising," "including," and / or "comprise" specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0018] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It will also be understood that, unless expressly defined herein, terms such as those defined in common dictionaries should be interpreted as having the same meaning as in the context of the relevant art and this disclosure, and should not be interpreted in an idealized or overly formal sense.

[0019] When the term “about” is used in conjunction with a numerical value in this document, it should be understood that the value may be in the range of 90% to 110% of the value, that is, the value may be + / - 10% of the value. For example, “about 1 kg” means from 0.90 kg to 1.1 kg.

[0020] In describing this invention, it will be understood that numerous techniques and steps are disclosed. Each of these has its own individual benefits, and each can also be used in conjunction with one or more other disclosed techniques, or in some cases, in combination with all other disclosed techniques. Therefore, for clarity, this description will avoid unnecessarily repeating every possible combination of the individual steps. However, it should be understood upon reading the specification and claims that such combinations are fully within the scope of this invention and the claims.

[0021] refer to Figure 1 and Figure 2 For both labeled and unlabeled local clients, the RSCFed method of this embodiment performs the following steps in each round. First, in step S110, local clients are randomly sampled. Then, in step S120, the current global model is assigned to the randomly sampled local clients for initialization, and local training is performed on the randomly sampled local clients. Next, in step S130, local models are collected from the randomly sampled local clients, and distance-reweighted model aggregation (DMA) is performed to obtain sub-consensus models. Then, in step S140, steps S110 to S130 are repeated multiple times to obtain a set of sub-consensus models. Next, in step S150, a new model is aggregated based on the sub-consensus models as the next global model.

[0022] 1.1. FSSL Settings

[0023] Suppose we have {C1,…,C} m There are m labeled local clients, and each client has a local dataset D. l Defined as Similarly, suppose we have denoted as {C} m+1 ,…,C m+n There are n unlabeled local clients, and each client has a set of N... u A local dataset D of unlabeled data u , The RSCFed method of this invention is configured to derive a global model θ by utilizing labeled and unlabeled local data in a distributed scheme. glob The labeled and unlabeled local datasets are distributed across different locations or devices. The RSCFed method of this invention is based on a general federated learning (FL) framework and can be applied to identify various types of local data, including but not limited to images of smart cities, mobile data packets, smart financial data, smart energy data, or traffic sign information.

[0024] 1.2. Local Training

[0025] All local models use the current global model at the start of the t-th synchronization round. This is used for initialization. The RSCFed method in this embodiment is configured to perform standard supervised training and unsupervised training on labeled local clients and unlabeled local clients, respectively. For illustrative purposes, all representations in this section are assumed to occur in the t-th synchronization round.

[0026] Mark local client

[0027] For local training performed on the local client, cross-entropy loss L is used. CE As the main objective defined by equation (1):

[0028]

[0029] in It's a prediction from local data derived from a local model. Then, the client, after training, will use θ... l Return it to the server.

[0030] Unmarked local client

[0031] For unlabeled clients, a mean-teacher-based consistency regularization framework is used, and the student model is treated as a local model. When this client is selected for the first time, [the following is used]: For the teacher model θ tea Initialization is performed. In each local iteration executed on the unlabeled local client, a batch of input images is augmented twice and fed into the student model and teacher model respectively. Their predictions are then generated. stu and p tea Then, the sharpening method defined in equation (1) is used to improve the teacher's predicted temperature, as shown in equation (2):

[0032]

[0033] Where pi and p refers to before and after sharpening, respectively. tea Each element in p, and τ is the heat parameter. Therefore, p tea "Sharpened" Furthermore, the samples are pushed away from the decision boundary to generate a better consistency alignment objective. For two predictions with different augmentation inputs, the mean squared error loss is used as the local objective for the unlabeled local client, as defined in equation (3):

[0034]

[0035] It should be noted that equation (3) only updates the student model, and the teacher model receives the student model parameters via an exponential moving average after each local iteration, as defined in equation (4):

[0036] θ tea =αθ stu +(1-α)θ tea (4)

[0037] Where α is the momentum parameter. The unlabeled local client ultimately uses the student model as its local model θ. u return.

[0038] 1.3. Random Sampling Consensus (FL)

[0039] The RSCFed method of this invention is an FSSL framework with random subset sampling and distance-reweighted model aggregation, configured to obtain a more robust global model from severely offset local models. Specifically, all clients are randomly quadratically sampled, and their uploaded models are collected to mine their basic consensus. Sub-consensus models are then obtained by aggregating the collected models, where a distance-reweighted model aggregation (DMA) strategy is introduced to dynamically adjust their weights. These two steps are repeated M times to obtain a set of sub-consensus models. The set of sub-consensus models is then aggregated to obtain a global model in each round.

[0040] Multiple random secondary sampling

[0041] Random quadratic sampling is used to extract sub-consensus models, and multiple random quadratic samplings are performed to obtain multiple sub-consensus models. To achieve this, at the start of synchronization round t, independent random quadratic sampling is performed M times to sample K clients. Then, the server distributes the global model... The model is sent to the sampled local client, and then local training is performed on the sampled local client. It should be noted that if the local client is sampled multiple times in a single round, it is not necessary to send the global model back to the local client for initialization, in order to minimize the communication cost between the server and the local client.

[0042] Distance reweighted model aggregation

[0043] To enhance the robustness of the sub-consensus model, unlike the aggregation of multiple selected clients as shown in the traditional FedAvg method

[23] , distance reweighted model aggregation (DMA) is used to dynamically increase the weights of models that are close to the average model, and vice versa. For the local models of the sampled local clients, a model aggregation strategy based on model distance is used to perform model aggregation. For each subset, the average model θ within the subset is first calculated according to the definition of equation (5). avg :

[0044] and

[0045] Where θ i N represents the i-th local model in the subset. i Let w represent the amount of local data, and K represent the number of clients in the subset. DMA does not simply average the local clients, but dynamically scales the weight w of the i-th client in each subset. i As defined by equation (6):

[0046] and

[0047] Where β is a hyperparameter, and ||θ i -θ avg ||2 refers to the L2 norm of the model gradient between the i-th local model and the time-averaged model in the subset. The model distance is divided by the amount of local data n. i To reduce the impact of local iterations on model drift, the model weights within the subset are then normalized to the range [0,1]. After obtaining a set of sub-consensus models, their equally weighted average is expressed as the final global model θ using equation (7). glob :

[0048]

[0049] in Let m represent the m-th sub-consensus model. Then, use... Execute the (t+1)th synchronization round As initialization, the entire update process of the RSCFed method in the t-th synchronization round is shown in Algorithm 1 below.

[0050]

[0051] 2. Experiment

[0052] To demonstrate the effectiveness and robustness of the RSCFed method of the present invention, experiments were conducted on multiple benchmark datasets, enabling the evaluation of RSCFed method embodiments with various settings (e.g., different unlabeled data ratios or limited communication costs).

[0053] 2.1. Dataset and Experiment Setup

[0054] benchmark dataset

[0055] The effectiveness of the RSCFed method in this embodiment of the invention is evaluated using two natural image classification datasets, SVHN and CIFAR-100. Furthermore, to simulate a decentralized-distributed scenario for real-world privacy data, the RSCFed method is also evaluated using the ISIC2018 dataset (skin lesion analysis for melanoma detection), which includes 10,015 dermoscopic images with seven skin lesions. For all three benchmark datasets, 80% of the images in each dataset are randomly selected for training, and the remaining images are reserved for testing. For the SVHN and CIFAR-100 datasets, the original 32×32 images are resized to 40×40 pixels, randomly cropped into 32×32 regions, and then normalized to generate the input to the RSCFed method's network. For the ISIC 2018 dataset, the spatial resolution of the original images is resized from 600×450 pixels to 240×240 pixels, then randomly cropped into 224×224 pixel regions, and the cropped regions are normalized to be input to the RSCFed method's network.

[0056] Feature extraction backbone

[0057] When training on the SVHN and CIFAR-100 datasets, a suitable network (e.g., a CNN network containing two 5×5 convolutional layers, a 2×2 max-pooling layer, and two fully connected layers) can be used as the feature extraction backbone. For the ISIC 2018 dataset, ResNet-18 was used as the feature extraction backbone. Furthermore, two MLP layers and a fully connected layer can be used to define the classification network for all datasets on each client. For comparison with the RSCFed method of this invention, the conventional FL method also uses the same classification network on each client.

[0058] federated learning environment

[0059] To simulate the joint learning setup, the number of clients was empirically set to 10, and a conventional approach of [16,26,33] with a Dirichlet distribution Dir(γ) (γ = 0.8 for all three benchmark datasets) was used to generate non-IID data partitions on the clients. With this non-IID data partitioning strategy, the number of classes and samples per client were different, and therefore not all clients contained samples from all classes. Clients with approximately 10% of the total data and samples from all classes were selected as labeled local clients for the Joint Semi-Supervised Learning (FSSL) method. In summary, the training dataset for the FSSL setup contained 10 clients: 1 labeled client with approximately 10% of the sample data with classification labels; and 9 unlabeled clients with only unlabeled samples. The same FSSL training dataset was used to train a network for the conventional FL method for comparison with the RSCFed method of this invention.

[0060] Implementation details

[0061] It is worth noting that when, Figure 3 As shown, when all local models are aggregated using the traditional FedAvg method

[23] , the original work in [21,28] produces a very limited number of results with sufficient labeled data. Therefore, the method in

[28] is reimplemented to increase the aggregation weights from labeled local clients in the sets {20%; 30%; 50%; and 70%}. Experimental results show that the best classification accuracy is achieved with the set of 50%. Therefore, the weights of labeled clients are empirically increased to about 50%, and the remaining 50% weights are shared by the other 9 unlabeled clients in each FSSL synchronization round. This aggregation weight is also applied when reimplementing the traditional FedIRM method

[21] and the RSCFed method of this invention to ensure the performance of the deep model. In the RSCFed method, the number of secondary sampling operations M and the number of local clients K used in each secondary sampling operation are set to: M = 3, K = 5.

[0062] The RSCFed method was implemented using the SGD optimizer and PyTorch. The learning rates for labeled and unlabeled local clients were empirically set to 0.03 and 0.021 for all methods executed on the SVHN and CIFAR-100 datasets, and empirically set to 0.002 and 0.001 for all methods executed on the ISIC 2018 dataset. The batch size for the SVHN and CIFAR-100 datasets was set to 64, and the batch size for the ISIC 2018 dataset was set to 12. Approximately 1000 synchronization epochs were trained on all datasets to allow the global model to converge stably, and the local training time (epoch) was set to 1. The RSCFed method of this invention has three parameters: the momentum parameter α of equation (4), the heat parameter τ of equation (2), and the scaling factor β of equation (6). The empirical settings of α = 0.001 and τ = 0.5 were used for all three benchmark datasets. The scaling factor β for the SVHN and CIFAR-100 datasets was set to 10000, and the scaling factor β for the ISIC 2018 dataset was set to 0.05.

[0063] 2.2. Comparison of RSCFed method with traditional methods

[0064] The RSCFed method of this invention is compared with the conventional FSSL method, which includes (1) FedIRM

[21] , which computes the inter-class relations of labeled clients and uses them as additional supervision for unlabeled clients; and (2) Fed-Consist

[28] , which computes the consistency loss of predictions from multiple augmented inputs of unlabeled data in the mean teacher framework

[25] . The RSCFed network of this invention is also compared with the conventional FedAvg method

[23] trained with all 10 labeled clients as upper bound classification results and with one labeled client as lower bound classification result. The results of the comparison are summarized in Table 1. In addition, four widely used metrics for evaluating these different methods include accuracy, area under the ROC curve (AUC), precision, and recall.

[0065] Specifically, Table 1 shows the results based on the SVHN, CIFAR-100, and ISIC 2018 datasets under heterogeneous data partitioning. It is worth noting that the traditional FedIRM method

[21] and the Fed-Consist method

[28] cannot be generalized in non-IID settings. Figure 3 As shown in Table 1, the results are adjusted for weights.

[0066] Table 1

[0067]

[0068] Quantitative comparison

[0069] Table 1 shows the quantitative results of the RSCFed method of the present invention and conventional methods, performed on three benchmark datasets and evaluated according to four metrics. The results of the two conventional FSSL methods (i.e., the FedIRM method

[21] and the Fed-Consist method

[28] ) and the RSCFed method of the present invention are observed to be between the upper and lower bounds obtained by the FedAvg method

[23] for all three benchmark datasets. In addition, the RSCFed method of the present invention is observed to exhibit superior metric performance to the two conventional methods on all three benchmark datasets. The superior performance of the RSCFed method of the present invention compared to the conventional Fed-Consist method indicates enhanced generalization ability of the aggregation strategy in the RSCFed method of the present invention. Furthermore, due to the consistency assumption that inter-class relationships between clients do not apply due to the distribution of non-IID data on all clients, the RSCFed method of the present invention also outperforms the FedIRM method in all three datasets in terms of the four metrics.

[0070] Evaluation based on the SVHN dataset

[0071] Compared to these two conventional methods, the Fed-Consist method achieves 66.94% accuracy, 94.19% AUC, 68.92% precision, and 66.75% recall. In contrast, the RSCFed method of this invention achieves 70.29% accuracy (a 3.32% improvement over the Fed-Consist method), 95.54% AUC (a 1.35% improvement over the Fed-Consist method), 73.36% precision (a 4.44% improvement over the Fed-Consist method), and 68.46% recall (a 1.71% improvement over the Fed-Consist method).

[0072] Evaluation based on the CIFAR-100 dataset

[0073] For the CIFAR-100 database, the FedIRM method achieved an accuracy score of 14.11%, an AUC score of 79.22%, and a recall score of 14.03%, while the Fed-Consist method achieved a precision score of 15.12%. In contrast, the RSCFed method of this invention achieved an accuracy score of 15.82%, an AUC score of 81.41%, a precision score of 15.85%, and a recall score of 16.37%.

[0074] Evaluation based on the ISIC 2018 dataset

[0075] Although the recall score achieved by the traditional Fed-Consist method is better than that of the RSCFed method of this invention, the RSCFed method achieves an accuracy score of 70.26%, an AUC score of 86.01%, and a precision score of 45.65%, indicating that the RSCFed method of this invention achieves higher classification accuracy than the traditional FL method when executed on the ISIC 2018 dataset.

[0076] 2.3. Ablation Studies

[0077] Ablation experiments were conducted to evaluate the effectiveness of the main components (secondary sampling and aggregation strategies) of the RSCFed method of this invention. The performance of the RSCFed method of this invention was analyzed in terms of parameters including the unlabeled ratio, communication cost constraints, and hyperparameters. Unless otherwise clarified, all experimental results are based on experiments performed on the SVHN dataset.

[0078] Effectiveness of SSO and DMA

[0079] To evaluate the effectiveness of Multiple Subsampling Operations (SSO) and Distance Reweighted Model Aggregation (DMA), ablation studies were performed on three benchmark datasets. Table 2 summarizes the accuracy and AUC scores of the RSCFed method of this invention compared with the quantitative results of two conventional baseline networks (i.e., "Basic+SSO" and "Basic").

[0080] Specifically, Table 2 presents the quantitative results of the RSCFed method of the present invention and the conventional backbone model

[28] , which does not have multiple subsampling operations and distance reweighted aggregation mechanism on three benchmark datasets. “SSO” indicates multiple subsampling operations with model aggregation, while “DMA” indicates distance reweighted model aggregation mechanism.

[0081] Table 2

[0082]

[0083] Based on the quantitative results, SSO and DMA were determined to make significant contributions to the success of the RSCFed method of this invention in the FSSL scenario. According to the quantitative results of "Basic+SSO" and "Basic", when performed on the SVHN dataset, SSO improved the accuracy score by 2.21% and the AUC score by 1.01%; when performed on the CIFAR-100 dataset, the accuracy score improved by 1.03% and the AUC score by 3.5%; and when performed on the ISIC 2018 dataset, the accuracy score improved by 1.11% and the AUC score by 0.8%. Furthermore, the DMA of the RSCFed method of the present invention was determined to be as follows: when executed on the SVHN dataset, the accuracy score was improved by 1.11% and the AUC score by 0.34%; when executed on the CIFAR-100 dataset, the accuracy score was improved by 0.9% and the AUC score decreased by 0.4%; and when executed on the ISIC 2018 dataset, the accuracy score was improved by 0.41% and the AUC score by 0.5%.

[0084] Unmarked client ratio

[0085] It is worth noting that the conventional FedIRM method

[21] uses 10 clients, with 1 labeled client and 9 unlabeled clients as set by FSSL. The RSCFed method of the present invention empirically divides the entire training data into ten clients, including one labeled client (or 10% labeled clients) and nine unlabeled clients (or 90% unlabeled clients). To evaluate the performance of the RSCFed method of the present invention at different unlabeled client ratios, ablation studies were conducted to compare different joint semi-supervised learning methods with different numbers of clients, wherein the number of labeled clients was empirically set to 1 for all methods. Here, in various experiments, the total number of clients was set to 5, 10, 15, 25, 35 and 50, and Table 3 shows the experimental results of the RSCFed method of the present invention and the conventional Fed-Consist method

[28] . As shown in the table, the RSCFed method of the present invention improves upon the conventional Fed-Consist method in terms of accuracy and AUC score as the number of unlabeled clients increases. As the number of unlabeled clients increased from 4 to 49, the accuracy improvement gradually increased from 1.51% to 3.98%, and the AUC improvement gradually increased from 0.5% to 3.5%.

[0086] In particular, Table 3 shows the results of the ablation study of the RSCFed method of the present invention compared with the results of the Fed-Consist method

[28] as a state-of-the-art FSSL method for different numbers of unlabeled clients.

[0087] Table 3

[0088]

[0089] Communication cost limitations

[0090] It is worth noting that the two conventional methods compared with the RSCFed method of the present invention pass through 10 local client models in each synchronization round, while the RSCFed method of the present invention considers 15 local models because it uses 3 subsampling operations and selects 5 local clients in each subsampling operation. Therefore, the communication cost of the RSCFed method of the present invention is 1.5 times that of the conventional Fed-Consist method

[28] . Ablation studies were conducted to evaluate the RSCFed method of the present invention under different communication cost constraints. Specifically, three other cases were considered, with 8 clients, 9 clients and 10 clients, respectively. Therefore, the communication cost was 0.8, 0.9 and 1.0 times that of the baseline communication cost. Table 4 shows the accuracy and AUC score of the RSCFed method of the present invention and the conventional Fed-Consist method under different communication costs. The results show that the RSCFed method of the present invention has a communication cost of 0.8 times that of the conventional Fed-Consist method in terms of accuracy and AUC score.

[0091] In particular, Table 4 summarizes the results of the ablation study of the RSCFed method of the present invention compared with the results of the conventional Fed-Consist method in terms of different communication costs. In Table 4, “Com.cost” represents the communication cost and indicates how many times that of the conventional Fed-Consist method

[28] .

[0092] Table 4

[0093]

[0094] Hyperparameters

[0095] The RSCFed method of this invention has two main hyperparameters: the number of subsampling operations (M) and the number of local clients used in each subsampling operation (K). The RSCFed method of this invention empirically sets M=3 and K=5. Here, ablation studies were conducted to investigate the effects of different choices of M and K, and the accuracy and AUC scores are summarized in Table 5. Based on the results, it was determined that the accuracy and AUC scores differ only slightly with different values ​​of M and K.

[0096] In particular, Table 5 presents the results of the ablation studies for different hyperparameter values. Here, M represents the number of subsamples, and K represents the number of clients in each subsample.

[0097] Table 5

[0098]

[0099] According to embodiments of the present invention, the Random Sampled Consensus Joint (RSCFed) learning method and system are configured to consider the uneven reliability between models from labeled and unlabeled clients in a non-IID setting, where there is no information leakage between clients. In the FSSL setting, labeled local clients tend to be biased towards local data, while unlabeled local clients struggle to achieve high accuracy, resulting in uneven model reliability among local clients. To achieve a robust global model, local models are treated as noisy models, and multiple consensus models are extracted through random sampling before aggregation into the global model, such as... Figure 4 As shown. Specifically, in each synchronization round, the client is randomly subsampled, and the average weight from the subsampled model is recorded as a sub-consensus model. By performing this operation multiple times, the global model is updated by aggregating multiple sub-consensus models. To extract robust sub-consensus models from randomly sampled local clients, a distance-reweighted model aggregation (DMA) module is used, which dynamically increases the weights of models that are closer to the sub-consensus models and vice versa. Extensive experiments were conducted on natural image classification datasets (e.g., SVHN and CIFAR-100) and medical datasets (e.g., ISIC 2018Skin), demonstrating the effectiveness of the RSCFed method of this invention.

[0100] Compared with traditional joint learning methods with non-IID, the RSCFed method of this invention offers many advantages:

[0101] 1. The RSCFed method solves the problem of uneven reliability of non-IID local clients, and focuses on solving the data heterogeneity problem under the FSSL scheme;

[0102] 2. Unlike traditional FSSL frameworks that directly aggregate local clients, the RSCFed method is configured to update the global model by aggregating multiple sub-consensus models;

[0103] 3. To improve the sub-consensus model, a distance reweighting model aggregation (DMA) module is adopted, which dynamically adjusts the weight of each sampled local client to the sub-consensus model.

[0104] 4. Experiments on three public datasets show that the RSCFed method significantly outperforms the traditional FSSL method, and the greater the proportion of unlabeled data, the greater the improvement that RSCFed can achieve.

[0105] All patents, patent applications, provisional applications and publications mentioned or cited herein are incorporated herein in their entirety by reference, including all figures and tables, without contradicting the express teachings of this specification.

[0106] It should be understood that the examples and embodiments described herein are for illustrative purposes only, and

[0107] Various modifications or variations will be made accordingly to those skilled in the art and will be included within the spirit and scope of this application and the appended claims. Furthermore, any element or limitation of any invention or embodiment thereof disclosed herein may be combined with any and / or all other elements or limitations disclosed herein (alone or in any combination) or any other invention or embodiment thereof, and all such combinations are included within the scope of this invention, but are not limited thereto.

[0108] References

[0109] [1]David Berthelot,Nicholas Carlini,Ian Goodfellow,Nicolas Papernot,Avital Oliver,and Colin Raffel.Mixmatch:A holistic approach to semi-supervised learning.arXiv preprint arXiv:1905.02249,2019.3,4

[0110] [2]Hong-You Chen and Wei-Lun Chao.Fedbe:Making bayesian model ensemble applicable to federated learning.In International Conference onLearning Representations,2021.3

[0111] [3]Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmentation with cross pseudo supervision. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages2613–2622, 2021.3

[0112] [4]Wenhui Cui,Yanlin Liu,Yuxing Li,Menghao Guo,Yiming Li,Xiuli Li,Tianle Wang,Xiangzhu Zeng,and Chuyang Ye.Semi-supervised brain lesionsegmentation with an adapted mean teacher model.In International Conferenceon Information Processing in Medical Imaging,pages 554–565.Springer,2019.3

[0113] [5]Martin A Fischler and Robert C Bolles.Random sample consensus:aparadigm for model fitting with applications to image analysis and automatedcartography.Communications of the ACM,24(6):381–395,1981.2

[0114] [6]Kaiming He,Xiangyu Zhang,Shaoqing Ren,and Jian Sun.Deep residuallearning for image recognition.In Proceedings of the IEEE conference oncomputer vision and pattern recognition,pages 770–778,2016.5

[0115] [7]Zijian Hu,Zhengyu Yang,Xuefeng Hu,and Ram Nevatia.Simple:Similarpseudo label exploitation for semi-supervised classification.In Proceedingsof the IEEE / CVF Conference on Computer Vision and Pattern Recognition,pages15099–15108,2021.3

[0116] [8]Wonyong Jeong,Jaehong Yoon,Eunho Yang,and Sung Ju Hwang.Federatedsemi-supervised learning with inter-client consistency&disjoint learning.InInternational Conference on Learning Representations,2021.1,3

[0117] [9]Peter Kairouz,H Brendan McMahan,Brendan Avent,Aur′elien Bellet,Mehdi Bennis,Arjun Nitin Bhagoji,Kallista Bonawitz,Zachary Charles,GrahamCormode,Rachel Cummings,et al.Advances and open problems in federatedlearning.arXiv preprint arXiv:1912.04977,2019.1

[0118]

[10] Georgios A Kaissis,Marcus R Makowski,Daniel R¨uckert,and RickmerFBraren.Secure,privacy-preserving and federated machine learning in medicalimaging.Nature Machine Intelligence,2(6):305–311,2020.1

[0119]

[11] Yan Kang,Yang Liu,and Tianjian Chen.Fedmvt:Semi-supervisedvertical federated learning with Multiview training.arXiv preprint arXiv:2008.10838,2020.3

[0120]

[12] Sai Praneeth Karimireddy,Satyen Kale,Mehryar Mohri,Sashank Reddi,Sebastian Stich,and

[0121] Ananda Theertha Suresh.Scaffold:Stochastic controlled averaging forfederated learning.In International Conference on Machine Learning,pages5132–5143.PMLR,2020.1,2

[0122]

[13] Jakub Koneˇcn`y,H Brendan McMahan,Felix X Yu,Peter Richt′arik,Ananda Theertha Suresh,and Dave Bacon.Federated learning:Strategies forimproving communication efficiency.arXiv preprint arXiv:1610.05492,2016.2

[0123]

[14] Rajesh Kumar,Abdullah Aman Khan,Jay Kumar,A Zakria,NoorbakhshAmiri Golilarz,Simin Zhang,Yang Ting,Chengyu Zheng,and WenYongWang.Blockchain-federated-learning and deep learning models for covid-19detection using ct imaging.IEEE Sensors Journal,2021.1

[0124]

[15] Qinbin Li,Yiqun Diao,Quan Chen,and Bingsheng He.Federatedlearning on non-iid data silos:An experimental study.arXiv preprint arXiv:2102.02079,2021.1

[16] Qinbin Li,Bingsheng He,and Dawn Song.Modelcontrastivefederated learning.In Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition,pages 10713–10722,2021.1,2,5

[0125]

[17] Tian Li,Anit Kumar Sahu,Manzil Zaheer,Maziar Sanjabi,AmeetTalwalkar,and Virginia Smith.Federated optimization in heterogeneousnetworks.arXiv preprint arXiv:1812.06127,2018.2

[0126]

[18] Xiaomeng Li,Lequan Yu,Hao Chen,Chi-Wing Fu,Lei Xing,and Pheng-AnnHeng.Transformationconsistent self-ensembling model for semisupervisedmedical image segmentation.IEEE Transactions on Neural Networks and LearningSystems,32(2):523–534,2020.3

[0127]

[19] Haowen Lin,Jian Lou,Li Xiong,and Cyrus Shahabi.Semifed:Semi-supervised federated learning with consistency and pseudo-labeling.arXivpreprint arXiv:2108.09412,2021.1,3

[0128]

[20] Quande Liu,Cheng Chen,Jing Qin,Qi Dou,and Pheng-Ann Heng.Feddg:Federated domain generalization on medical image segmentation via episodiclearning in continuous frequency space.In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition,pages 1013–1023,2021.2

[0129]

[21] Quande Liu,Hongzheng Yang,Qi Dou,and Pheng-Ann Heng.Federatedsemi-supervised medical image classification via inter-client relationmatching.arXiv preprint arXiv:2106.08600,2021.1,2,3,6,7

[0130]

[22] Yang Liu,Anbu Huang,Yun Luo,He Huang,Youzhi Liu,Yuanyuan Chen,Lican Feng,Tianjian Chen,Han Yu,and Qiang Yang.Fedvision:An online visualobject detection platform powered by federated learning.In Proceedings of theAAAI Conference on Artificial Intelligence,volume 34,pages 13172–13179,2020.1

[0131]

[23] Brendan McMahan,Eider Moore,Daniel Ramage,Seth Hampson,and BlaiseAguera y Arcas.Communication-efficient learning of deep networks fromdecentralized data.In Artificial Intelligence and Statistics,pages 1273–1282.PMLR,2017.1,2,5,6

[0132]

[24] Kihyuk Sohn,David Berthelot,Chun-Liang Li,Zizhao Zhang,NicholasCarlini,Ekin D Cubuk,Alex Kurakin,Han Zhang,and Colin Raffel.Fixmatch:Simplifying semi-supervised learning with consistency and confidence.arXivpreprint arXiv:2001.07685,2020.3

[0133]

[25] Antti Tarvainen and Harri Valpola.Mean teachers are better rolemodels:Weight-averaged consistency targets improve semi-supervised deeplearning results.arXiv preprint arXiv:1703.01780,2017.3,6

[0134]

[26] Hongyi Wang,Mikhail Yurochkin,Yuekai Sun,Dimitris Papailiopoulos,and Yasaman Khazaeni.Federated learning with matched averaging.InInternational Conference on Learning Representations,2020.3,5

[0135]

[27] Jianyu Wang,Qinghua Liu,Hao Liang,Gauri Joshi,and H VincentPoor.Tackling the objective inconsistency problem in heterogeneous federatedoptimization.arXiv preprint arXiv:2007.07481,2020.3

[0136]

[28] Dong Yang,Ziyue Xu,Wenqi Li,Andriy Myronenko,Holger R Roth,Stephanie Harmon,Sheng Xu,Baris Turkbey,Evrim Turkbey,Xiaosong Wang,etal.Federated semi-supervised learning for covid region segmentation in chestct using multi-national data from china,italy,japan.Medical image analysis,70:101992,2021.1,2,3,6,7,8

[0137]

[29] Qiang Yang,Yang Liu,Tianjian Chen,and Yongxin Tong.Federatedmachine learning:Concept and applications.ACM Transactions on IntelligentSystems and Technology(TIST),10(2):1–19,2019.2

[0138]

[30] Hongxu Yin,Arun Mallya,Arash Vahdat,Jose M Alvarez,Jan Kautz,andPavlo Molchanov.See through gradients:Image batch recovery viagradinversion.In Proceedings of the IEEE / CVF Conference on Computer Visionand Pattern Recognition,pages 16337–16346,2021.8

[0139]

[31] Tehrim Yoon,Sumin Shin,Sung Ju Hwang,and Eunho Yang.Fedmix:Approximation of mixup under mean augmented federated learning.InInternational Conference on Learning Representations,2021.2

[0140]

[32] Lequan Yu,Shujun Wang,Xiaomeng Li,Chi-Wing Fu,and Pheng-AnnHeng.Uncertainty-aware selfensembling model for semi-supervised 3d leftatrium segmentation.In International Conference on Medical Image Computingand Computer-Assisted Intervention,pages 605–613.Springer,2019.3

[0141]

[33] Mikhail Yurochkin,Mayank Agarwal,Soumya Ghosh,KristjanGreenewald,Nghia Hoang,and Yasaman Khazaeni.Bayesian nonparametric federatedlearning of neural networks.In International Conference on Machine Learning,pages 7252–7261.PMLR,2019.5

[0142]

[34] Michael Zhang,Karan Sapra,Sanja Fidler,Serena Yeung,and JoseM.Alvarez.Personalized federated learning with first order modeloptimization.In International Conference on Learning Representations,2021.3

[0143]

[35] Zhengming Zhang,Zhewei Yao,Yaoqing Yang,Yujun Yan,Joseph EGonzalez,and Michael W Mahoney.Benchmarking semi-supervised federatedlearning.arXiv preprint arXiv:2008.11364,17,2020.3

[0144]

[36] Yuliang Zou,Zizhao Zhang,Han Zhang,Chun-Liang Li,Xiao Bian,Jia-Bin Huang,and Tomas Pfister.Pseudoseg:Designing pseudo labels for semanticsegmentation.arXiv preprint arXiv:2010.09713,2020.3.

Claims

1. A classification method utilizing Random Sampling Consensus Joint Learning (RSCFed), comprising: Random sampling of local clients; At the start of the synchronization round, the current global classification model is assigned to the randomly sampled local client for initialization; The randomly sampled local client is trained locally using local data, including image data; Local classification models are collected from the randomly sampled local clients, and distance reweighted model aggregation (DMA) is performed on the collected local classification models to obtain a sub-consensus model, wherein the distance reweighted model aggregation (DMA) is configured to dynamically adjust the weights of the collected local classification models; Repeat the above steps multiple times to obtain a set of sub-consensus models; and The new model is aggregated based on the sub-consensus model and used as the next global classification model.

2. The method according to claim 1, wherein, The local clients include tagged local clients with tagged local data and untagged local clients with untagged local data.

3. The method according to claim 1, wherein, The step of assigning the current global classification model to the randomly sampled client for initialization includes: initializing the local classification model with the current global classification model to perform local training on the randomly sampled client.

4. The method according to claim 2, wherein, The steps for local training include: performing standard supervised training and unsupervised training on the labeled local client and the unlabeled local client, respectively.

5. The method according to claim 2, wherein, Utilizing the cross-entropy loss L as the primary objective defined by the following equation CE The local training is performed on the labeled local client: L CE = -y i log( i ); in i It is a prediction from a local client based on the random sample from the corresponding local classification model.

6. The method according to claim 2, wherein, The unlabeled local client is trained using a mean-teacher-based consistency regularization framework, treating the student model as the local classification model.

7. The method according to claim 2, wherein, During the local training on the unlabeled local client, after generating predictions from the student model and the teacher model, a sharpening method is configured to increase the heat of the teacher model's predictions.

8. The method according to claim 7, wherein, When the local training is complete, the student model is provided as the local classification model to the corresponding unlabeled local client.

9. The method according to claim 1, wherein, Performing distance-reweighted aggregation (DMA) includes: Calculate the in-subset average model for each subset; Scaling weights for the local client in each subset; and The model weights within the subset are normalized to the range [0, 1].

10. A classification system utilizing Random Sampling Consensus Joint Learning (RSCFed), comprising: A federated server, which is coupled to multiple local clients via a communication network; The joint server is configured to: randomly sample the plurality of local clients; at the start of a synchronization round, assign the current global classification model to the randomly sampled local clients for initialization; train the randomly sampled local clients locally using local data including image data; collect local classification models from the randomly sampled local clients; perform distance-reweighted model aggregation (DMA) on the collected local classification models to obtain sub-consensus models, wherein the distance-reweighted model aggregation (DMA) is configured to dynamically adjust the weights of the collected local classification models; repeat the above steps multiple times to obtain a set of sub-consensus models; and aggregate a new model based on the sub-consensus models as the next global classification model.

11. The system according to claim 10, wherein, The local clients include tagged local clients with tagged local data and untagged local clients with untagged local data.

12. The system according to claim 10, wherein, Assigning the current global classification model to the randomly sampled client for initialization includes: initializing a local classification model with the current global classification model for local training on the randomly sampled client.

13. The system according to claim 11, wherein, The local training includes: performing standard supervised training and unsupervised training on the labeled local client and the unlabeled local client, respectively.

14. The system according to claim 11, wherein, Utilizing the cross-entropy loss L as the primary objective defined by the following equation CE The local training is performed on the labeled local client: L CE = -y i log( i ); in i It is a prediction from a local client based on the random sample from the corresponding local classification model.

15. The system according to claim 11, wherein, The unlabeled local client is trained using a mean-teacher-based consistency regularization framework, treating the student model as the local classification model.

16. The system according to claim 11, wherein, During the local training on the unlabeled local client, after generating predictions from the student model and the teacher model, a sharpening method is configured to increase the heat of the teacher model's predictions.

17. The system according to claim 16, wherein, When the local training is complete, the student model is provided as the local classification model to the corresponding unlabeled local client.

18. The system according to claim 10, wherein, The distance reweighting model aggregation (DMA) is performed in the following manner: Calculate the in-subset average model for each subset; Scale the weights for local clients in each subset; and The model weights within the subset are normalized to the range [0, 1].

Images