Semi-supervised non-independent and identically distributed federated learning distillation method and device

Abstract

The present invention relates to the field of artificial intelligence technology, and provides a semi-supervised non-independent and identically distributed federated learning distillation method and device. This method uses the prediction results of the non-labeled data of the target terminal by the biased teacher model of other terminals to make the soft labels and hard labels of the non-labeled data more accurate and reliable. It can not only greatly improve the training efficiency of the basic model, but also make the generalization ability of the obtained student model stronger, thereby improving the accuracy of the aggregated model obtained by federated learning. In addition, this method combines knowledge distillation and federated learning, which can enable the student model to learn the knowledge of other terminals that does not exist at all, that is, its own data has no relevant labels, but it can learn relevant knowledge through federated learning. This is an extreme non-independent and identically distributed scenario for data labels. At the same time, this method can make the student model's existing fitting ability better through federated learning.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a semi-supervised non-independent and identically distributed federated learning distillation method and device. Background Art

[0002] Federated Learning (FL) is a new model training method that allows distributed terminal devices to initially train the global model sent by the server using local data. Each terminal device then uploads the preliminarily trained local model to the server. The server aggregates each uploaded local model and sends the aggregated model to each terminal device. Federated learning prevents local data from being leaked, effectively protecting the privacy and security of local data, while fully utilizing massive amounts of distributed local data for model training to obtain local models with better fitting performance. Because federated learning allows participants to collaboratively train models without sharing data, it effectively protects the privacy of local data and breaks down data silos. Therefore, federated learning has received widespread attention and is particularly widely used in distributed training scenarios.

[0003] In distributed training scenarios, many traditional distributed machine learning algorithms assume uniform data distribution, meaning that the data distribution between each terminal device must obey independent and identical distribution (IID). However, in real life, the generation of local data cannot be controlled, and local data is generated independently on different terminal devices. When multiple distributed terminal devices act as participants in federated learning, the local data on each terminal device may be non-independent and identically distributed (Non-IID), and even the labels of the local data may not be independent and identically distributed. This will lead to a significant decrease in model training efficiency in federated learning and the emergence of weak model generalization ability. Moreover, after federated learning, the accuracy of the aggregated model obtained by the participants in federated learning will not improve much, and may even decrease.

[0004] Therefore, it is crucial to improve the model training efficiency of federated learning in Non-IID scenarios, enhance the model generalization ability, and improve the accuracy of the aggregation model. Summary of the Invention

[0005] The present invention provides a semi-supervised non-independent and identically distributed federated learning distillation method and device to address the defects existing in the prior art.

[0006] The present invention provides a semi-supervised non-independent and identically distributed federated learning distillation method, which is applied to a target terminal, wherein the data and / or labels of each terminal under the target server to which the target terminal belongs satisfy non-independent and identically distributed data; the method comprises:

[0007] Determine the label data and unlabeled data of the target terminal, and perform label alignment on the target terminal and other terminals under the target server, and train the initial teacher model based on the label data and the label alignment result to obtain a first biased teacher model;

[0008] Based on the first subject-biased teacher model, label prediction is performed on the labeled data and the unlabeled data respectively to obtain a soft label for the labeled data and a first prediction result for the unlabeled data;

[0009] Receiving a second biased subject teacher model from the other terminal, performing label prediction on the unlabeled data based on the second biased subject teacher model to obtain a second prediction result for the unlabeled data, and generating a soft label and a hard label for the unlabeled data based on the first prediction result and the second prediction result;

[0010] Based on the labeled data and its soft labels, and the soft labels and hard labels of the unlabeled data, a base model is locally distilled to obtain a student model, and federated learning is performed based on the student model.

[0011] According to a semi-supervised non-IID federated learning distillation method provided by the present invention, generating a soft label and a hard label for the unlabeled data based on the first prediction result and the second prediction result includes:

[0012] Calculating the variance between the first prediction result and the second prediction result;

[0013] Generating a hard label for the unlabeled data based on a difference between a prediction result with a large variance and a prediction result with a small variance;

[0014] Calculate the mean of the first prediction result and the second prediction result, and use the mean as the soft label of the unlabeled data.

[0015] A semi-supervised non-IID federated learning distillation method provided by the present invention further includes:

[0016] Extracting part of the structure in the first subject-biased teacher model;

[0017] Performing differential privacy protection on the partial structure to obtain a target structure, and sending the target structure to the other terminal; or

[0018] Differential privacy protection is performed on part of the structure in the first subject-biased teacher model to obtain a target subject-biased teacher model, and the target subject-biased teacher model is sent to the other terminal.

[0019] According to a semi-supervised non-IID federated learning distillation method provided by the present invention, the second biased teacher model is a structure obtained by performing differential privacy protection on a part of the structure of the initial biased teacher model of the other terminal;

[0020] Accordingly, the label prediction is performed on the non-labeled data based on the second subject-biased teacher model to obtain a second prediction result of the non-labeled data, including:

[0021] splicing the difference structure between the second biased subject teacher model and the first biased subject teacher model to obtain a spliced model;

[0022] Based on the splicing model, label prediction is performed on the unlabeled data to obtain the second prediction result.

[0023] According to a semi-supervised non-IID federated learning distillation method provided by the present invention, the federated learning based on the student model includes:

[0024] Uploading the student model to the target server;

[0025] The target server receives an aggregate model obtained by performing federated average aggregation on the student models uploaded by each terminal, and uses the aggregate model as the basic model for local distillation in a loop until the federated learning is completed.

[0026] The present invention also provides a semi-supervised non-independent and identically distributed federated learning distillation device, which is applied to a target terminal, wherein the data and / or labels of each terminal under the target server to which the target terminal belongs satisfy non-independent and identically distributed conditions; the device comprises:

[0027] A determination module is used to determine the label data and unlabeled data of the target terminal, and perform label alignment on the target terminal with other terminals under the target server. Based on the label data and the label alignment result, an initial teacher model is trained to obtain a first biased teacher model;

[0028] A first prediction module is configured to perform label prediction on the labeled data and the unlabeled data based on the first subject-biased teacher model, to obtain a soft label for the labeled data and a first prediction result for the unlabeled data;

[0029] A second prediction module is configured to receive a second biased subject teacher model from the other terminal, perform label prediction on the unlabeled data based on the second biased subject teacher model, obtain a second prediction result for the unlabeled data, and generate a soft label and a hard label for the unlabeled data based on the first prediction result and the second prediction result;

[0030] A federated distillation module is used to locally distill the base model based on the labeled data and its soft labels, and the soft labels and hard labels of the unlabeled data to obtain a student model, and perform federated learning based on the student model.

[0031] A semi-supervised non-IID federated learning distillation device provided by the present invention further includes a sending module for:

[0032] Extracting part of the structure in the first subject-biased teacher model;

[0033] Performing differential privacy protection on the partial structure to obtain a target structure, and sending the target structure to the other terminal; or

[0034] Differential privacy protection is performed on part of the structure in the first subject-biased teacher model to obtain a target subject-biased teacher model, and the target subject-biased teacher model is sent to the other terminal.

[0035] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, it implements the semi-supervised non-independent and identically distributed federated learning distillation method as described in any one of the above.

[0036] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the semi-supervised non-independent and identically distributed federated learning distillation method as described in any one of the above.

[0037] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the semi-supervised non-independent and identically distributed federated learning distillation method as described in any one of the above.

[0038] The semi-supervised non-independent and identically distributed federated learning distillation method and device provided by the present invention, by means of the prediction results of the non-labeled data of the target terminal by the biased teacher model of other terminals, can make the soft labels and hard labels of the non-labeled data more accurate and reliable, which can not only greatly improve the training efficiency of the basic model, but also make the generalization ability of the obtained student model stronger, thereby improving the accuracy of the aggregated model obtained by federated learning. In addition, this method combines knowledge distillation and federated learning, which can enable the student model to learn the knowledge of other terminals that do not exist at all, that is, its own data has no relevant labels, but can learn relevant knowledge through federated learning. This is an extreme non-independent and identically distributed scenario for data labels. At the same time, this method can make the fitting ability that the student model already has better through federated learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on the drawings in the following description without any creative work.

[0040] Figure 1 Schematic diagram of the process of the semi-supervised non-independent and identically distributed federated learning distillation method provided by the present invention;

[0041] Figure 2 2 is a schematic diagram of the structure of the semi-supervised non-IID federated learning distillation device provided by the present invention;

[0042] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0044] Because in the prior art, when multiple dispersed terminal devices act as participants in federated learning, the local data on each terminal device may not be independent and identically distributed, and even the labels attached to the local data may not be independent and identically distributed. This will lead to a significant decrease in the efficiency of model training in federated learning and the problem of weak model generalization ability. Moreover, after the federated learning participants conduct federated learning, the accuracy of the aggregated model obtained will not be greatly improved, and may even be reduced. Therefore, an embodiment of the present invention provides a semi-supervised non-independent and identically distributed federated learning distillation method to improve the model training efficiency of federated learning in non-independent and identically distributed scenarios, improve the model generalization ability, and improve the accuracy of the aggregated model.

[0045] Figure 1 This is a flow chart of a semi-supervised non-independent and identically distributed federated learning distillation method provided in an embodiment of the present invention. The method is applied to a target terminal, and the data and / or labels of each terminal under the target server to which the target terminal belongs meet non-independent and identically distributed requirements. Figure 1 As shown, the method includes:

[0046] S1, determining the label data and unlabeled data of the target terminal, and performing label alignment on the target terminal and other terminals under the target server, and training the initial teacher model based on the label data and the label alignment result to obtain a first biased teacher model;

[0047] S2, based on the first subject-biased teacher model, performing label prediction on the labeled data and the unlabeled data respectively, to obtain a soft label for the labeled data and a first prediction result for the unlabeled data;

[0048] S3, receiving a second biased subject teacher model from the other terminal, performing label prediction on the unlabeled data based on the second biased subject teacher model, obtaining a second prediction result for the unlabeled data, and generating a soft label and a hard label for the unlabeled data based on the first prediction result and the second prediction result;

[0049] S4: Based on the labeled data and its soft labels, and the soft labels and hard labels of the unlabeled data, the base model is locally distilled to obtain a student model, and federated learning is performed based on the student model.

[0050] Specifically, the semi-supervised non-IID federated learning distillation method provided in the embodiments of the present invention is executed by a semi-supervised non-IID federated learning distillation device. This device can be configured in a target terminal, which can be every terminal in a target server. That is, every terminal in the target server executes the method. The target terminal can be a computer, which can be a local computer or a cloud computer. The local computer can be a computer, a tablet, etc., and is not specifically limited here.

[0051] The scenario in which this method is applied is a non-independent and identically distributed scenario, that is, the data and / or labels of each terminal including the target terminal under the target server to which the target terminal belongs meet non-independent and identically distributed requirements. The data meeting non-independent and identically distributed requirements means that the data label types of each terminal are the same, but the data distribution is inconsistent. The labels meeting non-independent and identically distributed requirements means that the data label types of each terminal are not exactly the same, and each terminal has unlabeled data. The fact that the data label types of each terminal are not exactly the same may include that the data label types of each terminal do not overlap or partially overlap. Label data refers to labeled data, and unlabeled data refers to unlabeled data. Semi-supervision refers to the presence of unlabeled data in the data of each terminal.

[0052] First, step S1 is executed to determine the labeled data and unlabeled data of the target terminal. Here, the target terminal can cluster the local data, specifically the labeled data and the unlabeled data in the local data, to obtain the labeled data and the unlabeled data. The local data can be images, private data, etc., and the labels carried by the local data can be object categories in the images, private data categories, etc., which are not specifically limited here.

[0053] For example, the target server includes terminals A and B. Terminal A's local data includes labeled data with the labels "cat" and "dog" and unlabeled data without at least one of the labels "cat," "dog," and "fish." The unlabeled data always contains the three types of data, but the specific data does not have a label. Terminal B's local data includes labeled data with the label "fish" and unlabeled data without at least one of the labels "cat," "dog," and "fish." The unlabeled data always contains the three types of data, but the specific data does not have a label.

[0054] The target terminal can perform label alignment on the target terminal and other terminals on the target server. The result of label alignment is all the labels involved in the local data of each terminal on the target server, thereby ensuring that the label categories involved are consistent across all terminals. This ensures that all terminals use the same loss function during federated learning, thus maintaining the consistency of predicted label types. For example, after label alignment, the labels involved in terminal A include cat, dog, and fish, but there is no corresponding data for the label fish. The labels involved in terminal B also include cat, dog, and fish, but there is no corresponding data for the labels cat and dog.

[0055] Furthermore, the target terminal can use the label data and the label alignment results to train the initial teacher model to obtain a first biased teacher model. It can be understood that due to the introduction of the label alignment results, the output items of the first biased teacher model include the prediction results corresponding to all labels, including the prediction results corresponding to labels that were not originally involved in the initial label data of the target terminal. For example, for terminal A, the first biased teacher model Ta obtained can better predict the data of cat and dog categories, but has no prediction ability for fish category data, but its prediction items still exist. For terminal B, the first biased teacher model Tb obtained can better predict the data of fish category, but has no prediction ability for cat and dog category data, but its prediction items still exist.

[0056] Here, the initial teacher model can be constructed based on a neural network. When training the initial teacher model, the label data can be input into the initial teacher model to obtain the output result of the initial teacher model. The loss function is calculated using the output result and the label carried by the label data, and the structural parameters of the initial teacher model are iteratively updated based on the loss function until the loss function converges, thereby obtaining the first biased teacher model.

[0057] Then, step S2 is executed. The target terminal uses the first biased teacher model to predict labels for the labeled data and unlabeled data. Specifically, the labeled data and unlabeled data are input into the first biased teacher model, and the first biased teacher model outputs a predicted result for the labeled data and a first predicted result for the unlabeled data. Here, the predicted result for the labeled data is the soft label of the labeled data, and the original label of the labeled data serves as its hard label.

[0058] For example, for terminal A, the first biased teacher model Ta is used to predict labels for its labeled data, obtaining the soft-label for the labeled data; the first biased teacher model Ta is used to predict labels for its unlabeled data, obtaining the first prediction result A-Ta-soft for the unlabeled data. For terminal B, the first biased teacher model Tb is used to predict labels for its labeled data, obtaining the soft-label for the labeled data; the first biased teacher model Tb is used to predict labels for its unlabeled data, obtaining the first prediction result B-Tb-soft for the unlabeled data.

[0059] Thereafter, step S3 is executed, where the target terminal receives the second subject-biased teacher model of the other terminal, and uses the second subject-biased teacher model to perform label prediction on the non-labeled data of the target terminal to obtain a second prediction result of the non-labeled data of the target terminal.

[0060] For example, for terminal A, a second biased teacher model Tb' of terminal B is received. The second biased teacher model Tb' may be the same as the first biased teacher model Tb of terminal B, or may be only a part of the first biased teacher model Tb, or may be obtained by encrypting the first biased teacher model Tb or a part thereof, which is not specifically limited here. The second biased teacher model Tb' is used to perform label prediction on the unlabeled data of terminal A, and a second prediction result A-Tb'-soft for the unlabeled data of terminal A can be obtained.

[0061] Terminal B receives a second biased teacher model Ta' of terminal A. This second biased teacher model Ta' may be the same as the first biased teacher model Ta of terminal A, or may be only a portion of the first biased teacher model Ta, or may be obtained by encrypting the first biased teacher model Ta or a portion thereof, without specific limitation herein. Using the second biased teacher model Ta', a label prediction is performed on the unlabeled data of terminal B to obtain a second prediction result B-Ta'-soft for the unlabeled data of terminal B.

[0062] The target terminal can then use the first and second prediction results to generate soft and hard labels for the unlabeled data. This completes the preparation of the data and labels required for the subsequent local distillation step. The hard label for the unlabeled data can be determined based on the difference between the first and second prediction results. For example, a difference threshold can be introduced. If the difference is less than the difference threshold, the hard label of the corresponding unlabeled data is determined to be 0. If the difference is greater than or equal to the difference threshold, the hard label of the corresponding unlabeled data is determined to be 1. The soft label for the unlabeled data can be determined based on the mean of the first and second prediction results.

[0063] Here, for terminal A, the first prediction result A-Ta-soft and the second prediction result A-Tb'-soft are used to generate the soft-label and hard-label for terminal A's unlabeled data. This completes the data and labels required for the subsequent local distillation step for terminal A. For terminal B, the first prediction result B-Tb-soft and the second prediction result B-Ta'-soft are used to generate the soft-label and hard-label for terminal B's unlabeled data. This completes the data and labels required for the subsequent local distillation step for terminal B.

[0064] Finally, step S4 is executed to perform local distillation on the base model using the labeled data and its soft labels, and the soft and hard labels of the unlabeled data, to obtain the student model. Multiple rounds of local distillation can be performed, and the end mark can be the convergence of the distillation loss.

[0065] For example, for terminal A, the base model is locally distilled using the labeled data and its soft-label, the soft-label and hard-label of the unlabeled data, and the student model Sa can be obtained; for terminal B, the base model is locally distilled using the labeled data and its soft-label, the soft-label and hard-label of the unlabeled data, and the student model Sb can be obtained.

[0066] Afterwards, the obtained student model can be used for federated learning, that is, the student model can be used to interact with the target server to achieve federated learning.

[0067] The semi-supervised non-independent and identically distributed federated learning distillation method provided in an embodiment of the present invention is applied to a target terminal, and the data and / or labels of each terminal under the target server to which the target terminal belongs satisfy non-independent and identically distributed. The method first determines the label data and unlabel data of the target terminal, and performs label alignment on the target terminal and other terminals under the target server. Based on the label data and the label alignment results, the initial teacher model is trained to obtain a first biased teacher model; then, the first biased teacher model is used to perform label prediction on the label data and the unlabel data respectively to obtain a soft label for the label data and a first prediction result for the unlabel data; thereafter, a second biased teacher model of other terminals is received, and based on the second biased teacher model, label prediction is performed on the unlabel data to obtain a second prediction result for the unlabel data, and based on the first prediction result and the second prediction result, a soft label and hard label for the unlabel data are generated; finally, the label data and its soft label, the soft label and hard label of the unlabel data are used to perform local distillation on the basic model to obtain a student model, and federated learning is performed based on the student model. This method leverages the prediction results of the unlabeled data of the target terminal by the biased teacher model of other terminals, making the soft and hard labels of the unlabeled data more accurate and reliable. This not only greatly improves the training efficiency of the basic model, but also makes the generalization ability of the resulting student model stronger, thereby improving the accuracy of the aggregated model obtained by federated learning. In addition, this method combines knowledge distillation and federated learning, allowing the student model to learn knowledge from other terminals that does not exist at all. In other words, the data itself has no relevant labels, but can learn relevant knowledge through federated learning. This is an extreme scenario for non-independent and identically distributed data labels. At the same time, this method can improve the fitting ability that the student model already has through federated learning.

[0068] Based on the above embodiment, the semi-supervised non-IID federated learning distillation method provided in the embodiment of the present invention, wherein the soft label and hard label of the unlabeled data are generated based on the first prediction result and the second prediction result, includes:

[0069] Calculating the variance between the first prediction result and the second prediction result;

[0070] Generating a hard label for the unlabeled data based on a difference between a prediction result with a large variance and a prediction result with a small variance;

[0071] Calculate the mean of the first prediction result and the second prediction result, and use the mean as the soft label of the unlabeled data.

[0072] Specifically, when generating soft and hard labels for the target terminal's unlabeled data, the variance between the first and second predictions is calculated. The difference between the prediction with the larger variance and the prediction with the smaller variance is then calculated. This difference is then used to generate the hard label for the target terminal's unlabeled data. Introducing the variance factor ensures that the resulting hard label is more consistent with actual conditions.

[0073] Thereafter, the mean of the first prediction result and the second prediction result is calculated, and the mean can be used as a soft label for the non-label data of the target terminal.

[0074] Based on the above embodiment, the semi-supervised non-IID federated learning distillation method provided in the embodiment of the present invention further includes:

[0075] Extracting part of the structure in the first subject-biased teacher model;

[0076] Performing differential privacy protection on the partial structure to obtain a target structure, and sending the target structure to the other terminal; or

[0077] Differential privacy protection is performed on part of the structure in the first subject-biased teacher model to obtain a target subject-biased teacher model, and the target subject-biased teacher model is sent to the other terminal.

[0078] Specifically, the target terminal may also extract a partial structure in the first subject-biased teacher model, and the partial structure may be half of the first subject-biased teacher model.

[0079] This partial structure is encrypted using differential privacy protection to obtain the target structure, which is then sent to other terminals. Due to the introduction of differential privacy protection, other terminals cannot recover the target terminal's data. Furthermore, encrypting only the first biased teacher model and sending it to other terminals further increases the difficulty for other terminals to recover the target terminal's data.

[0080] In an embodiment of the present invention, differential privacy protection can also be directly performed on part of the structure in the first subject-biased teacher model to obtain a target subject-biased teacher model, and the target subject-biased teacher model can be sent to other terminals, so that other terminals can directly use the target subject-biased teacher model, reducing the workload of other terminals in applying the target subject-biased teacher model.

[0081] Based on the above embodiment, in the semi-supervised non-IID federated learning distillation method provided in the embodiment of the present invention, the second biased teacher model is a structure obtained by the other terminal performing differential privacy protection on part of the structure of its initial biased teacher model;

[0082] Accordingly, the label prediction is performed on the non-labeled data based on the second subject-biased teacher model to obtain a second prediction result of the non-labeled data, including:

[0083] splicing the difference structure between the second biased subject teacher model and the first biased subject teacher model to obtain a spliced model;

[0084] Based on the splicing model, label prediction is performed on the unlabeled data to obtain the second prediction result.

[0085] Specifically, the second biased teacher model can be used by other terminals to perform differential privacy protection on part of the structure of the initial biased teacher model trained using labeled data and label alignment results. At this time, the second biased teacher model is incomplete and does not have a prediction function. Therefore, when the target terminal uses the second biased teacher model to perform label prediction on its unlabeled data, it can splice the second biased teacher model with the difference structure in the first biased teacher model to obtain a spliced model. It can be understood that the difference structure in the first biased teacher model refers to a structure that is different from the second biased teacher model, and the spliced model obtained by splicing the two is a model that can be used for prediction.

[0086] After that, the splicing model is used to directly predict the label of the non-label data to obtain the second prediction result.

[0087] Based on the above embodiment, the semi-supervised non-IID federated learning distillation method provided in the embodiment of the present invention, wherein the federated learning is performed based on the student model, includes:

[0088] Uploading the student model to the target server;

[0089] The target server receives an aggregate model obtained by performing federated average aggregation on the student models uploaded by each terminal, and uses the aggregate model as the basic model for local distillation in a loop until the federated learning is completed.

[0090] Specifically, when using the student model for federated learning, the student model can be first uploaded to the target server. The target server then receives the student models uploaded by all terminals belonging to the target server and performs federated average aggregation to obtain the aggregated model S. The federated average aggregation method can be to perform a weighted average of the structural parameters of all received student models.

[0091] The target terminal can then receive the aggregated model, which can be downloaded directly from the target server. The aggregated model can then be used as the base model again, and local distillation can be repeated repeatedly. This involves performing multiple rounds of local distillation to obtain a student model, which is then uploaded to the target server. The aggregated model from the target server is then received, and this process is repeated several times until federated learning is complete. This results in a student model that can be used for label category prediction on the target terminal, as well as an aggregated model on the target server—the federated model.

[0092] like Figure 2 As shown, based on the above embodiment, an embodiment of the present invention provides a semi-supervised non-IID federated learning distillation device, including:

[0093] A determination module 21 is configured to determine the label data and unlabeled data of the target terminal, perform label alignment on the target terminal and other terminals under the target server, and train an initial teacher model based on the label data and the label alignment result to obtain a first biased teacher model;

[0094] A first prediction module 22 is configured to perform label prediction on the labeled data and the unlabeled data based on the first subject-biased teacher model, and obtain a soft label for the labeled data and a first prediction result for the unlabeled data;

[0095] A second prediction module 23 is configured to receive a second biased subject teacher model from the other terminal, perform label prediction on the unlabeled data based on the second biased subject teacher model, obtain a second prediction result for the unlabeled data, and generate a soft label and a hard label for the unlabeled data based on the first prediction result and the second prediction result;

[0096] The federated distillation module 24 is used to perform local distillation on the base model based on the labeled data and its soft labels, the soft labels and hard labels of the unlabeled data, to obtain a student model, and perform federated learning based on the student model.

[0097] Based on the above embodiment, the semi-supervised non-IID federated learning distillation apparatus provided in the embodiment of the present invention further includes a sending module for:

[0098] Extracting part of the structure in the first subject-biased teacher model;

[0099] Performing differential privacy protection on the partial structure to obtain a target structure, and sending the target structure to the other terminal; or

[0100] Differential privacy protection is performed on part of the structure in the first subject-biased teacher model to obtain a target subject-biased teacher model, and the target subject-biased teacher model is sent to the other terminal.

[0101] Based on the above embodiment, in the semi-supervised non-IID federated learning distillation apparatus provided in the embodiment of the present invention, the second prediction module is specifically used to:

[0102] Calculating the variance between the first prediction result and the second prediction result;

[0103] Generating a hard label for the unlabeled data based on a difference between a prediction result with a large variance and a prediction result with a small variance;

[0104] Calculate the mean of the first prediction result and the second prediction result, and use the mean as the soft label of the unlabeled data.

[0105] Based on the above embodiment, in the semi-supervised non-IID federated learning distillation device provided in the embodiment of the present invention, the second biased teacher model is a structure obtained by the other terminal performing differential privacy protection on part of the structure of its initial biased teacher model;

[0106] Accordingly, the second prediction module is specifically configured to:

[0107] splicing the difference structure between the second biased subject teacher model and the first biased subject teacher model to obtain a spliced model;

[0108] Based on the splicing model, label prediction is performed on the unlabeled data to obtain the second prediction result.

[0109] Based on the above embodiments, the semi-supervised non-IID federated learning distillation device provided in the embodiments of the present invention, wherein the federated distillation module is specifically used for:

[0110] Uploading the student model to the target server;

[0111] The target server receives an aggregate model obtained by performing federated average aggregation on the student models uploaded by each terminal, and uses the aggregate model as the basic model for local distillation in a loop until the federated learning is completed.

[0112] Specifically, the functions of each module in the semi-supervised non-independent and identically distributed federated learning distillation device provided in the embodiment of the present invention correspond one-to-one to the operating procedures of each step in the above-mentioned method embodiment, and the effects achieved are also consistent. Please refer to the above-mentioned embodiment for details, and no further details will be given in the embodiment of the present invention.

[0113] Figure 3An example of a physical structure diagram of an electronic device is shown below. Figure 3 As shown, the electronic device may include: a processor (Processor) 310, a communication interface (Communications Interface) 320, a memory (Memory) 330 and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call the logic instructions in the memory 330 to execute the semi-supervised non-independent and identically distributed federated learning distillation method provided in the above embodiments, applied to the target terminal, and the data and / or labels of each terminal under the target server to which the target terminal belongs satisfy the non-independent and identically distributed method; the method includes: determining the label data and unlabeled data of the target terminal, and label aligning the target terminal with other terminals under the target server, training the initial teacher model based on the label data and the label alignment result to obtain a first biased teacher model; based on the first biased teacher model, performing label prediction on the label data and the unlabeled data respectively to obtain a soft label of the label data and a first prediction result of the unlabeled data; receiving the second biased teacher model of the other terminal, and based on the second biased teacher model, performing label prediction on the unlabeled data to obtain a second prediction result of the unlabeled data, and generating a soft label and a hard label of the unlabeled data based on the first prediction result and the second prediction result; locally distilling the basic model based on the label data and its soft label, the soft label and the hard label of the unlabeled data to obtain a student model, and performing federated learning based on the student model.

[0114] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0115] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the semi-supervised non-independent and identically distributed federated learning distillation method provided in the above embodiments, which is applied to a target terminal, and the data and / or labels of each terminal under the target server to which the target terminal belongs satisfy non-independent and identically distributed data; the method includes: determining the label data and non-label data of the target terminal, and aligning the label of the target terminal with other terminals under the target server, training the initial teacher model based on the label data and the label alignment result, and obtaining the first A biased teacher model; based on the first biased teacher model, performing label prediction on the labeled data and the unlabeled data respectively to obtain a soft label for the labeled data and a first prediction result for the unlabeled data; receiving a second biased teacher model from the other terminal, performing label prediction on the unlabeled data based on the second biased teacher model to obtain a second prediction result for the unlabeled data, and generating a soft label and a hard label for the unlabeled data based on the first prediction result and the second prediction result; performing local distillation on the basic model based on the labeled data and its soft label, and the soft label and hard label of the unlabeled data to obtain a student model, and performing federated learning based on the student model.

[0116] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the semi-supervised non-independent and identically distributed federated learning distillation method provided in the above embodiments, which is applied to a target terminal, and the data and / or labels of each terminal under the target server to which the target terminal belongs satisfy non-independent and identically distributed; the method comprises: determining the label data and non-label data of the target terminal, and aligning the label of the target terminal with other terminals under the target server, training the initial teacher model based on the label data and the label alignment result to obtain a first biased teacher model; based on the first The biased teacher model predicts labels for the labeled data and the unlabeled data respectively to obtain soft labels for the labeled data and a first prediction result for the unlabeled data; the second biased teacher model of the other terminal is received, and based on the second biased teacher model, the label prediction is performed on the unlabeled data to obtain a second prediction result for the unlabeled data, and the soft label and hard label of the unlabeled data are generated based on the first prediction result and the second prediction result; the basic model is locally distilled based on the labeled data and its soft label, and the soft label and hard label of the unlabeled data to obtain a student model, and federated learning is performed based on the student model.

[0117] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0118] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A semi-supervised non-IID federated learning distillation method, characterized by: Applied to a target terminal, the data and / or labels of each terminal under the target server to which the target terminal belongs satisfy non-independent and identically distributed requirements; the method includes: Determine the labeled data and unlabeled data of the target terminal, perform label alignment on the target terminal and other terminals under the target server, and train an initial teacher model based on the labeled data and the label alignment result to obtain a first biased teacher model; the labeled data and the unlabeled data are obtained by clustering local data by the target terminal, the local data including images or private data, and the labels carried by the labeled data include object categories in the images or private data categories; Based on the first subject-biased teacher model, label prediction is performed on the labeled data and the unlabeled data respectively to obtain a soft label for the labeled data and a first prediction result for the unlabeled data; Receiving a second biased subject teacher model from the other terminal, performing label prediction on the unlabeled data based on the second biased subject teacher model to obtain a second prediction result for the unlabeled data, and generating a soft label and a hard label for the unlabeled data based on the first prediction result and the second prediction result; Based on the labeled data and its soft labels, and the soft labels and hard labels of the unlabeled data, a base model is locally distilled to obtain a student model, and federated learning is performed based on the student model.

2. The semi-supervised non-IID federated learning distillation method according to claim 1, characterized in that Generating a soft label and a hard label for the unlabeled data based on the first prediction result and the second prediction result includes: Calculating the variance between the first prediction result and the second prediction result; Generating a hard label for the unlabeled data based on a difference between a prediction result with a large variance and a prediction result with a small variance; Calculate the mean of the first prediction result and the second prediction result, and use the mean as the soft label of the unlabeled data.

3. The semi-supervised non-IID federated learning distillation method according to claim 1, characterized in that Also includes: Extracting part of the structure in the first subject-biased teacher model; Performing differential privacy protection on the partial structure to obtain a target structure, and sending the target structure to the other terminal; or, Differential privacy protection is performed on part of the structure in the first subject-biased teacher model to obtain a target subject-biased teacher model, and the target subject-biased teacher model is sent to the other terminal.

4. The semi-supervised non-IID federated learning distillation method according to claim 1, characterized in that The second biased subject teacher model is a structure obtained by the other terminal performing differential privacy protection on a part of the structure of its initial biased subject teacher model; Accordingly, the label prediction is performed on the non-labeled data based on the second subject-biased teacher model to obtain a second prediction result of the non-labeled data, including: splicing the difference structure between the second biased subject teacher model and the first biased subject teacher model to obtain a spliced model; Based on the splicing model, label prediction is performed on the unlabeled data to obtain the second prediction result.

5. The semi-supervised non-IID federated learning distillation method according to any one of claims 1 to 4, characterized in that: The federated learning based on the student model includes: Uploading the student model to the target server; The target server receives an aggregate model obtained by performing federated average aggregation on the student models uploaded by each terminal, and uses the aggregate model as the basic model for local distillation in a loop until the federated learning is completed.

6. A semi-supervised non-IID federated learning distillation device, characterized by: Applied to a target terminal, the data and / or labels of each terminal under the target server to which the target terminal belongs satisfy non-independent and identically distributed requirements; the device comprises: A determination module is configured to determine the labeled data and unlabeled data of the target terminal, perform label alignment on the target terminal and other terminals under the target server, and train an initial teacher model based on the labeled data and the label alignment result to obtain a first biased teacher model; the labeled data and the unlabeled data are obtained by clustering local data by the target terminal, the local data including images or private data, and the labels carried by the labeled data include object categories in the images or private data categories; A first prediction module is configured to perform label prediction on the labeled data and the unlabeled data based on the first subject-biased teacher model, to obtain a soft label for the labeled data and a first prediction result for the unlabeled data; A second prediction module is configured to receive a second biased subject teacher model from the other terminal, perform label prediction on the unlabeled data based on the second biased subject teacher model, obtain a second prediction result for the unlabeled data, and generate a soft label and a hard label for the unlabeled data based on the first prediction result and the second prediction result; A federated distillation module is used to locally distill the base model based on the labeled data and its soft labels, and the soft labels and hard labels of the unlabeled data to obtain a student model, and perform federated learning based on the student model.

7. The semi-supervised non-IID federated learning distillation apparatus according to claim 6, characterized in that Also included is a sending module for: Extracting part of the structure in the first subject-biased teacher model; Performing differential privacy protection on the partial structure to obtain a target structure, and sending the target structure to the other terminal; or, Differential privacy protection is performed on part of the structure in the first subject-biased teacher model to obtain a target subject-biased teacher model, and the target subject-biased teacher model is sent to the other terminal.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the semi-supervised non-independent and identically distributed federated learning distillation method according to any one of claims 1 to 5 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the semi-supervised non-independent and identically distributed federated learning distillation method according to any one of claims 1 to 5 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the semi-supervised non-independent and identically distributed federated learning distillation method according to any one of claims 1 to 5 is implemented.

Images