Federal learning distillation method and device for non-independent and identically distributed scenarios

Abstract

The application relates to the technical field of artificial intelligence, and provides a federated learning distillation method and device for a non-independent and identically distributed scene, which can enrich the training samples of an initial teacher model and a base model by means of the similarity of non-label data of a target terminal and initial label data of the target terminal, the first feature similarity of the initial label data of the target terminal and initial label data of other terminals, and the second feature similarity of the initial label data of the target terminal and the initial label data of the other terminals, can greatly improve the training efficiency of the initial teacher model and the base model, and can make the generalization ability of a target teacher model and a student model obtained stronger, thereby improving the accuracy of an aggregated model obtained through federated learning. In addition, the method combines knowledge distillation and federated learning, so that the student model can learn the knowledge of other terminals which does not exist in the student model itself, that is, the data of the student model does not have relevant labels, but the student model can learn relevant knowledge through federated learning.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a federated learning distillation method and apparatus for non-independent and identically distributed scenarios. Background Technology

[0002] Federated Learning (FL) is a novel model training method that allows distributed terminal devices to initially train a global model distributed by a server using local data. Each terminal device then uploads its pre-trained local model to the server, where it aggregates all uploaded models and distributes the aggregated model back to the terminal devices. Federated Learning achieves both data security and privacy protection by preventing local data leakage, and the efficient use of massive amounts of distributed local data for model training, resulting in a more performant local model. Because federated learning allows participants to collaboratively train models without sharing data, effectively protecting local data privacy and breaking down data silos, it has attracted widespread attention, especially in distributed training scenarios.

[0003] In distributed training scenarios, many traditional distributed machine learning algorithms assume a uniform data distribution, meaning the data distribution across different terminal devices must follow an Independent-and-Identically-Distributed (IID) pattern. However, in reality, the generation of local data is uncontrollable, with different terminal devices generating local data independently. When multiple distributed terminal devices participate in federated learning, the local data on each device may be Non-Independent-and-Identically-Distributed (Non-IID), and even the labels on the local data may be non-independent-and-IID. This leads to a significant decrease in model training efficiency and weak model generalization ability in federated learning. Moreover, the accuracy of the aggregated model obtained after federated learning is not significantly improved, and may even decrease.

[0004] Therefore, it is crucial to improve the model training efficiency, enhance the model generalization ability, and increase the accuracy of the aggregated model in federated learning scenarios. Summary of the Invention

[0005] This invention provides a federated learning distillation method and apparatus for non-independent and identically distributed scenarios, in order to overcome the deficiencies in the prior art.

[0006] This invention provides a federated learning distillation method for scenarios with non-independent and identically distributed data, applied to a target terminal where the data and / or tags of each terminal under a target server satisfy the non-independent and identically distributed nature. The method includes:

[0007] The initial labeled data and unlabeled data of the target terminal are determined, and the unlabeled data is labeled based on the similarity between the unlabeled data and the initial labeled data of the target terminal to obtain the first labeled data;

[0008] Based on the first feature similarity between the unlabeled data and the initial label data of the other terminals, the unlabeled data is labeled to obtain the second label data. Based on the second feature similarity between the initial label data of the target terminal and the initial label data of the other terminals, the third label data is determined.

[0009] The target terminal is aligned with other terminals under the target server. Based on the initial label data of the target terminal, the first label data, the second label data, the third label data, and the label alignment result, the initial teacher model is trained to obtain the target teacher model. Based on the target teacher model, the initial label data of the target terminal is predicted to obtain the soft label of the initial label data of the target terminal.

[0010] Based on the soft labels of the initial label data of the target terminal, the initial label data of the target terminal, the first label data, the second label data, and the third label data, the base model is locally distilled to obtain the student model, and federated learning is performed based on the student model.

[0011] According to the federated learning distillation method for non-independent and identically distributed scenarios provided by the present invention, the first feature similarity is determined based on the following steps:

[0012] Determine an initial feature extraction model, and based on the initial feature extraction model, extract the first feature vector of the unlabeled data;

[0013] The preset structure in the initial feature extraction model is subjected to differential privacy protection to obtain the target feature extraction model;

[0014] The target feature extraction model is sent to the other terminal, and the second feature vector of its initial label data extracted by the other terminal based on the target feature extraction model is received.

[0015] The similarity between the first feature vector and the second feature vector is determined as the first feature similarity.

[0016] According to the federated learning distillation method for non-independent and identically distributed scenarios provided by the present invention, the second feature similarity is determined based on the following steps:

[0017] Based on the initial feature extraction model, the third feature vector of the initial label data of the target terminal is extracted;

[0018] The similarity between the third feature vector and the second feature vector is determined as the second feature similarity.

[0019] According to the present invention, a federated learning distillation method for non-independent and identically distributed scenarios is provided, wherein the federated learning based on the student model includes:

[0020] Upload the student model to the target server;

[0021] The system receives the aggregated model obtained by the target server based on the federated average aggregation of the student models uploaded by each terminal, and uses the aggregated model as the base model to perform local distillation in a loop until the federated learning ends.

[0022] According to the present invention, a federated learning distillation method for non-independent and identically distributed scenarios is provided, wherein the unlabeled data is labeled based on the similarity between the unlabeled data and the initial labeled data of the target terminal to obtain first labeled data, including:

[0023] The first similar data with the highest similarity to the unlabeled data in the initial label data of the target terminal is determined, and the unlabeled data is labeled based on the label of the first similar data to obtain the first labeled data.

[0024] According to a federated learning distillation method for non-independent and identically distributed scenarios provided by the present invention, the unlabeled data is labeled based on a first feature similarity between the unlabeled data and the initial labeled data of the other terminals to obtain second labeled data, including:

[0025] The second similar data with the highest similarity to the first feature of the unlabeled data in the initial label data of the other terminals is determined, and the unlabeled data is labeled based on the label of the second similar data to obtain the second labeled data.

[0026] According to a federated learning distillation method for non-independent and identically distributed scenarios provided by the present invention, the step of determining the third label data based on the second feature similarity between the initial label data of the target terminal and the initial label data of other terminals includes:

[0027] Calculate the average label value of the third similar data and the fourth similar data. The third similar data is the initial label data of the target terminal corresponding to the second feature similarity greater than a preset threshold. The fourth similar data is the initial label data of the other terminal corresponding to the second feature similarity greater than the preset threshold.

[0028] The average value of the labels is used as the label of the third similar data to obtain the third label data.

[0029] This invention also provides a federated learning distillation apparatus for non-independent and identically distributed scenarios, applied to a target terminal, wherein the data and / or tags of each terminal under the target server to which the target terminal belongs satisfy non-independent and identically distributed conditions; the apparatus includes:

[0030] The data aggregation module is used to determine the initial labeled data and unlabeled data of the target terminal, and to label the unlabeled data based on the similarity between the unlabeled data and the initial labeled data of the target terminal to obtain the first labeled data.

[0031] The data labeling module is used to align the target terminal with other terminals under the target server, and label the unlabeled data based on the first feature similarity between the unlabeled data and the initial label data of the other terminals to obtain second label data. Based on the second feature similarity between the initial label data of the target terminal and the initial label data of the other terminals, the module determines third label data.

[0032] The label prediction module is used to train an initial teacher model based on the initial label data of the target terminal, the first label data, the second label data, the third label data, and the label alignment result to obtain a target teacher model, and to perform label prediction on the initial label data of the target terminal based on the target teacher model to obtain soft labels for the initial label data of the target terminal.

[0033] The federated distillation module is used to perform local distillation on the base model based on the soft labels of the initial label data of the target terminal, the initial label data of the target terminal, the first label data, the second label data, and the third label data to obtain the student model, and to perform federated learning based on the student model.

[0034] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the federated learning distillation method for non-independent and identically distributed scenarios as described above.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the federated learning distillation method for non-independent and identically distributed scenarios as described above.

[0036] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the federated learning distillation method for non-independent and identically distributed scenarios as described above.

[0037] This invention provides a federated learning distillation method and apparatus for scenarios with non-independent and identically distributed data. This method leverages the similarity between the unlabeled data of the target terminal and its initial labeled data, the first feature similarity with the initial labeled data of other terminals, and the second feature similarity between the initial labeled data of the target terminal and the initial labeled data of other terminals. This enriches the training samples of the initial teacher model and the base model, significantly improving their training efficiency and enhancing the generalization ability of the target teacher model and student model, thereby improving the accuracy of the aggregated model obtained through federated learning. Furthermore, by combining knowledge distillation and federated learning, this method enables student models to learn knowledge from other terminals that are completely absent from their own data—that is, data without relevant labels can acquire relevant knowledge through federated learning. This addresses an extreme scenario of non-independent and identically distributed data labels. Simultaneously, this method can further improve the fitting ability already possessed by the student model through federated learning. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on the drawings described below without creative effort.

[0039] Figure 1 This is a flowchart illustrating the federated learning distillation method for non-independent and identically distributed scenarios provided by the present invention.

[0040] Figure 2 This is a schematic diagram of the structure of the federated learning distillation device for non-independent and identically distributed scenarios provided by the present invention.

[0041] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0043] In existing technologies, when multiple distributed terminal devices participate in federated learning, the local data on each device may be non-independent and identically distributed (i.i.d.), and even the labels on the local data may be non-independent and identically distributed. This leads to a significant decrease in model training efficiency and weak model generalization ability in federated learning. Moreover, the accuracy of the aggregated model obtained after federated learning is not significantly improved, and may even decrease. Therefore, this invention provides a federated learning distillation method for non-independent and identically distributed scenarios, which is crucial for improving model training efficiency, enhancing model generalization ability, and increasing the accuracy of the aggregated model in such scenarios.

[0044] Figure 1 This is a flowchart illustrating a federated learning distillation method for non-independent and identically distributed scenarios provided in an embodiment of the present invention. The method is applied to a target terminal, where the data and / or tags of terminals under the target server to which the target terminal belongs satisfy the non-independent and identically distributed nature. For example... Figure 1 As shown, the method includes:

[0045] S1, determine the initial labeled data and unlabeled data of the target terminal, and label the unlabeled data based on the similarity between the unlabeled data and the initial labeled data of the target terminal to obtain the first labeled data;

[0046] S2, based on the first feature similarity between the unlabeled data and the initial label data of the other terminals, the unlabeled data is labeled to obtain the second label data, and based on the second feature similarity between the initial label data of the target terminal and the initial label data of the other terminals, the third label data is determined;

[0047] S3, perform label alignment between the target terminal and other terminals under the target server, and train the initial teacher model based on the initial label data of the target terminal, the first label data, the second label data, the third label data and the label alignment result to obtain the target teacher model, and perform label prediction on the initial label data of the target terminal based on the target teacher model to obtain the soft label of the initial label data of the target terminal;

[0048] S4. Based on the soft labels of the initial label data of the target terminal, the initial label data of the target terminal, the first label data, the second label data, and the third label data, the base model is locally distilled to obtain the student model, and federated learning is performed based on the student model.

[0049] Specifically, the federated learning distillation method for non-independent and identically distributed scenarios provided in this embodiment of the invention is executed by a federated learning distillation device for non-independent and identically distributed scenarios. This device can be configured within a target terminal, which can be any terminal within a target server, meaning that each terminal within the target server executes the method. The target terminal can be a computer, which can be a local computer or a cloud computer. A local computer can be a PC, tablet, etc., without specific limitations here.

[0050] This method is applied in scenarios where data and / or tags of all terminals, including the target terminal, under the target server to which the target terminal belongs satisfy non-independent and identically distributed (IMDG) conditions. Non-independent and identically distributed data means that while the data tag types of each terminal are the same, their distribution is inconsistent. Non-independent and identically distributed tags mean that the data tag types of each terminal are not completely identical, and each terminal also possesses unlabeled data. The non-identical data tag types of each terminal can include non-overlapping or partially overlapping data tag types, while unlabeled data refers to data without tags.

[0051] First, step S1 is executed to determine the initial labeled data and unlabeled data of the target terminal. Here, the target terminal can cluster the local data, that is, cluster the labeled data and the unlabeled data in the local data separately to obtain the initial labeled data and unlabeled data. The local data can be images, privacy data, etc., and the tags it carries can be object categories in images, privacy data categories, etc., without specific limitations here.

[0052] All initially labeled data can form an initial labeled dataset, and all unlabeled data can form an unlabeled dataset. For example, the target server includes terminal A and terminal B. Terminal A's local data includes initially labeled data with labels 1, 2, and 3, and unlabeled data without labels 1 and 5. Terminal B's local data includes initially labeled data with labels 2, 3, 4, and 5, and unlabeled data without labels 1 and 4. For terminal A, its corresponding initial labeled dataset can be represented as LA, and its corresponding unlabeled dataset as UA; for terminal B, its corresponding initial labeled dataset can be represented as LB, and its corresponding unlabeled dataset as UB.

[0053] By utilizing the similarity between the unlabeled data of the target terminal and the initial labeled data of the target terminal, the unlabeled data of the target terminal can be labeled to obtain the first labeled data. For terminal A, all the first labeled data can constitute the first labeled dataset, which can be represented as U-Aa; for terminal B, all the first labeled data can constitute the first labeled dataset, which can be represented as U-Bb.

[0054] Here, the similarity between the unlabeled data and the initial labeled data of the target terminal can be characterized by Manhattan distance, Euclidean distance, Mahalanobis distance, or cosine distance. Using this similarity, and combining it with the labels attached to the initial labeled data of the target terminal, the unlabeled data of the target terminal can be labeled, thereby obtaining the first label of the unlabeled data of the target terminal, that is, obtaining the first labeled data consisting of the unlabeled data and the first label of the target terminal.

[0055] Here, you can select initial labeled data with a similarity greater than a first given threshold and use its labels as the first label of the unlabeled data, or you can select initial labeled data with the highest similarity and use its labels as the first label of the unlabeled data. The first given threshold can be set as needed and is not specifically limited here.

[0056] Then, step S2 is executed. By utilizing the first feature similarity between the unlabeled data of the target terminal and the initial labeled data of other terminals, the unlabeled data of the target terminal can be labeled to obtain the second labeled data. For example, for terminal A, all the second labeled data can constitute the second labeled dataset, which can be represented as U-Ab; for terminal B, all the second labeled data can constitute the second labeled dataset, which can be represented as U-Ba.

[0057] The first feature similarity between the unlabeled data of the target terminal and the initial labeled data of other terminals can be calculated using data features extracted by a unified feature extraction model. Then, using this first feature similarity and combining it with the labels attached to the initial labeled data of other terminals, the unlabeled data of the target terminal can be labeled, thereby obtaining the second label of the unlabeled data of the target terminal, i.e., the second labeled data consisting of the unlabeled data of the target terminal and its second label.

[0058] Here, initial labeled data from other terminals with a first feature similarity greater than a second given threshold can be selected, and their labels can be used as the labels for the unlabeled data of the target terminal. Alternatively, initial labeled data with the highest similarity from other terminals can be selected, and their labels can be used as the labels for the unlabeled data of the target terminal. The second given threshold can be equal to or different from the first given threshold, and can be determined according to the actual situation; no specific limitation is made here.

[0059] The third label data can be determined by using the second feature similarity between the initial label data of the target terminal and the initial label data of other terminals. This third label data consists of label data from the initial label data of the target terminal whose second feature similarity is greater than a preset threshold. This preset threshold can be determined based on actual circumstances and is not specifically limited here. For example, for terminals A and B, all the third label data can constitute a third label dataset, which can be represented as L-AB.

[0060] The second feature similarity between the initial label data of the target terminal and the initial label data of other terminals can be calculated using data features extracted by a unified feature extraction model. Furthermore, using this second feature similarity, similar data and their associated labels can be obtained from the initial label data of the target terminal and the initial label data of other terminals, thus obtaining the third label data.

[0061] Then, step S3 is executed, first aligning the labels of the target terminal with those of other terminals under the target server. The label alignment result is all the labels involved in the local data of each terminal under the target server, thus making the label categories involved in each terminal consistent. This ensures that each terminal uses a unified loss function during the federated learning process.

[0062] The initial label data, first label data, second label data, and third label data of the target terminal can be used together to form a new dataset. Using this new dataset and the label alignment results, the initial teacher model can be trained to obtain the target teacher model.

[0063] For example, for terminal A, the initial labeled dataset LA, the first labeled dataset U-Aa, the second labeled dataset U-Ab, and the third labeled dataset L-AB can be combined to form a new dataset NA. Using the new dataset NA and the label alignment results, the initial teacher model is trained to obtain the target teacher model TA. For terminal B, the initial labeled dataset LB, the first labeled dataset U-Bb, the second labeled dataset U-Ba, and the third labeled dataset L-AB can be combined to form a new dataset NB. Using the new dataset NB and the label alignment results, the initial teacher model is trained to obtain the target teacher model TB.

[0064] Understandably, due to the introduction of label alignment results, the output of the target teacher model includes predictions for all labels, including predictions for labels not originally present in the target terminal's initial label data. The introduction of the first label dataset U-Bb, the second label dataset U-Ba, and the third label dataset L-AB enhances the generalization ability of the resulting target teacher model.

[0065] Here, the initial teacher model can be built based on a neural network. When training the initial teacher model, data from the new dataset can be input into the initial teacher model to obtain the output of the initial teacher model. The loss function is calculated using the output and the labels of the input data, and the structural parameters of the initial teacher model are iteratively updated based on the loss function until the loss function converges, thus obtaining the target teacher model.

[0066] Subsequently, the target teacher model can be used to predict the labels of the initial label data of the target terminal, obtaining the soft labels for the initial label data of the target terminal. At this time, the initial label data of the target terminal carries hard labels. For example, for terminal A, the output result LA-soft of the target teacher model TA corresponding to the initial label dataset LA is used as the soft label of the initial label dataset LA, and the new dataset NA has no soft label; for terminal B, the output result LB-soft of the target teacher model TB corresponding to the initial label dataset LB is used as the soft label of the initial label dataset LB, and the new dataset NB has no soft label.

[0067] Finally, step S4 is executed, using the soft labels, initial label data, first label data, second label data, and third label data from the target terminal's initial label data to perform local distillation on the base model, obtaining the student model. Multiple rounds of local distillation can be performed here, with the end signal being convergence of the distillation loss.

[0068] For example, for terminal A, using the new dataset NA and the hard labels and soft labels of the initial label dataset LA, the base model can be locally distilled to obtain student model Sa; for terminal B, using the new dataset NB and the hard labels and soft labels of the initial label dataset LB, the base model can be locally distilled to obtain student model Sb.

[0069] Subsequently, the obtained student model can be used for federated learning, that is, the student model can be used to interact with the target server to achieve federated learning.

[0070] The federated learning distillation method for non-independent and identically distributed scenarios provided in this embodiment of the invention is applied to a target terminal. The data and / or labels of each terminal under the target server to which the target terminal belongs satisfy non-independent and identically distributed conditions. The method first determines the initial labeled data and unlabeled data of the target terminal. Based on the similarity between the unlabeled data and the initial labeled data of the target terminal, the unlabeled data is labeled to obtain first labeled data. Then, based on the first feature similarity between the unlabeled data and the initial labeled data of other terminals, the unlabeled data is labeled to obtain second labeled data. Finally, based on the initial labeled data of the target terminal and the initial labeled data of other terminals… The second feature similarity is used to determine the third label data. Subsequently, the target terminal is aligned with other terminals under the target server using labels. Based on the initial label data, the first label data, the second label data, the third label data, and the label alignment results, the initial teacher model is trained to obtain the target teacher model. Based on the target teacher model, label prediction is performed on the initial label data to obtain soft labels for the initial label data. Finally, based on the soft labels of the initial label data, the initial label data, the first label data, the second label data, and the third label data, the base model is locally distilled to obtain the student model. Federated learning is then performed based on the student model. This method, by leveraging the similarity between the unlabeled data of the target terminal and the initial label data of the target terminal, the first feature similarity with the initial label data of other terminals, and the second feature similarity between the initial label data of the target terminal and the initial label data of other terminals, can enrich the training samples of the initial teacher model and the base model. This not only significantly improves the training efficiency of the initial teacher model and the base model but also enhances the generalization ability of the obtained target teacher model and student model, thereby improving the accuracy of the aggregated model obtained through federated learning. Furthermore, this method combines knowledge distillation and federated learning, enabling the student model to learn knowledge from other terminals that are completely absent from its own data. This means that even if its own data lacks relevant labels, it can still acquire relevant knowledge through federated learning—an extreme scenario where data labels are not independently and identically distributed. Simultaneously, this method can further enhance the student model's existing fitting ability through federated learning.

[0071] Based on the above embodiments, the federated learning distillation method for non-independent and identically distributed scenarios provided in this embodiment of the invention determines the first feature similarity based on the following steps:

[0072] Determine an initial feature extraction model, and based on the initial feature extraction model, extract the first feature vector of the unlabeled data;

[0073] The preset structure in the initial feature extraction model is subjected to differential privacy protection to obtain the target feature extraction model;

[0074] The target feature extraction model is sent to the other terminal, and the second feature vector of its initial label data extracted by the other terminal based on the target feature extraction model is received.

[0075] The similarity between the first feature vector and the second feature vector is determined as the first feature similarity.

[0076] Specifically, calculating the first feature similarity between the unlabeled data of the target terminal and the initial labeled data of other terminals can be achieved through the following steps:

[0077] First, an initial feature extraction model can be determined. This initial feature extraction model can be determined by any terminal under the target server, or by a common node; no specific limitation is made here. This initial feature extraction model can be a convolutional neural network model.

[0078] Then, using this initial feature extraction model, the first feature vector of the unlabeled data of the target terminal can be extracted.

[0079] Subsequently, differential privacy protection can be applied to the preset structure in the initial feature extraction model to obtain the target feature extraction model. This preset structure can be a portion of the initial feature extraction model, such as the last few layers. Due to the introduction of differential privacy protection, other terminals cannot reconstruct the unlabeled data of the target terminal. Since differential privacy protection is applied only to a portion of the initial feature extraction model, the feature vectors calculated by different terminals are not significantly different.

[0080] Finally, the target feature extraction model can be sent to other terminals, which can then use it to extract the second feature vector from their own initial label data. Simultaneously, the first feature vector can also be sent to other terminals to enable them to calculate feature similarity.

[0081] After the target terminal receives the second feature vector, it can calculate the similarity between the first feature vector and the second feature vector. This similarity is the first feature similarity.

[0082] In this embodiment of the invention, the introduction of differential privacy protection prevents other terminals from reconstructing the unlabeled data of the target terminal, thus achieving the purpose of data privacy protection. Moreover, since differential privacy protection is applied only to a portion of the structure in the initial feature extraction model, the feature vectors calculated by different terminals are not significantly different, reducing the error caused by differential privacy protection.

[0083] Based on the above embodiments, the federated learning distillation method for non-independent and identically distributed scenarios provided in this embodiment of the invention determines the second feature similarity based on the following steps:

[0084] Based on the initial feature extraction model, the third feature vector of the initial label data of the target terminal is extracted;

[0085] The similarity between the third feature vector and the second feature vector is determined as the second feature similarity.

[0086] Specifically, calculating the initial tag data of the target terminal and the initial tag data of other terminals can be achieved through the following steps:

[0087] First, the initial feature extraction model can be used to extract the third feature vector of the initial label data of the target terminal.

[0088] Then, the similarity between the third feature vector and the second feature vector can be calculated, and this similarity is the second feature similarity.

[0089] In this embodiment of the invention, the introduction of differential privacy protection prevents other terminals from reconstructing the initial tag data of the target terminal, thus achieving the purpose of data privacy protection. Moreover, since differential privacy protection is applied only to a portion of the structure in the initial feature extraction model, the feature vectors calculated by different terminals are not significantly different, reducing the error caused by differential privacy protection.

[0090] Based on the above embodiments, the federated learning distillation method for non-independent and identically distributed scenarios provided in this embodiment of the invention, wherein the federated learning based on the student model includes:

[0091] Upload the student model to the target server;

[0092] The system receives the aggregated model obtained by the target server based on the federated average aggregation of the student models uploaded by each terminal, and uses the aggregated model as the base model to perform local distillation in a loop until the federated learning ends.

[0093] Specifically, when using student models for federated learning, the student models can first be uploaded to the target server. The target server then receives all student models uploaded by all terminals belonging to that target server and performs federated averaging aggregation to obtain the aggregated model S. The federated averaging aggregation method can be a weighted average of the structural parameters of all received student models.

[0094] Subsequently, the target terminal can receive the aggregated model, which can be achieved by directly downloading the aggregated model from the target server. Then, the aggregated model can be used as the base model again for iterative local distillation—that is, multiple rounds of local distillation to obtain student models, which are then uploaded to the target server. The target server's aggregated model is then received, and this process is repeated several times until the federated learning ends. At this point, a student model that can be used for label category prediction on the target terminal and an aggregated model on the target server are obtained, i.e., the federated model.

[0095] Based on the above embodiments, the federated learning distillation method for non-independent and identically distributed scenarios provided in this embodiment of the invention, wherein the unlabeled data is labeled based on the similarity between the unlabeled data and the initial labeled data of the target terminal to obtain first labeled data, includes:

[0096] The first similar data with the highest similarity to the unlabeled data in the initial label data of the target terminal is determined, and the unlabeled data is labeled based on the label of the first similar data to obtain the first labeled data.

[0097] Specifically, when determining the first tag data, we can first identify the first similar data in the initial tag data of the target terminal that has the highest similarity to the non-tag data of the target terminal. That is, for terminal A, find the first similar data of UA in LA, and for terminal B, find the first similar data of UB in LB.

[0098] Subsequently, using the labels carried by the first similar data, the unlabeled data of the target terminal is labeled to obtain the first labeled data corresponding to the unlabeled data of the target terminal. That is, for terminal A, UA is labeled using the labels carried by the first similar data of UA in LA to obtain U-Aa; for terminal B, UB is labeled using the labels carried by the first similar data of UB in LB to obtain U-Ab.

[0099] Based on the above embodiments, the federated learning distillation method for non-independent and identically distributed scenarios provided in this embodiment of the invention, wherein the unlabeled data is labeled based on the first feature similarity between the unlabeled data and the initial labeled data of the other terminals to obtain second labeled data, includes:

[0100] The second similar data with the highest similarity to the first feature of the unlabeled data in the initial label data of the other terminals is determined, and the unlabeled data is labeled based on the label of the second similar data to obtain the second labeled data.

[0101] Specifically, when determining the second tag data, we can first identify the second similar data among the initial tag data of other terminals that has the highest similarity to the first feature of the non-tag data of the target terminal. That is, for terminal A, we find the second similar data of UA in LB, and for terminal B, we find the second similar data of UB in LA.

[0102] Subsequently, using the labels carried by the second similar data, the unlabeled data of the target terminal is labeled to obtain the second labeled data corresponding to the unlabeled data of the target terminal. That is, for terminal A, UA is labeled using the labels carried by the second similar data of UA in LB to obtain U-Ab; for terminal B, UB is labeled using the labels carried by the second similar data of UB in LA to obtain U-Ba.

[0103] Based on the above embodiments, the federated learning distillation method for non-independent and identically distributed scenarios provided in this embodiment of the invention, wherein determining the third label data based on the second feature similarity between the initial label data of the target terminal and the initial label data of other terminals, includes:

[0104] Calculate the average label value of the third similar data and the fourth similar data. The third similar data is the initial label data of the target terminal corresponding to the second feature similarity greater than a preset threshold. The fourth similar data is the initial label data of the other terminal corresponding to the second feature similarity greater than the preset threshold.

[0105] The average value of the labels is used as the label of the third similar data to obtain the third label data.

[0106] Specifically, when determining the third label data, the labels of the third similar data and the fourth similar data can be determined first, and then the average label value of the two can be calculated. The third similar data refers to the initial label data of the target terminal corresponding to a second feature similarity greater than a preset threshold, and the fourth similar data refers to the initial label data of other terminals corresponding to a second feature similarity greater than a preset threshold. For example, for terminals A and B, LA and LB are compared for similarity to determine the labels of the third similar data in LA and the labels of the fourth similar data in LB. Then, the average label value is calculated, and the labels of the third similar data in LA and the labels of the fourth similar data in LB are replaced with this average label value. Afterwards, for terminal A, the third label data consisting of the third similar data and its corresponding average label value is obtained; for terminal B, the third label data consisting of the fourth similar data and its corresponding average label value is obtained.

[0107] like Figure 2As shown, based on the above embodiments, this embodiment of the invention provides a federated learning distillation device for non-independent and identically distributed scenarios, applied to a target terminal, wherein the data and / or tags of each terminal under the target server to which the target terminal belongs satisfy non-independent and identically distributed conditions; the device includes:

[0108] The data aggregation module 21 is used to determine the initial labeled data and unlabeled data of the target terminal, and to label the unlabeled data based on the similarity between the unlabeled data and the initial labeled data of the target terminal to obtain the first labeled data.

[0109] The data labeling module 22 is used to align the target terminal with other terminals under the target server, and label the unlabeled data based on the first feature similarity between the unlabeled data and the initial label data of the other terminals to obtain second label data, and determine third label data based on the second feature similarity between the initial label data of the target terminal and the initial label data of the other terminals.

[0110] The label prediction module 23 is used to train an initial teacher model based on the initial label data of the target terminal, the first label data, the second label data, the third label data, and the label alignment result to obtain a target teacher model, and to perform label prediction on the initial label data of the target terminal based on the target teacher model to obtain soft labels for the initial label data of the target terminal.

[0111] The federated distillation module 24 is used to perform local distillation on the base model based on the soft labels of the initial label data of the target terminal, the initial label data of the target terminal, the first label data, the second label data, and the third label data to obtain a student model, and to perform federated learning based on the student model.

[0112] Based on the above embodiments, the federated learning distillation apparatus for non-independent and identically distributed scenarios provided in this embodiment of the invention further includes a feature similarity determination module, used for:

[0113] Determine an initial feature extraction model, and based on the initial feature extraction model, extract the first feature vector of the unlabeled data;

[0114] The preset structure in the initial feature extraction model is subjected to differential privacy protection to obtain the target feature extraction model;

[0115] The target feature extraction model is sent to the other terminal, and the second feature vector of its initial label data extracted by the other terminal based on the target feature extraction model is received.

[0116] The similarity between the first feature vector and the second feature vector is determined as the first feature similarity.

[0117] Based on the above embodiments, the federated learning distillation apparatus for non-independent and identically distributed scenarios provided in this embodiment of the invention further includes a feature similarity determination module for:

[0118] Based on the initial feature extraction model, the third feature vector of the initial label data of the target terminal is extracted;

[0119] The similarity between the third feature vector and the second feature vector is determined as the second feature similarity.

[0120] Based on the above embodiments, the federated learning distillation apparatus for non-independent and identically distributed scenarios provided in this embodiment of the invention, wherein the federated distillation module is specifically used for:

[0121] Upload the student model to the target server;

[0122] The system receives the aggregated model obtained by the target server based on the federated average aggregation of the student models uploaded by each terminal, and uses the aggregated model as the base model to perform local distillation in a loop until the federated learning ends.

[0123] Based on the above embodiments, the federated learning distillation apparatus for non-independent and identically distributed scenarios provided in this embodiment of the invention, wherein the data aggregation module is specifically used for:

[0124] The first similar data with the highest similarity to the unlabeled data in the initial label data of the target terminal is determined, and the unlabeled data is labeled based on the label of the first similar data to obtain the first labeled data.

[0125] Based on the above embodiments, the federated learning distillation apparatus for non-independent and identically distributed scenarios provided in this embodiment of the invention further includes a data annotation module for:

[0126] The second similar data with the highest similarity to the first feature of the unlabeled data in the initial label data of the other terminals is determined, and the unlabeled data is labeled based on the label of the second similar data to obtain the second labeled data.

[0127] Based on the above embodiments, the federated learning distillation apparatus for non-independent and identically distributed scenarios provided in this embodiment of the invention further includes a data annotation module for:

[0128] Calculate the average label value of the third similar data and the fourth similar data. The third similar data is the initial label data of the target terminal corresponding to the second feature similarity greater than a preset threshold. The fourth similar data is the initial label data of the other terminal corresponding to the second feature similarity greater than the preset threshold.

[0129] The average value of the labels is used as the label of the third similar data to obtain the third label data.

[0130] Specifically, the functions of each module in the federated learning distillation apparatus for non-independent and identically distributed scenarios provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above-described method-like embodiments, and the achieved effects are also consistent. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0131] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute the federated learning distillation method for non-independent and identically distributed scenarios provided in the above embodiments. This method is applied to a target terminal, where the data and / or labels of terminals under the target server to which the target terminal belongs satisfy non-independent and identically distributed conditions. The method includes: determining the initial label data and unlabeled data of the target terminal, and labeling the unlabeled data based on the similarity between the unlabeled data and the initial label data of the target terminal to obtain first label data; labeling the unlabeled data based on the first feature similarity between the unlabeled data and the initial label data of other terminals to obtain second label data; and labeling the unlabeled data based on the similarity between the initial label data of the target terminal and the initial label data of other terminals. The second feature similarity is used to determine the third label data; the target terminal is aligned with other terminals under the target server in terms of label, and the initial teacher model is trained based on the initial label data of the target terminal, the first label data, the second label data, the third label data, and the label alignment result to obtain the target teacher model. Based on the target teacher model, the initial label data of the target terminal is predicted to obtain the soft label of the initial label data of the target terminal; based on the soft label of the initial label data of the target terminal, the initial label data of the target terminal, the first label data, the second label data, and the third label data, the base model is locally distilled to obtain the student model, and federated learning is performed based on the student model.

[0132] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the federated learning distillation method for non-independent and identically distributed scenarios provided in the above embodiments. This method is applied to a target terminal, where the data and / or labels of each terminal under the target server to which the target terminal belongs satisfy non-independent and identically distributed conditions. The method includes: determining the initial label data and unlabeled data of the target terminal, and labeling the unlabeled data based on the similarity between the unlabeled data and the initial label data of the target terminal to obtain first label data; labeling the unlabeled data based on the first feature similarity between the unlabeled data and the initial label data of the other terminals to obtain second label data; and based on... The third label data is determined by the second feature similarity between the initial label data of the target terminal and the initial label data of other terminals. The target terminal is then aligned with the labels of other terminals under the target server. Based on the initial label data of the target terminal, the first label data, the second label data, the third label data, and the label alignment result, an initial teacher model is trained to obtain a target teacher model. Based on the target teacher model, label prediction is performed on the initial label data of the target terminal to obtain soft labels for the initial label data of the target terminal. Based on the soft labels of the initial label data of the target terminal, the initial label data of the target terminal, the first label data, the second label data, and the third label data, the base model is locally distilled to obtain a student model. Federated learning is then performed based on the student model.

[0134] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, this computer program implements the federated learning distillation method for non-independent and identically distributed scenarios provided in the above embodiments. This method is applied to a target terminal, where the data and / or labels of terminals under a target server to which the target terminal belongs satisfy non-independent and identically distributed conditions. The method includes: determining initial labeled data and unlabeled data of the target terminal; labeling the unlabeled data based on the similarity between the unlabeled data and the initial labeled data of the target terminal to obtain first labeled data; labeling the unlabeled data based on the first feature similarity between the unlabeled data and the initial labeled data of other terminals to obtain second labeled data; and further labeling the unlabeled data based on the first feature similarity between the initial labeled data of the target terminal and the initial labeled data of other terminals to obtain second labeled data. The second feature similarity of the initial label data of the other terminals is used to determine the third label data; the target terminal is aligned with the other terminals under the target server, and the initial teacher model is trained based on the initial label data of the target terminal, the first label data, the second label data, the third label data, and the label alignment result to obtain the target teacher model; based on the target teacher model, the initial label data of the target terminal is predicted to obtain the soft label of the initial label data of the target terminal; based on the soft label of the initial label data of the target terminal, the initial label data of the target terminal, the first label data, the second label data, and the third label data, the base model is locally distilled to obtain the student model, and federated learning is performed based on the student model.

[0135] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A federated learning distillation method for non-independent and identically distributed scenarios, characterized in that, Applied to a target terminal, wherein the data and / or tags of each terminal under the target server to which the target terminal belongs satisfy the condition of non-independent and identically distributed; the method includes: The initial labeled data and unlabeled data of the target terminal are determined, and the unlabeled data is labeled based on the similarity between the unlabeled data and the initial labeled data of the target terminal to obtain the first labeled data; The target terminal is aligned with other terminals under the target server. Based on the first feature similarity between the unlabeled data and the initial label data of the other terminals, the unlabeled data is labeled to obtain the second label data. Based on the second feature similarity between the initial label data of the target terminal and the initial label data of the other terminals, the third label data is determined. Based on the initial label data of the target terminal, the first label data, the second label data, the third label data, and the label alignment results, the initial teacher model is trained to obtain the target teacher model. Based on the target teacher model, label prediction is performed on the initial label data of the target terminal to obtain the soft labels of the initial label data of the target terminal. Based on the soft labels of the initial label data of the target terminal, the initial label data of the target terminal, the first label data, the second label data, and the third label data, the base model is locally distilled to obtain the student model, and federated learning is performed based on the student model. The initial labeled data and the unlabeled data are obtained by clustering local data by the target terminal. The local data includes images or privacy data, and the labels carried by the initial labeled data include the object category in the image or the privacy data category.

2. The federated learning distillation method for non-independent and identically distributed scenarios according to claim 1, characterized in that, The first feature similarity is determined based on the following steps: Determine an initial feature extraction model, and based on the initial feature extraction model, extract the first feature vector of the unlabeled data; The preset structure in the initial feature extraction model is subjected to differential privacy protection to obtain the target feature extraction model; The target feature extraction model is sent to the other terminal, and the second feature vector of its initial label data extracted by the other terminal based on the target feature extraction model is received. The similarity between the first feature vector and the second feature vector is determined as the first feature similarity.

3. The federated learning distillation method for non-independent and identically distributed scenarios according to claim 2, characterized in that, The second feature similarity is determined based on the following steps: Based on the initial feature extraction model, the third feature vector of the initial label data of the target terminal is extracted; The similarity between the third feature vector and the second feature vector is determined as the second feature similarity.

4. The federated learning distillation method for non-independent and identically distributed scenarios according to any one of claims 1-3, characterized in that, The federated learning based on the student model includes: Upload the student model to the target server; The system receives the aggregated model obtained by the target server based on the federated average aggregation of the student models uploaded by each terminal, and uses the aggregated model as the base model to perform local distillation in a loop until the federated learning ends.

5. The federated learning distillation method for non-independent and identically distributed scenarios according to any one of claims 1-3, characterized in that, The step of labeling the unlabeled data based on the similarity between the unlabeled data and the initial label data of the target terminal to obtain the first label data includes: The first similar data with the highest similarity to the unlabeled data in the initial label data of the target terminal is determined, and the unlabeled data is labeled based on the label of the first similar data to obtain the first label data.

6. The federated learning distillation method for non-independent and identically distributed scenarios according to any one of claims 1-3, characterized in that, The step of labeling the unlabeled data based on the first feature similarity between the unlabeled data and the initial labeled data of the other terminals to obtain second labeled data includes: The second similar data with the highest similarity to the first feature of the unlabeled data in the initial label data of the other terminals is determined, and the unlabeled data is labeled based on the label of the second similar data to obtain the second label data.

7. The federated learning distillation method for non-independent and identically distributed scenarios according to any one of claims 1-3, characterized in that, The determination of the third tag data based on the second feature similarity between the initial tag data of the target terminal and the initial tag data of the other terminals includes: Calculate the average label value of the third similar data and the fourth similar data. The third similar data is the initial label data of the target terminal corresponding to the second feature similarity greater than a preset threshold. The fourth similar data is the initial label data of the other terminal corresponding to the second feature similarity greater than the preset threshold. The average value of the labels is used as the label of the third similar data to obtain the third label data.

8. A federated learning distillation apparatus for non-independent, identically distributed scenarios, characterized in that, Applied to a target terminal, wherein the data and / or tags of each terminal under the target server to which the target terminal belongs satisfy the condition of non-independent identical distribution; the device includes: The data aggregation module is used to determine the initial labeled data and unlabeled data of the target terminal, and to label the unlabeled data based on the similarity between the unlabeled data and the initial labeled data of the target terminal to obtain the first labeled data. The data labeling module is used to align the target terminal with other terminals under the target server, and label the unlabeled data based on the first feature similarity between the unlabeled data and the initial label data of the other terminals to obtain second label data. Based on the second feature similarity between the initial label data of the target terminal and the initial label data of the other terminals, the module determines third label data. The label prediction module is used to train an initial teacher model based on the initial label data of the target terminal, the first label data, the second label data, the third label data, and the label alignment result to obtain a target teacher model, and to perform label prediction on the initial label data of the target terminal based on the target teacher model to obtain soft labels for the initial label data of the target terminal. The federated distillation module is used to perform local distillation on the base model based on the soft labels of the initial label data of the target terminal, the initial label data of the target terminal, the first label data, the second label data, and the third label data to obtain a student model, and to perform federated learning based on the student model. The initial labeled data and the unlabeled data are obtained by clustering local data by the target terminal. The local data includes images or privacy data, and the labels carried by the initial labeled data include the object category in the image or the privacy data category.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the federated learning distillation method for non-independent and identically distributed scenarios as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the federated learning distillation method for non-independent and identically distributed scenarios as described in any one of claims 1-7.

Images