Privacy protection knowledge distillation-based single neural network federated learning method
By adopting the privacy protection knowledge distillation method in federated learning, using secret sharing and secure two-party computing technology, the large communication overhead, privacy issues and model integrity problems in traditional federated learning are solved, and efficient privacy protection and robust model training are achieved.
Patent Information
- Application Number
- CN202411683072.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-22
- Publication Date
- 2025-04-25
AI Technical Summary
Traditional federated learning has challenges with high communication overhead, privacy issues and model integrity issues, especially when using knowledge distillation technology, which makes it difficult to ensure robustness and privacy protection.
A single-time neural network federated learning method based on privacy protection knowledge distillation is adopted, and local model parameters are distributed to dual servers through secret sharing technology. The abnormal output results are removed using secure two-party computing technology and HDBScan clustering algorithm to form a semi-supervised learning data set and knowledge distillation is completed.
It effectively reduces the traffic of the client, protects the privacy of the local model, and improves the robustness and perfection of the global model.
Smart Images
Figure CN119886281A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of privacy computing technology, and specifically to a single-shot neural network federated learning method based on privacy-preserving knowledge distillation. Background Art
[0002] In recent years, data privacy issues have been increasingly valued, and the demand for collective and personal data to remain local has been increasing. On the other hand, machine learning and deep learning algorithms based on big data require a large amount of data empowerment, and the data island phenomenon has obviously hindered the current development of artificial intelligence. The introduction of federated learning breaks the shackles of data islands. This technology can realize machine learning or deep learning model training under the premise of ensuring the privacy of data not leaving the local area.
[0003] However, traditional federated learning still faces many challenges, including:
[0004] 1. High communication overhead: In the traditional federated learning training process, the client needs to communicate with the server for multiple rounds, and the overhead of each round of communication is very high. When the client's communication is unstable, it is easy to cause the knowledge of the client's local model to fail to be properly transferred to the global model;
[0005] 2. Privacy issues: The adversary can steal the client’s local dataset information by analyzing the local model submitted by the client;
[0006] 3. Model integrity issue: The adversary can bias the training of the global model in a direction selected by the adversary or disrupt model training by sending an abnormal local model to the server.
[0007] In order to solve the problem of high communication overhead in traditional federated learning, some single-shot federated learning schemes based on knowledge distillation technology have been proposed. This technology allows the client to complete federated learning with only one round of communication with the server, which greatly reduces the client's communication volume. However, knowledge distillation does not take into account model privacy and abnormal models, which leads to the security issues in traditional federated learning still existing in this scenario. In order to protect local data privacy, Qinbin Li et al. proposed the FedKT scheme, which uses two-layer knowledge distillation and differential privacy technology to protect the privacy of the data set; however, since knowledge distillation technology will reduce the model quality to a certain extent, and differential privacy technology also reduces the model quality to a certain extent, this leads to a certain gap between the global model and the ideal global model.
[0008] Therefore, a technology to improve the completeness of the global model is urgently needed. Summary of the invention
[0009] The purpose of this application is to provide a single neural network federated learning method based on privacy-preserving knowledge distillation, so as to solve the defects of the prior art that robustness evaluation and privacy leakage cannot be achieved in the scenario of using knowledge distillation to realize single federated learning.
[0010] To achieve the above objectives, the present application discloses the following technical solution: a single-shot neural network federated learning method based on privacy-preserving knowledge distillation, comprising the following steps:
[0011] S1. After the client completes local training using the local private data set, it distributes the local model parameters to the dual servers using secret sharing technology, and sends its own unique identification code to the dual servers as an identity identifier;
[0012] S2. After receiving the model share from the client, the two servers confirm the client identification code received by each other, and find the intersection. At the same time, remove the local model share of the remaining clients, use the secure two-party computing technology to predict the public data set for each local model, and obtain the output results of all models;
[0013] S3. The server removes the prediction results that do not match the prediction results of most models, vectorizes the remaining prediction results, and combines the result vector with the original public data set and the server's private data set to form a semi-supervised training data set. Based on the semi-supervised learning data set, the server completes knowledge distillation semi-supervised learning to obtain a global model.
[0014] Preferably, the S1 specifically includes:
[0015] S11. Before training begins, the client applies to the server for a unique identification code C i As an identity marker;
[0016] S12. Set up the client The local model parameter vector is w i , using arithmetic sharing to divide it into {[w i ]0,[w i ]1}, where w i =[w i ]0+[w i ]1. Client C i Send its model secret share [w i ] b Its unique identification code C i , where b=0 or 1.
[0017] Preferably, in S2, after receiving the model share from the client, the two servers mutually confirm the client identification code received by the other party, and find the intersection, and at the same time remove the local model share of the remaining clients, which specifically includes:
[0018] S21, server b sends the received client identification code set to server 1-b Find the intersection
[0019] S22. Server b removes the remaining model shares submitted by the client.
[0020] Preferably, in S2, the use of secure two-party computing technology to predict the public data set for each local model respectively and obtain the output results of all models specifically includes:
[0021] S23, the server uses secure two-party computing technology to Input to each local model w i , get the output results of each local model;
[0022] S24. For public datasets For each data point x in The predicted value of each local model is set to Y x ={y 1,x ,...,y n,x}, where y i,x is the predicted value of the i-th local model.
[0023] Preferably, in S3, the server removes prediction results that are inconsistent with the prediction results of most models, and performs vector aggregation on the remaining prediction results, specifically including:
[0024] S31, the server calculates y in turn i,x The Euclidean distance d to other predicted values i,j , forming a distance matrix D = (d i,j ), input the distance matrix into the HDBScan clustering algorithm, find and remove outliers, and average the remaining prediction values to get the final prediction value y x .
[0025] Preferably, in S3, the obtained result vector is combined with the original public data set and the private data set of the server to form a semi-supervised training data set. Based on the semi-supervised learning data set, the server completes knowledge distillation-type semi-supervised learning to obtain a global model, which specifically includes:
[0026] S32, the predicted value of the public data set is The server compares it with the original dataset Combined with the server's private dataset to form a semi-supervised learning dataset
[0027] S33, Server in Semi-Supervised Learning Dataset The knowledge distillation-style semi-supervised learning is completed to obtain the global model.
[0028] Preferably, a single-shot neural network federated learning system based on privacy-preserving knowledge distillation that applies the single-shot neural network federated learning method based on privacy-preserving knowledge distillation includes:
[0029] The distribution module is used to distribute the local model parameter vector to the dual servers using secret sharing technology after the client completes model training locally;
[0030] The acquisition module is used for the server to confirm the client identification code with another server after receiving the model share from the client and to obtain the intersection thereof;
[0031] The computing module is used by the server to input the public data set into each local model using secure two-party computing technology, calculate its output results, and remove those whose prediction results do not match those of most models;
[0032] The training module is used for the server to combine the output results with the public data set and its private data set to form a semi-supervised learning data set, and complete knowledge distillation-type semi-supervised learning based on the semi-supervised learning data set to obtain the global model and publish it.
[0033] Beneficial effects: In the single-shot neural network federated learning method based on privacy-preserving knowledge distillation of the present application, the client sends the local model to the dual servers using secret sharing. The dual servers use secure two-party computing technology to input the public data set into each local model and obtain its output results, and use the HDBScan clustering algorithm to remove abnormal output results, and then aggregate the remaining outputs and synthesize them with the public data set into a semi-supervised learning data set to complete knowledge distillation. This example can protect the privacy of the local model by using secret sharing. At the same time, the HDBScan clustering algorithm is used to remove abnormal output results, which can ensure the robustness of the global model. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.
[0035] Figure 1 A schematic diagram of the process flow of a single neural network federated learning method for privacy-preserving knowledge distillation provided in an embodiment of the present application;
[0036] Figure 2 A schematic diagram of dual-server interaction provided in an embodiment of the present application;
[0037] Figure 3 A schematic diagram of the server knowledge distillation process provided in an embodiment of the present application. DETAILED DESCRIPTION
[0038] The technical solutions in the embodiments of the present application are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.
[0039] In this article, the term "comprising" is intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprising..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements.
[0040] This embodiment provides a single-shot neural network federated learning method based on privacy-preserving knowledge distillation, involving participants including multiple clients and two servers.
[0041] The functions of the client are: to provide a trained local private data set, implement local model training, and secretly share local model parameters with two servers; in terms of security, the client may be controlled by malicious attackers, submit malicious model updates to the system, disrupt the model training process, and steal the private data of other clients. Due to network reasons or the attacker's reasons, the client may be disconnected.
[0042] Among them, the functions of the server are: using secure two-party computing technology, inputting the public data set into each local model to obtain its output, and using the HDBScan clustering algorithm to remove outliers, aggregating the remaining prediction values and combining them with the original data set into a semi-supervised learning data set, and completing knowledge distillation on this basis to obtain a global model; in terms of security, based on the assumption of non-collusion of the two servers (in reality, this may represent two companies. If the two companies declare non-collusion and collusion is found during an audit, it will have a serious impact on the integrity of the companies, so this assumption is established), and will follow the program steps to obtain a global model, but is curious about the local model submitted by the client.
[0043] Based on the above description of the client and server functions, refer to Figure 1-3 As shown, the single-shot neural network federated learning method based on privacy-preserving knowledge distillation in this embodiment specifically includes the following steps:
[0044] After the client completes local training using a local private data set, it distributes the local model parameters to the two servers using secret sharing technology, and sends its own unique identification code to the two servers as an identity;
[0045] After the server receives the model share from the client, the two servers confirm the client identification code received by each other and find the intersection. At the same time, the local model share of the remaining clients is removed, and the public data set is predicted for each local model using secure two-party computing technology to obtain the output results of all models.
[0046] If the server's prediction results do not match those of most models, they will be removed and the remaining prediction result vectors will be aggregated. The resulting result vectors will be combined with the original public data set and the server's private data set to form a semi-supervised training data set. Based on this data set, the server completes knowledge distillation-style semi-supervised learning to obtain a global model.
[0047] In this embodiment, the client sends the local model to the dual servers using secret sharing. The dual servers use secure two-party computing technology to input the public data set into each local model and obtain its output results. The HDBScan clustering algorithm is used to remove abnormal output results, and the remaining outputs are aggregated and combined with the public data set to form a semi-supervised learning data set to complete knowledge distillation. This example can protect the privacy of the local model by using secret sharing. At the same time, the HDBScan clustering algorithm is used to remove abnormal output results, which can ensure the robustness of the global model.
[0048] Specifically, after the client completes local training using a local private data set, it distributes the local model parameters to the two servers using a secret sharing technique, and simultaneously sends its own unique identification code to the two servers as an identity identifier, specifically including:
[0049] Before training begins, the client applies to the server for a unique identification code C i As an identity marker;
[0050] Set up client The local model parameter vector is w i , use arithmetic sharing to divide it into 2 parts: {[w i ]0,[w i ]1}, so that w i =[w i ]0+[w i ]1. Client C i Send its model secret sharing share [w i ] b and its unique identification code.
[0051] Specifically, after the server receives the model share from the client, the two servers confirm the client identification code received by each other, and find the intersection, and remove the local model share of the remaining clients, specifically including:
[0052] Server b sends the client identification code set it received to server 1-b Find the intersection
[0053]
[0054] Server b removes the remaining clients (i.e. ) submitted model sharing.
[0055] Specifically, the server uses secure two-party computing technology to predict the public data set for each local model and obtain the output results of all models, including:
[0056] The server uses secure two-party computing technology to convert public data sets Input to each local model w i , obtain its output value. Specifically, for a neural network, the server inputs data into each layer in turn, and shares the output of each layer with another server to restore its value, and inputs it into the next layer, and the cycle continues until the output of the last layer is obtained;
[0057] For each data point in the dataset The predicted value of each local model is set to Y x ={y 1,x,...,y n,x}, where y i,x is the predicted value of the i-th local model.
[0058] Specifically, the server removes the prediction results that do not match the prediction results of most models, and aggregates the remaining prediction result vectors, specifically including:
[0059] The server calculates y i,x The Euclidean distance d to other predicted values i,j , forming a distance matrix D = (d i,j ), input the distance matrix into the HDBScan clustering algorithm, find and remove outliers, and average the remaining prediction values to get the final prediction value y x .
[0060] Specifically, the result vector is combined with the original public data set and the server's private data set to form a semi-supervised training data set. Based on this data set, the server completes knowledge distillation-style semi-supervised learning to obtain a global model, which specifically includes:
[0061] The predicted value of the public dataset is The server compares it with the original dataset Combined with the server's private dataset to form a semi-supervised learning dataset
[0062] The server completes knowledge distillation-based semi-supervised learning on this dataset to obtain a global model.
[0063] Corresponding to the above-mentioned single-shot neural network federated learning method based on privacy-preserving knowledge distillation, this embodiment further provides a single-shot neural network federated learning system based on privacy-preserving knowledge distillation applying the method, including:
[0064] The distribution module is used to distribute the local model parameter vector to the dual servers using secret sharing technology after the client completes model training locally;
[0065] The acquisition module is used for the server to confirm the client identification code with another server after receiving the model share from the client and to obtain the intersection thereof;
[0066] The computing module is used by the server to input the public data set into each local model using secure two-party computing technology, calculate its output results, and remove those whose prediction results do not match those of most models;
[0067] The training module is used for the server to combine the output results with the public data set and its private data set to form a semi-supervised learning data set, and complete knowledge distillation-type semi-supervised learning based on the semi-supervised learning data set to obtain the global model and publish it.
[0068] In summary, the solution of this embodiment is mainly composed of two different entities: a client (which provides data and can implement local training of machine learning / deep learning) and a server (two non-collusive servers with a common public data set). After the client completes the training task locally, it shares the model parameters with the two servers through secret sharing technology; based on secure two-party computing technology, the server inputs the public data set into each local model in turn, obtains its prediction value (secret), removes abnormal prediction values, and calculates the average prediction value. The server then uses the prediction value combined with the public data set to complete knowledge distillation to obtain a global model. In the scenario where knowledge distillation technology is used, this invention uses secure two-party computing to ensure the privacy of the local model, so that the knowledge in the local model can be transferred to the global model to the maximum extent.
[0069] In the embodiments provided in the present application, it should be understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, code or any appropriate combination thereof. For hardware implementation, the processor can be implemented in one or more of the following units: application specific integrated circuit (ASIC), digital signal processor (DSP), digital signal processing device (DSPD), programmable logic device (PLD), field programmable gate array (FPGA), processor, controller, microcontroller, microprocessor, other electronic units designed to implement the functions described herein or their combination. For software implementation, part or all of the flow of the embodiment can be completed by instructing the relevant hardware through a computer program. When implemented, the above program can be stored in a computer-readable storage medium or transmitted as one or more instructions or codes on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein the communication medium includes any medium that is convenient for transmitting a computer program from one place to another. The storage medium can be any available medium that a computer can access. The computer-readable storage medium can include but is not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer.
[0070] Finally, it should be noted that the above are only preferred embodiments of the present application and are not intended to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.
Claims
1. A single-shot neural network federated learning method based on privacy-preserving knowledge distillation, characterized in that: The following steps are involved: S1. After the client completes local training using the local private data set, it distributes the local model parameters to the dual servers using secret sharing technology, and sends its own unique identification code to the dual servers as an identity identifier; S2. After receiving the model share from the client, the two servers confirm the client identification code received by each other, and find the intersection. At the same time, remove the local model share of the remaining clients, use the secure two-party computing technology to predict the public data set for each local model, and obtain the output results of all models; S3. The server removes the prediction results that do not match the prediction results of most models, vectorizes the remaining prediction results, and combines the result vector with the original public data set and the server's private data set to form a semi-supervised training data set. Based on the semi-supervised learning data set, the server completes knowledge distillation semi-supervised learning to obtain a global model.
2. The single-shot neural network federated learning method based on privacy-preserving knowledge distillation according to claim 1 is characterized in that: The S1 specifically includes: S11. Before training begins, the client applies to the server for a unique identification code C i As an identity marker; S12. Set up the client The local model parameter vector is w i , using arithmetic sharing to divide it into {[w i ]0,[w i ]1}, where w i =[w i ]0+[w i ]1. Client C i Send its model secret share [w i ] b Its unique identification code C i , where b=0 or 1.
3. The single-shot neural network federated learning method based on privacy-preserving knowledge distillation according to claim 2 is characterized in that: In S2, after receiving the model share from the client, the two servers mutually confirm the client identification code received by the other party, and find the intersection, while removing the local model share of the remaining clients, specifically including: S21, server b sends the received client identification code set to server 1-b Find the intersection S22. Server b removes the remaining model shares submitted by the client.
4. The single-shot neural network federated learning method based on privacy-preserving knowledge distillation according to claim 3 is characterized in that: In S2, the secure two-party computing technology is used to predict the public data set for each local model and obtain the output results of all models, specifically including: S23, the server uses secure two-party computing technology to Input to each local model w i , get the output results of each local model; S24. For public datasets For each data point x in The predicted value of each local model is set to Y x ={y 1,x ,...,y n,x }, where y i,x is the predicted value of the i-th local model.
5. The single-shot neural network federated learning method based on privacy-preserving knowledge distillation according to claim 4 is characterized in that: In S3, the server removes the prediction results that do not match the prediction results of most models, and performs vector aggregation on the remaining prediction results, specifically including: S31, the server calculates y in turn i,x The Euclidean distance d to other predicted values i,j , forming a distance matrix D = (d i,j ), input the distance matrix into the HDBScan clustering algorithm, find and remove outliers, and average the remaining prediction values to get the final prediction value y x .
6. The single-shot neural network federated learning method based on privacy-preserving knowledge distillation according to claim 5 is characterized in that: In S3, the obtained result vector is combined with the original public data set and the server's private data set to form a semi-supervised training data set. Based on the semi-supervised learning data set, the server completes knowledge distillation-type semi-supervised learning to obtain a global model, which specifically includes: S32, the predicted value of the public data set is The server compares it with the original dataset Combined with the server's private dataset to form a semi-supervised learning dataset S33, Server in Semi-Supervised Learning Dataset The knowledge distillation-style semi-supervised learning is completed to obtain the global model.
7. The single-shot neural network federated learning method based on privacy-preserving knowledge distillation according to any one of claims 1 to 6, characterized in that: A single-shot neural network federated learning system based on privacy-preserving knowledge distillation using the single-shot neural network federated learning method based on privacy-preserving knowledge distillation includes: The distribution module is used to distribute the local model parameter vector to the dual servers using secret sharing technology after the client completes model training locally; The acquisition module is used for the server to confirm the client identification code with another server after receiving the model share from the client and to obtain the intersection thereof; The computing module is used by the server to input the public data set into each local model using secure two-party computing technology, calculate its output results, and remove those whose prediction results do not match those of most models; The training module is used for the server to combine the output results with the public data set and its private data set to form a semi-supervised learning data set, and complete knowledge distillation-type semi-supervised learning based on the semi-supervised learning data set to obtain the global model and publish it.