Federal learning encryption system and method based on model structure transformation

Abstract

The invention provides a federated learning encryption system and method based on model structure transformation. The system comprises computing equipment participating in training and a central server, the computing devices participating in training are connected with the central server in a wireless communication mode. According to the method, the global model is converted into the private local model structure before federated learning local training through model conversion, and the private local model structure is converted back to the global structure after the local training is finished, so that the local model structure of the local training is highly confidential in the whole federated learning training process, and the accuracy of the federated learning training is improved. Therefore, the central server or an external attacker cannot reversely deduce the private data information of the local client through the uploaded gradient information; compared with a traditional federated learning encryption method, the method has the advantages that local model information is encrypted through lightweight model conversion, gradient information is encrypted in an implicit mode, the privacy of federated learning is remarkably improved, and the method has very high practical value.

Description

Technical Field

[0001] The present invention relates to a federated learning encryption system and method based on model structure transformation, and belongs to the technical field of data processing. Background Art

[0002] Federated Learning (FL) is a technique that allows multiple devices or servers to collaborate in training machine learning models without exchanging the local data they hold. This approach ideally protects data privacy because the original data does not leave its physical location. However, even though the data is not directly exchanged, it is still possible for a malicious server or attacker to infer or reconstruct the client's private data by analyzing the gradient information uploaded from the client, a method called a gradient attack.

[0003] Gradient attacks usually use the gradient information uploaded by participants to recover the original data or obtain other sensitive attributes through reverse engineering. This not only threatens personal privacy, but may also damage the trust foundation of the entire federated learning network. In the face of this problem, existing defense strategies mainly focus on encryption technology, differential privacy, secure multi-party computing and other fields, but these methods have their own limitations. For example, although encryption technology can protect data from unauthorized access, it usually increases computational complexity and communication costs. Differential privacy masks gradients by adding noise, which can effectively prevent data leakage, but may sacrifice the accuracy of the model. Secure multi-party computing can perform joint calculations without exposing the inputs of each party, but its implementation is complex and difficult to deploy efficiently in large-scale federated learning environments.

[0004] The present invention aims at the gradient attack problem in federated learning and provides a federated learning encryption system and method based on model structure transformation. By performing model structure transformation before and after local training, encryption of the training model on the local device is achieved, thereby effectively preventing gradient reconstruction attacks initiated by the central server and effectively protecting the data privacy of the device without affecting the accuracy of local model training. Summary of the invention

[0005] In order to solve the problems of low computational efficiency and loss of model performance in existing encryption methods, the present invention proposes a federated learning encryption system and method based on model structure transformation.

[0006] The technical solution adopted by the present invention to solve the above problems is: the present invention proposes a federated learning encryption system based on model structure transformation, comprising:

[0007] Computing devices and central servers involved in training;

[0008] The computing devices involved in the training are connected to the central server via wireless communication.

[0009] Optionally, the computing device involved in the training includes a computing resource estimation submodule, a network parameter privacy sensitivity estimation submodule, and a model structure transformation submodule;

[0010] The computing resource estimation submodule is used to estimate the currently available running memory, video memory, and computing speed of model training on the computing device;

[0011] The network parameter privacy sensitivity estimation submodule is used by the computing device to estimate the sensitivity of each layer in the network to gradient attacks, reflecting the risk of exposing data privacy at that layer;

[0012] The model structure transformation submodule is used to perform structural transformation on the network after the computing device completes the computing resource estimation and the network parameter privacy sensitivity estimation.

[0013] A federated learning encryption method based on model structure transformation, comprising:

[0014] Step 1: Obtain the initial model structure and parameters based on the central server;

[0015] Step 2: In each round of training, the central server broadcasts the initial global model to the devices participating in this round of training;

[0016] Step 3: After receiving the initial global model, the device participating in the training uses the initial global model as the local public network and converts the current local public network structure into a local private structure network through the model structure transformation submodule;

[0017] Step 4: The computing device participating in the training uses the local private structure network and parameters to perform local updates on the local data set;

[0018] Step 5: After the local update is completed, the computing device participating in the training converts the updated local private structure network into a network model with the same structure as the local public network through the model structure transformation submodule, and uploads the converted local public network to the central server;

[0019] Step 6: The central server performs global aggregation on the received local public networks;

[0020] Step 7: Repeat steps 2-6 until preset indicators such as the number of communications or model performance meet preset requirements, and then output the encryption result.

[0021] Optionally, in step 2, the number of devices participating in this round of training is less than or equal to all devices.

[0022] Optionally, during the conversion process of the model structure transformation submodule in step 3, the computing capacity of the device participating in the training is determined according to the computing resource estimation submodule, and the upper limit of the model structure transformation is determined;

[0023] According to the network parameter privacy sensitivity estimation submodule, the privacy requirements and computing power of the devices participating in the training are determined, and the magnitude of the model structure transformation in the current round and the model structure that needs to be transformed are determined.

[0024] Optionally, step 3 converts the current local public network structure into a local private structure network through a model structure transformation submodule, specifically including:

[0025] Convert a K×K convolutional layer into a series of K×K convolutional and BN layers;

[0026] Convert a K×K convolutional layer into multiple M×N convolutions in parallel, where M and N are both less than or equal to K;

[0027] Convert a K×K convolutional layer into multiple M×N convolutions connected by Concat on the features, where M and N are both less than or equal to K;

[0028] Convert a pooling layer to a K×K convolutional layer.

[0029] Optionally, in step 3, after the network structure is transformed, the network parameters also change accordingly, but before and after the network structure is transformed, if the network input remains unchanged, the network outputs of the two networks are consistent.

[0030] Optionally, when the structure transformation operation in step 3 is performed on different clients, the transformed structures do not need to remain consistent.

[0031] The beneficial effects of the present invention are:

[0032] 1. The present invention converts the global model into a private local model structure before local training of federated learning through model transformation, and converts it back to the global structure after the local training is completed, so that during the entire federated learning training process, the local model structure of local training is highly confidential, so that the central server or external attackers cannot infer the private data information of the local client through the uploaded gradient information.

[0033] 2. The present invention is used to improve the privacy of federated learning. For the network before and after the conversion, if the network input is the same, the output remains consistent. In this way, the privacy of the local model is guaranteed, and the accuracy of the federated learning training is guaranteed.

[0034] 3. Compared with the traditional federated learning encryption method, the present invention uses lightweight model conversion to encrypt local model information and then implicitly encrypts gradient information, which significantly improves the privacy of federated learning and has high practical value.

[0035] 4. Compared with the method based on homomorphic encryption, this method has higher computational efficiency because the model structure conversion adopted by this method only requires simple addition and subtraction.

[0036] 5. Compared with the method based on differential privacy, this method has almost no loss in model accuracy. This is because this method ensures that if the input of the model is the same before and after the model structure conversion, the output is exactly the same. The differential privacy method needs to add noise to the model parameters, which seriously affects the model accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A structural diagram of a federated learning encryption system based on model structure transformation provided by the present invention;

[0038] Figure 2 A flowchart of a federated learning encryption method based on model structure transformation provided by the present invention;

[0039] Figure 3 A schematic diagram of the model structure conversion provided by the present invention;

[0040] Figure 4 This is a schematic diagram of the model parameter conversion form provided by the present invention. DETAILED DESCRIPTION

[0041] Specific implementation method 1: Combination Figure 1 This embodiment is described as follows. Figure 1 As shown, the structure of a federated learning encryption system based on model structure transformation described in this embodiment includes:

[0042] Federated learning is a distributed training framework consisting of computing devices participating in training and a central server;

[0043] The computing devices involved in the training are connected to the central server via wireless communication.

[0044] The computing devices involved in the training include a computing resource estimation submodule, a network parameter privacy sensitivity estimation submodule, and a model structure transformation submodule; the computing resource estimation submodule is used by the computing device to estimate the currently available running memory, video memory, and computing speed of model training; the network parameter privacy sensitivity estimation submodule is used by the computing device to estimate the sensitivity of each layer in the network to gradient attacks, reflecting the risk of exposing data privacy at this layer; the model structure transformation submodule is used to perform structural transformation on the network after the computing device completes the computing resource estimation and the network parameter privacy sensitivity estimation.

[0045] Specific implementation method 2: Combination Figure 2-4 This embodiment is described. Figure 2As shown, this embodiment discloses a federated learning encryption method based on model structure transformation, and the specific steps include:

[0046] S1: Initial stage:

[0047] The central server obtains the initial global model structure and parameters

[0048] S2: Broadcasting phase:

[0049] In round t, the central server broadcasts the current global model to the clients participating in this round of training.

[0050] S3: Local training phase:

[0051] S301: After the client receives the global model sent by the central server, it randomly selects the K×K convolutional layers in the Z global models through the model structure transformation submodule and performs the following operations: Figure 2 The structure is transformed into 1 K×K convolutional layer and M-1 P×Q convolutional layers, where P,Q≤K, and the parameters are randomly initialized and recorded as {o (1) ,…,o (M)}, for input I, the output of the converted module is:

[0052]

[0053] For the same input I, the output of the original convolution module is:

[0054]

[0055] In formula (2), o (ori) is the parameter of the original K×K convolutional layer. Therefore, for the same input I, the output of the original convolutional module and the converted convolutional module are different. In order to maintain consistent output, this embodiment converts the parameter o of the converted K×K convolutional layer to (1) Change to:

[0056] o (1) =o (ori) -(o (2) +o (3) +…+o (M) ) (3);

[0057] S30101: Determine the computing capability of the device involved in the training according to the computing resource estimation submodule during the conversion process of the model structure transformation submodule, and determine the upper limit of the model structure transformation;

[0058] According to the network parameter privacy sensitivity estimation submodule, the privacy requirements and computing power of the devices participating in the training are determined, and the magnitude of the model structure transformation in the current round and the model structure that needs to be transformed are determined.

[0059] S30102: The form of structural transformation is as follows Figure 4 As shown in the figure: a K×K convolution layer is converted into a series of K×K convolutions and a BN layer. A K×K convolution layer is converted into multiple M×N convolutions in parallel, where M and N are both less than or equal to K; a K×K convolution layer is converted into multiple M×N convolutions connected by Concat on the features, where M and N are both less than or equal to K; a pooling layer is converted into a K×K convolution layer, completing the structural conversion, and the following is obtained: Figure 4 Six shown.

[0060] S302: At this time, for the same input I, the output of the converted convolutional module is exactly the same as the output of the original module. The local model after model conversion is recorded as After the client completes model conversion, perform local training:

[0061]

[0062] In formula (4), It is the client's local data set;

[0063] S303: After local training is completed, the client will update the local model Convert to a global model structure by summing the corresponding parameters of the converted M convolutional layers and calculating the missing parameters as zero to re-merge them into a K×K convolutional layer. The converted local model is recorded as After the conversion is completed, the client uploads the local model.

[0064] During the local training process, the local initial model and the trained model are completely known to the central server, so the client's local data set can be inferred in an optimized way.

[0065] S4: Central server aggregation stage:

[0066] After receiving the local models, the central server performs aggregation:

[0067]

[0068] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. A federated learning encryption system based on model structure transformation, characterized in that: The system structure of the federated learning encryption based on model structure transformation includes: Computing devices and central servers involved in training; The computing devices participating in the training are connected to the central server by wireless communication.

2. According to the system of federated learning encryption based on model structure transformation according to claim 1, it is characterized in that: The computing device involved in the training includes a computing resource estimation submodule, a network parameter privacy sensitivity estimation submodule, and a model structure transformation submodule; The computing resource estimation submodule is used for the computing device to estimate the currently available running memory, video memory and computing speed of model training; The network parameter privacy sensitivity estimation submodule is used for calculating the sensitivity of each layer in the network to gradient attacks, reflecting the risk of exposing data privacy at this layer; The model structure transformation submodule is used to perform a structure transformation on the network after the computing device completes the computing resource estimation and the network parameter privacy sensitivity estimation.

3. A federated learning encryption method based on model structure transformation, applied to a federated learning encryption system based on model structure transformation as claimed in any one of claims 1-2, characterized in that: The steps of the federated learning encryption method based on model structure transformation include: Step 1: Obtaining initial model structure and parameters based on the central server; Step 2: In each round of training, the central server broadcasts the initial global model to the devices participating in this round of training; Step 3: After receiving the initial global model, the device participating in the training uses the initial global model as the local public network and converts the current local public network structure into a local private structure network through the model structure transformation submodule; Step 4: The computing device participating in the training uses the local private structure network and parameters to perform local updates on the local data set; Step 5: After the local update is completed, the computing device participating in the training converts the updated local private structure network into a network model with the same structure as the local public network through the model structure transformation submodule, and uploads the converted local public network to the central server; Step 6: The central server performs global aggregation on the received local public network; Step 7: Repeat steps 2-6 until preset indicators such as the number of communications or model performance meet preset requirements, and then output the encryption result.

4. The federated learning encryption method based on model structure transformation according to claim 3 is characterized in that: In step 2, the number of devices participating in this round of training is less than or equal to the total number of devices.

5. The federated learning encryption method based on model structure transformation according to claim 3 is characterized in that: In the model structure transformation submodule conversion process in step 3, the computing capacity of the device participating in the training is determined according to the computing resource estimation submodule, and the upper limit of the model structure transformation is determined; The privacy requirements and computing power of the devices participating in the training are determined according to the network parameter privacy sensitivity estimation submodule, and the transformation amplitude of the model structure in the current round and the model structure that needs to be transformed are determined.

6. The federated learning encryption method based on model structure transformation according to claim 3 is characterized in that: Step 3 converts the current local public network structure into a local private structure network through the model structure transformation submodule, specifically including: Convert a K×K convolutional layer into a series of K×K convolutional and BN layers; Convert a K×K convolutional layer into multiple M×N convolutions in parallel, where M and N are both less than or equal to K; Convert a K×K convolutional layer into multiple M×N convolutions connected by Concat on the features, where M and N are both less than or equal to K; Convert a pooling layer to a K×K convolutional layer.

7. The method for federated learning encryption based on model structure transformation according to claim 3, characterized in that: In step 3, after the network structure is transformed, the network parameters also change accordingly. However, before and after the network structure is transformed, if the network input remains unchanged, the network outputs of the two networks are consistent.

8. The method for federated learning encryption based on model structure transformation according to claim 3, characterized in that: When the structure transformation operation in step 3 is executed on different clients, the transformed structures do not need to remain consistent.

Images