Privacy protection machine learning rapid calculation method and system based on sharing model

Abstract

The invention discloses a privacy protection machine learning rapid calculation method based on a sharing model, and the method comprises the following steps: employing a semi-honesty three-party calculation mechanism, enabling a data source to carry out the fragmentation of data in a secret sharing manner, and distributing the fragmented data to a server P0 and a server P1; the server P0 and the server P1 share parameters of an initial to-be-trained model, and when forward propagation is carried out on model training, all linear operations of a linear layer are locally calculated by the server P0 and the server P1 according to fragment data respectively; when the back propagation of the model is finished, the server P0 and the server P1 exchange respective calculated parameter gradients, locally calculate the sum of the parameter gradients, and carry out consistency updating on the model parameters; and the server P0 and the server P1 acquire fragment data of the next round of model training to carry out the next round of model training. The method can greatly improve the calculation efficiency of machine learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information security, in particular to a privacy protection machine learning fast calculation method and system based on a sharing model, which can be used for safe and fast machine learning. BACKGROUND

[0002] Multi-party machine learning algorithms can be divided into provably secure machine learning algorithms and non-provable secure machine learning algorithms. The provably secure machine algorithm usually adopts HE (homomorphic encryption), GC (garbled circuit), or ASS (arithmetic secret sharing) method. However, because of the high operation amount, the application of the algorithm is hindered, especially in the scene with high response time requirement.

[0003] NN (neural network) is an important machine learning algorithm, and a multi-layer NN network can be considered to be composed of three parts: an input layer, a hidden layer, and an output layer. The hidden layer can be composed of multiple sub-layers, and each sub-layer is composed of a full connection layer and an activation layer. The output of the previous layer is the input of the next layer. If it is a DNN network, there will also be a BN layer in front of the activation function to speed up the convergence of the model. If it is a CNN network, there will also be a pooling layer.

[0004] Patent 2024108929838 “Activation function calculation method based on secret sharing and mask” accelerates the operation speed of the activation function by using arithmetic secret sharing and random mask. As shown in Figure 1 The scheme adopts a semi-honest three-party computing mechanism, each data provider slices the data and distributes it to P0 and P1. The calculation of the full connection layer adopts the existing arithmetic secret sharing scheme. For the RELU activation function and maxpool non-linear operation, P0 and P1 mask the input slices of the activation function, P2 aggregates and calculates the activation function value, and the activation function value is arithmetic sliced and sent to P0 and P1 respectively.

[0005] Although the above measures accelerate the machine learning algorithm, there is still a big gap in the operation efficiency and direct machine learning algorithm. The efficiency loss mainly comes from the multiplication of model parameter slicing or convolution kernel slicing and data slicing, and each slicing multiplication causes a network communication. SUMMARY

[0006] The main purpose of the present application is to provide a privacy protection machine learning fast calculation method and system based on a sharing model, which can improve the calculation efficiency.

[0007] The technical scheme adopted by the present application is:

[0008] A privacy protection machine learning fast calculation method based on a shared model is provided, comprising the following steps:

[0009] A semi-honest three-party computer mechanism is adopted, and the data source shards the data in an arithmetic secret sharing manner and distributes the data to a first server P0 and a second server P1.

[0010] The first server P0 and the second server P1 share the parameters of an initial to-be-trained model, when the model training is performed in a forward propagation manner, all linear operations of a linear layer are calculated locally by the first server P0 and the second server P1 according to the shard data; when the model training is performed in a backward propagation manner, the first server P0 and the second server P1 exchange the calculated parameter gradients, calculate the sum of the parameter gradients locally, and update the model parameters uniformly.

[0011] The first server P0 and the second server P1 obtain shard data for the next round of model training to perform the next round of model training.

[0012] In combination with the above technical scheme, when the to-be-trained model has a BN layer and the forward propagation is performed, the mean E of batch data is calculated, and then the transformation of the BN layer of a data x is reduced to y = γ(x-E) + β, wherein γ and β are to-be-trained parameters; for the reduced BN transformation, since γ and β are shared parameters, the first server P0 and the second server P1 perform local calculation; when the backward propagation is performed, since the global mean and the global variance are known, the transformation of the BN layer is reduced to y = γx + β, and the parameters γ and β are known, so the first server P0 and the second server P1 still perform local calculation on the shard results of the BN layer.

[0013] In combination with the above technical scheme, in the model training process, the calculation of the activation function adopts a masking method and a random permutation method to prevent the third server P2 from obtaining the information of the input data and the information of the model parameters.

[0014] According to the technical scheme, when the to-be-trained model has a pooling layer, the calculation process of the pooling function of the pooling layer is as follows: the first server P0 and the second server P1 generate the same random number mask respectively, and perform linear mask processing on all data to be pooled by using the random number mask; the transformed result is sent to the third server P2 by using the same random permutation to transform all the pooled data; the third server P2 compares the sizes of the two pieces of received data after performing a preset arithmetic operation on the two pieces of data, and returns the arithmetic slices of the maximum value to the first server P0 and the second server P1 respectively, and returns the position information of the maximum value; the first server P0 and the second server P1 perform the de-masking operation on the slices of the maximum value to obtain the real maximum value slices, and then derive the real position corresponding to the maximum value according to the position information and the random permutation.

[0015] According to the technical scheme, the to-be-trained model is a DNN network, including a CNN network.

[0016] According to the technical scheme, the to-be-trained model includes an input layer, a hidden layer and an output layer, and the linear calculation of each layer is performed locally by the first server P0 and the second server P1 after sharing the data slices.

[0017] According to the technical scheme, when performing model reasoning, the linear operation of the reasoning process is independently calculated locally by the first server P0 and the second server P1.

[0018] According to the technical scheme, the calculation of the activation function and the pooling layer is performed by the first server P0, the second server P1 and the third server P2 through one round of collaborative operation, and the function of the third server P2 is independently managed and run on the device of each user.

[0019] The application also provides a privacy protection machine learning fast calculation system based on a shared model, which is used to execute the privacy protection machine learning fast calculation method based on the shared model.

[0020] The application also provides a computer storage medium, which stores a computer program that can be executed by a processor, and the computer program executes the privacy protection machine learning fast calculation method based on the shared model.

[0021] The beneficial effects generated by the present application are: the present application allows the first server P0 and the second server P1 to share the model, that is, when starting training, the model parameters of the first server P0 and the second server P1 are consistent; when forward propagation, the multiplication of the model weight parameters or the convolution kernel and the data slice can be performed locally, thereby reducing the operation amount; when back propagation, after all the gradient calculations are completed, the two servers exchange all the parameter gradients at one time, both sides sum the gradients, and the summed gradients are used to update the model parameters, thereby minimizing the slice multiplication operation and further accelerating the machine learning algorithm with the privacy protection function.

[0022] Further, in the model training process, the calculation of the activation function adopts a masking method and a random permutation method, which can prevent the third server P2 from obtaining the information of the input data and also prevent the P2 from stealing the model parameters.

[0023] Further, the linear operation (such as convolution, full connection, BN, and the BN layer can be reduced to y = γx + β in the inference stage, where γ and β are fixed constants) in the inference process can be independently calculated by the first server P0 and the second server P1 locally, and only the activation function and the pooling need to be cooperatively calculated by both sides for one round of light operation, thereby realizing an efficient inference algorithm.

[0024] Of course, any product implementing the present application does not necessarily need to achieve all the advantages described above. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0026] Figure 1 is a structural schematic diagram of the semi-honest three-party computer mechanism of the embodiment of the present application;

[0027] Figure 2 is a flowchart of the privacy protection machine learning fast computing method based on shared model of the embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present application more clear, the following will further describe the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0029] It should be noted that the diagrams provided in the embodiments of the present application only schematically illustrate the basic concepts of the present application, and therefore only the components related to the present application are shown in the diagrams, not the number, shape and size of the components when actually implemented, and the shape, number and ratio of the components when actually implemented can be arbitrarily changed, and the layout pattern of the components can be more complex.

[0030] In the present application, it should be noted that, when terms such as "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like appear, the indicated orientation or positional relationship is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, when the terms "first" and "second" appear, they are only for description and differentiation purposes, and cannot be understood as indicating or implying relative importance.

[0031] The arithmetic secret sharing scheme is a commonly used secure multi-party computation scheme, and the operation amount mainly comes from the piecewise multiplication operation of linear operation and from nonlinear operations such as BN, activation function and maximum pooling. The present patent reduces the operation amount of piecewise multiplication through the scheme of sharing model parameters, and further reduces the operation amount of nonlinear operation through the method of masking and random permutation.

[0032] As shown in Figure 2 The privacy protection machine learning fast calculation method based on the shared model in the embodiments of the present application mainly includes the following steps:

[0033] S1, a semi-honest three-party computing mechanism is adopted, data sources are divided into pieces in a secret sharing manner, and are distributed to a first server P0 and a second server P1; the data can be image data, voice data, text data, etc., and during model training, the data used for training is mainly divided and distributed to P0 and P1;

[0034] S2, the first server P0 and the second server P1 share the parameters of the initial model to be trained, when the model training is performed in the forward propagation, all linear operations of the linear layer are calculated by the first server P0 and the second server P1 according to the piecewise data respectively; when the model is trained in the backward propagation, the first server P0 and the second server P1 exchange the calculated parameter gradients, respectively calculate the sum of the parameter gradients in the local, and update the consistency of the model parameters;

[0035] S3, the first server P0 and the second server P1 obtain the piecewise data of the next round of model training to perform the next round of model training.

[0036] Specifically, when performing operations on fully connected layers or convolutional layers, such as Figure 1 As shown, the first server P0 and the second server P1 share the initial model parameters. During forward propagation, because of the shared model parameters, each server can compute fully connected layers or convolutional layers locally. At the end of a backpropagation cycle, the first server P0 and the second server P1 swap their computed parameter gradients, compute the sum of their parameter gradients locally, and perform a consistent update of the model parameters in preparation for the next round of model training.

[0037] When performing Batch Normalization (BN) layer operations, during forward propagation, the mean E of the batch data is calculated. For a single data point x, the BN layer transformation can be reduced to y = γ(xE) + β, where γ and β are parameters that need to be trained. In the actual training phase, to accelerate convergence, the idea of ​​transfer learning can be adopted, and the initial model parameters can be initialized to similar, fully trained model parameters. For the reduced BN transformation, since γ and β are shared parameters, P0 and P1 can be computed locally. When used for model inference, since the global mean and global variance are known, the BN layer transformation can be reduced to y = γx + β, and the parameters γ and β are known. Therefore, P0 and P1 can compute the BN slice results locally.

[0038] When performing activation function calculations, the ReLU function is a commonly used activation function. Even replacing GELU in large models with ReLU does not result in a loss of model accuracy. For ReLU calculations, masking and random permutation methods are used to prevent a third-party server (P2) from obtaining information about the input data and model parameters. The activation function calculation process mainly includes the following steps:

[0039] 1) The first server P0 and the second server P1 flatten the first and second shard values ​​they obtained into a one-dimensional shard vector, and the two shard vectors have the same number of elements n;

[0040] 2) P0 and P1 generate n random numbers using the same seed and random number generator, where each random number is greater than 0, and use the random numbers as masks to perform masking calculations on the two slice vectors respectively;

[0041] 3) P0 and P1 each use the same seed and random number generator to generate n random permutations π, and apply them to the results of their respective mask calculations, and then send them to the third server P2.

[0042] 4) After receiving the mask with random permutation, P2 performs mathematical reconstruction to reconstruct the complete input vector of the activation function, and uses the activation function to calculate its activation function value;

[0043] 5) P2 generates n random numbers, performs arithmetic calculations on the activation function value using the newly generated n random numbers, sends the newly generated n random numbers to P0, and sends the arithmetic calculation results to P1 respectively, which is equivalent to redistributing the slice values.

[0044] 6) P0 and P1 respectively transform the order of the received data using the inverse permutation corresponding to the random permutation, in order to restore the original order of the activation function output values;

[0045] 7) P0 and P1 respectively use the original mask to calculate and restore the redistributed fragment values;

[0046] Furthermore, a specific example of the above method is as follows:

[0047] 1) The first server P0 and the second server P1 flatten the shard values ​​X and Y (which may be high-dimensional data) into one-dimensional shard vectors x and y, and denote their common dimension as size(x);

[0048] 2) P0 and P1 each use the same random number generator to generate size(x) masks {k}. i Let i = 1, 2, ..., size(x)), where each random number is greater than 0. Then calculate X' = {x'}. i =k i *x i ,i=1,2,...,size(x)},Y'={y' i =k i *y i ,i=1,2,...,size(x)};

[0049] 3) P0 and P1 each use the same random number generator to generate a random permutation π with size(x) elements, and apply it to X' and Y' respectively to obtain π(X') and π(Y'), and send them to P2 respectively;

[0050] 4) After receiving π(X') and π(Y'), P2 reconstructs the complete input vector {π(x')} of the activation function by arithmetic summation. i +y′ i ), i = 1, 2, ..., size(x)), and calculate its activation function value {ReLU(π(x′)} using conventional methods. i +y′ i )), i=1, 2,..., size(x)};

[0051] 5) P2 generates random numbers {r} i , i = 1, 2, ..., size(x)), calculate {Relu(π(x′)} i +y′i ))-r i ), i = 1, 2, ..., size(x)}, and set {r i , i=1, 2,..., size(x)} and {Relu(π(x′ i +y′ i ))-r i ), i = 1, 2, ..., size(x)} are sent to P0 and P1 respectively;

[0052] 6) P0 and P1 are respectively replaced by the inverse permutation pairs {r} corresponding to random permutations. i ,i=1,2,...,size(x)},{Relu(π(x′ i +y′ i ))-r i The order of the values ​​i = 1, 2, ..., size(x) is transformed to restore the original order of the activation function output values;

[0053] 7) P0 and P1 respectively calculate {r} using the original mask. i / k i ,i=1,2,...,size(x)),{Relu(x′ i +y′ i ) / k i -r i / k i , i = 1, 2, ..., size(x)}. Since Relu(x′) i +y′ i ) / k i -r i / k i =max(0, (x′) i +y′ i )) / k i -r i / k i =max(0, x) i +y i )-r i / k i That is, the arithmetic sum of the shares of P0 and P1 is xi + y. i That is, P0 and P1 each obtained the slice value of the activation function ReLU(X+Y).

[0054] Other activation functions similar to ReLU, such as LReLU and Parametric ReLU, are also applicable to the processing procedure described in the above embodiments; while softmax can be processed in a similar manner to the above embodiments, but the data x can be masked in the form of x+b.

[0055] When performing pooling layer operations, the calculation process of the pooling function is similar to that of the activation function. P0 and P1 generate identical random number masks a and b (where a > 0) and linearly mask all the data to be pooled using a and b. Then, they transform all the masked pooled data using the same random permutation, sending the result to P2. P2 performs an arithmetic sum operation on the two received data sets, compares their sizes, and returns the arithmetic slice of the maximum value to P0 and P1, along with its position information. P0 and P1 then demask the slice of the maximum value based on the original mask, obtaining the true maximum value slice. Finally, based on the returned position information, they perform an inverse permutation to obtain the original position information. Here, the result of linearly masking a data set x is ax + b. Max pooling performs better than average pooling in some models.

[0056] Based on the above method implementation, a DNN can be trained, and the main process is as follows:

[0057] (1) P0 and P1 use the sliced ​​input data and the operation of the convolutional layer in the above embodiment to calculate the local convolution slice result locally.

[0058] (2) Optionally, P0 and P1 use the convolution slicing results and the BN layer operation in the above embodiment to calculate their local BN operation slicing results locally.

[0059] (3) P0 and P1 use the locally computed fragmentation results and the operation of the activation function in the above embodiment to jointly compute the activation function value fragmentation;

[0060] (4) P0 and P1 use the obtained activation function values ​​to divide the data and the pooling layer operation in the above embodiment to jointly calculate the pooled result;

[0061] (5) P0 and P1 use the pooling sharding results and the operation of the fully connected layer in the above embodiment to calculate the sharding results of the fully connected layer locally;

[0062] (6) P0 and P1 each calculate the loss function locally based on the label sharding data and the sharding results of the fully connected layer, and perform backpropagation to calculate the gradient of each parameter. Then, the gradients of all parameters are swapped, and the gradients of the two sets of parameters are arithmetically summed for consistent updating of model parameters.

[0063] Based on the above method implementation examples, the main process is given below using CNN inference as an example:

[0064] like Figure 1As shown, logically it is still a three-server architecture, but in actual deployment, the P2 function runs on each user's device. Since the data belongs to the user, the user does not need to worry about data leakage. Therefore, the masking operation to protect data privacy can be removed, and only the issue of preventing users from obtaining model parameters needs to be considered.

[0065] The user segments the data to be inferred and sends it to P0 and P1 respectively. After receiving the data segments, P0 and P1 each use the convolution operation described in the previous embodiment to calculate their local convolution segmentation results. Optionally, P0 and P1 use the convolution segmentation results and the BN operation described in the previous embodiment to calculate their local BN operation segmentation results. P0 and P1 use their locally calculated segmentation results and the activation function operation described in the previous embodiment to collaboratively calculate the activation function value segmentation. Here, since each user runs the P2 function independently on their own device, there is no need to worry about data confidentiality. Therefore, the activation function calculation process can only include random permutation to prevent users from stealing model parameters, without the need for a masking mechanism. P0 and P1 use the data segments to be pooled and the max pooling operation described in the previous embodiment to calculate the max pooling value segmentation. P0 and P1 use the pooling segmentation results and the fully connected layer operation described in the previous embodiment to calculate the fully connected layer segmentation results locally, and then send the segmentation results to the user. The user performs arithmetic summation of the segmentation results to obtain the inference result.

[0066] As can be seen, the linear operations (convolution, fully connected layers, batch normalization) in the inference process can be computed independently locally. Only the activation function and pooling require a lightweight collaborative computation between the two parties, thus resulting in a practical and efficient inference algorithm. During the training phase, due to the large amount of training data from different users, P2 must process all the data and operate as an honest third party. However, during the inference phase, a piece of inference data definitely belongs to the inference user, and P2 can be run by the inference user themselves. Therefore, the computation of the activation function and pooling layer involves a collaborative computation between the first server P0, the second server P1, and the third server P2, with the functionality of the third server P2 being independently managed and operated by each user in an appropriate manner. Therefore, while the forward processes of inference and training are similar, there are still considerable differences.

[0067] Furthermore, if the user's inference speed requirements are not particularly high, P2 can be deployed on the user's computing device. If the user's inference speed requirements are particularly high, P0, P1, and P2 can be deployed within three TEEs (Trusted Execution Environments) on the same local area network. This way, the inference speed will be further accelerated due to intranet communication between the three TEEs. Because the threat model is a semi-honest model, even if the TEEs are subjected to side-channel attacks and the adversary obtains some data, as long as the execution logic within the TEEs is not changed, the entire system remains secure and private. Moreover, TEEs from different manufacturers can be used, preventing simultaneous exploitation by attackers and collusion. If there are concerns about the model being completely possessed by P0 or P1 and thus misused, a small number of hidden layers can be randomly selected, and their model parameters can be arithmetically sharded. These selected hidden layers can then be trained using existing methods.

[0068] The present invention also provides a privacy-preserving machine learning fast computation system based on a shared model, which is used to execute the privacy-preserving machine learning fast computation method based on a shared model described in the above embodiments.

[0069] This application also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, app store, etc., which stores a computer program, and the program implements corresponding functions when executed by a processor. The computer-readable storage medium of this embodiment implements a privacy-preserving machine learning fast computation method based on a shared model when executed by a processor.

[0070] It should be noted that, depending on the implementation needs, the various steps / components described in this application can be broken down into more steps / components, or two or more steps / components or parts of the operation of steps / components can be combined into new steps / components to achieve the purpose of this invention.

[0071] The order of the steps in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0072] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A privacy-preserving machine learning fast computing method based on a shared model, characterized in that, The method comprises the following steps: The data source shards the data in a secret sharing manner and distributes the data to the first server P0 and the second server P1 by using a semi-honest three-party computer mechanism; The first server P0 and the second server P1 share the parameters of the initial trained model, and when the model training is performed in a forward propagation manner, all linear operations of the linear layer are calculated locally by the first server P0 and the second server P1 according to the shard data; when the model is performed in a backward propagation manner, the first server P0 and the second server P1 exchange the calculated parameter gradients, calculate the sum of the parameter gradients locally, and update the model parameters uniformly; The first server P0 and the second server P1 obtain the shard data of the next round of model training to perform the next round of model training.

2. The privacy-preserving machine learning fast computing method based on a shared model according to claim 1, characterized in that, When the trained model has a BN layer and is performed in a forward propagation manner, the mean E of the batch data is calculated, and then the transformation of the BN layer of a data x is reduced to y = γ(x-E) + β, where γ and β are the parameters to be trained; for the reduced BN transformation, since γ and β are shared parameters, the first server P0 and the second server P1 perform local calculation; when the model is performed in a backward propagation manner, since the global mean and the global variance are known, the transformation of the BN layer is reduced to y = γx + β, and the parameters γ and β are known, so the first server P0 and the second server P1 still perform local calculation on the shard results of the BN. 3.The privacy-preserving machine learning fast computing method based on a shared model according to claim 1, wherein, During the model training process, the calculation of the activation function adopts a masking method and a random permutation method to prevent the third server P2 from obtaining the information of the input data.

4. The privacy-preserving machine learning fast computing method based on a shared model according to claim 1, characterized in that, When the trained model has a pooling layer, the calculation process of the pooling function is as follows: the first server P0 and the second server P1 respectively generate the same random number mask, and perform linear masking on all data to be pooled by using the random number mask; The transformed results are sent to the third server P2; the third server P2 performs a preset arithmetic operation on the received two data and compares the sizes, returns the arithmetic shard of the maximum value to the first server P0 and the second server P1 respectively, and returns the position information of the maximum value; the first server P0 and the second server P1 perform the demasking operation on the shard of the maximum value to obtain the real shard of the maximum value, and then derive the real position of the maximum value according to the position information and the random permutation. 5.The privacy-preserving machine learning fast computing method based on a shared model according to claim 1, wherein, The trained model is a DNN network, including a CNN network. 6.The privacy-preserving machine learning fast computing method based on a shared model according to claim 1, wherein, The trained model comprises an input layer, a hidden layer and an output layer, and the linear calculation of each layer is performed locally by the first server P0 and the second server P1. 7.The privacy-preserving machine learning fast computing method based on a shared model according to claim 1, wherein, When the model is inferred, the linear operation of the inference process is independently calculated locally by the first server P0 and the second server P1. 8.The privacy-preserving machine learning fast computing method based on a shared model according to claim 7, wherein, The calculation of the activation function and the pooling layer is performed by one round of collaborative operation of the first server P0, the second server P1 and the third server P2, and the function of the third server P2 is independently managed and operated by each user. 9.A privacy-preserving machine learning fast computing system based on a shared model, characterized in that, The system is used to perform the privacy protection machine learning fast calculation method based on a shared model according to any one of claims 1-8.

10. A computer storage medium, characterized in that, The memory stores a computer program executable by the processor, and the computer program performs the privacy protection machine learning fast calculation method based on a shared model according to any one of claims 1-8.

Images