Privacy computing method based on secure differentially private stochastic gradient descent, and system

Abstract

A privacy computing method based on secure differentially private stochastic gradient descent, and a system, relating to the field of privacy computing. According to the method, data features are extracted locally, so that significant accuracy loss and huge communication overhead caused by securely training a differential privacy model can be reduced for all parties; then, a global model is initialized by utilizing local data held by all parties, so that the number of iterations during a global model training process is remarkably reduced and the rate of convergence of model training is accelerated, thereby improving the efficiency of secure multi-party computing; finally, a privacy computing task is completed by utilizing randomly sampled global data and the initialized global model. According to the described method, privacy, efficiency, and accuracy are balanced.

Description

Privacy computing method and system based on secure differential privacy stochastic gradient descent Technical Field

[0001] The present invention belongs to the field of privacy computing, and specifically relates to a privacy computing method and system based on secure differential privacy stochastic gradient descent. Background Art

[0002] Massive amounts of data and powerful computing resources have become the primary drivers of today's intelligent technology development. With the increasing maturity of large-scale machine learning algorithms and big data processing techniques, as well as the continuous improvement of computer hardware performance, the vast amounts of data collected can be more efficiently used to model and compute real-world problems. However, the increasing number of data misuse and privacy breaches in recent years has reminded us that the inappropriate use of big data can have disastrous consequences.

[0003] Machine learning based on secure multi-party computation (MPC), or MML for short, has become an important technology for leveraging multi-party data while protecting privacy. While MML provides strict security guarantees for the computational process, models trained using MML remain vulnerable to attacks that rely solely on access to the model. Differential privacy can help defend against such attacks. However, the accuracy loss associated with differential privacy and the significant communication overhead of secure MML protocols make balancing the three-way trade-off between privacy, efficiency, and accuracy extremely challenging.

[0004] Therefore, we propose a secure differentially private gradient descent algorithm. Furthermore, to reduce the accuracy loss caused by differential privacy noise and the huge communication overhead of multi-party machine learning, we propose a feature extraction method and a global model initialization method based on local data. The former aims to simplify the training model structure, while the latter aims to accelerate the convergence of model training.

[0005] Summary of the Invention

[0006] Based on the above background and problems existing in the prior art, the present invention adopts the following technical solutions: First, a privacy computing method based on secure differential privacy stochastic gradient descent is provided, which can utilize multi-party data through the secure differential privacy gradient descent algorithm and complete machine learning computing tasks efficiently and accurately while protecting privacy.

[0007] A privacy computing method based on secure differential privacy stochastic gradient descent includes the following steps:

[0008] Performing feature extraction on the local data using kernel principal component analysis to obtain feature vectors of the local data, wherein the local data originates from each participant;

[0009] Input the feature vector into the secure differential privacy gradient descent model for training to obtain the local model of each participant;

[0010] Aggregating the local models of the participants to obtain an initialized global model;

[0011] In a trusted blockchain environment, the local data of multiple participants are randomly sampled to obtain a global data set.

[0012] Inputting the initialized global model and the global dataset into a differentially private gradient descent model for training to obtain a global model;

[0013] The global model is used to perform machine learning tasks based on secure multi-party computing and output calculation results.

[0014] As an implementable method, the method of extracting features from local data using kernel principal component analysis to obtain a feature vector of the local data includes the following steps:

[0015] Use the Gaussian kernel function to map each sample point of the local data into the high-dimensional feature space to obtain a kernel matrix;

[0016] Subtract the mean of the corresponding row and the corresponding column from each element in the kernel matrix to obtain a centralized kernel matrix;

[0017] Performing eigenvalue decomposition on the centralized kernel matrix to obtain eigenvectors and eigenvalues;

[0018] According to the size of the eigenvalue, the largest k eigenvectors are selected as the new feature space after dimensionality reduction;

[0019] Mapping the local data sample points to the new feature space, that is, performing an inner product operation on the row vector corresponding to each sample point of the kernel matrix and the selected k feature spaces to obtain a matrix of the new feature space;

[0020] The matrix of the new feature space is reversely mapped back to the feature space of the local data to obtain the feature vector of the local data.

[0021] As an implementation method, the feature vectors of the local data are input into a secure differentially private gradient descent model for pre-training to obtain a local model of each participant, including the following steps:

[0022] Calculating a gradient vector using the eigenvector;

[0023] Calculating the reciprocal of the norm of the gradient vector by a reciprocal square root protocol, and clipping the norm of the gradient vector according to a comparison result between the reciprocal of the norm of the gradient vector and 1, so that the norm of the gradient vector is less than or equal to a given constant;

[0024] Updating the gradient vector according to the comparison result between the inverse of the norm of the gradient vector of each participant and 1;

[0025] Each participant generates Gaussian noise with a mean of 0 and a standard deviation of σ, and applies the Gaussian noise to perturb the gradient vector;

[0026] Each participant updates the model parameters according to the gradient descent process to obtain the local model of each participant.

[0027] As an implementation method, the calculating the inverse of the norm of the gradient vector by the inverse square root protocol comprises the following steps:

[0028] The local data x is represented in binary code form, and then the local data x is transformed into x', so that x'∈[0.5,1) and x=x'*2 exp ;

[0029] calculate The fixed-point representation of

[0030] Calculate approximations by approximating polynomials;

[0031] By the approximation and Multiply to get the output result.

[0032] As an implementable method, the local models of the participants are aggregated to obtain an initialized global model, which includes the following steps:

[0033] Each participant generates a local model and evaluates the accuracy of each local model separately;

[0034] Aggregate the local models using different aggregation strategies to obtain an initial global model.

[0035] Among them, the aggregation strategy includes the average strategy and the accuracy strategy.

[0036] The averaging strategy is to average the local model parameters as the parameters of the initial global model;

[0037] The accuracy strategy is to select the most accurate local model as the initial global model.

[0038] A privacy computing system based on secure differential privacy stochastic gradient descent, including a feature extraction module, a pre-training module, a secure differential privacy gradient descent module, and an output module:

[0039] The feature extraction module extracts features from local data using kernel principal component analysis to obtain feature vectors of the local data, wherein the local data originates from each participant;

[0040] In the pre-training module, the feature vectors of local data are input into the secure differential privacy gradient descent model for pre-training to obtain the local model of each participant;

[0041] Aggregating the local models of the participants to obtain an initialized global model;

[0042] The secure differentially private gradient descent module randomly samples local data from multiple participants in a trusted blockchain environment to obtain a global dataset.

[0043] Inputting the initialized global model and the global dataset into a differentially private gradient descent model for training to obtain a global model;

[0044] The output module performs machine learning tasks based on secure multi-party computing through a global model and outputs calculation results.

[0045] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following method:

[0046] Performing feature extraction on the local data using kernel principal component analysis to obtain feature vectors of the local data, wherein the local data originates from each participant;

[0047] Input the feature vector into the secure differential privacy gradient descent model for training to obtain the local model of each participant;

[0048] Aggregating the local models of the participants to obtain an initialized global model;

[0049] In a trusted blockchain environment, the local data of multiple participants are randomly sampled to obtain a global data set.

[0050] Inputting the initialized global model and the global dataset into a differentially private gradient descent model for training to obtain a global model;

[0051] Perform machine learning tasks based on secure multi-party computing through a global model and output the calculation results.

[0052] A privacy computing device based on secure differentially private stochastic gradient descent includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the following method:

[0053] Performing feature extraction on the local data using kernel principal component analysis to obtain feature vectors of the local data, wherein the local data originates from each participant;

[0054] Input the feature vector into the secure differential privacy gradient descent model for training to obtain the local model of each participant;

[0055] Aggregating the local models of the participants to obtain an initialized global model;

[0056] In a trusted blockchain environment, the local data of multiple participants are randomly sampled to obtain a global data set.

[0057] Inputting the initialized global model and the global dataset into a differentially private gradient descent model for training to obtain a global model;

[0058] Perform machine learning tasks based on secure multi-party computing through a global model and output the calculation results.

[0059] (1) A privacy computing method and system based on secure differential privacy stochastic gradient descent is proposed. By using feature extraction and a global model initialization method based on local data, the training model structure is simplified and the convergence speed of model training is accelerated, thus reducing the accuracy loss caused by differential privacy noise and the huge communication overhead of multi-party machine learning.

[0060] (2) A privacy computing method and system based on secure differential privacy stochastic gradient descent is proposed. Through the secure differential privacy stochastic gradient descent algorithm, the privacy, efficiency and accuracy in the security model training of machine learning based on secure multi-party computing are balanced.

[0061] Other advantages, objectives and features of the present invention will be reflected in part from the following description and will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] FIG1 is a schematic diagram of the steps of the privacy calculation method based on secure differential privacy stochastic gradient descent of the present invention;

[0063] FIG2 is a flowchart of the privacy calculation method based on secure differential privacy stochastic gradient descent of the present invention. DETAILED DESCRIPTION

[0064] In order to clearly illustrate the present invention and make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described below in combination with the drawings in the embodiments of the present invention, so that those skilled in the art can implement them according to the text of the description.

[0065] In the context of the present invention, a trusted execution environment (TEE) provides an isolated runtime environment from the perspective of the underlying hardware and operating system, protecting the code and data running within it from external attacks, including attacks from the operating system, hardware, and other applications. This technology has been used in some fields to achieve the objectives described above, and some of the basic principles of this technology are also known to those skilled in the art. However, after reading this application, those skilled in the art will understand how to apply this technology in this context and will clearly understand the novelty of this technology when combined with other features in a specific context.

[0066] The technology of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0067] Example 1:

[0068] The present invention is a privacy computing method based on secure differential privacy stochastic gradient descent. The steps of the invention are shown in Figure 1. The specific steps are as follows:

[0069] Performing feature extraction on the local data using kernel principal component analysis to obtain feature vectors of the local data, wherein the local data originates from each participant;

[0070] Input the feature vector into the secure differential privacy gradient descent model for training to obtain the local model of each participant;

[0071] Aggregating the local models of the participants to obtain an initialized global model;

[0072] In a trusted blockchain environment, the local data of multiple participants are randomly sampled to obtain a global data set.

[0073] Inputting the initialized global model and the global dataset into a differentially private gradient descent model for training to obtain a global model;

[0074] The global model is used to perform machine learning tasks based on secure multi-party computing and output calculation results.

[0075] (1) Using kernel principal component analysis to extract features from local data to obtain feature vectors of local data, including the following steps:

[0076] Use the Gaussian kernel function to map each sample point of the local data into the high-dimensional feature space to obtain a kernel matrix;

[0077] The Gaussian kernel function is defined as follows:

[0078] The above formula involves the calculation of the Euclidean distance between two vectors. Moreover, the Gaussian kernel function is a monotonic function of the Euclidean distance between two vectors. σ is the bandwidth, which controls the radial range of action, that is, σ controls the local range of action of the Gaussian kernel function.

[0079] Subtract the mean of the corresponding row and the corresponding column from each element in the kernel matrix to obtain a centralized kernel matrix;

[0080] Performing eigenvalue decomposition on the centralized kernel matrix to obtain eigenvectors and eigenvalues;

[0081] According to the size of the eigenvalue, the largest k eigenvectors are selected as the new feature space after dimensionality reduction;

[0082] Mapping the local data sample points to the new feature space, that is, performing an inner product operation on the row vector corresponding to each sample point of the kernel matrix and the selected k feature spaces to obtain a matrix of the new feature space;

[0083] The matrix of the new feature space is reversely mapped back to the feature space of the local data to obtain the feature vector of the local data.

[0084] Among them, after kernel principal component analysis maps the data into a high-dimensional space, it can better distinguish different categories of data and improve the separability of the data.

[0085] (2) The feature vectors of the local data are input into the secure differential privacy gradient descent model for pre-training to obtain the local model of each participant, as shown in Figure 2, including the following steps:

[0086] Calculating a gradient vector using the eigenvector;

[0087] Calculating the reciprocal of the norm of the gradient vector by a reciprocal square root protocol, and clipping the norm of the gradient vector according to a comparison result between the reciprocal of the norm of the gradient vector and 1, so that the norm of the gradient vector is less than or equal to a given constant;

[0088] Updating the gradient vector according to the comparison result between the inverse of the norm of the gradient vector of each participant and 1;

[0089] Each participant generates Gaussian noise with a mean of 0 and a standard deviation of σ, and applies the Gaussian noise to perturb the gradient vector;

[0090] Each participant updates the model parameters according to the gradient descent process to obtain the local model of each participant.

[0091] Among them, in order to clip the L2 norm of a gradient vector g, each party needs to calculate the inverse of ||g||2, that is, Since this calculation involves two functions, square root and division, both of which are nonlinear, directly implementing them using a secret sharing protocol means huge communication overhead. To reduce the overhead, one feasible approach is to use polynomials to approximate nonlinear functions.

[0092] The inverse square root protocol consists of the following steps:

[0093] The i-th participant P i First, the local data <x> i Decompose it into binary code form and set all bits after the first 1 in the binary code to 1. Finally, P i Will <x> i Transformed into<x'> i , and get <exp> i , so that x'∈[0.5,1) and x=x'*2 exp ;

[0094] P i calculate The fixed-point representation of First, P i Get the least significant bit of exp (ie <lsb> i ), which indicates the parity of exp. Subsequently, P i Calculate separately and Among them, if exp is an odd number, then Finally, P i according to <lsb> i Choose the correct value to get

[0095] P i Using the polynomial 0.8277x' 2 -2.046x'+2.223 approximation (For x'∈[0.5,1),. Next, P i Subtract an approximation with an error bound of 0.0048 to ensure that the approximation is strictly smaller than the true value, thus maintaining strict privacy guarantees;

[0096] P i Multiply the approximate value with Get output <y> i .

[0097] (3) Aggregating the local models of the participants to obtain an initialized global model, including the following steps:

[0098] Each participant generates a local model and evaluates the accuracy of each local model separately;

[0099] Aggregate the local models using different aggregation strategies to obtain an initial global model.

[0100] Among them, the aggregation strategy includes the averaging strategy and the accuracy strategy. The averaging strategy is to average the local model parameters as the parameters of the initial global model; the accuracy strategy is to select the most accurate local model as the initial global model.

[0101] Using the local data held by each party to initialize an accurate global model will significantly reduce the number of iterations in the global model training process, thereby improving the efficiency of model training. In addition, since the distribution state of data is difficult to measure directly, the initialization of the global model is selected by selecting different aggregation strategies to generate the most accurate model.

[0102] (4) In a trusted blockchain environment, the local data of multiple participants are randomly sampled to obtain a global dataset. The global dataset integrates random samples of different data distributions from multiple parties, thereby enhancing the privacy protection of the training process.

[0103] The initialized global model and the global data set are input into a differential privacy gradient descent model for training to obtain a global model.

[0104] (5) Inputting the initialized global model and the global data set into a differential privacy gradient descent model for training to obtain a global model.

[0105] (6) Performing machine learning tasks based on secure multi-party computation using the global model and outputting the computation results. Each participant obtains the global model and uses it to perform their own machine learning tasks and output their own computation results.

[0106] In summary, the present invention provides a privacy-preserving computing method and system based on secure differentially private stochastic gradient descent. This method extracts data features locally, enabling each party to mitigate the significant accuracy loss and communication overhead associated with securely training a differentially private model. Initializing the global model using the local data held by each party significantly reduces the number of iterations during global model training, accelerating the convergence of model training and thus improving the efficiency of secure multi-party computation. Finally, the randomly sampled global data and the initialized global model are used to complete the privacy-preserving computing task. This privacy-preserving computing method based on secure differentially private stochastic gradient descent balances privacy, efficiency, and accuracy in secure model training for machine learning based on secure multi-party computation.

[0107] Example 2:

[0108] A privacy computing system based on secure differential privacy stochastic gradient descent, including a feature extraction module, a pre-training module, a secure differential privacy gradient descent module, and an output module:

[0109] The feature extraction module extracts features from local data using kernel principal component analysis to obtain feature vectors of the local data, wherein the local data originates from each participant;

[0110] In the pre-training module, the feature vectors of local data are input into the secure differential privacy gradient descent model for pre-training to obtain the local model of each participant;

[0111] Aggregating the local models of the participants to obtain an initialized global model;

[0112] The secure differentially private gradient descent module randomly samples local data from multiple participants in a trusted blockchain environment to obtain a global dataset.

[0113] Inputting the initialized global model and the global dataset into a differentially private gradient descent model for training to obtain a global model;

[0114] The output module performs machine learning tasks based on secure multi-party computing through a global model and outputs calculation results.

[0115] Various changes and modifications can be made without departing from the spirit and scope of the present invention, and all equivalent technical solutions also fall within the scope of the present invention.

[0116] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referenced to each other.

[0117] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device produce a device for implementing the functions specified in one or more processes in the flowcharts and / or one or more blocks in the block diagrams.

[0119] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0121] It should be noted that:

[0122] References in this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Therefore, appearances of the phrases "one embodiment" or "an embodiment" in various places throughout this specification do not necessarily refer to the same embodiment.

[0123] The above description of the embodiments is intended to facilitate understanding and application of the present invention by those skilled in the art. It will be apparent that those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without requiring inventive effort. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made by those skilled in the art based on the disclosure of the present invention should fall within the scope of protection of the present invention.< / y> < / lsb> < / lsb> < / exp> < / x> < / x>

Claims

1. A privacy computing method based on secure differential privacy stochastic gradient descent, characterized in that: The steps include: Performing feature extraction on local data using kernel principal component analysis to obtain feature vectors of local data, wherein the local data originates from each participant; Input the feature vector into the secure differential privacy gradient descent model for training to obtain a local model of each participant; Aggregating the local models of the participants to obtain an initialized global model; In the blockchain trusted environment, the local data of multiple participants are randomly sampled to obtain a global data set; Inputting the initialized global model and the global data set into a differential privacy gradient descent model for training to obtain a global model; The global model is used to perform machine learning tasks based on secure multi-party computing and output calculation results.

2. The privacy computing method based on secure differential privacy stochastic gradient descent according to claim 1, characterized in that: The method of extracting features from local data using kernel principal component analysis to obtain a feature vector of the local data includes the following steps: Use the Gaussian kernel function to map each sample point of the local data into the high-dimensional feature space to obtain a kernel matrix; Subtract the mean of the corresponding row and the corresponding column from each element in the kernel matrix to obtain a centralized kernel matrix; Performing eigenvalue decomposition on the centralized kernel matrix to obtain eigenvectors and eigenvalues; According to the size of the eigenvalue, select the largest k eigenvectors as the new feature space after dimensionality reduction; Mapping the local data sample points to the new feature space, that is, performing an inner product operation on the row vector corresponding to each sample point of the kernel matrix and the selected k feature spaces to obtain a matrix of the new feature space; The matrix of the new feature space is reversely mapped back to the feature space of the local data to obtain the feature vector of the local data.

3. The privacy computing method based on secure differential privacy stochastic gradient descent according to claim 1, characterized in that: The feature vector of the local data is input into the secure differential privacy gradient descent model for training to obtain the local model of each participant, including the following steps: Calculate a gradient vector using the eigenvector; Calculating the inverse of the norm of the gradient vector by a reciprocal square root protocol, and clipping the norm of the gradient vector according to a comparison result between the inverse of the norm of the gradient vector and 1, so that the norm of the gradient vector is less than or equal to a given constant; Update the gradient vector according to the value of the comparison result between the inverse of the norm of the gradient vector of each participant and 1; Each participant generates Gaussian noise with a mean of 0 and a standard deviation of σ, and applies the Gaussian noise to perturb the gradient vector; Each participant updates the model parameters according to the gradient descent process to obtain the local model of each participant.

4. The privacy computing method based on secure differential privacy stochastic gradient descent according to claim 3 is characterized in that: The step of calculating the inverse of the norm of the gradient vector by the inverse square root protocol comprises the following steps: The local data x is represented in binary code form, and then the local data x is transformed into x', so that x'∈[0.5,1) and x=x'*2 exp ; calculate The fixed-point representation of Calculate approximations through approximating polynomials; By the approximation Multiply to get the output result.

5. The privacy computing method based on secure differential privacy stochastic gradient descent according to claim 1, characterized in that: Aggregating the local models of the participants to obtain an initialized global model includes the following steps: Each participant generates a local model and evaluates the accuracy of each local model separately; The local models are aggregated using different aggregation strategies to obtain an initial global model, wherein the aggregation strategy includes an average strategy and an accuracy strategy. The average strategy is to average the local model parameters as the parameters of the initial global model; the accuracy strategy is to select the most accurate local model as the initial global model.

6. A privacy computing system based on secure differential privacy stochastic gradient descent, characterized in that: Including feature extraction module, pre-training module, secure differential privacy gradient descent module and output module: The feature extraction module uses kernel principal component analysis to extract features from local data to obtain feature vectors of local data, wherein the local data comes from each participant; In the pre-training module, the feature vector of local data is input into the secure differential privacy gradient descent model for pre-training to obtain the local model of each participant; Aggregating the local models of the participants to obtain an initialized global model; The secure differential privacy gradient descent module randomly samples local data of multiple participants in a blockchain trusted environment to obtain a global data set; Inputting the initialized global model and the global data set into a differential privacy gradient descent model for training to obtain a global model; The output module performs a machine learning task based on secure multi-party computing through a global model and outputs a calculation result.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

8. A privacy computing device based on secure differential privacy stochastic gradient descent, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

Images