Privacy-preserving neural network training method and device based on function secret sharing
Through a privacy protection neural network training method based on function secret sharing, combined with additive arithmetic secret sharing and function secret sharing technology, a secure two-party computing protocol is designed, which solves the existing technology for large communication overhead, complex computing and dependence on trusted third parties. The problem of this paper realizes an efficient and practical neural network training framework.
Patent Information
- Application Number
- CN202410074807.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-18
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-01-18
AI Technical Summary
The existing privacy protection neural network training methods have the disadvantages of large communication overhead, complex calculations, inefficient calculations of nonlinear functions and relying on trusted third parties, which limit their practicality.
Using a privacy protection neural network training method based on function secret sharing, combined with additive arithmetic secret sharing and function secret sharing technology, a secure two-party computing protocol was designed to reduce the communication cost through offline-online computing paradigm and do not rely on credible Three parties to achieve efficient neural network training.
Without reducing model accuracy, communication and offline costs are reduced, distributed comparison functions and bit-integer multiplication sub-protocol are optimized, constant round online communication complexity is reduced, and pseudo-random generation is friendly through secure two-party calculations The device replaces trusted third parties and improves practicality.
Smart Images

Figure CN117592527B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a privacy-preserving neural network training method and device based on function secret sharing. Background Art
[0002] Privacy-preserving neural network training enables multiple parties to jointly train neural network models without leaking sensitive data, but its practicality is limited by high communication costs and high reliance on trusted third parties. This paper proposes a secure two-party neural network training framework with efficient communication. The framework can design efficient computational protocols for the linear and nonlinear layers of the neural network by combining additive secret sharing and functional secret sharing techniques, reducing the communication cost during training, and is independent of trusted third parties, thus realizing a practical and efficient privacy-preserving neural network training framework.
[0003] In recent years, privacy-preserving neural network training methods have emerged, including:
[0004] (1) Privacy-preserving neural network training based on obfuscated circuits: By obfuscating the computational process of the neural network model, model training can be performed while protecting the privacy of the original data. During the training process, participants encrypt and obfuscate the input data to avoid the exchange of plaintext data, thereby ensuring the privacy of the data and model during the model training process.
[0005] (2) Privacy-preserving neural network training based on homomorphic encryption: In a multi-party collaborative scenario, privacy protection of input data is achieved during the training of the neural network. During the training process, this method homomorphically encrypts the input data, allowing multiple parties to perform calculations in an encrypted state while maintaining the privacy of the data.
[0006] (3) Privacy-preserving neural network training based on additive secret sharing: In a multi-party collaborative environment, the privacy of input data involved in the training process of the neural network model is protected. During the training process, the original data is divided into multiple secret shares and distributed to each party for calculation, ensuring that each party can only obtain partial information and cannot restore the complete data.
[0007] The above-mentioned prior art has the following disadvantages:
[0008] (1) Privacy-preserving neural network training based on obfuscated circuits: Since obfuscated circuit technology is only applicable to Boolean circuits and obfuscation tables are needed for encryption and obfuscation in the process of implementing secure computing, the Boolean circuits that need to be calculated in the training process of neural network models with a large number of arithmetic operations are more complex and require more obfuscation tables, which makes this technology have the disadvantages of high communication overhead and poor practicality.
[0009] (2) Privacy-preserving neural network training based on homomorphic encryption: Since the data is calculated in an encrypted state in the homomorphic encryption method, it involves complex encryption and decryption operations, which makes the training process slower, resulting in the disadvantages of the technology's computational overhead and implementation complexity.
[0010] (3) Privacy-preserving neural network training based on additive secret sharing: Since the number of communication rounds required by the additive secret sharing method is proportional to the circuit depth of the function calculated by the neural network, this technology has the disadvantages of inefficient nonlinear function calculation and a large number of communication rounds. In addition, this technology relies on a trusted third party to generate protocol components such as multiplication triplets, which makes the cryptographic assumptions required by this technology relatively strong. Summary of the invention
[0011] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, provide a privacy-preserving neural network training method and device based on function secret sharing, adopt an offline-online computing paradigm, design a communication-efficient neural network training secure computing sub-protocol, and use a secure two-party computing protocol to realize a trusted third party, thereby realizing a practical and efficient secure two-party neural network training framework.
[0012] In order to achieve the above object, the present invention adopts the following technical solutions:
[0013] One aspect of the present invention provides a privacy-preserving neural network training method based on function secret sharing, comprising the following steps:
[0014] Given a training set D ={( x 1, y 1),…,( x N , y N )},in x i =( x i1 , x i2 ,…, x id ) T , y i =( y i1 , y i2 ,…, y im ) T ,in N Indicates the number of training samples, the superscript T Represents the transpose of a matrix or vector. d Input neurons,m output neurons and L - A multi-layer feedforward neural network with 1 hidden layer. The specific training process is as follows:
[0015] S1. Participants P 0 and P 1Determine batch size | B |, and based on their respective data, the training data D ={( x 1, y 1),…,( x N , y N )} is divided into t batch training data, denoted as ( X B1 , Y B1 ),( X B2 , Y B2 ),…,( X Bt , Y Bt ),in N =| B |• t ; Then, P 0 distributes the batch training data it holds to P 1, P 1Distribute the batch training data you hold to P 0, making P 0 and P 1 holds the secret share of training data respectively ([ X B1 ]0,[ Y B1 ]0),([ X B2 ]0,[ Y B2 ]0),…,([ X Bt ]0, Y Bt ]0) and ([ X B1 ]1,[ Y B1 ]1),([ X B2 ]1,[ Y B2 ]1),…,([ X Bt ]1,Y Bt ]1), where ([ X Bj ] b ,[ Y Bj ] b ), j =1,2,…, t Denotes the secret share of batch training data, and the secret share is denoted by [•] b express, b Indicates the index of the participant. b =0 means yes P A secret share of 0, b =1 means yes P A secret share of 1, when the subscript is omitted, indicates the secret share that needs to be used simultaneously;
[0016] S2. Participants P 0 and P 1First, the weight parameters of each neural network [ W ]0 and [ W ]1 is initialized to 0, and a batch of training data ([ X Bj ],[ Y Bj ]), j =1,2,…, t , from batch training data ([ X Bj ],[ Y Bj ]) select a sample ([ x i ],[ y i ]), perform the following steps to set the round:
[0017] S21, forward propagation stage: for b =0,1,Participants P b Set the initial input to [ x (0) ] b =[ x i ] b ,right l =1,2,…, L , L is the total number of neural network layers, iterative calculation [ z (l) ] b =MatMul( W (l) ,x (l-1) ),and[ x (l) ] b =ReLU( z (l) );in W (l) For the l The neural network weight parameters of the layer, where MatMul(•) is a secure multiplication subprotocol that calculates the product of two matrices or vectors in the form of secret shares; ReLU(•) is a secure ReLU subprotocol that calculates the ReLU values of all elements in a matrix or vector in the form of secret shares. x The ReLU value is equal to max(0, x ), where max(0, x ) represents 0 and x The maximum value of
[0018] S22, back propagation stage: for b =0,1,Participants P b Let the predicted output [ ]=[ x (L) ], calculate the output layer error [ d (L) ] b ←Hadamard( - y i ,DReLU( z (L) )), then calculate the hidden layer error: [ d i (l’) ] b ←HadamProd(MatMul(( W (l’+1) ) T , d i (l’+1) ),DReLU( z (l’) ));in l’=L -1,…,1, Hadamard(•) is a secure Hadamard product subprotocol, which calculates the element-by-element product of two matrices or vectors in the form of secret shares, that is, the corresponding elements are directly multiplied; DReLU(•) is a secure DReLU subprotocol, which calculates the DReLU values of all elements in the matrix or vector in the form of secret shares, and the elements x The DReLU value is equal to 1{ x ≥0}, where 1{ x ≥0} meansx When ≥0, the output value is 1, otherwise it is 0;
[0019] S23, gradient descent stage, l =1,2,…, L , b =0,1,Participants P b calculate ;
[0020] S3. Finally, P 0The neural network weight parameters[ W ]0=([ W (1) ]0,[ W (2) ]0,…,[ W (L) ]0)Send to P 1. Set the neural network weight parameters [ W ]1=([ W (1) ]1,[ W (2) ]1,…,[ W (L) ]1)Send to P 1, then P 0 and P 1 Each is calculated locally W =[ W ]0+[ W ]1. Reconstruct the neural network weight parameters W .
[0021] As a preferred technical solution, the secure multiplication sub-protocol is specifically:
[0022] for b =0,1,Participants P b Holds secret share[ X ] b ,[ Y 1] b ,…,[ Y k ] b , the relevant triples generated in the offline phase { A j’ ,{ B j’ , C j’} j’=1,…,k},in A j’ for d 1× d 2 matrices,B j’ for d 2× d 3 matrices, C j’ for d 1× d 3 matrices, and satisfy C j’ = A j’ × B j’ ; Subsequently, the participants P b Locally compute secret shares[ XA j’ ] b ,[ Y 1 -B 1] b ,…,[ Y k -B k ] b , and send it to the participants P 1-b , Participants P 0 and P 1 Reconstruction XA j’ , Y 1 -B 1,…, Y k -B k ; Finally, for j’ =1 to k , P b Local Computing Z j’ ] b = b •( X - A j’ )( Y j’ - B j’ )+( X - A j’ )[ B j’ ] b +[ A j’ ] b ( Y j’ - B j’ )+[ Cj’ ] b Get the secret share Z j’ ] b ,…,[ Z k ] b .
[0023] As a preferred technical solution, the secure Hadamard product protocol is specifically:
[0024] Define the bit integer multiplication protocol BitXA(•), BitXA( y,x The specific calculation process of ) is as follows: Given n Bit length input x ∈{0,1} n and 1-bit input y ∈{0,1}, participants P 0 and P 1 respectively hold secret shares [ x ]0,[ y ]0 and [ x ]1,[ y ]1, and execute the offline phase and online phase described below;
[0025] H1. Offline stage: P 0, P 1 Generate relevant randomness, the steps are as follows:
[0026] H11, P b Random Sampling ,Then P 0 and P 1 Follow the steps below to convert it into the corresponding secret share [ d y ] b ∈{0,1} n :
[0027] H111, P 0 order , [ f ]0=0, P 1 order[ e ]1=0, ;
[0028] H112, P 0 and P 1 Call the secure multiplication subprotocol to calculate ([ if ]0,[ if ]1)←MatMul([ e ] b ,[ f ]b ), where MatMul([ e ] b ,[ f ] b )express P 0 and P 1 respectively [ e ]0,[ f ]0 and [ e ]1,[ f ]1 as the input of the secure multiplication sub-protocol MatMul(•), and finally P 0, P 1 respectively obtain the output [ if ]0,[ if ]1;
[0029] H113, P b calculate[ d y ] b =[ e ] b +[ f ] b -2•[ if ] b ;
[0030] H12, P b Random sampling d x ] b ∈{0,1} n , and calculate ([ d z ]0,[ d z ]1)←MatMul([ d x ],[ d y ]);
[0031] H13. After the above steps, P 0 and P 1Hold ,[ d x ]0,[ d y ]0,[ d z ]0 and ,[ d x ]1,[ d y ]1,[ d z ]1;
[0032] H2, Online Stage: P 0, P 1. Perform bit-integer multiplication based on the relevant randomness in the offline phase. The steps are as follows:
[0033] H21, P b Compute locally x ] b +[ d x ] b , , where ⊕ represents an XOR operation, and sends it to P 1-b ;
[0034] H22, P b Refactoring D x = x + d x , , and set D y ’ = D y ,in D y ’ ∈{0,1} n ;
[0035] H23, P b Compute locally z ] b =b • D y ’ • D x +[ d y ] b • D x -2• D y ’ • D x •[ d y ] b - D y ’ •[ d x ] b -[ dz ] b +2• D y ’ •[ d z ] b ;
[0036] Converting the bit integer multiplication protocol BitXA(•) for scalar input into a vector form yields the secure Hadamard product subprotocol Hadamard(•).
[0037] As a preferred technical solution, the specific calculation steps of the secure DReLU sub-protocol are as follows:
[0038] Assuming there are participants P 0 and P 1 respectively hold secret shares [ x ]0 and [ x ]1, which calculates the secret share [ v ]0,[ v ]1Satisfy[ v ]0⊕[ v ]1=1{ x ≥0}, where 1{ x ≥0} means x ≥0, its value is 1, otherwise it is 0, as follows:
[0039] D1. In the offline phase, a trusted third party generates a secret shared key and sends it to P 0 and P 1. The steps are as follows:
[0040] D11. Calculation ( k 01 , k 11 )←Gen n Comp (1 λ , r 1 in , r 2 in , r out ), where the superscript Comp Represents a comparison function, subscript n Indicates the input length, 1 λ is a safety parameter, r 1 in , r 2 in Represents the masks of the two input values respectively, r out Represents a mask of an output value, used to xIn the case of elements x Compare with 0, the output value is x The comparison result with 0;
[0041] D12. k 01 ,[ r 1 in ]0,[ r 2 in ]0,[ r out ]0 and k 11 ,[ r 1 in ]1,[ r 2 in ]1,[ r out ]1Send to P 0 and P 1;
[0042] D2. In the online stage, P 0 and P 1 Calculate the output, the steps are as follows:
[0043] D21, P b Local Computing x + r 1 in ] b =[ x ] b +[ r 1 in ] b and send it to P 1-b ;
[0044] D22, P b Local reconstruction x + r 1 in ;
[0045] D23, P b Local Computing v b ← b ⊕Eval n Comp ( b,k b1 ,x + r 1 in ,r 2 in )⊕[ rout ] b ;in b Indicates the index of the participant. k b1 Indicates Gen n Comp (1 λ , r 1 in , r 2 in , r out ) output value, Eval n Comp Leveraging the Party Index b and the corresponding function secret shared key k b1 To calculate the elements x The result of comparing with 0 is calculated in the process of x and a mask value of 0 x + r 1 in , r 2 in As input to generate x The comparison result with 0 and the value that will not be leaked during the calculation process are guaranteed, and [ r out ] b XOR the comparison result to ensure the output v b privacy.
[0046] As a preferred technical solution, for Gen n Comp (1 λ , r 1 in , r 2 in , r out )The specific execution steps are as follows:
[0047] G1, order r =(2 n -( r 1 in - r 2 in )),and α (n-1) = r [0,n-1) ,in r [0,n-1) Representation value r 0th to n -2 digits to form the value of the string;
[0048] G2, calculate ( k 02 (n-1) , k 12 (n-1) )←Gen n-1 DDCF (1 λ , α (n-1) , β 0, β 1), where β 0=1⊕ r [n-1] , β 1= r [n-1] , r [n-1] Representation value r No. n -1 bit, which is the most significant bit; Gen n-1 DDCF Used to obtain x and α (n-1) The result of the comparison is x > α (n-1) ,but( k 02 (n-1) , k 12 (n-1) ) will lead to the result value β 0, otherwise it leads to the result value β 1;
[0049] G3, Random Selection r 01 , r 11 ∈{0,1}, which satisfies r 01 ⊕ r 11 = r out ;
[0050] G4, remember k b1 = k b2 (n-1) | r b1 ;
[0051] G5, output ( k 01 , k 11 );
[0052] For Eval n Comp ( b,k b1 ,x + r 1 in ,r 2 in ),in b Indicates the index of the participant. k b1 Indicates Gen n Comp (1 λ , r 1 in , r 2 in , r out ), x Represents the input value. The specific execution steps are as follows:
[0053] E1, P b Analysis k b1 = k b2 (n-1) | r b1 , and order z = x + r 1 in -r 2 in , x + r 1 in ,r 2 in Represents the input value;
[0054] E2, P b calculate v b2 ←Eval n-1 DDCF ( b , k b2 (n-1) , z (n-1) ),in z (n-1) = z n-1 - z [0,n-1) -1, Eval n-1 DDCF Indicates that based on inputb , k b2 (n-1) , z (n-1) To generate x and α (n-1) The results of the comparison;
[0055] E3, P b make v b1 =( b • z [n-1] )⊕( v b2 )⊕ r b1 ;
[0056] E4. Output v b1 .
[0057] As a preferred technical solution, for Gen n DDCF (1 λ , α , β 0, β 1), superscript DDCF denotes a dual distributed comparison function, subscript n Indicates the input length, 1 λ is a safety parameter, α , β 0, β 1 indicates input. The specific execution steps are as follows:
[0058] G21, Calculation ( k 03 (n) , k 13 (n) )←Gen n < (1 λ , α , β 0⊕ β 1); among which Gen n < Used to obtain x and α The result of the comparison is x > α ,but( k 03 (n) , k 13 (n)) will lead to the result value β 0⊕ β 1, otherwise the result value is 0;
[0059] G22, random sampling r 02 , r 12 ∈{0,1}, which satisfies r 02 ⊕ r 12 = β 1;
[0060] G23, remember k b2 = k b3 (n) | r b2 ;
[0061] G24, output ( k 02 , k 12 );
[0062] For Eval n DDCF ( b , k b2 , x ), b Indicates the index of the participant. k b2 Indicates Gen n DDCF (1 λ , α , β 0, β 1) output value, x Represents the input value. The specific execution steps are as follows:
[0063] E21, P b Analysis k b2 = k b3 (n) | r b2 ;
[0064] E22, P b calculate y b (n-1) ←Eval n < (b , k b3 (n) , x );Eval n < Leveraging the Party Index b and the corresponding function secret shared key k b3 (n) , to calculate x ≤ α (n-1) The results of the comparison;
[0065] E23, P b make v b2 = y b (n-1) ⊕ r b2 ;
[0066] E24, Output v b2 .
[0067] As a preferred technical solution, for the dual distributed comparison function Gen n < (1 λ , α , β ), where the superscript < represents a less than sign, and the subscript n Indicates the input length, 1 λ is a safety parameter, α , β Represents input, the specific execution steps are as follows:
[0068] G211, Order α 1|…| α n yes α The bit decomposition of α i’ express α No. i’ Bit, | represents the string concatenation operator;
[0069] G212, Random Sampling s 0 (0) ∈{0,1} λ and s 1 (0) ∈{0,1} λ , v 0 (0) =0, v 1 (0) =0 andt 0 (0) =0, t 1 (0) =1;
[0070] G213, yes i’ =1 to n , repeatedly perform the following steps:
[0071] G2131, Calculation s 0 L | v 0 L | t 0 L | s 0 R | v 0 R | t 0 R = G ( s 0 (i’-1) )and s 1 L | v 1 L | t 1 L | s 1 R | v 1 R | t 1 R = G ( s 1 (i ’-1) ),in G It is a random number generator. The L in the superscript indicates the left direction, and the R indicates the right direction.
[0072] G2132, if α i’ =0, let Keep = L , Lose = R Otherwise, Keep = R , Lose = L ;
[0073] G2133, Calculation s CW ← s 0 Lose ⊕ s 1 Lose , v CW L ← v 0L ⊕ v 1 L ⊕( α i’ , β );
[0074] G2134, Calculation t CW L ← t 0 L ⊕ t 1 L ⊕ α i’ ⊕1, t CW R ← t 0 R ⊕ t 1 R ⊕ α i’ ;
[0075] G2135, Calculation CW (i’) = s CW | v CW L | t CW L | t CW R ;
[0076] G2136, Calculation s 0 (i’) = s 0 Keep ⊕ t 0 (i’-1) • s CW , s 1 (i’) = s 1 Keep ⊕ t 1 (i’-1) • s CW ;
[0077] G2137, Calculation t 0 (i’) = t 0 Keep ⊕ t 0 (i’-1) • t CW Keep , t 1(i’) = t 1 Keep ⊕ t 1 (i’-1) • t CW Keep ;
[0078] G214, Order k b3 = s b3 (0) | CW (1) | … | CW (n) ;
[0079] G215, Return ( k 03 , k 13 ).
[0080] As a preferred technical solution, for the distributed comparison function Eval n < ( b,k b3 ,x ), b Indicates the index of the participant. k b3 Indicates Gen n < (1 λ , α , β ), x Represents the input value. The specific execution steps are as follows:
[0081] E221, Analysis k b3 = s b3 (0) | CW (1) | … | CW (n) ,make x = x 1|…| x n , v =0 and t (0) = b ;
[0082] E222, Yes i’ =1 to n , repeatedly perform the following steps:
[0083] E2221, analysis CW (i’) = s CW | v CW L | t CW L | t CW R ;
[0084] E2222, Calculation ;
[0085] E2223, Calculation ;
[0086] E2224, command t (i’) = s L | v L | t L | s R | v R | t R ;
[0087] E2225, if x i’ =0, let y b (n-1) ← v ⊕ v L , s (i’) ← s L , t (i’) ← t L Otherwise, s (i’) ← s R , t (i’) ← t R ;
[0088] E223, Output y b (n-1) .
[0089] As a preferred technical solution, the secure ReLU sub-protocol is specifically:
[0090] According to ReLU( x )= x •DReLU( x ),when P 0 and P 1 respectively obtain DReLU( x )'s Boolean secret share v 0 and v 1. Using ReLU( x )=BitXA(DReLU( x ), x ) can be used to calculate ReLU( x ), where BitXA(•) is a secure bit-integer multiplication protocol.
[0091] Another aspect of the present invention provides a privacy-preserving neural network training device based on function secret sharing, the training device comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the privacy-preserving neural network training method based on function secret sharing as described above.
[0092] The present invention proposes a two-party neural network training framework with efficient communication and privacy protection to support practical secure neural network training. The framework has a small constant round online communication complexity, reduces offline communication costs without reducing model accuracy, and does not rely on a trusted third party in the offline stage. Compared with the prior art, the advantages, beneficial effects and implementation methods of the present invention are as follows:
[0093] (1) Communication-optimized distributed comparison function: The prior art implements the distributed comparison function through a binary tree, but this construction requires a large function secret sharing key size. The present invention proposes a communication-optimized distributed comparison function with a minimum key size by designing a more compact key generation algorithm.
[0094] (2) Bit-integer multiplication sub-protocol: The existing technology does not support bit-integer multiplication. Usually, bits are converted into integers before multiplication calculation, and two rounds of communication are required in the online stage. The present invention designs an offline stage to generate relevant randomness in the offline stage for function calculation in the online stage, so that only one round of communication is required in the online stage, thereby greatly reducing the amount of communication.
[0095] (3) Distributed comparison function key generation algorithm. The existing technology uses a trusted third party to generate relevant randomness, but the trusted third party assumption is strong and is not suitable for practical scenarios. The present invention proposes a distributed comparison function key generation scheme to replace the trusted third party by using a pseudo-random generator that is friendly to secure two-party computing. BRIEF DESCRIPTION OF THE DRAWINGS
[0096] Figure 1 is a flow chart of a privacy-preserving neural network training method based on function secret sharing according to an embodiment of the present invention;
[0097] Figure 2 It is a schematic diagram of an example of a distributed comparison function key generation scheme according to an embodiment of the present invention. DETAILED DESCRIPTION
[0098] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.
[0099] Example
[0100] Secret sharing is one of the basic modules of secure multi-party computing, and secure multi-party computing based on secret sharing is one of the basic paradigms for solving secure multi-party computing problems.
[0101] There are two main stages of secret sharing, namely the secret distribution stage and the secret reconstruction stage:
[0102] 1) Secret distribution phase: The secret owner distributes the secret x Distribute to M Participants P 1, P 2,…, P M , each secret participant will hold a part of the secret, called the secret share, denoted as [ x ], Participants P q The secret share in hand is [ x ] q ,in q ∈{1,.., M};
[0103] 2) Secret reconstruction phase: All secret participants will hold their secret shares [ x ]1,[ x ]2,…,[ x] M Sent to the secret reconstruction party, who reconstructs the secret based on all the secret shares x .
[0104] The secret sharing scheme needs to satisfy the following two properties:
[0105] Privacy: Secret Share[ x ] q Will not reveal secrets x any information;
[0106] Correctness: During the secret reconstruction process, the secret share can be correctly reconstructed to obtain the secret x .
[0107] Additive secret sharing is a very simple and widely used secret sharing scheme. In additive secret sharing, the secret shares satisfy additive homomorphism, that is, ([ x ]0+[ x ]1)mod2 n = x , superscript n is the bit width of the corresponding value, and mod is the modulus operation. Next, we take additive secret sharing as an example to formally define the secret distribution operation and secret reconstruction operation of secret sharing. In the additive secret sharing scheme, it is often defined on a ring, and the secret value is an element on the ring, and there are modular arithmetic addition and modular arithmetic multiplication operations on the ring. In addition, in the additive secret sharing scheme, the secret value s After being distributed to each participant, each participant can perform calculations based on the secret share in hand to obtain the secret share of the function result (usually requires interaction), and then reconstruct the secret to obtain the function result. The following definition will also include the intermediate calculation process.
[0108] With ring , secret value , whose length is n bits. In particular, we assume M =2, that is, there are only two secret parties P 0 and P 1.
[0109] 1) Secret distribution Shr (•)): The secret owner randomly selects , then [ x ]0= r Send to P 0, will [ x ]1=( x - r )mod 2 n Send to P 1;
[0110] 2) Function evaluation Eval (•)): P 0 holds secret shares[ x ]0, and P 1 holds a secret share[ x ]1. The two jointly calculate the function f (•), respectively obtain the secret share of the function result [ y ]0 and [ y ]1. In additive secret sharing, it is usually required that [ y ]0 and [ y ]1 also satisfies additive homomorphism;
[0111] 3) Secret reconstruction Rec (•)): Secret participant P 0 will secret share [ y ]0Sent to the secret reconstruction party, secret participant P 1 The secret share [ y ]1 is sent to the secret reconstruction party, who calculates y =([ y ]0+[ y ]1) mod 2 n , reconstruct the function result y .
[0112] Function Secret Sharing (FSS) is a cryptographic primitive that considers such a scenario. Suppose there is a class of functions F , in this class function f satisfy f :{0,1} n → ,in is an Abelian group (also known as a commutative group). The following is the definition of function secret sharing, which mainly includes three algorithms: Gen,Eval,Rec ):
[0113] 1) is a probabilistic polynomial time key generation algorithm whose input is 1 λ (safety parameters) and , whose output M Keys ( k 1,…, k M ). We assume Explicitly include an input length of 1 n ,group Description and size parameters S .
[0114] 2) Eval ( j'',k j'' ,x ) is a polynomial-time evaluation algorithm whose input is j'' ∈[ M ]([ M ]={1,…, M}), j'' represents the index of the participant), k j'' (defined as f i :{0,1} n → ) and x ∈{0,1} n ( f j'' ), which outputs a group element y j'' ∈ , that is f ( x ) j'' Secret Share f j'' ( x );
[0115] 3) Rec ( f 1( x ) ,…,f M ( x )) is the secret share reconstruction algorithm, whose input is the secret share f j'' ( x ), output the result f ( x ).
[0116] Function secret sharing can be used in secure two-party computations in offline and online models, supporting efficient computational comparisons, piecewise functions and other nonlinear functions. The algorithm Gen (•)and Eval (•) correspond to the offline and online stages respectively. Let the participant index b =0,1, two parties P b = P 0. P 1. In the offline phase, a trusted third party checks each input line in the computing circuit. w in Mask r in or each output line w out of rout For each w in and w out Door g , a trusted third party constructs an offset function , and run Gen (•) to generate the function Function Secret Shared Key( k 0, k 1) Then, the trusted third party will k b Send to P b , and P b Possess circuit input and output lines w The corresponding mask r Send to P b In the online stage, P b Calculate each w in Shield value (in P b hold r in ) and sends it to P 1-b . Starting from the input gate, P 0 and P 1Evaluate the gates in topological order to obtain the masked output line values. To use w in and w out Processing Gate g , P b Local call Eval (•) Algorithm, using a function to secretly share a key k b and shield value As input, get the shield output line value g ( x )+ r out .at last, P 0 and P 1 minus the corresponding mask received from the dealer to get the final output value.
[0117] Distributed Comparison Function (DCF) is a comparison function class for f α,β <(x )(when x < α , output β ; otherwise output 0) of the two-party function secret sharing scheme. Among them, the key generation algorithm Gen (•) Using a Pseudo-Random Generator (PRG) G :{0,1}→{0,1} 2(λ+2) Generate two keys ( k 0, k 1), where k b Includes PRG seeds s b ’ and n Shared correction words ( CW 1 ,…,CW (n) ), b = 0, 1 is the index of the participant. This key implicitly defines a Goldreich-Goldwasser-Micali (GGM) binary tree, which has 2 n leaf nodes, where each leaf node is marked as a n Bit length input x Each node in the binary tree is associated with a label tuple ( s',v,t' )∈{0,1} λ ×{0,1}×{0,1}, where s’ Represents seeds, v’ Represents the result bit, t’ Represents the state bit. The label tuple of each node is completely determined by the label of its parent node. This construction ensures that for a certain input x For all nodes, the sum of the result bits is v 0⊕ v 1 is exactly equal to f α,β <( x ). Therefore, when entering x Evaluation Key k b (i.e., evaluation algorithm Eval (•)) Need to go from root to representative x Leaf node traversal k b The binary tree represented by , and calculates each node in turn ( s b ’,v b ’, t b’ ) and value v b ’ , and finally get the result of the comparison function.
[0118] Neural network model, let D ={( x i ,y i )| i ∈{0,1,…, m’}} represents the training data set, where each data sample x i Contains the corresponding output label y i of d dimensional features. Neural networks are learning functions g’ The calculation process makes g’ ( x i )≈ y i ,in g’ Can be expressed as the neural network weight parameter W and input data x i The training process of a neural network consists of two stages, forward propagation and backward propagation. The stage is called forward propagation, which consists of linear operations and nonlinear activation functions. f One of the most popular activation functions is the ReLU function. f ( u )=max(0, u ).
[0119] To learn the neural network weight parameters W , defines a quantitative prediction value and actual value y i The cost function of the error between C ( W ), and by solving argmin W C ( W ) optimization problem to calculate and update W This optimization problem can be solved using the stochastic gradient descent algorithm, which is an efficient approximation algorithm that gradually approaches the local minimum of a function. The stochastic gradient descent algorithm works as follows: W is initialized to a vector of random values or all 0s. In each iteration, a sample is randomly selected ( x i ,y i ), and through W := W - α C ( W ) Update the neural network weight parameters W ,in α is the learning rate, C ( W ) is the partial derivative, i.e. the cost with respect to the change in weight. Calculate the change α C ( W ) is called back propagation, where the error rate is fed back through the neural network to update the neural network weight parameters W .
[0120] In practice, instead of selecting a data sample at each iteration, a small batch of samples is randomly selected and averaged over all samples in the current iteration. W Update the partial derivatives on W This is called the mini-batch stochastic gradient descent algorithm, and its advantage is to allow the use of vectorized libraries to speed up the calculations, resulting in faster calculations.
[0121] like Figure 1 As shown, this embodiment provides a privacy-preserving neural network training method based on function secret sharing, which can support two participants P 0, P 1 (denoted as P b and b =0,1) to achieve joint neural network model training without revealing the participating parties P 0, P 1. Privacy of training data held, including the following steps:
[0122] During the training process, P 0, P 1 respectively hold the training data and input the training data (such as Figure 1 In [ x i ]0,[ x i ]1), forward propagation calculation, error function calculation and backward propagation calculation are performed, and the weights are updated. In this process, the secure linear layer protocol (secure multiplication ( Figure 1 × in), secure Hadamard product ( Figure 1 ⊙ in)) and secure nonlinear layer operations ( Figure 1 ReLU and DReLU in ), these operations requireP 0, P 1 is calculated through a secure two-party computation (2PC) protocol. Specifically, the secure linear layer protocol is implemented by an additive secret sharing protocol, and the secure nonlinear layer protocol is implemented by a functional secret sharing protocol. The specific protocols are as follows.
[0123] 1. Secure Linear Layer Subprotocol
[0124] The secure linear layer sub-protocol is introduced below, including the secure multiplication sub-protocol and the secure Hadamard product sub-protocol.
[0125] (1) Secure Multiplication Sub-Protocol
[0126] The present invention utilizes multiplication triplets in matrix form to improve matrix operation efficiency, and utilizes related multiplication triplets to reduce the number of multiplication triplets used in matrix operation.
[0127] Features d 1× d 2 Matrix X Respectively d 2× d 3 Matrix Y 1,…, Y k Multiplication, independent multiplication triples are required when calculating with independent multiplication triples { A j’ , B j’ , C j’} j’=1,…,k ,in A j’ for d 1× d 2 matrices, B j’ for d 2× d 3 matrices, C j’ for d 1× d 3 matrices, and satisfy C j’ = A j’ × B j’ . But considering that only the same A j’ To cover X Then, using this technique, only the relevant triples { A j’ ,{ Bj’ , C j’} j’=1,…,k},in C j’ = A j’ × B j’ , thereby reducing k -1 d 1× d 2 matrices.
[0128] The secure multiplication protocol is denoted as MatMul(•), and the specific calculation process is as follows: b =0,1,Participants P b Holds secret share[ X ] b ,[ Y 1] b ,…,[ Y k ] b , the relevant triples generated in the offline phase { A j’ ,{ B j’ , C j’} j’=1,…,k}. Subsequently, the participants P b Locally compute secret shares[ XA j’ ] b ,[ Y 1 -B 1] b ,…,[ Y k -B k ] b , and send it to the participants P 1-b , Participants P 0 and P 1 Reconstruction XA j’ , Y 1 -B 1,…, Y k -B k Finally, for j’ =1 to k , P b Local Computing Z j’ ] b =b •( X - A j’ )( Y j’ - B j’ )+( X - A j’ )[ B j’ ] b +[ A j’ ] b ( Y j’ - B j’ )+[ C j’ ] b Get the secret share Z j’ ] b ,…,[ Z k ] b .
[0129] (2) Secure Hadamard Product Protocol
[0130] The Hadamard product, also known as the element-by-element product, is very similar to matrix multiplication and can be implemented by modifying MatMul(•). However, in the neural network training protocol, the Hadamard product calculation X ⊙ Y hour, Y Medium Element y ∈{0,1} is a bit value, so it cannot be directly calculated using multiplication triples. The present invention designs a bit integer multiplication protocol, denoted as BitXA(•), which can be easily extended to vector form to obtain Hadamard(•).
[0131] ( y,x ) The specific calculation process of the protocol is as follows: Given n Bit length input x ∈{0,1} n and 1-bit input y ∈{0,1}, participants P 0 and P 1 respectively hold secret shares [ x ]0,[ y ]0 and [ x ]1,[ y ]1, and execute the offline phase and online phase described below.
[0132] H1. Offline stage: P 0, P 1 Generate relevant randomness, the steps are as follows:
[0133] H11, P b Random Sampling ,Then P 0 and P 1 Follow the steps below to convert it into the corresponding secret share [ d y ] b ∈{0,1} n :
[0134] H111, P 0 order , [ f ]0=0, P 1 order[ e ]1=0, .
[0135] H112, P 0 and P 1 Call the secure multiplication subprotocol to calculate ([ if ]0,[ if ]1)←MatMul([ e ] b ,[ f ] b ), where MatMul([ e ] b ,[ f ] b )express P 0 and P 1 respectively [ e ]0,[ f ]0 and [ e ]1,[ f ]1 as the input of the secure multiplication sub-protocol MatMul(•), and finally P 0, P 1 respectively obtain the output [ if ]0,[ if ]1.
[0136] H113, P b calculate[ d y ] b =[ e ] b +[ f ] b -2•[ if ] b .
[0137] H12, P b Random sampling d x ] b ∈{0,1} n , and calculate ([ d z ]0,[ d z ]1)←MatMul([ d x ],[ d y ]).
[0138] H13. After the above steps, P 0 and P 1Hold ,[ d x ]0,[ d y ]0,[ d z ]0 and ,[ d x ]1,[ d y ]1,[ d z ]1.
[0139] H2, Online Stage: P 0, P 1. Perform bit-integer multiplication based on the relevant randomness in the offline phase. The steps are as follows:
[0140] H21, P b Compute locally x ] b +[ d x ] b , , where ⊕ represents an XOR operation, and sends it to P 1-b .
[0141] H22, P b Refactoring D x = x + d x , , and set D y ’ = D y ,in D y ’ ∈{0,1} n
[0142] H23, P b Compute locally z ] b =b • D y ’ • D x +[ d y ] b • D x -2• D y ’ • D x •[ d y ] b - D y ’ •[ d x ] b -[ d z ] b +2• D y ’ •[ d z ] b .
[0143] The above process is for scalar input, which can be easily converted into vector form to obtain the secure Hadamard product protocol Hadamard(•).
[0144] 2. Secure Nonlinear Layer Protocol
[0145] The ReLU function is a commonly used activation function in the neural network training process as follows:
[0146]
[0147] Its derivative is denoted as DReLU and is defined as:
[0148] Among them, ReLU( x )= x •DReLU( x ).
[0149] The following first gives a function secret sharing scheme for the distributed comparison function, and then gives a secure computation sub-protocol for the ReLU function and the DReLU function based on the distributed comparison function.
[0150] (1) Distributed comparison function
[0151] Integer comparison function class Including all groups arrive Function. Formally, the class of integer comparison functions have:
[0152]
[0153] Among them, the function f α,β : → satisfy:
[0154]
[0155] The present invention mainly uses f α,1 { x},middle , ={0,1}. The basic idea is to construct a function that can represent the comparison f α,1 { x}, the binary tree has n +1 layer nodes, the root node is at the first layer, and the two outgoing edges of each node in the tree are bit values 0 and 1 respectively (without loss of generality, let the left outgoing edge correspond to 0 and the right outgoing edge correspond to 1). Each node in the tree is associated with a tuple s’ || v’ || t’ ,in s’ ∈{0,1} λ It's a seed. v’ ∈{0,1}, t’ ∈{0,1}.
[0156] definition x = x 1,…, x n ∈{0,1} n The corresponding path is the calculation path, and the definition α = α 1,…, α n ∈{0,1} n The corresponding calculation path is a special calculation path. For example, Figure 2 middleα 1 α 2=01 corresponds to the special calculation path. x ≠ α ,but x The corresponding calculation path and α The corresponding calculation path (special calculation path) will start to branch after a certain node, which is called x Relative to α The bifurcation node of is also called x Notice that when x < α Assuming that x The bifurcation node is at i’ (1≤ i’ ≤ n ) layer, then there is x 1,…, x n = α 1,…, α n ,and x i’+1 =0, α i’+1 =1, this situation is called x Relative to α The corresponding bifurcation node is called the left bifurcation node; when x ≥ α When x 1,…, x i’ = α 1,…, α i’ ,and x i’+1 =1, α i’+1 =0, this situation is called x Relative to α Right bifurcation, the corresponding bifurcation node is called right bifurcation node. Figure 2 Middle Node l’’ That is x =00 relative to α =01 left bifurcation node.
[0157] The reason why the tuples associated with any node in the tree are required to satisfy the above relationship is that we hope x The corresponding calculation path passes through the nodes v XOR of value equal to f α,1 { x}. Note that for any x = x 1,…,x n ∈{0,1} n , the corresponding calculation path passes through the nodes v The value is either 0 or 1, and only if a node on the path is x Only when the left child node of the bifurcation node v =1. And because for any x , which is relative to α There is at most one bifurcation node, and only if x < α , it has a left bifurcation node, the left child node of this node v The value is 1. Therefore, if and only if x < α When the corresponding calculation path passes through the nodes v The sum of the values is equal to 1, and the rest are 0, which is consistent with f α,1 { x}results.
[0158] The following is a distributed comparison function construction, whose scheme is given by (Gen n < ,Eval n < ), and its specific structure is as follows:
[0159] Gen n < (1 λ , α , β ): Gen n < The superscript < represents the less-than sign (because it is used to construct a less-than relationship), and the subscript n Indicates the input length, 1 λ is a safety parameter, α , β Represents input, the specific execution steps are as follows:
[0160] G211, Order α 1|…| α n yes α The bit decomposition of α i’ express α No. i’ Position, | represents the string concatenation operator.
[0161] G212, Random Sampling s 0 (0) ∈{0,1} λ and s 1(0) ∈{0,1} λ , v 0 (0) =0, v 1 (0) =0 and t 0 (0) =0, t 1 (0) =1.
[0162] G213, yes i’ =1 to n , repeatedly perform the following steps:
[0163] G2131, Calculation s 0 L | v 0 L | t 0 L | s 0 R | v 0 R | t 0 R = G ( s 0 (i’-1) )and s 1 L | v 1 L | t 1 L | s 1 R | v 1 R | t 1 R = G ( s 1 (i ’-1) ),in G It is a random number generator. The L in the superscript indicates the left direction, and the R indicates the right direction.
[0164] G2132, if α i’ =0, let Keep = L , Lose = R Otherwise, Keep = R , Lose = L ;
[0165] G2133, Calculation s CW ← s 0 Lose⊕ s 1 Lose , v CW L ← v 0 L ⊕ v 1 L ⊕( α i’ , β );
[0166] G2134, Calculation t CW L ← t 0 L ⊕ t 1 L ⊕ α i’ ⊕1, t CW R ← t 0 R ⊕ t 1 R ⊕ α i’ ;
[0167] G2135, Calculation CW (i’) = s CW | v CW L | t CW L | t CW R ;
[0168] G2136, Calculation s 0 (i’) = s 0 Keep ⊕ t 0 (i’-1) • s CW , s 1 (i’) = s 1 Keep ⊕ t 1 (i’-1) • s CW ;
[0169] G2137, Calculation t 0 (i’) = t 0 Keep⊕ t 0 (i’-1) • t CW Keep , t 1 (i’) = t 1 Keep ⊕ t 1 (i’-1) • t CW Keep ;
[0170] G214, Order k b3 = s b3 (0) | CW (1) | … | CW (n) ;
[0171] G215, Return ( k 03 , k 13 ).
[0172] n < ( b,k b3 ,x ):in b Indicates the index of the participant. k b3 Indicates Gen n < (1 λ , α , β ), x Represents the input value. The specific steps are as follows:
[0173] E221, Analysis k b3 = s b (0) | CW (1) | … | CW (n) ,make x = x 1|…| x n , v =0 and t (0) = b ;
[0174] E222, Yes i’ =1 to n , repeatedly perform the following steps:
[0175] E2221, analysis CW (i’) = s CW | v CW L | t CW L | t CW R ;
[0176] E2222, Calculation ;
[0177] E2223, Calculation ;
[0178] E2224, command t (i’) = s L | v L | t L | s R | v R | t R ;
[0179] E2225, if x i’ =0, let y b (n-1) ← v ⊕ v L , s (i’) ← s L , t (i) ← t L Otherwise, s (i’) ← s R , t (i’) ← t R ;
[0180] E223, Output y b(n-1) .
[0181] In Gen n < (1 λ , α , β ) During the implementation process, the present invention implements the algorithm based on a pseudo-random number generator, thereby eliminating the need to rely on a trusted third party for execution.
[0182] (2) Secure DReLU sub-protocol
[0183] The secure computation subprotocol of the DReLU function is given below. First, a variant of the distributed comparison function is given. Then, a function secret sharing construction scheme for the signed function comparison gate is given. Finally, a secure DReLU subprotocol is designed and implemented based on this scheme.
[0184] 1) Dual distributed comparison function
[0185] The dual distributed comparison function is a variant of the distributed comparison function, whose scheme is given by (Gen n DDCF ,Eval n DDCF ), and its specific structure is as follows:
[0186] Gen n DDCF (1 λ , α , β 0, β 1):Gen n DDCF Superscript DDCF represents the dual distributed comparison function, subscript n Indicates the input length, 1 λ is a safety parameter, α , β 0, β 1 indicates input. The specific execution steps are as follows:
[0187] G21, Calculation ( k 03 (n) , k 13 (n) )←Gen n < (1 λ , α , β 0⊕ β 1)
[0188] G22, random sampling r 02, r 12 ∈{0,1}, which satisfies r 02 ⊕ r 12 = β 1.
[0189] G23, remember k b2 = k b3 (n) | r b2 .
[0190] G24, output ( k 02 , k 12 ).
[0191] n DDCF ( b , k b2 , x ):in b Indicates the index of the participant. k b2 Indicates Gen n DDCF (1 λ , α , β 0, β 1) output value, x Represents the input value. The specific steps are as follows:
[0192] E21, P b Analysis k b2 = k b3 (n) | r b2 .
[0193] E22, P b calculate y b (n-1) ←Eval n < ( b , k b3 (n) , x ).
[0194] E23, Pb make v b2 = y b (n-1) ⊕ r b2 .
[0195] E24, Output v b2 .
[0196] 2) Signed function comparison gate
[0197] The signed function comparison gate function secret sharing scheme is given by (Gen n Comp ,Eval n Comp ), and its specific structure is as follows:
[0198] Gen n Comp (1 λ , r 1 in , r 2 in , r out ): Gen n Comp Superscript Comp Represents a comparison function, subscript n Indicates the input length, 1 λ is a safety parameter, r 1 in , r 2 in Represent the masks of the two input lines respectively, r out Represents the mask of an output line. The specific execution steps are as follows:
[0199] G1, order r =(2 n -( r 1 in - r 2 in )),and α (n-1) = r [0,n-1) ,in r [0,n-1) Representation value r 0th to n -2 digits that make up the value of the string.
[0200] G2, calculate ( k 02 (n-1), k 12 (n-1) )←Gen n-1 DDCF (1 λ , α (n-1) , β 0, β 1), where β 0=1⊕ r [n-1] , β 1= r [n-1] , r [n-1] Representation value r No. n -1 bit, which is the most significant bit.
[0201] G3, Random Selection r 01 , r 11 ∈{0,1}, which satisfies r 01 ⊕ r 11 = r out .
[0202] G4, remember k b1 = k b2 (n-1) | r b1 .
[0203] G5, output ( k 01 , k 11 ).
[0204] n Comp ( b,k b1 ,x + r 1 in ,r 2 in )Eval n Comp ( b,k b1 ,x + r 1 in ,r 2 in ):in b Indicates the index of the participant. kb1 Indicates Gen n Comp (1 λ , r 1 in , r 2 in , r out ), x + r 1 in ,r 2 in Represents the input value. The specific steps are as follows:
[0205] E1, P b Analysis k b1 = k b2 (n-1) | r b1 , and order z = x + r 1 in -r 2 in .
[0206] E2, P b calculate v b2 ←Eval n-1 DDCF ( b , k b2 (n-1) , z (n-1) ),in z (n-1) = z n-1 - z [0,n-1) -1.
[0207] E3, P b make v b1 =( b • z [n-1] )⊕( v b2 )⊕ r b1 .
[0208] E4. Output v b1 .
[0209] 3) Secure DReLU sub-protocol based on signed function comparison gate
[0210] The following is a detailed implementation of the secure DReLU subprotocol. P 0 and P 1 respectively hold secret shares [ x ]0 and [ x ]1, which calculates the secret share [ v ]0,[ v ]1Satisfy[ v ]0⊕[ v ]1=1{ x ≥0}, where 1{ x ≥0} means x ≥0, its value is 1, otherwise it is 0. The specific calculation steps are as follows:
[0211] D1. In the offline phase, a trusted third party generates a secret shared key and sends it to P 0 and P 1. The steps are as follows:
[0212] D11. Calculation ( k 01 , k 11 )←Gen n Comp (1 λ , r 1 in , r 2 in , r out ). n Comp Superscript Comp Represents a comparison function, subscript n Indicates the input length, 1 λ is a safety parameter, r 1 in , r 2 in Represents the masks of the two input values respectively, r out Represents a mask of an output value, used to x In the case of elements x Compare with 0 to protect x The privacy of x The comparison result with 0.
[0213] D12. k 01 ,[ r 1 in ]0,[r 2 in ]0,[ r out ]0 and k 11 ,[ r 1 in ]1,[ r 2 in ]1,[ r out ]1Send to P 0 and P 1.
[0214] D2. In the online stage, P 0 and P 1 Calculate the output, the steps are as follows:
[0215] D21, P b Local Computing x + r 1 in ] b =[ x ] b +[ r 1 in ] b and send it to P 1-b .
[0216] D22, P b Local reconstruction x + r 1 in .
[0217] D23, P b Local Computing v b ← b ⊕Eval n Comp ( b,k b1 ,x + r 1 in ,r 2 in )⊕[ r out ] b .
[0218] (3) Secure ReLU protocol
[0219] According to ReLU( x )= x •DReLU( x) relationship, when P 0 and P 1 respectively obtain DReLU( x )'s Boolean secret share v 0 and v 1. Using ReLU( x )=BitXA(DReLU( x ), x ) can be used to calculate ReLU( x ), where BitXA(•) is the secure bit-integer multiplication protocol introduced earlier.
[0220] 3. Privacy-preserving neural network training protocol
[0221] Based on the above secure linear layer sub-protocol and nonlinear layer sub-protocol, the present invention proposes a privacy-preserving neural network training protocol. D ={( x 1, y 1),…,( x N , y N )},in x i =( x i1 , x i2 ,…, x id ) T , y i =( y i1 , y i2 ,…, y im ) T ,in N Indicates the number of training samples, the superscript T Represents the transpose of a matrix or vector, considering having d Input neurons, m output neurons and L -Multi-layer feedforward neural network with 1 hidden layer. The specific process of neural network training is as follows:
[0222] S1. Participants P 0 and P 1Determine batch size in advance | B | (for example, let the batch size be 128), and divide the training data into D ={( x 1, y 1),…,( xN , y N ))}divided into t batch training data, denoted as ( X B1 , Y B1 ),( X B2 , Y B2 ),…,( X Bt , Y Bt ), and distribute it secretly to P 0 and P 1, where N =| B |• t . Then, P 0 distributes the batch training data it holds to P 1, P 1Distribute the batch training data you hold to P 0, making P 0 and P 1 holds the secret share of training data respectively ([ X B1 ]0,[ Y B1 ]0),([ X B2 ]0,[ Y B2 ]0),…,([ X Bt ]0, Y Bt ]0) and ([ X B1 ]1,[ Y B1 ]1),([ X B2 ]1,[ Y B2 ]1),…,([ X Bt ]1, Y Bt ]1), where ([ X Bj ] b ,[ Y Bj ] b ), j =1,2,…, t Denotes the secret share of batch training data, and the secret share is denoted by [•]b express, b =0 means yes P A secret share of 0, b =1 means yes P A secret share of 1, when the subscript is omitted, indicates the secret share that needs to be used simultaneously;
[0223] S2. Participants P 0 and P 1First, the weight parameters of each neural network [ W ]0 and [ W ]1 is initialized to 0, and the participants select a batch of training data ([ X Bj ],[ Y Bj ]), j =1,2,…, t , from batch training data ([ X Bj ],[ Y Bj ]) select a sample ([ x i ],[ y i ]), execute the following steps to set the rounds (in this embodiment, execute 10 or 15 times):
[0224] S21, forward propagation stage: for b =0,1,Participants P b Set the initial input to [ x (0) ] b =[ x i ] b ,right l =1,2,…, L , L is the total number of neural network layers, iterative calculation [ z (l) ] b =MatMul( W (l) , x (l-1) ),and[ x (l) ] b =ReLU( z (l) );in W (l) For the lThe neural network weight parameters of the layer, where MatMul(•) is a secure multiplication subprotocol that calculates the product of two matrices or vectors in the form of secret shares; ReLU(•) is a secure ReLU subprotocol that calculates the ReLU values of all elements in a matrix or vector in the form of secret shares. x The ReLU value is equal to max(0, x ), where max(0, x ) represents 0 and x The maximum value of
[0225] S22, back propagation stage: for b =0,1,Participants P b Let the predicted output [ ]=[ x (L) ], calculate the output layer error [ d (L) ] b ←Hadamard( - y i ,DReLU( z (L) )), then calculate the hidden layer error:[ d i (l’) ] b ←HadamProd(MatMul(( W (l’+1) ) T , d i (l’+1) ),DReLU( z (l’) ));in l’=L -1,…,1; Hadamard(•) is a secure Hadamard product subprotocol, which calculates the element-by-element product of two matrices or vectors in the form of secret shares, that is, the corresponding elements are directly multiplied; DReLU(•) is a secure DReLU subprotocol, which calculates the DReLU values of all elements in the matrix or vector in the form of secret shares. x The DReLU value is equal to 1{ x ≥0}, where 1{ x ≥0} means x The output value is 1 when ≥0, otherwise it is 0.
[0226] S23, gradient descent stage, l =1,2,…, L , Participants P b calculate ;
[0227] S3. Finally, P 0The neural network weight parameters[ W ]0=([ W (1) ]0,[ W (2) ]0,…,[ W (L) ]0)Send to P 1. Set the neural network weight parameters [ W ]1=([ W (1) ]1,[ W (2) ]1,…,[ W (L) ]1)Send to P 1, then P 0 and P 1Reconstructing neural network weight parameters W .
[0228] In another embodiment of the present application, a privacy-preserving neural network training device based on function secret sharing is provided, the device comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the privacy-preserving neural network training method based on function secret sharing as described in the above embodiment.
[0229] It should be noted here that the system / device provided in the above embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure is divided into different functional modules to complete all or part of the functions described above. The system / device is a privacy-preserving neural network training method based on function secret sharing applied to the above embodiment.
[0230] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0231] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.
Claims
1. A privacy-preserving neural network training method based on function secret sharing, characterized in that: The steps include: Given a training set D ={( x 1, y 1),…,( x N , y N )},in x i =( x i1 , x i2 ,…, x id ) T , y i =( y i1 , y i2 ,…, y im ) T ,in N Indicates the number of training samples, the superscript T Represents the transpose of a matrix or vector. d Input neurons, m output neurons and L’ - A multi-layer feedforward neural network with 1 hidden layer. The specific training process is as follows: S1. Participants P 0 and P 1Determine batch size | B |, and based on their respective data, the training data D ={( x 1, y 1),…,( x N , y N )} is divided into t batch training data, denoted as ( X B1 , Y B1 ),( X B2 , Y B2 ),…,( X Bt , Y Bt ),in N =| B | t ; Then, P 0 distributes the batch training data it holds to P 1, P 1Distribute the batch training data you hold to P 0, making P 0 and P 1 holds the secret share of training data respectively ([ X B1 ]0,[ Y B1 ]0),([ X B2 ]0,[ Y B2 ]0),…,([ X Bt ]0, Y Bt ]0) and ([ X B1 ]1,[ Y B1 ]1),([ X B2 ]1,[ Y B2 ]1),…,([ X Bt ]1, Y Bt ]1), where ([ X Bj ] b ,[ Y Bj ] b ), j =1,2,…, t Represents the secret share of batch training data, and the secret share is denoted by [·] b express, b Indicates the index of the participant. b =0 means yes P A secret share of 0, b =1 means yes P A secret share of 1, when the subscript is omitted, indicates the secret share that needs to be used simultaneously; S2. Participants P 0 and P 1First, the weight parameters of each neural network [ W ]0 and [ W ]1 is initialized to 0, and a batch of training data ([ X Bj ],[ Y Bj ]), j =1,2,…, t , from batch training data ([ X Bj ],[ Y Bj ]) select a sample ([ x i ],[ y i ]), perform the following steps to set the round: S21, forward propagation stage: for b =0,1,Participants P b Set the initial input to [ x (0) ] b =[ x i ] b ,right l =1,2,…, L’ , L’ is the total number of neural network layers, iterative calculation [ z (l) ] b =MatMul( W (l) , x (l-1) ),and[ x (l) ] b =ReLU( z (l) );in W (l) For the l The neural network weight parameters of the layer, where MatMul(·) is a secure multiplication subprotocol that calculates the product of two matrices or vectors in the form of secret shares; ReLU(·) is a secure ReLU subprotocol that calculates the ReLU values of all elements in a matrix or vector in the form of secret shares. x The ReLU value is equal to max(0, x ), where max(0, x ) represents 0 and x The maximum value of The secure multiplication sub-protocol is specifically: for b =0,1,Participants P b Holds secret share[ X ] b ,[ Y 1] b ,…,[ Y k ] b , the relevant triples generated in the offline phase { A j’ ,{ B j’ , C j’ } j’=1,…,k },in A j’ for d 1× d 2 matrices, B j’ for d 2× d 3 matrices, C j’ for d 1× d 3 matrices, and satisfy C j’ = A j’ × B j’ ; Subsequently, the participants P b Locally compute secret shares[ XA ] b ,[ Y 1 -B 1] b ,…,[ Y k -B k ] b , and send it to the participants P 1-b , Participants P 0 and P 1 Reconstruction XA j’ , Y 1 -B 1,…, Y k -B k ; Finally, for j’ =1 to k , P b Local Computing Z j’ ] b = b •( X - A j’ )( Y j’ - B j’ )+( X - A j’ )[ B j’ ] b +[ A j’ ] b ( Y j’ - B j’ )+[ C j’ ] b Get the secret share Z j’ ] b ,…,[ Z k ] b ; S22, back propagation stage: for b =0,1,Participants P b Let the predicted output [ ]=[ x (L’) ], calculate the output layer error [ δ (L’) ] b ←Hadamard( - y i ,DReLU( z (L’) )), then calculate the hidden layer error: [ δ i (l’) ] b ←HadamProd(MatMul(( W (l’+1) ) T , δ i (l’+1) ),DReLU( z (l’) ));in l’=L’ -1,…,1, Hadamard(•) is a secure Hadamard product subprotocol, which calculates the element-by-element product of two matrices or vectors in the form of secret shares, that is, the corresponding elements are directly multiplied; DReLU(•) is a secure DReLU subprotocol, which calculates the DReLU values of all elements in the matrix or vector in the form of secret shares, and the elements x The DReLU value is equal to 1{ x ≥0}, where 1{ x ≥0} means x When ≥0, the output value is 1, otherwise it is 0; The secure Hadamard product protocol is specifically: Define the bit integer multiplication protocol BitXA(•), BitXA( y,x ) is as follows: Given n Bit length input x ∈{0,1} n and 1-bit input y ∈{0,1}, participants P 0 and P 1 respectively hold secret shares [ x ]0,[ y ]0 and [ x ]1,[ y ]1, and execute the offline phase and online phase described below; H1. Offline stage: P 0, P 1 Generate relevant randomness, the steps are as follows: H11, P b Random Sampling ,Then P 0 and P 1 Follow the steps below to convert it into the corresponding secret share [ δ y ] b ∈{0,1} n : H111, P 0 order , [ f ]0=0, P 1 order[ e ]1=0, ; H112, P 0 and P 1 Call the secure multiplication subprotocol to calculate ([ ef ]0,[ ef ]1)←MatMul([ e ] b ,[ f ] b ), where MatMul([ e ] b ,[ f ] b )express P 0 and P 1 respectively [ e ]0,[ f ]0 and [ e ]1,[ f ]1 as the input of the secure multiplication subprotocol MatMul(•), and finally P 0, P 1 respectively obtain the output [ ef ]0,[ ef ]1; H113, P b calculate[ δ y ] b =[ e ] b +[ f ] b -2•[ ef ] b ; H12, P b Random sampling δ x ] b ∈{0,1} n , and calculate ([ δ z ]0,[ δ z ]1)←MatMul([ δ x ],[ δ y ]); H13. After the above steps, P 0 and P 1Hold ,[ δ x ]0,[ δ y ]0,[ δ z ]0 and ,[ δ x ]1,[ δ y ]1,[ δ z ]1; H2, Online Stage: P 0, P 1. Perform bit-integer multiplication based on the relevant randomness in the offline phase. The steps are as follows: H21, P b Compute locally x ] b +[ δ x ] b , , where ⊕ represents an XOR operation, and sends it to P 1-b ; H22, P b Refactoring Δ x = x + δ x , , and set Δ y ’ = Δ y ,in Δ y ’ ∈{0,1} n ; H23, P b Compute locally z ] b =b • Δ y ’ • Δ x +[ δ y ] b • Δ x -2• Δ y ’ • Δ x •[ δ y ] b - Δ y ’ •[ δ x ] b -[ δ z ] b +2• Δ y ’ •[ δ z ] b ; Convert the bit integer multiplication protocol BitXA(•) for scalar input into vector form, thus obtaining the secure Hadamard product subprotocol Hadamard(•); The specific calculation steps of the secure DReLU subprotocol are as follows: Assuming there are participants P 0 and P 1 respectively hold secret shares [ x ]0 and [ x ]1, which calculates the secret share [ v ]0,[ v ]1Satisfy[ v ]0⊕[ v ]1=1{ x ≥0}, where 1{ x ≥0} means x ≥0, its value is 1, otherwise it is 0, as follows: D1. In the offline phase, a trusted third party generates a secret shared key and sends it to P 0 and P 1. The steps are as follows: D11. Calculation ( k 01 , k 11 )←Gen n Comp (1 λ , r 1 in , r 2 in , r out ), where the superscript Comp Represents a comparison function, subscript n Indicates the input length, 1 λ is a safety parameter, r 1 in , r 2 in Represents the masks of the two input values respectively, r out Represents a mask of an output value, used to x In the case of elements x Compare with 0, the output value is x The comparison result with 0; D12. k 01 ,[ r 1 in ]0,[ r 2 in ]0,[ r out ]0 and k 11 ,[ r 1 in ]1,[ r 2 in ]1,[ r out ]1Send to P 0 and P 1; D2. In the online stage, P 0 and P 1 Calculate the output, the steps are as follows: D21, P b Local Computing x + r 1 in ] b =[ x ] b +[ r 1 in ] b and send it to P 1-b ; D22, P b Local reconstruction x + r 1 in ; D23, P b Local Computing v b ← b ⊕Eval n Comp ( b,k b1 ,x + r 1 in ,r 2 in )⊕[ r out ] b ;in b Indicates the index of the participant. k b1 Indicates Gen n Comp (1 λ , r 1 in , r 2 in , r out ), Eval n Comp Leveraging the Party Index b and the corresponding function secret shared key k b1 To calculate the elements x The result of comparing with 0 is calculated in the process of x and a mask value of 0 x + r 1 in ,r 2 in As input to generate x The comparison result with 0 and the value that will not be leaked during the calculation process are guaranteed, and [ r out ] b XOR the comparison result to ensure the output v b privacy; S23, gradient descent stage, l =1,2,…, L’ , b =0,1,Participants P b calculate ; S3. Finally, P 0The neural network weight parameters[ W ]0=([ W (1) ]0,[ W (2) ]0,…,[ W (L’) ]0)Send to P 1. Set the neural network weight parameters [ W ]1=([ W (1) ]1,[ W (2) ]1,…,[ W (L’) ]1)Send to P 1, then P 0 and P 1 Each is calculated locally W =[ W ]0+[ W ]1. Reconstruct the neural network weight parameters W .
2. The privacy-preserving neural network training method based on function secret sharing according to claim 1, characterized in that: For Gen n Comp (1 λ , r 1 in , r 2 in , r out )The specific execution steps are as follows: G1, order r =(2 n -( r 1 in - r 2 in )),and α (n-1) = r [0,n-1) ,in r [0,n-1) Representation value r 0th to n -2 digits to form the value of the string; G2, calculate ( k 02 (n-1) , k 12 (n-1) )←Gen n-1 DDCF (1 λ , α (n-1) , β 0, β 1), where β 0=1⊕ r [n-1] , β 1= r [n-1] , r [n-1] Representation value r No. n -1 bit, which is the most significant bit; Gen n-1 DDCF Used to obtain x and α (n-1) The result of the comparison is x > α (n-1) ,but( k 02 (n-1) , k 12 (n-1) ) will lead to the result value β 0, otherwise it leads to the result value β 1; G3, Random Selection r 01 , r 11 ∈{0,1}, which satisfies r 01 ⊕ r 11 = r out ; G4, remember k b1 = k b2 (n-1) | r b1 ; G5, output ( k 01 , k 11 ); For Eval n Comp ( b,k b1 ,x + r 1 in ,r 2 in ),in b Indicates the index of the participant. k b1 Indicates Gen n Comp (1 λ , r 1 in , r 2 in , r out ), x Represents the input value. The specific execution steps are as follows: E1, P b Analysis k b1 = k b2 (n-1) | r b1 , and order z = x + r 1 in -r 2 in , x + r 1 in ,r 2 in Represents the input value; E2, P b calculate v b2 ←Eval n-1 DDCF ( b , k b2 (n-1) , z (n-1) ),in z (n-1) = z n-1 - z [0,n-1) -1, Eval n-1 DDCF Indicates that based on input b , k b2 (n-1) , z (n-1) To generate x and α (n-1) The results of the comparison; E3, P b make v b1 =( b • z [n-1] )⊕( v b2 )⊕ r b1 ; E4. Output v b1 .
3. The privacy-preserving neural network training method based on function secret sharing according to claim 2 is characterized in that: For Gen n DDCF (1 λ , α , β 0, β 1), superscript DDCF represents the dual distributed comparison function, subscript n Indicates the input length, 1 λ is a safety parameter, α , β 0, β 1 indicates input. The specific execution steps are as follows: G21, Calculation ( k 03 (n) , k 13 (n) )←Gen n < (1 λ , α , β 0⊕ β 1); among which Gen n < Used to obtain x and α The result of the comparison is x > α ,but( k 03 (n) , k 13 (n) ) will lead to the result value β 0⊕ β 1, otherwise the result value is 0; G22, random sampling r 02 , r 12 ∈{0,1}, which satisfies r 02 ⊕ r 12 = β 1; G23, remember k b2 = k b3 (n) | r b2 ; G24, output ( k 02 , k 12 ); For Eval n DDCF ( b , k b2 , x ), b Indicates the index of the participant. k b2 Indicates Gen n DDCF (1 λ , α , β 0, β 1) output value, x Represents the input value. The specific execution steps are as follows: E21, P b Analysis k b2 = k b3 (n) | r b2 ; E22, P b calculate y b (n-1) ←Eval n < ( b , k b3 (n) , x );Eval n < Leveraging the Party Index b and the corresponding function secret shared key k b3 (n) , to calculate x ≤ α (n-1) The results of the comparison; E23, P b make v b2 = y b (n-1) ⊕ r b2 ; E24, Output v b2 .
4. The privacy-preserving neural network training method based on function secret sharing according to claim 3 is characterized in that: For the dual distributed comparison function Gen n < (1 λ , α , β ), where the superscript < represents a less than sign, and the subscript n Indicates the input length, 1 λ is a safety parameter, α , β Represents input, the specific execution steps are as follows: G211, Order α 1|…| α n yes α The bit decomposition of α i’ express α No. i’ Bit, | represents the string concatenation operator; G212, Random Sampling s 0 (0) ∈{0,1} λ and s 1 (0) ∈{0,1} λ , v 0 (0) =0, v 1 (0) =0 and t 0 (0) =0, t 1 (0) =1; G213, yes i’ =1 to n , repeatedly perform the following steps: G2131, Calculation s 0 L | v 0 L | t 0 L | s 0 R | v 0 R | t 0 R = G ( s 0 (i’-1) )and s 1 L | v 1 L | t 1 L | s 1 R | v 1 R | t 1 R = G ( s 1 (i’-1) ),in G It is a random number generator. The L in the superscript indicates the left direction, and the R indicates the right direction. G2132, if α i’ =0, let Keep = L , Lose = R ; Otherwise, Keep = R , Lose = L ; G2133, Calculation s CW ← s 0 Lose ⊕ s 1 Lose , v CW L ← v 0 L ⊕ v 1 L ⊕( α i’ , β ); G2134, Calculation t CW L ← t 0 L ⊕ t 1 L ⊕ α i’ ⊕1, t CW R ← t 0 R ⊕ t 1 R ⊕ α i’ ; G2135, Calculation CW (i’) = s CW | v CW L | t CW L | t CW R ; G2136, Calculation s 0 (i’) = s 0 Keep ⊕ t 0 (i’-1) • s CW , s 1 (i’) = s 1 Keep ⊕ t 1 (i’-1) • s CW ; G2137, Calculation t 0 (i’) = t 0 Keep ⊕ t 0 (i’-1) • t CW Keep , t 1 (i’) = t 1 Keep ⊕ t 1 (i’-1) • t CW Keep ; G214, Order k b3 = s b3 (0) | CW (1) | … | CW (n) ; G215, Return ( k 03 , k 13 ).
5. The privacy-preserving neural network training method based on function secret sharing according to claim 4 is characterized in that: For the distributed comparison function Eval n < ( b,k b3 ,x ), b Indicates the index of the participant. k b3 Indicates Gen n < (1 λ , α , β ), x Represents the input value. The specific execution steps are as follows: E221, Analysis k b3 = s b3 (0) | CW (1) | … | CW (n) ,make x = x 1|…| x n , v =0 and t (0) = b ; E222, Yes i’ =1 to n , repeatedly perform the following steps: E2221, analysis CW (i’) = s CW | v CW L | t CW L | t CW R ; E2222, Calculation ; E2223, Calculation ; E2224, will τ (i’) Recorded as s L | v L | t L | s R | v R | t R form; E2225, if x i’ =0, let y b (n-1) ← v ⊕ v L , s (i’) ← s L , t (i’) ← t L Otherwise, s (i’) ← s R , t (i’) ← t R ; E223, Output y b (n-1) .
6. The privacy-preserving neural network training method based on function secret sharing according to claim 1, characterized in that: The secure ReLU subprotocol is specifically: According to ReLU( x )= x •DReLU( x ),when P 0 and P 1 respectively obtain DReLU( x )'s Boolean secret share v 0 and v 1. Using ReLU( x )=BitXA(DReLU( x ), x ) can be used to calculate ReLU( x ), where BitXA(•) is a secure bit-integer multiplication protocol.
7. A privacy-preserving neural network training device based on function secret sharing, characterized in that: The training device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the privacy-preserving neural network training method based on function secret sharing as described in any one of claims 1 to 5.
