Transformer model secret reasoning method and system based on secret sharing
By constructing secure Softmax, LayerNorm, and GELU functions, the nonlinear layer computation of the Transformer model is optimized, solving the problems of high computational cost, low accuracy, and privacy leakage in existing Transformer models. This achieves efficient data privacy protection and accurate inference operations, and supports GPU computing and switching of cryptographic primitives.
Patent Information
- Application Number
- CN202410221322.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-28
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-02-28
AI Technical Summary
Existing Transformer models suffer from high computational and communication costs, low computational accuracy, privacy leaks, and inability to support GPU computing and cryptographic primitive switching when processing user data, making them difficult to meet the needs of practical applications.
We employ a secret-sharing-based approach to construct secure Softmax, secure LayerNorm, and secure GELU functions. By utilizing secure exponential, secure division, and improved secure inverse square root functions, combined with piecewise polynomials and secret-sharing techniques, we optimize the nonlinear layer computation of the Transformer model.
While protecting data privacy, it improves the computational efficiency and accuracy of Transformer inference operations, reduces communication consumption, supports GPU computing, and can switch between various cryptographic primitives to meet practical application needs.
Smart Images

Figure CN118174856B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of data security technology, and specifically relates to a Transformer model secret-state reasoning method and system based on secret sharing. Background Art
[0002] Transformer is a revolutionary neural network architecture designed for processing serialized data. Its biggest feature is that it is completely based on the attention mechanism, without the need for traditional recursive or convolutional structures, which enables Transformer to process all elements in the sequence in parallel and efficiently capture long-distance dependencies between elements. This architecture performs well in natural language processing tasks, and its influence has expanded to other fields such as computer vision and audio processing. In today's real-world applications, Transformer models are widely used to process large amounts of user data for prediction and generation of related data. However, user data may contain sensitive information such as medical information, trajectory information, and family information. As users' demand for privacy protection increases, it is urgent to design a secure Transformer model inference method that can efficiently perform operations such as prediction and generation of related data while protecting data privacy.
[0003] Homomorphic encryption and obfuscation circuits provide a feasible way to solve the above problems. For example, the paper "Pang Q, Zhu J, H, et al. BOLT: Privacy-Preserving, Accurate and Efficient Inference for Transformers [J]. Cryptology ePrint Archive, 2023.》 and the second paper "Hao M, Li H, Chen H, et al. Iron: Private inference on transformers [J]. Advances in Neural Information Processing Systems, 2022, 35: 15718-15731." The framework proposed can realize secure matrix multiplication, secure GELU function, secure Softmax function, secure LayerNorm function and other operations based on homomorphic encryption and various operations of obfuscated circuits.
[0004] However, the above methods have problems such as high computation and communication costs, low computational accuracy, and privacy information leakage. In addition, the existing methods cannot support GPU (Graphics Processing Unit) calculations and cannot switch between multiple cryptographic primitives, making it difficult to meet practical application needs. Summary of the invention
[0005] In order to solve the above problems existing in the prior art, the present invention proposes a Transformer model secret reasoning method and system based on secret sharing, which is optimized for nonlinear function operations and provides higher computing efficiency while protecting user data privacy.
[0006] The technical problem to be solved by the present invention is achieved through the following technical solutions:
[0007] In the first aspect, the present invention proposes a Transformer model secret state reasoning method based on secret sharing, which is applied to a dual-cloud outsourcing server, including:
[0008] Construct a secure Softmax function based on the secure exponential function and the secure division function;
[0009] Construct a secure LayerNorm function based on the improved secure inverse square root function;
[0010] Construct secure GELU function based on piecewise polynomial and secret sharing technology;
[0011] Based on the Transformer model, the secure Softmax function, secure LayerNorm function and secure GELU function are used to perform secure protocol calculations of nonlinear functions on the encrypted shared data to obtain the shared results.
[0012] In a second aspect, the present invention proposes a Transformer model secret state reasoning system based on secret sharing, which is configured on a dual-cloud outsourcing server side and is used to implement the method proposed in the first aspect of the present invention, including:
[0013] A data acquisition module, used to obtain encrypted shared data from the data owner;
[0014] Softmax function building module, used to build a safe Softmax function based on the safe exponential function and the safe division function;
[0015] LayerNorm function building module, used to build a secure LayerNorm function based on the improved secure inverse square root function;
[0016] GELU function building module, used to build secure GELU functions based on piecewise polynomials and secret sharing techniques;
[0017] The data processing module is configured with a Transformer model, which includes multiple nonlinear layers. The data processing module is used to perform security protocol calculations of nonlinear functions on the acquired encrypted shared data based on the Transformer model using a secure Softmax function, a secure LayerNorm function, and a secure GELU function to obtain a shared result.
[0018] Beneficial effects of the present invention:
[0019] 1. The Transformer model secret-sharing based reasoning method proposed in the present invention improves the nonlinear layer functions such as the secure GELU function, secure Softmax function, and secure LayerNorm function in the Transformer model secret-state reasoning process, thereby realizing efficient Transformer reasoning operations while protecting data privacy. Compared with the prior art, this method can realize the prediction and generation of relevant data functions with privacy protection on an outsourced cloud server, while protecting data privacy and model security, reducing communication consumption and improving the outsourced computing efficiency and accuracy of the reasoning process;
[0020] 2. The Transformer model secret reasoning method based on secret sharing proposed in the present invention supports GPU computing, can switch between multiple cryptographic primitives, and can meet practical application needs.
[0021] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 It is a flowchart of a Transformer model secret state reasoning method based on secret sharing provided by an embodiment of the present invention;
[0023] Figure 2 It is a schematic diagram of the process of dense state reasoning of the Transformer model provided by an embodiment of the present invention;
[0024] Figure 3 It is a structural block diagram of a Transformer model secret-state reasoning system based on secret sharing provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0026] Embodiment 1
[0027] See also Figure 1 , Figure 1 The figure is a schematic diagram of a process of a Transformer model secret state reasoning method based on secret sharing provided by an embodiment of the present invention. The Transformer model secret state reasoning method based on secret sharing provided by this embodiment specifically includes the following contents:
[0028] Construct a secure Softmax function based on the secure exponential function and the secure division function;
[0029] Construct a secure LayerNorm function based on the improved secure inverse square root function;
[0030] Construct secure GELU function based on piecewise polynomial and secret sharing technology;
[0031] Based on the Transformer model, the secure Softmax function, secure LayerNorm function and secure GELU function are used to perform secure protocol calculations of nonlinear functions on the encrypted shared data to obtain the shared results.
[0032] The present invention mainly improves the outsourced computing efficiency and computing accuracy of the reasoning process by performing key calculations on nonlinear layers such as the secure GELU function, secure Softmax function and secure LayerNorm function in the dense state reasoning process of the Transformer model.
[0033] Specifically, the Transformer reasoning method designed in the present invention mainly includes the following seven components: generating auxiliary triples, secure basic operations, secure multiplication protocol, secure comparison protocol, secure GELU function, secure Softmax function and secure LayerNorm function.
[0034] The following is a detailed introduction to the seven components included in the Transformer reasoning method designed by the present invention.
[0035] It should be noted that, in the present invention, unless otherwise specified, the present invention and the protocol are all run on a dual cloud outsourcing computing architecture, with servers P0 and P1. All data operations are matrix operations, and the individual data in the matrix are in the loop. In , l represents the number of digits. If not otherwise specified, l=32 in the present invention. In the present invention, the size of the matrix X is N×M. In order to more clearly explain the algorithm proposed in the present invention, Table 1 provides a detailed list of key symbols and definitions used in the present invention.
[0036] Table 1 Key symbols and definitions
[0037]
[0038] in, <x>The symbol represents the additive secret sharing form of X, which is <x> 0, <x>1. When server P0 uses <x>When calculating, the data actually operated by server P0 is <x>0; Similarly, when server P1 uses <x>When performing calculations, the data actually operated by server P1 is <x>1.
[0039] Dual-cloud outsourced computing architecture: The data owner encrypts the data locally and then uploads it to two independent cloud servers. The cloud servers execute a series of secure computing protocols based on the specific tasks and finally send the encrypted results to the designated users.
[0040] Semi-honest threat model: the server will faithfully perform various operations in accordance with the protocol, but will attempt to spy on or infer private information related to the original data.
[0041] 1. Generate auxiliary triples
[0042] Some random multiplication triplets are pre-generated before the protocol starts: , , <c>}, also called auxiliary triples, where< / c> and Keep all servers confidential. <c>satisfy <c> =< / c> < / c> The auxiliary triplets can be generated in an offline phase by homomorphic encryption or oblivious transfer. Therefore, the present invention assumes that these triplets are pre-computed and can be used.
[0043] Specifically, the main steps of using homomorphic encryption to generate multiplication triples in this embodiment can be as follows:
[0044] Servers Pii (i∈0, 1) each generate a random matrix i , i , P0 will 0, 0 is homomorphically encrypted and sent to P1. P1 generates a random matrix R and calculates C1= 1 1-R, P1 calculation data V = 0 1+ 1 0+R, encrypt V homomorphically and send it to P0, P0 decrypts (Decode) V and calculates <c> 0=< / c> 0· 0+V, finally, P0 gets 0, 0, <c> 0, P1 gets< / c> 1, 1, <c>1. Algorithm 1 in Table 2 below shows this in more detail:
[0045] Table 2 Generate multiplication triples
[0046]
[0047] 2. Basic safety operations
[0048] The present invention realizes security basic operation based on arithmetic secret sharing. Let the secret sharing data be X, satisfying <x> 0+ <x>1=X.
[0049] Share(X): P i Select a random number R and calculate <x> i =XR, and send it to P 1-i ,Right now <x> 1-i =R.
[0050] ReconstructionReconst <x>): P 1-i Will <x> 1-i Send to P i , by P i Calculate X = <x> 0+ <x>1.
[0051] Specifically, the secure basic operations include a secure matrix addition algorithm and a secure matrix multiplication algorithm.
[0052] The present invention implements a secure matrix addition algorithm (SecMatAdd) based on arithmetic secret sharing. Suppose the secret shared data is X and Y, and the input of this algorithm is the shared value of X and Y. <x>and <y>.P i (i∈0,1) is calculated locally <z> i = <x> i + <y> i ,Z= <z> 0+ <z>1.
[0053] Secure matrix multiplication (SecMatMul) assumes two matrices of additive secret sharing form <x>and <y>Stored on two servers P0 and P1. In order to calculate the product of two matrices <xy> , a secret sharing multiplication triplet needs to be calculated in advance< / xy> < / y> < / x> < / z> < / z> < / y> < / x> < / z> < / y> < / x> < / x> < / x> < / x> < / x> < / x> < / x> < / x> < / x> < / c> , , <c>. Among them, the shapes of matrix A and matrix X are the same, the shapes of matrix B and matrix Y are the same, and matrix C = A·B. The function of Algorithm 2 is that P0 and P1 jointly calculate the product of two matrices <xy>, input is the shared value of X and Y <x>and <y>, the output is the shared value of XY <z> i , and Z=XY= <z> 0+ <z>1. (If there is only one input in the diagram, the other input is the model parameter stored locally on the server) The specific calculation steps are:
[0054] 1. P0 and P1 use shared values <x> and< / x> < / z> < / z> < / z> < / y> < / x> < / xy> < / c> calculate <e> = <x> -< / x> < / e> , using the shared value <y>and calculate <f> = <y> - ;
[0055] 2.P0 and P1 will <e>and <f>Reconstruct E = Reconst( <e>) and F = Reconst( <f>);
[0056] 3. P0 is calculated locally <z> 0=< / z> < / f> < / e> < / f> < / e> < / y> < / f> < / y> 0·F+E· 0+ <c>0, P1 is calculated locally <z> 1=E·F+< / z> < / c> 1·F+E· 1+ <c>1.
[0057] Algorithm 2 in Table 3 below shows the secure matrix multiplication in more detail:
[0058] Table 3 Security matrix multiplication
[0059]
[0060] 3. Safe Ride-Through Agreement
[0061] Since the processing of nonlinear functions often involves continuous multiplication of multiple shared values, and in some cases, the approximate implementation of nonlinear functions may need to be fitted through piecewise polynomial functions, this embodiment proposes a key component: a secure and efficient continuous multiplication protocol (SecCMul).
[0062] The secure multiplication protocol aims to calculate <z> ←f( <x1> , <x2>, …, <X n >), where Z = X1·X2·…·X n , including offline stage and online stage, among which,
[0063] SecCMul off ) is used to generate auxiliary parameters AP0 and AP1 based on the input pseudo-random keys k0 and k1.
[0064] Specifically, the offline calculation process can be expressed by Algorithm 3, and the specific steps are as follows:
[0065] 1.P0 generates a random matrix using key k0 <T j >0, j >0, P1 generates a random matrix using key k1 <T j >1, j > 1, where j∈[0,n];
[0066] 2. P0 and P1 are calculated locally <V j >= <T j > j >, then reconstruct V j , V j =Reconst( <V j >)=T j U j , where j∈[0,n];
[0067] 3. P0 and P1 calculation <T j >The inverse of where j∈[0,n];
[0068] 4.P0 generates a random matrix using key k0 j >0, P1 generates a random matrix using key k1 j > 1, where j∈[0,n];
[0069] 5. P0 and P1 calculation and j P j >= j > <P j >, where j∈[0,n];
[0070] 6. P0 and P1 calculation and
[0071] 7.P0 sets AP0 = {( j >0, <P j >0, j P j >0) j∈[1,n] , <q>0}, P1 sets AP1 = {( j >1, <P j >1, j P>1) j∈[1,n] , <q>1}.
[0072] Table 4 below provides a more detailed description of the offline phase of the secure multiplication protocol described in Algorithm 3:
[0073] Table 4 Offline phase of the secure multi-ride protocol
[0074]
[0075]
[0076] SecCMul on )The main function is to input <x1> , <x2>, …, <X n >, AP0 and AP1, get X1 to X n The specific steps are:
[0077] 1. P0 and P1 calculation <X j >- j >, then refactor (X j -A j )←Reconst( <X j >- j >), where j∈[0,n];
[0078] 2. P0 and P1 calculation <D j >= <X j -A j > <P j >+ j P j >, then reconstruct D j =Reconst( <D j >);
[0079] 3. P0 calculation <z>0=D1·D2…D n · <q>0, P1 calculation <z>1=D1·D2…D n · <q>1.
[0080] Table 5 below provides a more detailed description of the online phase of the secure multiplication protocol described in Algorithm 4:
[0081] Table 5 Online phase of the secure multi-ride protocol
[0082]
[0083] 4. Security Comparison Protocol
[0084] Since the nonlinear algorithm in Transformer involves a large number of comparison operations, the present invention needs to implement a safe and effective comparison protocol. In order to implement this protocol, the present invention converts it into comparing the most significant bit (MSB) of the data on the ring. More specifically, negative numbers and non-negative numbers are separated in the upper half of the ring [2 l-1 , 2 l -1] and the lower half of the ring [0, 2 l-1 -1]. Therefore, the MSB of the value on the ring corresponds to 0 for non-negative values and 1 for negative values. Based on the MSB, it is easy to compare the size of secret shared data. The specific secure MSB algorithm (SecMSB) can be expressed as:
[0085]
[0086] For the detailed implementation process, please refer to the existing related technical implementation.
[0087] Specifically, the secure comparison protocol includes: a secure greater than or equal to algorithm (SecGE) and a secure less than algorithm (SecLT).
[0088] In order to determine whether x is greater than or equal to y, the secure greater than or equal to algorithm (SecGE) Pii (i∈0, 1) is calculated separately. <x-y> i Calculate SecMSB together <x-y>) to get the return value <z>, if x ≥ y, then z = 0; if x < y, then z = 1.
[0089] The Secure Less Than algorithm (SecLT) is used to determine whether x is less than y. The present invention calls 1 - SecGE( <x> , <y>) to get the result. And this protocol can be extended to matrix data. In order to determine whether the corresponding position element in the X matrix is less than the corresponding position element in the Y matrix, the SecMatLT algorithm can be called to obtain the return value. , assuming that the matrix X is an N×M matrix, that is, if X[i][j]<Y[i][j] then B[i][j]=1, if X[i][j]≥Y[i][j] then B[i][j]=0, where i∈[0,N], j∈[0,M].
[0090] 5. Secure GELU Function
[0091] As an excellent activation function, the GELU function is not only widely used in various deep learning algorithms, but also the default activation function of the Transformer model. However, the nonlinearity of the GELU function leads to very low computational efficiency and is one of the main bottlenecks of privacy reasoning in the Transformer model. In order to solve such problems, the present invention designs an efficient privacy-preserving GELU algorithm based on piecewise polynomials and secret sharing technology.
[0092] The secure GELU function designed by the present invention is implemented based on the above secure multiplication protocol and secure comparison protocol. Specifically, the definition of the GELU function is in,
[0093]
[0094] f(x)=-0.003212x 7 +0.04972x 5 -0.2984x 3 +1.1x;
[0095] In the formula, x represents a numerical value;
[0096] The specific calculation process is shown in Algorithm 5.
[0097] The function of Algorithm 5 is to share the value of input x <x>0 and <x>1. Get the shared value of Z = GELU (X) <z>0 and <z>1. The specific steps are:
[0098] 1. P0 and P1 calculate the value of the part of X that is less than -2.5, that is <c0>=SecMatLT( <x>, -2.5);
[0099] 2. P0 and P1 calculate the value of the part of X that is greater than 2.5, that is <c1>=SecMatLT(2.5, <x>);
[0100] 3. P0 and P1 calculate the value of the part of X that is greater than or equal to -2.5 and less than or equal to 2.5, that is, <c2> =1- <c0> - <c1>;
[0101] 4. P0 and P1 calculation
[0102] 5. P0 and P1 calculate the shared value The seventh power Fifth Power and cubic
[0103] 6. P0 and P1 calculation Function, that is
[0104] 7. P0 and P1 calculate erf(X> function, that is
[0105] 8. P0 and P1 are calculated <z>=SecMatMul( <x 2>,(1+erf <x>)), where Z = GELU(X).
[0106] Table 6 below shows the secure GELU function described in Algorithm 5 in more detail:
[0107] Table 6. Secure GELU functions
[0108]
[0109]
[0110] 6. Secure Softmax Function
[0111] Transformer uses the Softmax function to calculate the attention score. X is an N×M matrix. For each row X[i] in X, the output of the Softmax function is a vector Z[i] of the same length, where each element Z[i][j] is calculated as follows
[0112]
[0113] In order to effectively implement a secure Softmax function, the present invention designs two key protocols: a secure exponential function protocol and a secure division protocol. These two protocols play a core role in ensuring the privacy and security of the computation process.
[0114] First, the safety index function Z = e is calculated using the limit approximation method. X .
[0115] The present invention utilizes the limit approximation to calculate the exponential function, which can be specifically expressed as e X =lim n→∞ (1+X / n) n In the present invention, unless otherwise specified, n in the safety index function is 8. The specific calculation process of the safety index function is shown in Algorithm 6.
[0116] The function of Algorithm 6 is to input the shared value of X <x>0 and <x>1, we get Z = e X Share value <z>0 and <z>1. The specific steps are:
[0117] 1. P0 and P1 calculation <e X >=1+ <x> / 8;
[0118] 2. P0 and P1 calculate the shared value <e X > to the eighth power, we get <z>=<(e X ) 8 >=SecCMul( <e X >, <e X >, …, <e X >).
[0119] Table 7 below shows the security index function described in Algorithm 6 in more detail:
[0120] Table 6 Safety index function
[0121]
[0122]
[0123] Then, the safe division function Z=U / X is calculated using Newton's iteration method.
[0124] In order to implement a secure division function, the present invention adopts Newton's iteration method to design a secure division function. The specific implementation steps follow the description of Algorithm 7, as shown below:
[0125] The function of Algorithm 7 is to input the shared value of X <x>0 and <x>1. U’s sharing value 0 and 1. Get the sharing value of Z = U / X <y>0 and <y>1. The specific steps are:
[0126] 1. P0 and P1 calculation 2 j - <x>The most significant bit of <D j >←SecMSB(2 j - <x>), where j∈[1,l-1];
[0127] 2. P0 and P1 calculation <D j > and And reconstruct K = Reconst( <k>);
[0128] 3. P0 and P1 calculate Y0=2 locally -K , Y1=2 -K-1 ;
[0129] 4. P0 and P1 calculation< / k> < / x> < / x> < / y> < / y> < / x> < / x> < / z> < / x> < / z> < / z> < / x> < / x> < / x> < / x> < / z> < / c0> < / c2> < / x> < / x> < / z> < / z> < / x> < / x> < / y> < / x> < / z> < / q> < / z> < / q> < / z> < / x1> < / q> < / q> < / x1> < / z> < / c> =<U / 2 K >= ·Y0;
[0130] 5. P0 and P1 calculation =<X / 2 K+1 >= <x>·Y1;
[0131] 6. P0 and P1 calculation <w> =2.9142-2· ;
[0132] 7. P0 and P1 calculation <e0>=1-SecMatMul( , <w> ), <e1> = <e0> · <e0>;
[0133] 8. P0 and P1 calculation <z> = =SecCMul( < / z> < / e0> < / e1> < / w> < / w> < / x> , <w> ,1+ <e0> ,1+ <e1>).
[0134] Table 8 below shows the safe division function described in Algorithm 7 in more detail:
[0135] Table 8 Safe division function
[0136]
[0137]
[0138] Finally, the secure exponential function and the secure division function are combined to construct a secure Softmax function.
[0139] Specifically, based on the secure exponential function and the secure division function, a secure Softmax function can be constructed to operate on the secret shared value. The specific implementation steps are shown in Algorithm 8. X is an N×M matrix, i represents the i-th row in the matrix X, and j represents the i-th column in the matrix X.
[0140] The function of Algorithm 8 is to input the shared value of X <x>0 and <x>1. Get the shared value of Z = Softmax(Z) <z>0 and <z>1. The specific steps are:
[0141] 1. P0 and P1 calculate the shared value <z>Exponential function <e X >=SecExp( <x>);
[0142] 2. P0 and P1 calculation vectors <e X >The sum of each component
[0143] 3. P0 and P1 calculation <e X > and <y>Division <z>=SecDiv( <e X >, <y>).
[0144] Table 9 below shows the secure softmax function described in Algorithm 8 in more detail:
[0145] Table 9 Security softmax function
[0146]
[0147] 7. Secure LayerNorm Function
[0148] For a vector X, the LayerNorm function is expressed as LayerNorm(X)=γ(X[i][j]-μ) / σ+β, where Compared with the MPC-friendly Batch Normalization (BN) in Convolutional Neural Networks, LayerNorm requires an inverse square root operation.
[0149] First, in the present invention, a new secure inverse square root function is designed, and the specific steps are shown in Algorithm 9.
[0150] The function of Algorithm 9 is to share the value of input x <x>0 and <x>1. Get Share value <z>0 and <z>1. The specific steps are:
[0151] 1. P0 and P1 calculation< / z> < / z> < / x> < / x> < / y> < / z> < / y> < / x> < / z> < / z> < / z> < / x> < / x> < / e0> < / w> =0.2- <x> / 2;
[0152] 2. P0 and P1 calculation =2.2·SecExp( < / x> )+0.2;
[0153] 3. P0 and P1 calculation <c> = - <x> / 1024; and <D0= <c>>
[0154] 4. Calculation <y>=3.0-SecCMul( <x> , <c> , <c>);
[0155] 5. Loop three times, P0 and P1 calculation <D j >=SecMatMul( <D j -1>, <y>)and <f>=SecMatMul( <c>, <D j >) / 2;
[0156] 6.P0 and P1 get <z> = <f>.
[0157] Table 10 below shows the inverse square root function described in Algorithm 9 in more detail:
[0158] Table 10 Inverse square root function
[0159]
[0160] Then, the secure LayerNorm function is implemented through the secure inverse square root function. The specific steps are shown in Algorithm 10.
[0161] The function of Algorithm 10 is to input the shared value of X <x>0 and <x>1, the shared values of γ <γ>0 and <γ>1 and the shared values of β <β>0 and <β>1 are the parameters of the model, which have been saved on the server in the offline stage, and the shared value of Z = LayerNorm(X) is obtained <y>0 and <y>1. The specific steps are:
[0162] 1. P0 and P1 calculation <z>The sum of each component
[0163] 2. P0 and P1 calculation <y> = <x>-<μ> and <y>square <Y 2 >=SecMatMul( <y> , <y>);
[0164] 3. P0 and P1 calculation The inverse square root of
[0165] 4. P0 and P1 calculation <z>=SeccMul(<γ>, <y>, <σ>)+<β>.
[0166] Table 11 below shows the secure LayerNorm function described in Algorithm 10 in more detail:
[0167] Table 11 Security LayerNorm Function
[0168]
[0169] The Transformer model secret-sharing based reasoning method proposed in the present invention improves the nonlinear layer functions such as the secure GELU function, secure Softmax function, and secure LayerNorm function in the Transformer model secret-state reasoning process, thereby realizing efficient Transformer reasoning operations while protecting data privacy. Compared with the prior art, this method can realize the prediction and generation of relevant data functions with privacy protection on an outsourced cloud server, while protecting data privacy and model security, reducing communication consumption and improving the outsourced computing efficiency and computing accuracy of the reasoning process.
[0170] The following is combined with< / y> < / z> < / y> < / y> < / y> < / x> < / y> < / z> < / y> < / y> < / x> < / x> < / f> < / z> < / c> < / f> < / y> < / c> < / c> < / x> < / y> < / c> < / x> < / c> Figure 2 The dense-state reasoning process of the base Transformer model proposed in the present invention is introduced in detail. Figure 2 It is a schematic diagram of the process of dense state reasoning of the Transformer model provided by an embodiment of the present invention.
[0171] The present invention is based on the Transformer model, and uses a secure Softmax function, a secure LayerNorm function, and a secure GELU function to perform a secure protocol calculation of a nonlinear function on secret sharing data to obtain an encryption result, which specifically includes the following steps:
[0172] Step 1: Perform a security basic operation on the input data based on arithmetic secret sharing to obtain a first shared value.
[0173] First, the encrypted shared data is used as the input data of the Transformer model to perform three secure matrix multiplication operations to obtain three sets of sharing results.
[0174] Then, the first two groups of sharing results are subjected to a security matrix addition operation, and the added sharing results are subjected to a security matrix multiplication operation again, and the obtained sharing result is recorded as a first sharing value.
[0175] Step 2: Use the secure Softmax function to process the first shared value to obtain the second shared value.
[0176] The first shared value is used as input data of the secure Softmax function, the secure Softmax function is calculated, and the obtained shared result is recorded as the second shared value.
[0177] Step 3: Use the secure LayerNorm function to process the second shared value to obtain a third shared value.
[0178] It should be noted that after obtaining the second shared value, the following operations are required:
[0179] Perform a security matrix addition operation on the second shared value and the third group of shared results obtained in step 1, and perform a security matrix multiplication operation on the added shared results again; input the obtained result into the Concat component for calculation, perform a security matrix multiplication operation on the output result of the Concat component again, and then perform a security matrix addition operation on the input data of the Transformer model as input to the secure LayerNorm function.
[0180] The third shared value is obtained through processing by the secure LayerNorm function.
[0181] Step 4: Use the secure GELU function to process the third shared value to obtain the fourth shared value.
[0182] The third shared value is subjected to a secure matrix multiplication operation, and the obtained result is used as input data of the secure GELU function. The fourth shared value is obtained through processing by the secure GELU function.
[0183] Step 5: Use the secure LayerNorm function to process the fourth shared value to obtain the fifth shared value.
[0184] The fourth shared value is subjected to a security matrix multiplication operation, and the obtained result is subjected to a security matrix addition operation with the third shared value. The added result is used as input data of a security LayerNorm function, and the fifth shared value is obtained through processing by the security LayerNorm function.
[0185] It should be noted that after obtaining the fifth share value, it also includes:
[0186] Step 6: Use the fifth shared value as input data of the Transformer model, use the Transformer model to perform repeated calculations multiple times, and output the fifth shared value obtained at the last time as the final shared result.
[0187] Optionally, as an implementation method, this embodiment performs 12 cycles of calculation. Figure 2 As shown in the figure, the solid line represents the data flow process, and the dotted line represents that the output of the last SecLayerNorm calculation process in the first 11 cycles will be used as the input of the next cycle, and in the last cycle, the SecLayerNorm calculation result will be directly output as the output of the entire reasoning process.
[0188] The present invention realizes efficient Transformer reasoning operation under the premise of protecting data privacy. Compared with the existing work, it realizes the protection of data in the calculation process, and further improves the calculation efficiency of the protocol and reduces the communication consumption. In addition, the method supports GPU computing and can switch between multiple cryptographic primitives, which can meet the needs of practical applications.
[0189] In order to verify the effectiveness of the method proposed in the present invention, a comparative explanation is given below through simulation experiments.
[0190] 1. Experimental environment
[0191] The performance evaluation of the secure computing protocol of the present invention is performed on two servers (P0 and P1) using Ubuntu 20.04 and equipped with a quad-core 2.5GHz Intel Xeon processor, 128GB memory and Nvidia 2080ti GPU, where the communication protocol is the TCP protocol in the local area network (LAN) between the two servers. The average network delay is 0.8ms and the bandwidth is 3GB / s.
[0192] 2. Experimental content and results analysis
[0193] Experiment 1:
[0194] In this experiment, the method of the present invention is compared with each nonlinear layer protocol in Iron based on the calculation time and communication volume. The unit of calculation time is second and the unit of communication volume is GB. The experimental results are shown in Table 12.
[0195] Table 12 Comparison results of nonlinear layer protocols of different methods
[0196]
[0197]
[0198] It can be seen from Table 2 that the running time of all the basic operation protocols of the present invention is better than that of the protocols in Iron. However, the division function in the SecSoftmax function of the present invention uses the Newton iteration method, resulting in a much larger communication volume than the protocols in Iron.
[0199] Experiment 2:
[0200] This experiment performed performance evaluation on three BERT models based on the Transformer model. The specific results are shown in Table 13 below. The unit of calculation time is seconds and the unit of communication volume is GB.
[0201] Table 13 Performance evaluation results of different models
[0202]
[0203] As can be seen from Table 13, as the model size increases, the time and communication overhead also increase accordingly. In terms of secure computing protocols, SecGELU has relatively low time overhead in each model, while SecSoftmax performs better in terms of communication overhead. The BERT-TINY model can run and complete the inference in only 11.14 seconds to get the final result, and even the large model BERT-BASE can complete the inference in 67 seconds.
[0204] Therefore, the effectiveness of the present invention is verified.
[0205] Embodiment 2
[0206] Based on the above embodiment 1 and the same inventive concept, this embodiment provides a Transformer model secret reasoning system based on secret sharing. Figure 3 , Figure 3 : is a structural block diagram of a Transformer model secret state reasoning system based on secret sharing provided by an embodiment of the present invention. The system is configured on a dual-cloud outsourcing server side and specifically includes:
[0207] A data acquisition module, used to obtain encrypted shared data from the data owner;
[0208] Softmax function building module, used to build a safe Softmax function based on the safe exponential function and the safe division function;
[0209] LayerNorm function building module, used to build a secure LayerNorm function based on the improved secure inverse square root function;
[0210] GELU function building module, used to build secure GELU functions based on piecewise polynomials and secret sharing techniques;
[0211] The data processing module is configured with a Transformer model, which includes multiple nonlinear layers. The data processing module is used to perform security protocol calculations of nonlinear functions on the acquired encrypted shared data based on the Transformer model using a secure Softmax function, a secure LayerNorm function, and a secure GELU function to obtain a shared result.
[0212] The system provided by the present invention can implement the method provided in the above embodiment 1, and the detailed process can refer to the above embodiment 1. Therefore, the system can also realize the prediction and generation of relevant data functions with privacy protection on the outsourced cloud server, while protecting data privacy and model security, reducing communication consumption and improving the outsourced computing efficiency and computing accuracy of the reasoning process.
[0213] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention. < / x> < / x> < / x> < / x> < / x> < / x> < / x>
Claims
1. A Transformer model secret state reasoning method based on secret sharing, applied to a dual-cloud outsourcing server, characterized in that: include: Construct a secure Softmax function based on the secure exponential function and the secure division function; Specifically include: Calculate the safety index function using the limit approximation method; Use Newton's method to calculate safe division functions, including: Calculation 2 j - <x>The most significant bit of <D j >←SecMSB(2 j - <x>), where j∈[1,l-1]; <x> Indicates encrypted shared data. The symbol "<>" indicates the encrypted shared form of the data inside. l indicates the number of digits. MSB indicates the most significant bit. SecMSB indicates the secure MSB algorithm.< / x> < / x> < / x> calculate And reconstruct K = Reconst( <k> ); Reconst() represents the secret sharing reconstruction operation;< / k> Calculate Y0 = 2 -K , Y1=2 -K-1 ; calculate =<U / 2 K >= ·Y0; where U = e X ; calculate =<X / 2 K+1 >= <x> ·Y11;< / x> calculate <w> =2.9142-2· ; < / w> calculate <e0>=1-SecMatMul( , <w> ), <e1> = <e0> · <e0> ; SecMatMul represents secure matrix multiplication; < / e0> < / e1> < / w> calculate <z> = =SecCMul( < / z> , <w> ,1+ <e0> ,1+ <e1> );SecCMul represents the secure multiplication protocol; < / e0> < / w> The secure Softmax function is constructed by combining the secure exponential function and the secure division function, and its expression is: Where Z[i][j] represents the shared value output by the secure Softmax function; X represents an N×M matrix, [i] represents the i-th row in the matrix, and [j] and [k] represent the j-th column or the [k-th] column in the matrix; Construct a secure LayerNorm function based on an improved secure inverse square root function; wherein the improved secure inverse square root function The specific implementation steps are: calculate <A=0.2- <x> / 2;< / x> calculate =2.2·SecExp( )+0.2; SecExp represents the security exponential function; calculate <c> = - <x> / 1024, and make <d0> = <c> ;< / c> < / d0> < / x> < / c> calculate <y>=3.0-SecCMul( <x> , <c> , <c> )< / c> < / c> < / x> < / y> Loop three times and calculate <D j >=SecMatMul( <D j-1 >, <y>)and <f>=SecMatMul( <c> ,<D j >) / 2;< / c> < / f> < / y> make <z> = <f> , and obtain the calculation result of the safe inverse square root function;< / f> < / z> The expression of the secure LayerNorm function is: LayerNorm(X)=γ(X[i][j]-μ) / σ+β; in, X represents an N×M matrix, [i] represents the i-th row in the matrix, [j] represents the j-th column in the matrix, d represents the number of all elements in the matrix X, and γ and β represent the server local model parameters; Construct a secure GELU function based on piecewise polynomials and secret sharing technology, including: Design safe ride-along protocols; Design a secure comparison protocol based on MSB; Based on the secure multiplication protocol and the secure comparison protocol, a piecewise GELU function is designed, and its expression is: in, f(x)=-0.003212x 7 +0.04972x 5 -0.2984x 3 +1.1x; In the formula, x represents a numerical value; Based on the Transformer model, the secure Softmax function, the secure LayerNorm function and the secure GELU function are used to perform secure protocol calculations of nonlinear functions on the encrypted shared data to obtain a shared result.
2. The Transformer model secret state reasoning method based on secret sharing according to claim 1 is characterized in that: The secure multi-ride protocol includes an offline phase and an online phase; wherein, The offline phase is used to generate auxiliary parameters based on the input pseudo-random key; The online stage is used to obtain continuous multiplication results based on the input data and auxiliary parameters.
3. The Transformer model secret state reasoning method based on secret sharing according to claim 1, characterized in that: Based on the Transformer model, the secure Softmax function, the secure LayerNorm function, and the secure GELU function are used to perform a secure protocol calculation of a nonlinear function on the secret sharing data to obtain an encryption result, specifically including: Performing a security basic operation on the input data based on arithmetic secret sharing to obtain a first shared value; Processing the first shared value using the secure Softmax function to obtain a second shared value; Processing the second shared value using the secure LayerNorm function to obtain a third shared value; Processing the third shared value using the secure GELU function to obtain a fourth shared value; The fourth shared value is processed using the secure LayerNorm function to obtain a fifth shared value.
4. The Transformer model secret state reasoning method based on secret sharing according to claim 3 is characterized in that: After obtaining the fifth shared value, the method further includes using the fifth shared value as input data of the Transformer model, performing multiple repeated calculations using the Transformer model, and outputting the fifth shared value obtained at the last time as the final shared result.
5. A Transformer model secret state reasoning system based on secret sharing, configured on a dual cloud outsourcing server, for implementing the method described in any one of claims 1 to 4, characterized in that: include: A data acquisition module, used to obtain encrypted shared data from the data owner; Softmax function building module, used to build a safe Softmax function based on the safe exponential function and the safe division function; LayerNorm function building module, used to build a secure LayerNorm function based on the improved secure inverse square root function; GELU function building module, used to build secure GELU functions based on piecewise polynomials and secret sharing techniques; The data processing module is configured with a Transformer model, wherein the Transformer model includes multiple nonlinear layers. The data processing module is used to perform a security protocol calculation of a nonlinear function on the acquired encrypted shared data based on the Transformer model using the secure Softmax function, the secure LayerNorm function, and the secure GELU function to obtain a shared result.
