Safe multi-party privacy protection reasoning method, device and system for GPT

Abstract

The invention discloses a safe multi-party privacy protection reasoning method, device and system for GPT (General Purpose Transfer), and the method explores a conversion protocol between additive secret sharing and multiplicative secret sharing, and designs a series of sub-protocols for achieving the calculation of a safe multi-party nonlinear activation function through the conversion protocol. In addition, a multiplication protocol for realizing additive secret sharing by utilizing a Beaver triple aiming at security multi-party computing protocols of different layers of the Transform in the GPT model is designed, so that a complete multi-party privacy protection Transform reasoning framework is realized; and based on the protocol, the plurality of servers and the user carry out security multi-party privacy protection calculation by adopting GPT to obtain a GPT reasoning result. On the premise that privacy is met, compared with an existing similar scheme, the method has the advantages that efficiency is improved, communication overhead is reduced, and the method is more suitable for a local area network with low communication delay and adapts to a large-model secure multi-party cloud reasoning application scene.

Description

Technical Field

[0001] The present invention relates to the field of information security technology, and in particular, to a secure multi-party privacy protection inference method, device, and system for GPT. Background Art

[0002] With the continuous improvement of pre-trained generative artificial intelligence models (such as GPT, Bert), more and more companies provide cloud inference services for pre-trained models, such as ChatGPT of OpenAI and Bard of Google. More and more tasks can be completed by these large pre-trained language models, and Transformer, as an important part of the GPT model structure, has also attracted more and more attention from artificial intelligence researchers.

[0003] GPT (Generative Pretrained Transformer) is a pre-trained language model based on the Transformer architecture. It uses a large amount of text data for pre-training to learn the general representation of language, and then can be fine-tuned for specific downstream tasks. One of the characteristics of the GPT series of models is that it uses the Transformer architecture as the basis for generating text, which allows GPT to effectively process various language tasks and generate coherent and relevant text. Transformer is the basic architecture of GPT. GPT learns the deep features of language by pre-training the Transformer model on a large-scale dataset and then uses it for various text generation tasks.

[0004] However, due to the high computing power required to train these large pre-trained language models (such as GPT and Bert) and the complexity of inference calculation, users cannot perform the training and inference of these large language models by themselves. When users use these cloud services to perform cloud inference on a third-party cloud server with their own inputs, the privacy and security of the input and output data are difficult to guarantee.

[0005] Currently, there is more privacy protection for private data in the cloud inference environment. Many efforts have been made in protecting the privacy of language model inference (for example, RNN-based models and Transformer-based models). The common method is to use homomorphic encryption (HE) to encrypt private data and perform arithmetic operations in the encrypted state. Multi-party computation (MPC) is another method to achieve privacy protection for language model inference. By secret sharing, multiple parties can cooperate to achieve large language model inference, but the efficiency of existing multi-party collaborative inference schemes is relatively low. Summary of the Invention

[0006] The objective of the present invention is to propose a secure multi-party privacy-preserving Transformer inference framework for GPT, which protects the sensitive input data of users in the scenario of large model cloud inference and improves the inference efficiency.

[0007] The technical solution adopted by the present invention is as follows:

[0008] In the first aspect, a secure multi-party privacy-preserving inference method for GPT is provided, including:

[0009] Utilize the Beaver triple to implement the multiplication protocol of additive secret sharing, and extend it to the vector inner product protocol and the matrix multiplication protocol;

[0010] Based on the multiplication protocol of additive secret sharing implemented by the Beaver triple and the conversion protocol between additive secret sharing and multiplicative secret sharing, design the secure DReLU protocol, the secure division protocol, the secure Softmax protocol, and the secure GELU protocol;

[0011] Adopt the GPT model to perform secure multi-party privacy-preserving calculations based on the multiplication protocol, the vector inner product protocol, the matrix multiplication protocol, the secure DReLU protocol, the secure division protocol, the secure Softmax protocol, and the secure GELU protocol to obtain the GPT inference result.

[0012] In an implementation manner, the multiplication protocol of additive secret sharing includes:

[0013] In the initialization stage, each participating party prepares a Beaver triple { , , <c>}, where \(c = a\cdot b\), and \(a\), \(b\), and \(c\) are the elements of the triple respectively, < / c> , , <c>Represents the share of the triple;

[0014] In the online phase, each participant locally computes the additive secret sharing of the intermediate value e <e> ＝ <x> -< / x> < / e> < / c> Additive secret sharing with another intermediate value f <f> ＝ <y> - ， <x> 、 <y>For secret sharing;

[0015] Each participant discloses <e>and <f>Obtain the public values e and f;

[0016] Each party calculates the additive secret sharing of the output result <z>= e·f + e· +f· < / z> < / f> < / e> < / y> < / x> < / y> < / f> + <c>。

[0017] In one implementation, the conversion protocol between additive secret sharing and multiplicative secret sharing includes:

[0018] Each participant P i holds its own multiplicative secret sharing distributes the multiplicative secret sharing it holds to other participants P j , j≠i, in the form of additive secret sharing, so as to construct an additive secret sharing vector <x> i = {<x 1 > i , … <x n > i};

[0019] Use the multiplication protocol of additive secret sharing for iterative calculation to obtain the output additive secret sharing <z> i , where the true output value z satisfies

[0020] In one implementation, the user and multiple servers use the Transformer model to perform secure multi-party privacy-preserving calculations based on the multiplication protocol, vector inner product protocol, matrix multiplication protocol, secure DReLU protocol, secure division protocol, secure Softmax protocol, and secure GELU protocol to obtain the GPT inference result, including:

[0021] Use the token embedding layer to map the user's private word embedding matrix and position matrix into vector representations;

[0022] Use the layer normalization layer to standardize the input matrix so that its range is specified in (0, 1) and it follows a normal distribution;

[0023] Use the multi-head attention layer to perform calculations based on the additive secret sharing matrix and pre-trained model parameters to obtain the secret sharing of the output result of the multi-head self-attention layer;

[0024] Use the feed-forward neural network to perform calculations based on the additive secret sharing matrix, the secret sharing of the output result of the multi-head self-attention layer, the weight matrix, and the bias matrix to obtain the secret sharing of the output result of the feed-forward neural network;

[0025] Obtain the final GPT inference result based on the secret sharing of the output result of the multi-head self-attention layer and the secret sharing of the output result of the feed-forward neural network.

[0026] In one implementation, using the layer normalization layer to standardize the input matrix so that its range is specified in (0, 1) and it follows a normal distribution, including:

[0027] Each participant P i Locally calculate the average of each row secret sharing of and for m ∈ [1, d model calculate the secret sharing of the difference between the matrix element and the mean V represents the size of the input vocabulary, and d model represents the size of the one-dimensional feature vector of a word;

[0028] All participants jointly execute the secure vector inner product multiplication protocol to obtain the secret sharing of z squared <z 2 >] where z m is the m-th element of the vector z;

[0029] Each participant locally calculates the noisy variance ∈ represents the error value;

[0030] All participating parties jointly execute the secure multi-party square root reciprocal function calculation protocol to obtain and locally calculate the intermediate variable and

[0031] Each participating party jointly executes the secure matrix multiplication protocol to obtain <Y T > where Y T = Z T ° V and

[0032] For m ∈ [1, d model , each participating party jointly executes the secure multiplication protocol to calculate the output result of layer normalization where Both γ and β are model pre-training parameters, and the subscript m represents the m-th parameter.

[0033] In one implementation, the multi-head attention layer is used to perform calculations based on the additive secret sharing matrix and pre-trained model parameters to obtain the secret sharing of the output result of the multi-head self-attention layer, including:

[0034] All parties jointly execute the secure matrix multiplication protocol to obtain the intermediate variable matrix of GPT model parameters and V represents the size of the input vocabulary, d model represents the size of the one-dimensional feature vector of a word, X represents the input matrix of the multi-head self-attention layer, W K , W Q , W V respectively represent the model pre-training parameters for generating the K, Q, V matrices, and ° represents matrix multiplication;

[0035] Each participating party P i locally performs matrix partitioning:

[0036]

[0037] where M is the number of heads, represents the K matrix of the first attention head and the K matrix of the M-th attention head, represents the Q matrix of the first attention head and the Q matrix of the M-th attention head, represents the Q matrix of the first attention head and the V matrix of the M-th attention head;

[0038] For j ∈ [M], the following steps are calculated in parallel:

[0039] All parties jointly execute the secure matrix multiplication protocol to obtain the secret sharing of the intermediate matrix variable <T j >Among them

[0040] All parties jointly execute the secure Softmax function calculation protocol to obtain the output <S of the Softmax function calculation protocol j >Among them

[0041] All parties jointly execute the secure matrix multiplication protocol to obtain Among them

[0042] Each participant P i Performs matrix combination locally and obtains the secret sharing of the output result of the multi-head self-attention layer Among them

[0043] In one implementation, the feed-forward neural network is used to calculate based on the additive secret sharing matrix, the secret sharing of the output result of the multi-head self-attention layer, the weight matrix, and the bias matrix to obtain the secret sharing of the output result of the feed-forward neural network, including:

[0044] All parties jointly execute the secure matrix multiplication protocol to calculate the secret sharing of the intermediate variable <t1>Among them Among them, X represents the input matrix of the feed-forward neural network FFN, W1 represents the weight matrix of the first linear layer in the FFN, V represents the size of the vocabulary in the input matrix, w represents the dimension of the weight matrix, and ° represents matrix multiplication;

[0045] All parties jointly execute the secure GELU function calculation protocol to obtain the secret sharing of the second intermediate variable <t2>, where where T1 represents the output matrix of the first linear layer in the FFN, and B1 represents the bias matrix of the first linear layer in the FFN,

[0046] All parties jointly execute the secure GELU function calculation protocol to calculate the secret sharing of the third intermediate variable <t3>, where Obtain the secret sharing of the output result of the feedforward neural network W2 T represents the weight matrix of the second linear layer in the FFN, and B2 represents the bias matrix of the second linear layer in the FFN.

[0047] Based on the same inventive concept, the second aspect of the present invention provides a secure multi-party privacy protection inference device for GPT, which is applied to users and multiple servers, including:

[0048] A protocol construction module, which is used to use Beaver triples to implement a multiplication protocol for additive secret sharing and extend it to a vector inner product protocol and a matrix multiplication protocol;

[0049] A secure computing protocol design module, which is used to design a secure DReLU protocol, a secure division protocol, a secure Softmax protocol, and a secure GELU protocol based on the multiplication protocol for additive secret sharing implemented by Beaver triples and the conversion protocol between additive secret sharing and multiplicative secret sharing;

[0050] A secure inference module, which is used to perform secure multi-party privacy protection calculations based on the GPT model using the multiplication protocol, the vector inner product protocol, the matrix multiplication protocol, the secure DReLU protocol, the secure division protocol, the secure Softmax protocol, and the secure GELU protocol to obtain the GPT inference result.

[0051] Based on the same inventive concept, the third aspect of the present invention provides a secure multi-party privacy protection inference system for GPT, including the secure multi-party privacy protection inference device for GPT described in the second aspect and a user terminal, where the user terminal performs token embedding locally and secretly shares private inputs with the server.

[0052] Based on the same inventive concept, the fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the secure multi-party privacy protection inference method for GPT described in the first aspect.

[0053] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0054] Based on the existing multiplicative secret sharing, additive secret sharing, and the conversion methods between the two types of secret sharing, combined with Beaver triples, the present invention realizes a more efficient M2A protocol, and uses this protocol to implement secure computing protocols for non-linear activation functions such as GELU and Softmax used in the GPT model. And based on the above protocol, a secure multi-party privacy-preserving Transformer inference framework (Transformer model) for the GPT model is adopted for inference calculation, and the efficiency is higher than existing similar solutions. Brief Description of the Drawings

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0056] < / z> < / x> < / c> Figure 1 Flow chart of the secure multi - party privacy - preserving inference method for GPT provided by the embodiments of the present invention;

[0057] Figure 2 Schematic diagram of the calculation method of the M2A protocol (conversion protocol between additive secret sharing and multiplicative secret sharing) provided by the embodiments of the present invention;

[0058] Figure 3 Architecture diagram of the secure multi - party privacy - preserving inference system for GPT provided by the embodiments of the present invention;

[0059] Figure 4 Flow chart of the GPT operation provided by the embodiments of the present invention;

[0060] Figure 5 Structure diagram of the secure multi - party privacy - preserving inference device for GPT provided by the embodiments of the present invention. Detailed implementation manners

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of the present invention.

[0062] Embodiment 1

[0063] The present invention discloses a secure multi - party privacy - preserving inference method for GPT, which is applied to a user and multiple servers. Please refer to Figure 1 , including:

[0064] S1: Implement a multiplication protocol for additive secret sharing using Beaver triples, and extend it to a vector inner - product protocol and a matrix multiplication protocol;

[0065] S2: Design a secure DReLU protocol, a secure division protocol, a secure Softmax protocol, and a secure GELU protocol based on the multiplication protocol for additive secret sharing implemented using Beaver triples and the conversion protocol between additive secret sharing and multiplicative secret sharing;

[0066] S3: The user and multiple servers perform secure multi - party privacy - preserving calculations based on the multiplication protocol, vector inner - product protocol, matrix multiplication protocol, secure DReLU protocol, secure division protocol, secure Softmax protocol, and secure GELU protocol using the Transformer model to obtain the GPT inference result.

[0067] For a better understanding of the present invention, the specific symbolic descriptions are as follows:

[0068] [n]: represents the set {1, …, n}.

[0069] [a, b]: For it is represented as the set {a, …, b}.

[0070] x: Vectors are represented by bold lowercase letters, for example

[0071] X: Matrices are represented by bold uppercase letters, for example

[0072] x i : represents the i-th element of the vector x.

[0073] x[i]: represents the i-th bit of x.

[0074] represents the set of participating parties, where it is indicated that the number of participating parties is n.

[0075] P i : represents the i-th participating party.

[0076] represents multiplicative secret sharing.

[0077] <x>: Represents additive secret sharing.

[0078] Represents P i The multiplicative secret sharing held by

[0079] <x> i : Representing P i 's held additive secret sharing.

[0080] The following is the secure multi-party computation protocol proposed in this scheme:

[0081] Π MUL : Secure multiplication protocol.

[0082] Π SMUL : Secure vector inner product multiplication protocol.

[0083] Π MMUL : Secure matrix multiplication protocol.

[0084] Π DReLU : Secure DReLU protocol.

[0085] Π Div : Secure division protocol.

[0086] Π Softmax : Secure Softmax function computation protocol.

[0087] Π GELU : Secure GELU function computation protocol.

[0088] The following are the secure multi-party computation protocols using the existing schemes:

[0089] Π sigm : Secure multi-party sigmoid function computation protocol.

[0090] Π sqrinv : Secure multi-party square root reciprocal function computation protocol.

[0091] °: Matrix multiplication.

[0092] Specifically, this scheme uses two different types of secret sharing, additive secret sharing <x>With multiplicative secret sharing For additive secret sharing, x = <x> 1 + <x> 2 +…+ <x> n mod 2 l ; for multiplicative secret sharing, Construct the operation of a non-linear function by using the conversion between these two different secret sharings. At the same time, use the Beaver triple to implement the multiplication protocol Π of additive secret sharing MUL for calculating <z>where z = x·y, and it can be extended to a vector inner product protocol and a matrix multiplication protocol.

[0093] The multiplication protocol based on Beaver triples can include the following steps. In the initialization phase, each participant prepares a Beaver triple { , , <c>}, where \(c = a\cdot b\). In the online phase, each participant locally computes the additive secret sharing of the intermediate value \(e\). <e> ＝ <x> -< / x> < / e> < / c> Additive secret sharing with another intermediate value f <f> ＝ <y> - 。Afterwards, each participating party makes it public <e>and <f>Obtain the public values e and f, and then each party calculates the additive secret sharing of the output result <z>= e·f + e· +f· < / z> < / f> < / e> < / y> < / f> + <c>。We can easily extend the multiplication protocol to the vector inner product protocol Π SMUL and the matrix multiplication protocol Π MMUL 。

[0094] Meanwhile, this scheme uses the conversion between additive secret sharing and multiplicative secret sharing to construct the operation of non-linear functions. This conversion protocol Π M2A is also implemented based on Beaver triples. The specific execution steps are as follows: Each participant P i holds its own multiplicative secret sharing and distributes its own multiplicative secret sharing share in the form of additive secret sharing to other participants P , j≠i. For each participant P j can also obtain the additive secret sharing shared by other participants, so as to construct an additive secret sharing vector i <x> i = {<x 1 > i , … <x n > i}, using the multiplication protocol Π MUL , continuously iteratively calculate as< / x> ​< / c> Figure 2 as shown, so as to obtain the output additive secret sharing <z> i , where the true value z output satisfies

[0095] This solution is based on the Beaver triple to implement the multiplicative protocol Π for additive secret sharing MUL and the secret sharing conversion protocol Π M2A . On this basis, a series of sub-protocols are designed for the secure multi-party privacy-preserving Transformer inference framework for GPT: 1) The secure DReLU protocol Π DReLU

[0096] Each participant P i holds the secret sharing of the input value , Secret sharing of the calculated output result <z>, whose output value satisfies z = DReLU(u).

[0097] Preprocessing: Each participating party prepares in advance <β> and { <r>, <r[0]>, …, <r[l x -1]>} (where α * and α k is a random number on, β a random bit, and r is a random number on and r[k] is a random l u is the number of bits of the most significant bit of u)

[0098] (a) All parties jointly execute the multiplication protocol Π MUL , and calculate <(2β - 1)·u>.

[0099] (b) Each party P i locally calculates the intermediate variable <u′> = <(2β - 1)· <x> + <r>and disclose it, thereby obtaining u′ = (-1) β+1 ·u + r.

[0100] (c) Each participant P i locally calculates the intermediate variable <m * > = (2<β> - 1), and the intermediate variable <m 0 > = u′ - <r>, for all k ∈ [1, l x , calculate the intermediate variable

[0101] (d) For all k ∈ [1, l x , each participating party P i locally calculates the discriminant variable <v * > = <m * > + 3·<m 0 > - 1, and the discriminant variable for subsequent sign determination of the input value u.

[0102] (e) All participating parties jointly execute the multiplication protocol Π MUL to calculate and publicly disclose the random variable and w * = α * ·v * .

[0103] (f) If there is a 0 among these random variables and w * = α * ·v * , then each participating party P i locally sets the output <z>= -<β>, otherwise output <z>= <β>, where z = DReLU(x).

[0104] 2) Secure division protocol Π Div

[0105] Each participant P i holds the secret sharing of the input value <x>And <y>, Secret sharing of the calculation output result <z> ＝ <x y>, z = x / y.

[0106] (a) Each participating party P i Randomly selects an additive secret sharing <r>= r i 。

[0107] (b) All participating parties jointly execute the multiplication protocol Π MUL , calculate the additive secret sharing <x·r> and <y·r> of the intermediate values, and publicly disclose

[0108] (c) Each participating party P i locally and non-interactively calculates the result of the division <z>= <x·r> / y·r

[0109] 3) Secure Softmax protocol ∏ Softmax

[0110] Each participant P i holds a secret - shared vector of input values i ={<u 1 > i ,…<u k > i}, computing the secret sharing of the output result <z>, whose output value satisfies z = Softmax(u).

[0111] (a) Each party P i randomly selects multiplicative secret sharing and for m ∈ [k], each party locally computes the secret sharing of the intermediate value

[0112] (b) For m ∈ [k], all parties jointly execute Π M2A , to obtain the additive secret sharing <φ m > of the M2A conversion result, where

[0113] (c) Each party P i locally computes the accumulated value <Φ> = ∑ m∈[k] <φ m >, and publicly discloses <Φ> to obtain Φ.

[0114] (d) Each party P i locally computes the output of Softmax <z>= {<z m >} m∈[k] = {<φ m > / Φ} m∈[k]

[0115] 4) Secure GELU Protocol Π GELU

[0116] Each participant P i holds a secret share of the input value , Secret sharing of the calculated result <z>, whose output value satisfies z = GELU(u).

[0117] (a) Each participating party P i Locally computes an approximate estimated secret sharing of the input value <k> ＝1.7017·2 16 · 。

[0118] (b) All participating parties jointly execute Π sigm to obtain Π sigm the secret sharing of the output value <s>, satisfying s = Sigmoid(1.7017·u).

[0119] (c) All parties jointly execute the multiplication protocol Π MUL , calculate that all parties jointly execute the multiplication protocol Π MUL , to obtain the protocol Π GELU The secret sharing of the output result <z>= <u, s>.

[0120] The secure multi-party privacy-preserving Transformer inference framework for GPT proposed by the present invention is as Figure 3 shown, and specifically includes four-layer structures as Figure 4 shown: Token Embedding, LayerNormalization, Multi-Head Self-Attention, and Feed-Forward Network.

[0121] 1) Token Embedding

[0122] The token embedding layer is used to map tokens from the input sequence to vector representations. We only consider the scenario involving single users and multiple servers. The user can perform token embedding locally and secretly share the private input with the server. Therefore, this step can be regarded as a process of data sharing. The user secretly shares its private word embedding matrix and position matrix will <t>and Distributed to n servers. d model is the size of the one-dimensional feature vector corresponding to a word, and V is the size of the vocabulary. Each server can calculate by <e> ＝ <t> +< / t> < / e> Compute the share of the embedding matrix locally.

[0123] 2) Layer Normalization

[0124] In the Transformer model, layer normalization is usually performed after matrix operations. We need to normalize each row vector of the input matrix to have a range of (0, 1) and follow a normal distribution. The main challenge of layer normalization is to calculate the reciprocal of the square root To compute layer normalization under secret sharing, we use the Π protocol to achieve it. It should be noted that we use the Π sqrinv protocol proposed above to replace the comparison protocol in the existing Π DReLU protocol and combine it with the above multiplication protocol Π sqrinv to complete the secure implementation of layer normalization MUL (The following symbols are consistent with those in the formula, and y represents the output).

[0125] (a) Each party P i locally computes the secret share of the mean of each row of the matrix and for m ∈ [1, d computes the secret share of the difference between the matrix element and the mean model

[0126] (b) All parties jointly execute the protocol Π SMUL to obtain the secret share of z squared <z 2 > where z m is the m-th element of the vector z.

[0127] (c) Each party locally computes the noisy variance

[0128] (d) All parties jointly execute the protocol Π sqrinv to obtain and locally compute the intermediate variables

[0129] and (e) All parties jointly execute Π MMUL to obtain <Y T > where

[0130] and

[0130] (f) For m ∈ [1, d model , all parties jointly execute Π MUL The output result of layer normalization in the calculation layer where

[0131] 3) Multi-Head Self-Attention

[0132] Input additive secret sharing matrix and pre-trained model parameters All parties execute the following steps:

[0133] (a) All parties jointly execute Π MMUL , to obtain the intermediate variable matrix of GPT model parameters and

[0134] (b) Each participating party P i Performs matrix partitioning locally

[0135]

[0136] where M is the number of heads.

[0137] (c) For j ∈ [M], calculate the following steps in parallel:

[0138] a) All parties jointly execute Π MMUL , to obtain the secret sharing <T j > of the intermediate matrix variable where

[0139] b) All parties jointly execute Π Softmax , to obtain the output <S Sodtmax > of Π j where

[0140] c) All parties jointly execute Π MMUL to obtain the output of a single attention mechanism where

[0141] (a) Each participating party P i Performs matrix combination locally and obtains the secret sharing of the output result of the multi-head self-attention layer where

[0142] 4) Feed-Forward Network

[0143] Input additive secret sharing matrix Weight matrix and bias matrix Each party performs the following steps:

[0144] (a) Each participating party jointly executes Π MMUL to calculate the secret sharing of intermediate variables <t1>Among them

[0145] (b) Then, all parties jointly execute Π GELU Obtain the secret sharing of the second intermediate variable <t2>, where

[0146] (c) Finally, all parties jointly execute Π MMUL Calculate the secret sharing of the third intermediate variable <t3>, where Obtain the secret sharing of the output result of the feedforward neural network

[0147] The present invention designs a Transformer inference framework for multi-party privacy protection in GPT and designs a series of sub-protocols for this framework. Specifically, we further explore the conversion protocol Π between additive secret sharing and multiplicative secret sharing M2A , using Π M2A Design a series of sub-protocols to implement the secure multi-party calculation of non-linear activation functions, and design the secure multi-party calculation protocol for different layers of Transformer in the GPT model, thus realizing a complete multi-party privacy protection Transformer inference framework. On the premise of satisfying privacy, compared with existing similar solutions, it improves efficiency, reduces communication overhead, is more suitable for local area networks with low communication latency, and adapts to the secure multi-party cloud inference application scenario of large models.

[0148] Embodiment 2

[0149] Based on the same inventive concept, this embodiment discloses a secure multi-party privacy protection inference device for GPT, which is applied to multiple servers. Please refer to Figure 5 , including:

[0150] The protocol construction module 201 is used to implement the multiplication protocol of additive secret sharing by using Beaver triples and expand it into a vector inner product protocol and a matrix multiplication protocol;

[0151] The secure computing protocol design module 202 is used to design the secure DReLU protocol, the secure division protocol, the secure Softmax protocol, and the secure GELU protocol based on the multiplication protocol of additive secret sharing implemented by Beaver triples and the conversion protocol between additive secret sharing and multiplicative secret sharing;

[0152] The secure inference module 203 is used to perform secure multi-party privacy protection calculations based on the GPT model using the multiplication protocol, the vector inner product protocol, the matrix multiplication protocol, the secure DReLU protocol, the secure division protocol, the secure Softmax protocol, and the secure GELU protocol to obtain the GPT inference result.

[0153] Since the device introduced in Embodiment 2 of the present invention is the device used to implement the secure multi-party privacy protection inference method for GPT in Embodiment 1 of the present invention, based on the method introduced in Embodiment 1 of the present invention, those skilled in the art can understand the specific structure and deformation of the device, so it will not be repeated here. Any device used in the method of Embodiment 1 of the present invention belongs to the scope of protection of the present invention.

[0154] Embodiment III

[0155] Based on the same inventive concept, this embodiment discloses a secure multi-party privacy protection inference system for GPT, including the secure multi-party privacy protection inference device for GPT described in Embodiment II and a client. Among them, the client performs token embedding locally and secretly shares private inputs with the server.

[0156] Figure 3 It is the architecture diagram of the secure multi-party privacy protection inference system for GPT provided by the embodiments of the present invention.

[0157] Since the system introduced in Embodiment III of the present invention is the system adopted for implementing the secure multi-party privacy protection inference method for GPT in Embodiment I of the present invention, based on the method introduced in Embodiment I of the present invention, those skilled in the art can understand the specific structure and variations of this system, so it will not be elaborated here. Any system adopted by the method in Embodiment I of the present invention falls within the scope of protection of the present invention.

[0158] Embodiment IV

[0159] The present invention also provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method described in Embodiment I is implemented.

[0160] Since the computer device introduced in Embodiment IV of the present invention is the computer device adopted for implementing the secure multi-party privacy protection inference method for GPT in Embodiment I of the present invention, based on the method introduced in Embodiment I of the present invention, those skilled in the art can understand the specific structure and variations of this computer device, so it will not be elaborated here. Any computer device adopted by the method of Embodiment I of the present invention falls within the scope of protection of the present invention.

[0161] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] The present invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each flow and / or block in the flowchart illustrations and / or block diagrams, and combinations of flows and / or blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus create means for implementing the functions specified in the flow Figure 1 one or more of the flows and / or Figure 1 blocks or combinations of blocks.

[0163] Although the preferred embodiments of the present invention have been described, additional changes and modifications to these embodiments can be made by those skilled in the art once they learn of the basic inventive concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art can make various changes and variations to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations. < / t> < / z> < / s> < / k> < / z> < / z> < / z> < / z> < / r> < / x> < / z> < / y> < / x> < / z> < / z> < / r> < / r> < / x> < / r> < / z> < / z> < / z> < / x> < / x> < / x> < / x> < / x> < / x>

Claims

1. A secure multi-party privacy-preserving reasoning method for GPT, characterized in that: Applied to multiple servers, including: The multiplication protocol of additive secret sharing is implemented using Beaver triples, and is extended to vector inner product protocol and matrix multiplication protocol; Based on the multiplication protocol of additive secret sharing implemented by Beaver triples and the conversion protocol between additive secret sharing and multiplicative secret sharing, we designed the secure DReLU protocol, secure division protocol, secure Softmax protocol and secure GELU protocol. The GPT model is used to perform secure multi-party privacy protection calculations based on the multiplication protocol, vector inner product protocol, matrix multiplication protocol, secure DReLU protocol, secure division protocol, secure Softmax protocol, and secure GELU protocol to obtain the GPT inference results.

2. The secure multi-party privacy-preserving reasoning method for GPT as claimed in claim 1, characterized in that: A multiplication protocol for additive secret sharing, including: In the initialization phase, each participant prepares a Beaver triplet { ， ， <c>}, where c = a·b, a, b, c are the elements of a triple,< / c> ， ， <c> represents the share of a triple;< / c> In the online phase, each participant calculates the additive secret sharing of the intermediate value e locally <e> ＝ <x> -< / x> < / e> Additive secret sharing with another intermediate value f <f> ＝ <y> - ， <x> 、 <y> For secret sharing;< / y> < / x> < / y> < / f> Public <e>and <f> Get the public values ​​e and f;< / f> < / e> Additive secret sharing of computation outputs between parties <z>=e·f+e· +f· < / z> + <c> 。< / c> 3. The secure multi-party privacy-preserving reasoning method for GPT as claimed in claim 2, characterized in that: The conversion protocol between additive secret sharing and multiplicative secret sharing includes: Each participant P i Hold your own multiplicative secret sharing Share your own multiplication secrets Distribute to other participants P in the form of additive secret sharing j , j≠i, thus constructing the additive secret sharing vector <x> i ＝{ <x1> i ，…<x n > i}；< / x1> < / x> The multiplication protocol of additive secret sharing is used for iterative calculation to obtain the output additive secret sharing <z> i , where the true value z of the output satisfies < / z> 4. The secure multi-party privacy-preserving reasoning method for GPT as claimed in claim 2, characterized in that: Users and multiple servers use the Transformer model to perform secure multi-party privacy protection calculations based on the multiplication protocol, vector inner product protocol, matrix multiplication protocol, secure DReLU protocol, secure division protocol, secure Softmax protocol, and secure GELU protocol to obtain GPT inference results, including: Use the word embedding layer to map the user's private word embedding moment and position matrix into a vector representation; The input matrix is ​​normalized using the layer normalization layer so that its range is set to (0,1) and it obeys the normal distribution. The multi-head attention layer is used to calculate based on the additive secret sharing matrix and the pre-trained model parameters to obtain the secret sharing of the output results of the multi-head self-attention layer; The feedforward neural network is used to calculate the secret sharing of the output results of the feedforward neural network based on the additive secret sharing matrix, the secret sharing of the output results of the multi-head self-attention layer, the weight matrix and the bias matrix to obtain the secret sharing of the output results of the feedforward neural network; The final GPT inference result is obtained based on the secret sharing of the output results of the multi-head self-attention layer and the secret sharing of the output results of the feedforward neural network.

5. The secure multi-party privacy-preserving reasoning method for GPT as claimed in claim 1, characterized in that: The input matrix is ​​standardized using the layer normalization layer so that its range is set to (0,1) and it obeys the normal distribution, including: Each participant P i Compute the matrix locally Each line of Secret Sharing of Average Values And for m∈[1,d model ]Compute the secret sharing of the difference between matrix elements and the mean V represents the size of the input vocabulary, d model The size of the one-dimensional feature vector representing a word; All participants jointly execute the secure vector inner product multiplication protocol to obtain the secret sharing of z squared <z 2 >, where z m is the mth element of vector z; Each participant calculates the variance with noise locally ∈ represents the error value; All participants jointly execute the secure multi-party inverse square root function calculation protocol to obtain And calculate intermediate variables locally as well as Each participant jointly executes the secure matrix multiplication protocol to obtain <Y T > Among them and For m∈[1,d model ], all parties jointly execute the normalized output of the secure multiplication protocol computing layer in Both γ and β are model pre-training parameters, and the subscript m represents the mth parameter.

6. The secure multi-party privacy-preserving reasoning method for GPT as claimed in claim 1, characterized in that: The multi-head attention layer is used to calculate based on the additive secret sharing matrix and the pre-trained model parameters to obtain the secret sharing of the output results of the multi-head self-attention layer, including: All parties jointly execute the secure matrix multiplication protocol to obtain the intermediate variable matrix of GPT model parameters as well as V represents the size of the input vocabulary, d model represents the size of a one-dimensional feature vector of a word, X represents the input matrix of the multi-head self-attention layer, and W K , W Q , W V Respectively represent the model pre-training parameters used to generate K, Q, and V matrices; Each participant P i Perform matrix partitioning locally: Where M is the number of heads, represents the K matrix of the first attention head and the K matrix of the Mth attention head, represents the Q matrix of the first attention head and the Q matrix of the Mth attention head, Represents the Q matrix of the first attention head and the V matrix of the Mth attention head; For j∈[M], the following steps are computed in parallel: All parties jointly execute the secure matrix multiplication protocol to obtain the secret sharing of the intermediate matrix variables <T j > Among them All parties jointly execute the secure Softmax function calculation protocol and obtain the output of the Softmax function calculation protocol j > Among them ​ The parties jointly execute the secure matrix multiplication protocol to obtain.. in Each participant P i Perform matrix combination locally and obtain secret sharing of the output results of the multi-head self-attention layer in 7. The secure multi-party privacy-preserving reasoning method for GPT as claimed in claim 1, characterized in that: The feedforward neural network is used to calculate the secret sharing of the output results of the feedforward neural network based on the additive secret sharing matrix, the secret sharing of the output results of the multi-head self-attention layer, the weight matrix, and the bias matrix, and the secret sharing of the output results of the feedforward neural network is obtained, including: Each participant jointly executes the secure matrix multiplication protocol to calculate the secret sharing of the intermediate variables <t1>in Where X represents the input matrix of the feed-forward neural network FFN, W1 represents the weight matrix of the first linear layer in FFN, V represents the size of the vocabulary in the input matrix, and w represents the dimension of the weight matrix; All parties jointly execute the secure GELU function calculation protocol to obtain the secret sharing of the second intermediate variable <t2>,in Among them, T1 represents the output matrix of the first linear layer in FFN, B1 represents the bias matrix of the first linear layer in FFN, All parties jointly execute the secure GELU function calculation protocol to calculate the secret sharing of the third intermediate variable <t3>,in, Get secret sharing of the output of a feedforward neural network W2 T represents the weight matrix of the second linear layer in FFN, and B2 represents the bias matrix of the second linear layer in FFN.

8. A secure multi-party privacy-preserving reasoning device for GPT, characterized in that: Applies to users and multiple servers, including: Protocol building blocks for implementing a multiplication protocol for additive secret sharing using Beaver triples, and extending it to vector inner product protocol and matrix multiplication protocol; The secure computing protocol design module is used for the multiplication protocol of additive secret sharing based on Beaver triples and the conversion protocol between additive secret sharing and multiplicative secret sharing. It designs secure DReLU protocol, secure division protocol, secure Softmax protocol and secure GELU protocol. The secure reasoning module is used to use the GPT model to perform secure multi-party privacy protection calculations based on the multiplication protocol, vector inner product protocol, matrix multiplication protocol, secure DReLU protocol, secure division protocol, secure Softmax protocol, and secure GELU protocol to obtain GPT reasoning results.

9. A secure multi-party privacy-preserving reasoning system for GPT, characterized in that: It includes a secure multi-party privacy-preserving reasoning device for GPT as described in claim 8 and a user end, wherein the user end performs token embedding locally and secretly shares private inputs with a server.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the secure multi-party privacy-preserving reasoning method for GPT as described in any one of claims 1 to 7.

Images