A lightweight privacy-preserving NLP prediction method and system

Abstract

The application discloses a lightweight NLP prediction method and system for protecting privacy, wherein a model trainer sends trained parameters to two cloud servers in the form of additive secret sharing, and assumes that the two cloud servers do not collude with each other. Since the two servers do not collude, the parameters of the model cannot be recovered by collusion. In the prediction process, a user sends own input to the two cloud servers in the form of additive secret sharing for prediction. When the result is recovered, the user adds the results returned by the two servers to recover the result. The user only needs to perform some lightweight operations to protect the privacy of user data from being leaked to the cloud server.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information security, and more particularly to a lightweight NLP prediction method and system for protecting privacy. BACKGROUND

[0002] At present, natural language processing (NLP) is a kind of artificial intelligence for analyzing human language. Its basic working principle can be summarized as follows: accepting self-language, translating natural language based on a probability algorithm, and finally outputting natural language analysis results. NLP problems play an important role in machine translation, speech recognition, sentiment analysis and email classification. Transformer is a classic model for solving NLP problems, which uses attention mechanism to improve the training speed of the model. Transformer is a deep learning model based entirely on self-attention mechanism and is suitable for parallel computing. Compared with RNN recurrent neural network, it has higher accuracy and performance. In the present application, we focus on the application of Transformer in machine translation (Chinese translation into English), and the parameters in the model have been trained in advance.

[0003] However, in real life, the model trainer and the user who uses the model to make predictions are usually different types of entities. For example, in the field of machine translation, the model trainer is usually some big companies such as Google or Baidu, etc., while the model user is usually some individuals or enterprises. Some companies or entities use the collected data to train the model, while some individuals or entities hold their own data and want to make predictions. Then, this two-way data interaction prediction has some problems. First, the model training party saves all the data parameters, which increases the storage overhead of the model training party; second, if you want to respond to the user's input data in a timely manner, the model training party needs to be online in real time. In addition, for the user, the input data contains their own privacy and confidential data, which cannot be disclosed to other entities.

[0004] Therefore, how to design a system and method that can make real-time NLP prediction and protect the privacy of both parties is a problem that needs to be solved by those skilled in the art. SUMMARY

[0005] In view of this, the present application provides a lightweight NLP prediction method and system for protecting privacy, a model trainer sends the trained parameters to two cloud servers in the form of additive secret sharing, assuming that the two cloud servers do not collude with each other, since the two servers do not collude, the parameters of the model cannot be recovered by collusion, in the prediction process, the user sends the input to the two cloud servers for prediction in the form of additive secret sharing. When the result is recovered, the user adds the results returned by the two servers to recover them. The user only needs to perform some lightweight operations to protect the privacy of the user data from being leaked to the cloud server.

[0006] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0007] A lightweight NLP prediction method for protecting privacy comprises the following steps:

[0008] S1, split the model parameters based on additive secret sharing, and send the split model parameters to server CSP1 and server CSP2 respectively;

[0009] S2, split the input X based on additive secret sharing to obtain <x>1 and <x>2, are sent to the server CSP1 and to the server CSP2, respectively, which send <x>1 and <x>2 Projecting into the embedding space, get the secret share <W semb • X>1 and <W semb • X>2, and based on the secret share and the position encoding matrix, respectively get the input matrix <s>1 and <s>2;

[0010] S3, the server CSP1 and the server CSP2 are based on the input matrix <s>1 and <s>2, respectively, to obtain a query matrix <q>1 and <q>2, key matrix <k>1 and <k>2, value matrix <v>1 and <v>2, respectively, to the query matrix <q>1 and <q>2. Key matrix <k>1 and <k>2 Dot product operation is performed to obtain <K T ·Q>1 and <K T ·Q>1, and the operation results are respectively normalized, and the normalized data is processed through an activation function, and combined with the value matrix <v>1 and <v>2, respectively, attention function 1 and 2;

[0011] S4, the server CSP1 and the server CSP2 respectively transform the input matrix <s>1 and <s>2, attention function< / s> < / s> <s> <s> 1 and 2 add, resulting in a matrix <r>1 and <r>2, matrix <r>1 and <r>2normalizing, respectively, to obtain an output matrix <l>1 and <l>2;

[0012] S5, for the output matrix <l>1 and <l>each column <l i >1 and <l i >2, compute <f i >1 and <f i >2, and further obtain the output of the forward propagation layer as <f>1 and <f>2;

[0013] S6, repeat steps S3-S5, after the calculation by the N layers of encoders and decoders, the server CSP1 obtains <h>1. The server CSP2 gets <h>2. After the linear layer, compute and get matrix 1 and 2, two servers interact to compute the matrix 1 and The softmax value of 2 is subtracted from the probability corresponding to each word, and the word with the maximum probability is obtained through a secure comparison protocol, denoted as the corresponding prediction result.

[0014] Preferably, the step S1 specifically comprises:

[0015] The model parameters include: a word embedding matrix W in a source vocabulary semb , a position encoding matrix PE, a query matrix parameter W Q , a key matrix parameter W K , a value matrix parameter W V , learning parameters γ, β, W1 and W2 are two weight matrices in a forward propagation layer, b1 and b2 are two vectors in the forward propagation layer, W temb is an output encoding matrix.

[0016] Based on the form of additive secret sharing, it is <W semb >1, <pe>1, <W Q >1, <W K >1, <W V >1, <γ>1, <β>1, <w1> 1， <w2> 1， <b1> 1， <b2>1, <W semb >1 and <W semb >2, <pe>2, <W Q >2, <W K >2, <W V >2, <γ>2, <β>2, <w1> 2， <w2> 2， <b1> 2， <b2>2, <W temb >2 and sends them to the server CSP1 and the server CSP2 respectively.

[0017] Preferably, the step S2 specifically comprises:

[0018] S21, the server CSP1 holds <W semb >1 and <pe>1. The server CSP2 holds <W semb >2 and <pe>2. Split the input X into the form of additive secret sharing as <x>1 and <x>2 and sent to the server CSP1 and the server CSP2 respectively;

[0019] S22, generating two matrices calculating C = F · D ∈ Z 512×n , performing additive secret sharing on F, D, C and sending <f> 1， <d> 1， <c>1 Send to server CSP1, will <f> 2， <d> 2， <c>2 is sent to the server CSP2. The server CSP1 computes 1＝<W semb >1- <f> 1， <v> 1＝ <x> 1- <d>1, and 1 and <v>1 sent to the server CSP2, which computes 2＝<W semb >2- <f> 2， <v> 2＝ <x> 2- <d>2, and 2 and <v>2 sent to the server CSP1 ;

[0020] S23, the server CSP1 reconstructs U= 1+ 2 and V = 0 <v> 1+ <v>2. The server CSP2 reconstructs U and V in the same way, the server CSP1 computes <f>1 V + U <d> 1+ <d>1 get W semb • a secret share <W semb • X > 1, the server CSP2 computes <f>2V+U <d> 2+ <c>2 get W semb • a secret share <W semb • X > 2;

[0021] S24, the server CSP1 based on <pe>1 and the <W semb • X > 1, compute <s>1＝ <pe>1+<W semb • X > 1, server CSP2 based on <pe>2 and the <W semb • X2calculation <s>2＝ <pe>2 + <W semb • X > 2.

[0022] Preferably, the step S3 specifically comprises:

[0023] S31, the server CSP1 calculates the input matrix <s>1 and the query matrix parameter <W Q >1, the key matrix parameter <W K >1, the value matrix parameter <W V >1 calculates the query matrix <q>1, key matrix <k>1 and value matrix <v> 1：

[0024] <q>1=<W Q • S>1

[0025] <k>1=<W K • S>1

[0026] <v>1 = <W V • S > 1

[0027] By analogy, the server CSP2 bases the input matrix <s>2 and the query matrix parameter <W Q >2, the key matrix parameter <W K >2, the value matrix parameter <W V >2 calculates the query matrix <q>2, key matrix <k>2 and the value matrix <v>2;

[0028] S32, the server CSP1 queries the matrix <q>1 and key matrix <k>1 Dot product operation is performed and divided by Normalization is performed to obtain The server CSP2 performs a query matrix <q>2 and key matrix <k>2 dot product operation and divide by Normalization is performed to obtain

[0029] S33, in the server CSP1, for the matrix Each list is represented as [<a 1,i >1...<a n,i >1] T ; in the server CSP2, for the matrix Each list is represented as [<a 1,i >2...<a n,i >2] T , the server CSP1 and the server CSP2 interact to calculate the softmax function, for the ith column of the tth row, t∈[1,n], the softmax value is represented as:

[0030]

[0031] The server CSP1 calculates The server CSP2 calculates is a multiplication secret sharing value;

[0032] A randomly generated non-zero real number c is obtained, which is divided into <c> 1, <c>2 sum General <c>1 and send to the server CSP1, to <c>2 and Sends to server CSP2, server CSP1 computes Server CSP2 computes Server CSP1 and server CSP2 interactively recover Server CSP1 computes Server CSP2 computes Based on the same method as above, server CSP1 gets Server CSP2 gets

[0033] Randomly select a non-zero number c, and divide it into <c>1 and <c>2, distributed to two servers, server CSP1 owns <c>1 and <τ i >1, the server CSP2 owns <c>2 and <τ i >2, two servers respectively calculate and c·τ i >1, server CSP1 owns and <c·τ i >2, server CSP2 owns and <c·τ i >2, two servers communicate with each other to reconstruct c·τ i = <c·τ i >1 + <c·τ i >2, server CSP1 calculates server CSP2 calculates

[0034] S34, server CSP1 calculates< / c> < / c> < / c> < / c> < / c> < / c> < / c> < / c> < / k> < / q> < / k> < / q> < / v> < / k> < / q> < / s> < / v> < / k> < / q> < / v> < / k> < / q> < / s> < / pe> < / s> < / pe> < / pe> < / s> < / pe> < / c> < / d> < / f> < / d> < / d> < / f> < / v> < / v> < / v> < / d> < / x> < / v> < / f> < / v> < / d> < / x> < / v> < / f> < / c> < / d> < / f> < / c> < / d> < / f> < / x> < / x> < / pe> < / pe> < / b1> < / w2> < / w1> < / pe> < / b1> < / w2> < / w1> < / pe> < / h> < / h> < / f> < / f> < / l> < / l> < / l> < / l> < / r> < / r> < / r> < / r> <s> <s> <s> 1, server CSP2 computes 2:

[0035]

[0036] Preferably, the step S4 specifically comprises:

[0037] S41, the server CSP1 determines the first user U1's first user identifier UID1 based on <s>1 and< / s> <s> 1, obtain <r> 1＝ <s>1+< / s> < / r> <s> 1. The server CSP2 is based on <s>2 and< / s> <s> 2, obtain <r> 2＝ <s>2+< / s> < / r> <s> 2;

[0038] S42, the server CSP1 calculates and the server CSP2 calculates and wherein the server CSP1 obtains from the server CSP2 i >1 the server CSP2 obtains from the server CSP1 i >2

[0039] S421, the server CSP1 calculates

[0040] <m> 1＝<u i >1·2 -λ

[0041] <z0>1 = ξ· <m>1 + ω

[0042]

[0043] Server CSP2 computes

[0044] <m> 2＝<u i >2·2 -λ

[0045] <z0>2 = ξ · <m>2

[0046]

[0047] where ξ and ω are two constants, ξ = -0.8099868542, ω = 1.787727479;

[0048] S422、based on <m> 1， <m> 2， <z0>1 and <z0>2. Both servers interact with each other to get

[0049] <t0>1 = <m · z0>1

[0050] <t0>2=<m·z0> 2

[0051] t0= <t0> 1+ <t0>2 = m · z0;

[0052] S423, for i = 1, 2, 3,..., λ, servers CSP1 and CSP2 interact with each other based on <t where is the number of cycles, both servers perform S424-S426 cyclically;

[0053] S424, based on <t j >1, <t j >2, <o j >1 and <o j >2, both servers interact with each other, run a secure multiplication operation to get <d j >1 and <d j >2, d j = <d j >1 + <d j >2 = t j · o j ;

[0054] S425, server CSP1 computes server CSP2 computes <d j >2 = - <d j >2;

[0055] S426, based on <t j >1, <t j >2, <o j >1, <o j >2, <d j >1 and <d j >2, both servers interact with each other to get <t j+1 >1, <t j+1 >2, <o j+1 >1 and <o j+1 >2, where t j+1 = <t j+1 >1 + <t j+1 >2 = t j · d j , o j+1 = <o j+1 >1 + <o j+1 >2 = o j · d j .

[0056] S427, after the cycle, server CSP1 gets <t j >1, server CSP2 gets <t j >2;

[0057] S428, if λ is even, servers CSP1 and CSP2 set a = 1; otherwise, set

[0058] S429Server CSP1 and server CSP2 compute Server CSP1 compute Server CSP2 compute Based on <σ i >1,<σ i >2 and column vector r i =[r 1,i r 2,i …r 512,i ] T The additive secret sharing values [<r 1,i >1<r 2,i >1…<r 512,i >1] T and [<r 1,i >2<r 2,i >2…<r 512,i >2] T of column vector r i =[r 1,i r 2,i …r 512,i ] T are normalized by two servers:

[0059] Server CSP1 compute Server CSP2 compute Two servers interact with each other to get and where

[0060] Based on <γ>1, <γ>2, <β>1 and <β>2, two servers interact with each other to get and

[0061] Server CSP1 gets <l> 1＝( <l1> 1， <l2>1,..., <l n >1), the server CSP2 gets <l> 2＝( <l1> 2， <l2>2,..., <l n >2).

[0062] Preferably, the step S5 specifically comprises:

[0063] S51, for each column l of the matrix L i , the server CSP1 possesses <l i >1 and the parameters <w1> 1， <w2> 1， <b1> 1， <b2>1 ; the server CSP2 owns <l i >2 and parameters <w1> 2， <w2> 2， <b1> 2， <b2>2. Both server computers compute the output of the forward propagation layer:

[0064] Based on <w1> 1，<l i >1， <b1>1 and <w1> 2，<l i >2， <b1>2, two servers interact with each other to get

[0065] <θ i >1 = <W1 · l i >1+ <b1>1

[0066] <θ i >2=<W1·l i >2+ <b1>2

[0067] θ i = <θ i >1+ <θ i >2= W1·l i +b1

[0068] Based on <θ i >1 and <θ i >2, two servers interact, compare the size of each element in each vector θ i with 0, k∈[1, 512], judge the sign of the highest bit, 1 for negative, 0 for positive.

[0069] S52, server CSP1 sets According to the binary bit string, it is expressed as λ is the length of the bit string, and similarly, server CSP2 sets According to the binary bit string, it is expressed as

[0070] For each bit CSP1 sets CSP2 sets

[0071] For each bit CSP1 sets CSP2 sets Both parties set an initial carry bit

[0072] S53, the carry of the j+1 position can be expressed as:

[0073] S54, based on and Server CSP1 and server CSP2 calculate step S54λ-1 times to get and

[0074] S55, server CSP1 calculates Server CSP2 calculates

[0075] S56, server CSP1 calculates Server CSP2 calculates

[0076] S57, server CSP1 sets and​​​ Server CSP2 settings and

[0077] S58. Both parties use a safe multiplication protocol to calculate. and in

[0078] S59. Both parties use a safe multiplication protocol to calculate. and in

[0079] S510, for θ i Each element All steps S53-S59 are executed, and finally server CSP1 obtains a column vector. <ff i >1, Server CSP2 receives a column vector <ff i >2, based on <w2>1, <ff i >1, <b2>1 and <w2>2, <ff i >2, <b2>2, both servers interact to obtain <f i >1 and <f i >2, f i = <f i >1 + <f i >2 = W2·ff i +b2, for each column in the matrix L, the server CSP1 obtains <f>1. The server CSP2 gets <f>2.

[0080] A lightweight NLP prediction system for protecting privacy, comprising:

[0081] A user end, a cloud server and a model providing end, the user end and the model providing end are connected with the cloud server, the cloud server is at least 2, and the cloud servers do not collude with each other;

[0082] The model providing end sends the trained model parameters to the cloud server in the form of additive sharing, and the user end also sends the to-be-predicted data to the cloud server in the form of additive sharing, multiple cloud servers interact with each other to perform prediction.

[0083] Compared with the prior art, the technical scheme can know that the present application provides a lightweight NLP prediction method and system for protecting privacy, the model trainer sends the trained parameters to two cloud servers in the form of additive secret sharing, assuming that the two cloud servers do not collude with each other, since the two servers do not collude, the parameters of the model cannot be restored by collusion, in the prediction process, the user also sends the input to the two cloud servers in the form of additive secret sharing for prediction. When the result is restored, the user adds the results returned by the two servers to restore them. The user only needs to perform some lightweight operations to protect the privacy of the user data from being leaked to the cloud server. BRIEF DESCRIPTION OF DRAWINGS

[0084] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0085] < / f> < / f> < / w1> < / w1> < / b1> < / w2> < / w1> < / b1> < / w2> < / w1> < / l1> < / l> < / l1> < / l> < / t0> < / m> < / m> < / m> < / m> < / m> < / m> Figure 1 Figure is an encoder structure schematic diagram provided by the embodiment of the present application.

[0086] Figure 2 Figure is a self-attention layer data operation schematic diagram provided by the embodiment of the present application.

[0087] Figure 3 Figure is a system model structure schematic diagram provided by the embodiment of the present application.

[0088] Figure 4 Figure is a secure comparison protocol process schematic diagram provided by the embodiment of the present application. DETAILED DESCRIPTION

[0089] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0090] The embodiment of the present application discloses a kind of

[0091] 1, initialization phase:

[0092] First, the integer and decimal involved in model parameters and user input can be converted into a fixed λ-bit signed integer by a method. Then, for a λ-bit number x, addition sharing can be expressed as <x> 1+ <x>2 = x (mod 2 λ ), where <x>1 is a shared value of the server CSP1 and <x>2 is the share value of server CSP2. After that, for a λ-bit number x, the multiplication share can be expressed as where is the share value of server CSP1 and is the share value of server CSP2. < > represents the secret value under the additive share, represents the secret value under the multiplication share. The model trainer sends W semb , PE, W Q , W K , W V , γ, β, W1, W2, b1, b2, W temb and other parameters to server CSP1 and server CSP2 respectively according to the method of additive share.

[0093] 2, input stage:

[0094] Server CSP1 holds <W semb >1 and <pe>1. The server CSP2 holds <W semb >2 and <pe>2. The user who makes the prediction divides his input X into <x>1 and <x>2 and send to server CSP1 and server CSP2 respectively. Server CSP1 can get <W semb ·X>1 according to the following secure multiplication protocol and server CSP2 can get <W semb ·X>2:

[0095] Secure multiplication protocol: First a trusted third party generates two matrices and computes C = A · D ∈ Z 512×n . This third party additively shares A, D, C and sends 1, <d> 1, <c>1 to server CSP1, which computes< / c> < / d> 2, <d> 2, <c>2 is sent to the server CSP2. The server CSP1 computes 1＝<W semb >1- < / c> < / d> 1， <v> 1＝ <x> 1- <d>1, and 1 and <v>1 is sent to the server CSP2. The server CSP2 computes 2＝<W semb >2- < / v> < / d> < / x> < / v> 2， <v> 2＝ <x> 2- <d>2, and 2 and <v>2 is sent to the server CSP1. The server CSP1 reconstructs U = U1 + U2 1+ 2 and V = 0 <v> 1+ <v>2. Server CSP2 can reconstruct U and V in the same way. Server CSP1 computes< / v> < / v> < / v> < / d> < / x> < / v> 1 V + U <d> 1+ <c>1 get W semb • a secret share < W semb • X > 1. Server CSP2 computes< / c> < / d> 2V+U <d> 2+ <c>2 get W semb • a secret share of X < W semb • X > 2.

[0096] This secure multiplication protocol can be used not only to compute matrix multiplication, but also to compute the product of two numbers.

[0097] The position matrix can be computed by the model provider and sent to the two servers separately using encrypted secret sharing. Server CSP1 owns <pe>1 and the <W semb • X > 1, the server CSP2 owns <pe>2 and the <W semb • X>2. The server CSP1 computes <s>1＝ <pe>1 + <W semb • X > 1, the server CSP2 computes <s>2＝ <pe>2 + <W semb • X > 2.

[0098] 3. Encoder (decoder):

[0099] From the self-attention layer, the server CSP1 has the result of the previous step <s>1 and the model provider submits a <W Q >1, <W K >1 and the model provider submits a <W V >1. The server CSP1 has the <W <s>2 and the model provider submits <W Q >2, <W K >2 and <W V >2. The two servers can run a secure multiplication protocol to get the additive sharing of the three matrices. Server CSP1 gets <q>1， <k> 1， <v>1. The server CSP2 gets <q>2， <k> 2， <v>2. Similarly, server CSP1 bases its decision on <q>1， <k>1 and server CSP2 are based on <q>2， <k>2, two interactive running security multiplication protocol, respectively, can get <K T ·Q>1 and <K T ·Q>2. After two servers simultaneously divide get and

[0100] In the server CSP1 end, the matrix it owns Each column can be expressed as [<a 1,i >1...<a n,i >1] T ; In the server CSP2 end, the matrix it owns Each column can be expressed as [<a 1,i >2...<a n,i >2] T ; After that, two servers interactive computing softmax function, for the ith column of the t row, t∈[1,n], its softmax value can be expressed as:

[0101]

[0102] First describe how two servers calculate Server CSP1 owns <a t,i >1, server CSP2 owns <a t,i >2. Server CSP1 calculates Server CSP2 calculates Representing multiplication secret sharing value. After that, this multiplication sharing value can be converted into additive sharing value according to the following method:

[0103] Multiplication sharing value conversion to additive sharing value protocol: a trusted third party randomly generates a non-zero real number c, and then divides it into <c> 1, <c>2 and Afterwards <c>1 and Send to server CSP1, to <c>2 and to server CSP2. Server CSP1 computes Server CSP2 computes After the two servers interact, the Finally, server CSP1 computes Server CSP2 computes

[0104] The two servers can compute each of the algorithms in the same way Server CSP1 can get Server CSP2 can get Now, server CSP1 has and <τ i >1, and server CSP2 has and <τ i >2.

[0105] The two servers can compute the additive secret sharing value of P(a t,i ) in the following way:

[0106] Secure division protocol: A trusted third party randomly chooses a non-zero number c and divides it into <c>1 and <c>2, is distributed to two servers. Server CSP1 owns <c>1 and <τ i >1, the server CSP2 owns <c>2 and <τ i >2. Two servers run the secure multiplication protocol computation and c·τ i . Server CSP1 owns and <c·τ i >1. Server CSP2 owns and <c·τ i >2. After that, two servers communicate with each other to reconstruct c·τ i . Server CSP1 computes Server CSP2 computes

[0107] After that, two servers compute the attention function

[0108] Server CSP1 owns <v>1 and the one obtained in the previous step The server CSP2 owns <v>2 and the result of the previous step Both parties can run a secure multiplication protocol to obtain< / v> < / v> < / c> < / c> < / c> < / c> < / c> < / c> < / c> < / c> < / k> < / q> < / k> < / q> < / v> < / k> < / q> < / v> < / k> < / q> < / s> < / s> < / pe> < / s> < / pe> < / s> < / pe> < / pe> < / c> < / d> <s> <s> <s> 1 and 2.

[0109] In the residual connection and normalization layers, the server CSP1 has <s>1 and< / s> <s> 1 ; the server CSP2 owns <s>2 and< / s> <s> 2. Both parties can directly perform the addition of both to get the result of the residual link. Server CSP1 gets <r> 1＝ <s>1+< / s> < / r> <s> 1 ; server CSP2 gets <r> 2＝ <s>2+< / s> < / r> <s> 2. Next, the two servers can normalize each column of the matrix R in the following way:

[0110] Server CSP1 computes Server CSP2 computes Similarly, server CSP1 and server CSP2 can obtain and where Server CSP1 and server CSP2 run the secure root protocol on <u i >1 and <u i >2 to obtain and

[0111] (1) Server CSP1 computes <m>1 = <u i >1·2 -λ ; server CSP2 computes <m>2 = <u i > 2 · 2 -λ .

[0112] (2) Server CSP1 computes <z0>1 = ξ· <m>1 + ω and where ξ and ω are two constants, where ξ = -0.8099868542 and ω = 1.787727479.

[0113] (3) Server CSP2 computes <z0>2 = ξ · <m>2 and

[0114] (4) based on <m> 1, <m> 2, <z0>1 and <z0>2. Both servers interact with each other, run a secure multiplication operation to get <t0>1 and <t0> 2，t0＝ <t0> 1+ <t0>2 = m · z0.

[0115] (5) For where is the number of cycles, both servers perform (6)-(8) in a loop

[0116] (6) Based on <t j >1, <t j >2, <o j >1, and <o j >2, both servers interact with each other to run a secure multiplication operation to obtain <d j >1 and <d j >2, d j = <d j >1 + <d j >2 = t j · o j .

[0117] (7) Server CSP1 computes Server CSP2 computes <d j >2 = -<d j >2.

[0118] (8) Based on <t j >1, <t j >2, <o j >1, <o j >2, <d j >1, and <d j >2, both servers interact with each other to run a secure multiplication operation to obtain <t j+1 >1, <t j+1 >2, <o j+1 >1, <o j+1 >2. Where t j+1 = <t j+1 >1 + <t j+1 >2 = t j · d j , o j+1 = <o j+1 >1 + <o j+1 >2 = o j · d j .

[0119] (9) After the loop ends, server CSP1 obtains <t j >1, and server CSP2 obtains <t j >2.

[0120] (10) If λ is even, server CSP1 and server CSP2 set a = 1; otherwise, set

[0121] (11) Calculation of servers CSP1 and CSP2

[0122] (12) Server CSP1 calculation Server CSP2 Computing

[0123] based on <σ i >1,<σ i >2 and column vector r i =[t 1,i r 2,i …r 512,i ] T Addition secret shared value[ <r 1,i >1 <r 2,i >1… <r 512,i >1] T and[ <r 1,i >2 <r 2,i >2… <r 512,i >2] T Two servers for column vector r i =[r 1,i r 2,i …r 512,i ] T Perform normalization:

[0124] (1) Server CSP1 calculation Server CSP2 Computing

[0125] (2) Compared with the previous calculation of P(a) t,i The method is similar; the two servers interact and perform a safe division operation to obtain... and in

[0126] (3) Based on <γ>1,<γ>2,<β>1, and<β>2, the two servers interact with each other, running a secure multiplication protocol to obtain... <l i >1 and <l i >2, of which

[0127] Both servers perform the above operation on each column of matrix L, and server CSP1 finally obtains... <l>1. The server CSP2 finally obtains <l>2.

[0128] 5. Forward propagation layer:

[0129] For each column l of the matrix L i , the server CSP1 possesses <l i >1 and the corresponding parameters <w1> 1， <w2> 1， <b1>1 and <b2>1;

[0130] The server CSP2 owns <l i >2 and the corresponding parameters <w1> 2， <w2> 2， <b1>2 and <b2>2. The two servers compute the output of the forward propagation layer as follows: first, based on <w1> 1，<l i >1， <b1>1 and <w1> 2，<l i>2 ， <b1>2. The two servers interact with each other and run the secure multiplication protocol to obtain <θ i >1 and <θ i >2, θ i =<θ i >1+<θ i >2=W1·l i +b1. Based on <θ i >1 and <θ i >2. The two servers interact, comparing each vector θ. i Each element The relationship between the value of k and 0 is defined as k∈[1,512]. This determines the sign of the most significant bit: 1 represents a negative number, and 0 represents a positive number. A full adder method is used to calculate from the least significant bit to the most significant bit. A secure comparison protocol is as follows: < / w1> < / w1> < / w2> < / w1> < / w2> < / w1> < / l> < / l> < / t0> < / t0> < / m> < / m> < / m> < / m> < / m> < / m> Figure 4 as shown.

[0131] The details of the secure comparison protocol are as follows:

[0132] (1) Specifically, server CSP1 sets which can be expressed as a binary bit string λ is the length of the bit string, and server CSP2 sets which can be expressed as a binary bit string For each bit CSP1 sets CSP2 sets For each bit CSP1 sets CSP2 sets Both parties set an initial carry bit

[0133] (2) The carry for the j+1 position can be expressed as:

[0134] (3) Based on and Both parties run a secure multiplication and secure addition operation to calculate and

[0135] (4) Server CSP1 calculates Server CSP2 calculates

[0136] (5) Server CSP1 calculates Server CSP2 calculates

[0137] (6) Server CSP1 sets and Server CSP2 sets and

[0138] (7) Both parties run a secure multiplication protocol to calculate and where

[0139] (8) Both parties run a secure multiplication protocol to calculate and where

[0140] for each element of θ i (2)-(8) are performed, and finally server CSP1 gets a column vector <ff i >1 and server CSP2 gets a column vector <ff i >2. Based on <w2>1,<ff i >1, <b2>1 and <w2>2, <ff i >2, <b2>2, both servers interact with each other, run a secure multiplication operation to get <f i >1 and <f i >2, f i = <f i >1 + <f i >2 = W2·ff i +b2. For each column in the matrix L, the above operation is performed, and finally the server CSP1 gets <f>1. Finally, the server CSP2 gets <f>2.

[0141] 6. Output result:

[0142] After the calculation of N layers of encoders and decoders, the final server CSP1 gets <h>1. Finally, the server CSP2 gets <h>2. Based on the system parameters <W temb >1, <W temb >2, two servers interact with each other, run secure multiplication operation to get 1 and 2, where B = 0 1+ 2 = W temb ·H.

[0143] Finally, using the method of calculating the softmax function before, two servers interact with each other to calculate the softmax value of the matrix B, and by performing the difference operation on the probability corresponding to each word, the word with the maximum probability is obtained through the secure comparison protocol, which is the corresponding translation result.

[0144] The beneficial effects of the present application are:

[0145] (1) The model trainer can outsource the trained parameters to the cloud server, which can not only reduce the computational and storage overheads of the model trainer, but also ensure real-time prediction of user input. In addition, model training can use additive secret sharing to protect the privacy of cloud training parameters.

[0146] (2) The user only needs to communicate with the cloud server once to obtain the prediction result. In addition, the user only needs to perform some lightweight operations to protect the privacy of user data from being leaked to the cloud server.

[0147] (3) In the present system, multiple privacy-protecting, additive secret sharing-based secure computing methods are designed, such as secure multiplication protocol, secure division protocol, secure comparison protocol, etc. Two servers can interact with each other to securely and efficiently complete the NLP prediction process.

[0148] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0149] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein. < / h> < / h> < / f> < / f> ​< / s> < / s> < / s> < / s> < / s> < / s> < / s> < / x> < / x> < / pe> < / pe> < / x> < / x> < / x> < / x> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / v> < / v> < / k> < / k> < / q> < / q> < / v> < / v> < / k> < / k> < / q> < / q> < / s> < / s> < / s> < / s> < / x> < / x> < / x> < / x>

Claims

1. A lightweight NLP prediction method for privacy protection, characterized in that, The method comprises the following steps: S1, splitting the model parameters based on additive secret sharing, and sending the split model parameters to servers CSP1 and CSP2 respectively; S2, based on additive secret sharing, splits the input X to obtain <x>1 and <x>2, are sent to the server CSP1 and to the server CSP2, respectively, which send <x>1 and <x>2 Projecting into the embedding space, get the secret share <W semb • X>1 and <W semb • X>2, and based on the secret share and the position encoding matrix, respectively get the input matrix <s>1 and <s>2；< / s> < / s> < / x> < / x> < / x> < / x> <s> <s> S3, the server CSP1 and the server CSP2 based on the input matrix <s>1 and <s>2, respectively, to obtain a query matrix <q>1 and <q>2, key matrix <k>1 and <k>2, value matrix <v>1 and <v>2, respectively, to the query matrix <q>1 and <q>2. Key matrix <k>1 and <k>2 Dot product operation is performed to obtain <K T ·Q>1 and <K T ·Q>1, and the operation results are respectively normalized, and the normalized data is processed through an activation function, and combined with the value matrix <v>1 and <v>2, respectively, get attention function 1 and 2； < / v> < / v> < / k> < / k> < / q> < / q> < / v> < / v> < / k> < / k> < / q> < / q> < / s> < / s> <s> <s> S4, the server CSP1 and the server CSP2 respectively transform the input matrix <s>1 and <s>2, the softmax value of the attention function< / s> < / s> <s> <s> 1 and 2 add, resulting in a matrix <r>1 and <r>2, matrix <r>1 and <r>2 are normalized separately to obtain an output matrix <l>1 and <l> 2；< / l> < / l> < / r> < / r> < / r> < / r> < / s> < / s> <s> <s> S5, for the output matrix <l>1 and <l>each column <l i >1 and <l i >2, compute <f i >1 and <f i >2, and further obtain the output of the forward propagation layer as <f>1 and <f> 2；< / f> < / f> < / l> < / l> S6, repeating steps S3-S5, after the N-layer encoder and decoder calculation, the server CSP1 gets <h>1. The server CSP2 gets <h>2. After the linear layer, compute and get matrix 1 and 2, two servers interact to compute the matrix 1 and 2, and the probability of each word is calculated by difference operation, and the word with the maximum probability is obtained through a secure comparison protocol, which is recorded as the corresponding prediction result. < / h> < / h> 2. The lightweight NLP prediction method for privacy protection according to claim 1, wherein, The step S1 specifically comprises: The model parameters include: a word embedding matrix W in a source vocabulary semb , a position encoding matrix PE, a query matrix parameter W Q , a key matrix parameter W K , a value matrix parameter W V , learning parameters γ, β, W1 and W2 are two weight matrices in a forward propagation layer, b1 and b2 are two vectors in the forward propagation layer, and W temb is an output encoding matrix Formal based on additive secret sharing is divided into <W semb >1, <pe>1, <W Q >1, <W K >1, <W V >1, <γ>1, <β>1, <w1> 1， <w2> 1， <b1> 1， <b2>1, <W temb >1 and <W semb >2, <pe>2, <W Q >2, <W K >2, <W V >2, <γ>2, <β>2, <w1> 2， <w2> 2， <b1> 2， <b2>2, <W temb >2 and sends them to the server CSP1 and the server CSP2, respectively. < / b1> < / w2> < / w1> < / pe> < / b1> < / w2> < / w1> < / pe> 3. The lightweight NLP prediction method for privacy protection of claim 1, wherein, The step S2 specifically comprises: S21, the server CSP1 holds <W semb >1 and <pe>1. The server CSP2 holds <W semb >2 and <pe>2; split input X into additive secret shares <x>1 and <x>2, and are sent to servers CSP1 and CSP2 respectively;< / x> < / x> < / pe> < / pe> S22, generating two matrices Computing C = F · D ∈ Z 512×n Additively secret share F, D, C and send <f> 1， <d> 1， <c>1 Send to server CSP1, will <f> 2， <d> 2， <c>2 sent to server CSP2; server CSP1 computes 1＝<W semb >1- <f> 1， <v> 1＝ <x> 1- <d>1, and 1 and <v>1 sent to the server CSP2, which computes 2＝<W semb >2- <f> 2， <v> 2＝ <x> 2- <d>2, and 2 and <v>2, and are sent to server CSP1;< / v> < / d> < / x> < / v> < / f> < / v> < / d> < / x> < / v> < / f> < / c> < / d> < / f> < / c> < / d> < / f> S23, the server CSP1 reconstructs U= 1+ 2 and V = 0 <v> 1+ <v>2, the server CSP2 reconstructs U and V in the same way, the server CSP1 computes <f>1 V + U <d> 1+ <c>1 get W semb • a secret share <W semb • X > 1, the server CSP2 computes <f>2V+U <d> 2+ <c>2 get W semb • a secret share <W semb • X > 2;< / c> < / d> < / f> < / c> < / d> < / f> < / v> < / v> S24, the server CSP1 based on <pe>1 and the <W semb • X > 1, compute <s>1＝ <pe>1+<W semb • X > 1, server CSP2 based on <pe>2 and the <W semb • X2calculation <s>2＝ <pe>2 + < W smb • X > 2.< / pe> < / s> < / pe> < / pe> < / s> < / pe> <s> <s> 4. The lightweight NLP prediction method for privacy protection of claim 1, wherein, The step S3 specifically comprises: S31, the server CSP1 based on the input matrix <s>1 and the query matrix parameter <W Q >1, the key matrix parameter <W K >1, the value matrix parameter <W V >1 calculates the query matrix <q>1, key matrix <k>1 and value matrix <v> 1：< / v> < / k> < / q> < / s> <s> <q>1 = < W Q • S > 1< / q> <k>1=<W K • S>1< / k> <v>1=<W V • S>1< / v> By analogy, the server CSP2 bases the input matrix <s>2 and the query matrix parameter <W Q >2, the key matrix parameter <W K >2, the value matrix parameter <W V >2 calculates the query matrix <q>2, key matrix <k>2 and the value matrix <v> 2；< / v> < / k> < / q> < / s> <s> S32, the server CSP1 queries the matrix <q>1 and key matrix <k>1 Dot product operation and divide by Normalization is performed to obtain The server CSP2 performs a query matrix <q>2 and key matrix <k>2 dot product operation is performed and divided by Normalization is performed to obtain < / k> < / q> < / k> < / q> S33, in the server CSP1, the matrix owned by it Each list is represented as [<a 1,i >1...<a n,i >1] T ; in the server CSP2, the matrix owned by it Each list is represented as [<a 1,i >2...<a n,i >2] T , the server CSP1 and the server CSP2 interact to calculate the softmax function, and the softmax value of the i-th column and the t-th row is represented as: Server CSP1 computes Server CSP2 computes for multiplication secret share value; obtaining a randomly generated non-zero real number c, which is divided into <c> 1, <c>2 sum General <c>1 and send to the server CSP1, to <c>2 and Sends to server CSP2, server CSP1 computes Server CSP2 computes Server CSP1 and server CSP2 interact to recover Server CSP1 computes Server CSP2 computes Based on the same method as above, server CSP1 gets Server CSP2 gets < / c> < / c> < / c> < / c> A non-zero number c is randomly chosen and is split into <c>1 and <c>2, distributed to two servers, server CSP1 owns <c>1 and <τ i >1, the server CSP2 owns <c>2 and <τ i >2, two servers respectively calculate and c·τ i >1, server CSP1 owns and <c·τ i >2, two servers respectively calculate and <c·τ i >2, two servers respectively calculate i = <c·τ i >1 + <c·τ i >2, server CSP1 calculates server CSP2 calculates < / c> < / c> < / c> < / c> < / s> < / s> < / s> < / s> <s> <s> <s> S34, server CSP1 calculates 1, and server CSP2 calculates 2： 5. The lightweight NLP prediction method for privacy protection of claim 1, wherein, The step S4 specifically comprises: S41, Server CSP1 based on <s>1 and< / s> <s> 1, obtain <r> 1＝ <s>1+< / s> < / r> <s> 1. The server CSP2 is based on <s>2, and< / s> <s> 2, obtain <r> 2＝ <s>2+< / s> < / r> <s> 2； < / s> < / s> < / s> < / s> <s> <s> <s> S42, the server CSP1 calculates and the server CSP2 calculates and wherein the server CSP1 obtains <u i >1 from the server CSP2 obtains <u i >2 from S421, server CSP1 calculates <m> 1＝<u i >1·2 -λ < / m> <z0>1 = ξ· <m>1+ω< / m> Server CSP2 calculates <m> 2＝<u i >2·2 -λ < / m> <z0>2 = ξ · <m> 2< / m> Wherein, ξ and ω are two constants, ξ = -0.8099868542, ω = 1.787727479, and λ is the length of the bit string; S422, based on <m> 1， <m> 2， <z0>1 and <z0>2, two servers interact with each other to obtain< / z0> < / m> < / m> <t0>1 = <m·z0>1< / t0> <t0>2 = <m·z0>2< / t0> t0＝ <t0> 1+ <t0>2 = m·z0;< / t0> < / t0> S423、For wherein is the number of cycles, both servers cycle through S424-S426; S424、based on <t j >1, <t j >2, <o j >1 and <o j >2, two servers interact with each other, run a secure multiplication operation to obtain <d j >1 and <d j >2, d j = <d j >1 + <d j >2 = t j · o j ; S425, the server CSP1 computes the server CSP2 computes <d j >2 = -<d j >2; S426 based on <t j >1, <t j >2, <o j >1, <o j >2, <d j >1, and <d j >2, two servers interact with each other to obtain <t j+1 >1, <t j+1 >2, <o j+1 >1, and <o j+1 >2, where t j+1 = <t j+1 >1 + <t j+1 >2 = t j · d j , o j+1 = <o j+1 >1 + <o j+1 >2 = o j · d j ; S427, after the loop ends, the server CSP1 gets <t j >1, the server CSP2 gets <t j >2; S428、if λ is even, the server CSP1 and the server CSP2 set a = 1; otherwise, set S429 Server CSP1 and server CSP2 compute Server CSP1 computes Server CSP2 computes Based on <σ i >1, <σ i >2 and column vector r i = [r 1,ir2,i …r 512,i ] T The additive secret sharing values [<r 1,i >1 <r 2,i >1 … <r 512,i >1]T and [<r 1,i >2 <r 2,i >2, …, <r 512,i >2] T of the column vector r i = [r 1,i r 2,i , …, r 512,i ] T are normalized by the two servers: Server CSP1 computes Server CSP2 computes Both servers interact with each other to obtain and where Based on <γ>1, <γ>2, <β>1 and <β>2, two servers interact with each other to obtain and The server CSP1 gets <l> 1＝( <l1> 1， <l2>1,..., <l m >1), the server CSP2 gets <l> 2＝( <l1> 2， <l2> 2，…，<l n >2)。< / l2> < / l1> < / l> < / l1> < / l> 6. The lightweight NLP prediction method for privacy protection of claim 2, wherein, The step S5 specifically comprises: S51, for each column I of the matrix L i , the server CSP1 owns <l i >1 and the parameters <w1> 1， <w2> 1， <b1> 1， <b2>1 ; the server CSP2 owns <l i >2 and parameters <w1> 2， <w2> 2， <b1> 2， <b2>2, two servers calculate the output of the forward propagation layer:< / b2> < / b1> < / w2> < / w1> < / b1> < / w2> < / w1> Based on <w1> 1，<l i >1， <b1>1 and <w1> 2，<l i >2， <b1>2, two servers interact with each other to obtain< / b1> < / w1> < / w1> <θ i >1= <W1 · l i >1+ <b1> 1< / b1> <θ i >2= <W1·l i >2+ <b1> 2< / b1> θ i = <θ i > 1 + <θ i > 2 = W1 · l i + b1 Based on <θ i >1 and <θ i >2, two servers interact, compare each vector θ i Each element in the vector The size relationship with 0, k ∈ [1, 512], judge the sign of the highest bit, 1 is negative, 0 is positive; S52, the server CSP1 for expressed as a binary bit string λ is the length of the bit string, and the same server CSP2 for expressed as a binary bit string For Each bit CSP1 settings CSP2 settings For Each bit CSP1 setting CSP2 setting Both parties set an initial carry bit S53, The carry for the j+1 position can be expressed as: S54、based on and The server CSP1 and the server CSP2 calculate step S54 λ-1 times and S55, the server CSP1 calculates the server CSP2 calculates S56, the server CSP1 calculates the server CSP2 calculates S57, the server CSP1 sets and the server CSP2 sets and S58, both parties run a secure multiplication protocol to compute and where S59, both parties run a secure multiplication protocol to compute and where S510, for θ i each element in the middle S53-S59, and finally the server CSP1 gets a column vector <ff i >1, and the server CSP2 gets a column vector <ff i >2, based on <w2>1, <ff i >1, <b2>1 and <w2>2, <ff i >2, <b2>2, both servers interact to obtain <f i >1 and <f i >2, f i = <f i >1 + <f i >2 = W2·ff i +b2, for each column in the matrix L, the server CSP1 obtains <f>1. The server CSP2 gets <f> 2。< / f> < / f> 7. A lightweight NLP prediction system for privacy protection, characterized in that, The system is suitable for the lightweight NLP prediction method for protecting privacy according to any one of claims 1-6, and the system comprises: A user terminal, a cloud server and a model providing terminal, the user terminal and the model providing terminal are connected with the cloud server, the cloud server is at least two, and the cloud servers do not collude with each other; The model providing terminal sends the trained model parameters to the cloud server in an additive sharing manner, the user terminal also sends the data to be predicted to the cloud server in an additive sharing manner, and multiple cloud servers interact with each other to perform prediction. < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s>

Images