A novel privacy activation function design method based on secure multi-party computation

Abstract

This invention provides a novel privacy-preserving activation function design method based on secure multi-party computation, comprising: analyzing the activation function in a neural network to determine its mathematical operation steps and algebraic properties; selecting a corresponding basic secure multi-party computation component based on the mathematical operation steps and algebraic properties; modifying the activation function using the selected basic secure multi-party computation component to adapt the modified activation function to the computational framework required for secure multi-party computation, thereby enabling computation on secret input data; and ensuring that each participant in the modified activation function holds additive slices of secret data during input and output, so that the modified activation function can connect to other linear or nonlinear modules in the neural network before and after it. This invention protects data privacy while ensuring data availability, promoting the development of data sharing and privacy-preserving computation.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and information security technology, specifically to a novel privacy activation function design method based on secure multi-party computation. Background Technology

[0002] Neural networks (NNs) are a prime example of artificial intelligence technology, widely used across various industries to perform tasks such as classification, regression, and prediction. However, using neural networks typically requires inputting data in plaintext. This data often contains sensitive personal information such as medical records and financial data, as well as internal company information like financial details and source code. Therefore, protecting the privacy of data within neural networks is crucial for individuals and businesses while performing calculations.

[0003] Secure Multiparty Computation (MPC) is a cryptographic technique used to solve the problem of a group of distrustful parties, each holding secret data, collaboratively computing a predetermined function. Therefore, this technique provides a mechanism for protecting data privacy. Specifically, the data holder secretly shares the data, splitting it into multiple parts and storing them with different participants. Each part is shaped like a random number. Secure multiparty computation allows these participants to jointly perform neural network calculations without disclosing their respective data shares. This method enables parties to share data and collaborate on computations using a model without worrying about revealing sensitive information contained within the complete data, achieving "usable but invisible" data.

[0004] Furthermore, in the structure of neural networks, activation functions, as key nonlinear components, determine the network's ability to learn and represent complexity. However, achieving cooperative computation of nonlinear functions in secure multi-party computation is a major challenge. In particular, the emergence of novel activation functions with complex forms in recent years has introduced complexity into linear approximation or decomposition, making traditional secure multi-party computation methods difficult to apply directly.

[0005] To address this challenge, there is an urgent need to develop privacy-preserving computation methods designed for novel activation function structures, enabling them to adapt to the computational frameworks required for secure multi-party computation. Summary of the Invention

[0006] In view of this, the present invention provides a novel privacy activation function design method based on secure multi-party computation, which at least solves the above-mentioned technical problems, promotes a wider range of data sharing and privacy protection applications, and provides stronger support for the application of secure multi-party computation in the field of neural networks.

[0007] The first aspect of this invention provides a novel privacy activation function design method based on secure multi-party computation, comprising: analyzing the activation function in a neural network to determine the mathematical operation steps and algebraic properties of the activation function; selecting a corresponding basic secure multi-party computation component based on the mathematical operation steps and algebraic properties; modifying the activation function using the selected basic secure multi-party computation component to adapt the modified activation function to the computation framework required for secure multi-party computation, thereby realizing the computation of secret input data; and maintaining that each participant holds an additive slice of secret data during input and output of the modified activation function, so that the modified activation function can connect to other linear or nonlinear modules in the neural network before and after it.

[0008] According to a second aspect of the present invention, a privacy-preserving neural network computing system is provided, comprising a data holder, a model holder, and multiple participants, characterized in that: the data holder is used to secretly share data, split it into multiple additive fragments, and send the fragments to each participant; the model holder is used to determine the neural network structure, train the neural network, obtain the parameters of each layer, and publish the parameters to the data holder; the multiple participants are used to receive the data fragments sent by the data holder, and according to the neural network structure and parameters published by the model holder, perform inference computation of the neural network using the novel privacy activation function design method based on secure multi-party computation as described in the first aspect above, including linear layer computation and nonlinear layer computation, and finally send the computation results back to the data holder.

[0009] In summary, the present invention enables parties to share data and collaborate on calculations using the model without worrying about leaking sensitive information contained in the complete data, achieving "usable but invisible" data and improving data security. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the privacy neural network calculation process of the present invention.

[0011] Figure 2 This is a schematic diagram of the traditional neural network computation process.

[0012] Figure 3 This is a schematic diagram of the input layer, hidden layer, and output layer of the privacy neural network of the present invention.

[0013] Figure 4 This is a flowchart of the steps of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] To facilitate understanding, before describing the specific embodiments of the present invention in detail, the technical terms involved in the present invention will be explained as follows:

[0016] 1. Representation of the secret shared value:

[0017] For each secret value u that needs to be shared, n participants share the secret, using This represents the state of secret u after it has been shared with the addition secret. i Indicates participant P i The held addition fragment, denoted by [u], represents the state after the secret u is shared with the multiplication secret. i Indicates participant P i The multiplication slices held. At this point, we have:

[0018] u= 1+ 2+…+ i And u=[u]1*[u]2*...*[u] i ,

[0019] i ±<v> i =<u±v> i c* i =<c*u> i

[0020] [u] i *[v] i =[u*v] i ,

[0021] 2. Element-wise multiplication protocol in addition-shared state:

[0022] Suppose n participants want to jointly compute the result z of x*y without directly disclosing their inputs x and y. Using the Beaver Triple protocol, they can achieve element-wise multiplication in a shared addition state. Specifically, participant P... i Addition slices holding x <x> i Addition of y in slices <y> i Perform the following operations:

[0023] (1) Each participant P i Get a set of Beaver triples of fragments ( i ， i ， <c> i ), where a and b are random numbers, and c = a * b. In this case, we have

[0024] (2) Each participant P i Calculated locally <e> i ＝ <x> i -< / x> < / e> < / c> i and <f> i ＝ <y> i - i They then send their calculations to other participants, after which each participant can calculate... and

[0025] (3) Setting the last participant P1 <z> 1 = e*f + f*< / z> < / y> < / f> 1+e* 1+ <c>1. Participant P i i = 2, 3, ..., n, set <z> i ＝f*< / z> < / c> i +e* i + <c> i That is, to obtain the final calculation result, which satisfies

[0026] 3. Element-wise multiplication protocol in addition-shared state:

[0027] Suppose n participants want to convert multiplicative partitions of the secret value K into additive partitions. Using the protocol proposed by Josef Pieprzyk, Hossein Ghodosi, Ron Steinfeld, et al. in "Multi-Party Computation with Conversion of Secret Sharing" (2011), they can achieve the computation of additive partitions in a multiplicative-shared state. Specifically, participant P... j Holding the multiplication piece [m] of K j That is, at this point, we have [m]1*[m]2*...*[m] n =K, where the value of K is not publicly disclosed, and j = 1, 2, ..., n. In the computation steps of this protocol, let m be... i =[m] i .

[0028] The protocol requires the following auxiliary inputs: each participant P j Having n elements a 1，j , ..., a n，j Its satisfaction in The above a i，j Generate in pre-computation as follows:

[0029] (1) Select independent and uniform random numbers u1, u2, ..., u n-1 and calculate

[0030] (2) For each i = 1, ..., n, select n-1 independent uniform random numbers {a i，j } i≠j And calculate a i，i =u i *(∏ j≠i a i，j ) -1 .

[0031] Once the conversion protocol begins, each participant P j (j = 1, ..., n) send v i，j =a i，j m j (i = 1, ..., n) are given to participant P. i Afterwards, participant P i Calculate (i = 1, ..., n) At this time s i That is, the additive partitioning of the secret value K, because we have as well as

[0032] 4. Security Comparison Protocol:

[0033] This protocol is used for secure multi-party computation of piecewise functions involved in this invention, and it has different implementations, such as the truncation method proposed by Mohassel and Rindal et al. in their 2018 paper "ABY3: A Mixed Protocol Framework for Machine Learning". Specifically, the protocol input is a shared secret value. <z>, of which -2 k <<Z<<2 k ,return- ,in Protocol Analysis: If -2 k If ≤Z<0, then so If 0 ≤ Z < 2 k ,but so In other words, the security comparison protocol can be input. <z>Calculation output If Z < 0, then b = 1; if Z ≥ 0, then b = 0.

[0034] 5. ReLU activation function and Private ReLU:

[0035] The ReLU (Rectified Linear Unit) activation function is a classic non-linear activation function. Its widespread application in deep learning began around 2010, primarily proposed and studied by Hinton et al. in their paper "Rectified Linear Units Improve Restricted Boltzmann Machines". It is defined as ReLU(x) = max(0, x). That is, when the input x ≤ 0, the output is 0; when x > 0, the output is x.

[0036] The ReLU activation function is useful for introducing nonlinearity, helping neural network models learn complex nonlinear relationships. The calculation of the ReLU activation function is very simple, requiring only a threshold determination, which makes neural network training faster and alleviates the vanishing gradient problem, making gradient propagation more efficient and making deep neural networks easier to train. At the same time, the properties of the ReLU function make neuron activation more sparse, which can help the network generalize and reduce overfitting.

[0037] The ReLU activation function is widely used in deep learning, especially for processing large-scale data and deep neural networks, which can accelerate the training process and improve model performance.

[0038] In 2020, Kumar et al. proposed the concept of Private ReLU in their paper "Cryptflow: Secure Tensorflow Inference":

[0039] Suppose n participants want to collaboratively compute the result of a neural network containing the non-linear activation function ReLU, without disclosing their individual inputs. By using the Private ReLU protocol shown below, they can achieve privacy-preserving ReLU function computation, thus realizing complete privacy-preserving neural network computation. Specifically, when the neural network reaches the ReLU function computation step, participant P... i The addition slice holding the function input x <x> i At this point, it is satisfied. <x> 1+ <x> 2+…+ <x> n =x, and x is a secret value for the participants, perform the following operations:

[0040] (1) Using a secure comparison protocol, input <x>Get output At this point, n participants each hold Fragments 1， 2，...， n ;

[0041] (2) All participants utilize the Beaver Triple protocol to <x>Given <1-b> as input, calculate<x*(1-b)> Slicing, i.e., participant P i get <z> i ,make <z> 1+ <z> 2+…+ <z> n = x*(1-b).

[0042] The specific implementation of the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0043] See< / z> < / z> < / z> < / z> < / x> < / x> < / x> < / x> < / x> < / x> < / z> < / z> < / c> Figure 4 This invention provides a novel privacy activation function design method based on secure multi-party computation, comprising:

[0044] S1. Analyze the activation functions in the neural network to determine the mathematical operation steps and algebraic properties of the activation functions;

[0045] S2. Based on the mathematical operation steps and algebraic properties, select the corresponding basic secure multi-party calculation component;

[0046] S3. Modify the activation function using the selected basic secure multi-party computation component to adapt the modified activation function to the computation framework required for secure multi-party computation, thereby enabling the computation of secret input data.

[0047] S4. Ensure that each participant holds an additive slice of secret data during the input and output of the modified activation function, so that the modified activation function can connect to other linear or nonlinear modules in the neural network.

[0048] Optionally, the basic secure multi-party computation component includes one or more of the following: a representation of a secret shared value, an element-wise multiplication protocol in an addition-shared state, an addition fragmentation conversion protocol in an addition-shared state, and a secure comparison protocol.

[0049] Optionally, the activation function is one or more of Leaky ReLU, Elu, Celu, Hardsigmoid, Hardtanh, SiLU, GeLU, and Mish, and the corresponding modified activation functions are Private Leaky ReLU, Private Elu, Private Celu, Private Hardsigmoid, Private Hardtanh, Private SiLU, Private GeLU, and Private Mish, respectively.

[0050] Optionally, for the Leaky ReLU activation function, the specific steps for implementing the Private Leaky ReLU function are as follows: when the neural network reaches the Leaky ReLU function calculation step, participant P... i The addition slice holding the function input x <x> i ,Right now <x> 1+ <x> 2+…+ <x> n =x, where x is a secret value for the participants, and the following operations are performed: using a secure comparison protocol, input... <x>Get output At this time, each participant holds Fragments 1， 2，...， n The Beaver Triple protocol was used twice to calculate <0.1x*b> and <0.1x*b> respectively.<x*(1-b)> Sharding, followed by local computation by each participant. <z> i =<b*0.1x> i +<x*(1-b)> i .

[0051] Optionally, for the Elu activation function, the specific steps to implement the Private Elu function are as follows: when the neural network reaches the Elu function calculation step, participant P... i The addition slice holding the function input x <x> i ,Right now <x> 1+ <x> 2+…+ <x> n =x, where x is a secret value for the participants, and the following operation is performed: Participant P i Local computing get For e x Multiplication by pieces, notation All participants P i The protocol of converting multiplication fragmentation to addition fragmentation will be used together to convert [e] x ] i Convert to <e x > i Then each participant completes a linear computation locally, obtaining... <v>=a*(e <x>< / x> -1); Using a secure comparison protocol, input... <x>Get output Finally, using the BeaverTriple protocol twice and addition, we obtain... <z> ＝ * <v> +(1- )* <x>.

[0052] Optionally, for the SiLU activation function, the specific steps to implement the Private SiLU function are as follows: when the neural network reaches the SiLU function calculation step, participant P... i The addition slice holding the function input x <x> i ,Right now <x> 1+ <x> 2+…+ <x> n =x, where x is a secret value for the participants, and the following operation is performed: call with <x> i For the input PrivateSigmoid protocol, participant P i get i ,make 1+ 2+…+ n =sigmoid(x); using the Beaver Triple protocol, find<x*a> Slicing, i.e., participant P i get <z> i ,make <z> 1+ <z> 2+…+ <z> n = x*sigmoid(x).

[0053] Optionally, the specific execution steps of the Private Sigmoid protocol include: each participant generating a random number, which is used as a multiplication slice of the random number ρ, i.e., participant P i Holding [ρ] i This satisfies [ρ]1*[ρ]2*...*[ρ] n =ρ, and then use the multiplication-based fragmentation protocol to convert to addition-based fragmentation to calculate <ξ> = [ρ], that is, at this time ξ = <ξ>1 + <ξ>2 + ... + <ξ> n =ρ; Participant P i Local computing Then all participants will convert [k] by calling the multiplication-addition conversion protocol. i Convert to <φ> i That is, we get e x *ρ addition piecewise <φ> i Each participating party P i Local calculation<χ> i =<φ> i -<ξ> i And obtain χ through the reconstruction algorithm, at which point χ=φ-ξ=e x *ρ-ρ; Each participating party P i Local computing The output result is <y> i .

[0054] Optionally, when calculating the privacy activation function, the participants may include one or more of the following: cloud servers, cloud storage devices, and distributed nodes.

[0055] Another aspect of this invention provides a privacy-preserving neural network computing system, comprising:

[0056] The data holder is responsible for secretly sharing the data, splitting it into multiple additive shards, and sending the shards to each participating party.

[0057] The model holder is responsible for determining the neural network structure, training the neural network, obtaining the parameters of each layer, and publishing the parameters to the data holder.

[0058] Multiple parties are involved in receiving data fragments sent by the data holder. Based on the neural network structure and parameters published by the model holder, they perform inference computation of the neural network using the novel privacy activation function design method based on secure multi-party computation as described in the first aspect above, including linear layer computation and nonlinear layer computation. Finally, the computation results are sent back to the data holder.

[0059] Specifically, the solution of the present invention is further described with reference to the following examples:

[0060] This invention presents a novel privacy-preserving activation function design method based on secure multi-party computation. Addressing the different structures and computational steps of several representative novel activation functions in the field of artificial intelligence, it utilizes the five protocols listed above as fundamental secure multi-party computation components. Table 1 summarizes the correspondence between the improved activation function and the original function presented in this invention:

[0061] Table 1

[0062] Existing activation functions This invention proposes Leaky ReLU Private Leaky ReLU Elu Private Elu Celu PrivateCelu Hardsigmoid Private Hardsigmoid Hardtanh Private Hardtanh SiLU Private SiLU GeLU Private GeLU Mish Private Mish

[0063] The innovations of this method can be summarized as follows: (1) This method can maintain the mathematical operation steps and algebraic properties of each activation function, thus inheriting the design advantages of each activation function. At the same time, based on secure multi-party computation technology, it realizes the computation of secret input data; (2) This method maintains that each participant holds the addition slice of secret data when the function is input and output, so that it can be connected to other linear or nonlinear modules in the neural network before and after; (3) The new activation function contains nonlinear function components, such as logarithmic function, exponential function, reciprocal function, etc., while the classic secure multi-party computation protocol can only complete the calculation of addition and multiplication. Therefore, the secure multi-party computation of these nonlinear functions is still a challenge. In recent years, some scholars have proposed to use the idea of ​​approximate fitting to transform the secure computation problem of nonlinear functions into the secure computation problem of low-order polynomials. However, such methods will cause a loss of function accuracy. In contrast, the method proposed in this invention does not contain approximation in the calculation steps, and is an accurate calculation without any loss of accuracy.

[0064] The specific design and implementation methods for each function are as follows:

[0065] Leaky ReLU activation function

[0066] The Leaky ReLU activation function (Leaky Rectified Linear Unit) is a variant of the ReLU activation function, first proposed in 2013 by Maas et al. in a paper titled "Rectifier Nonlinearities Improve Neural Network Acoustic Models". This function sets the negative region of the ReLU function as a gradient-sloping line; when the input x ≤ 0, the output is 0.1x; when x > 0, the output is x. Its mathematical expression can be represented as:

[0067] The Leaky ReLU activation function was designed primarily to address the issue of gradients vanishing to zero in the negative regions of the ReLU function. This helps prevent neuron "death," where some neurons are never activated during training. Because it has a constant slope, its gradient does not become zero in the negative regions, potentially making neural network training more stable. Compared to some traditional activation functions like Sigmoid or Tanh, the Leaky ReLU activation function is linear in most regions, thus reducing the risk of gradient vanishing. It is also relatively simple to compute and typically performs well in practice.

[0068] This invention proposes a Private Leaky ReLU function, which is a privacy-preserving Leaky ReLU function that inherits the design advantages of the Leaky ReLU function. Furthermore, it utilizes secure multi-party computation technology to perform computation on secret input data. In addition, this method ensures that each participant holds an additive slice of the secret data during both input and output, allowing it to be connected to other linear or nonlinear modules in the neural network.

[0069] Private Leaky ReLU

[0070] When the neural network reaches the Leaky ReLU function calculation step, participant P... i The addition slice holding the function input x <x> i ,Right now <x> 1+ <x> 2+…+ <x> n =x, where x is a secret value for the participants, and the following operations are performed:

[0071] (1) Using a secure comparison protocol, input <x>Get output At this time, each participant holds Fragments 1, 2,..., n

[0072] (2) Calculate <0.1x*b> and <0.1x*b> twice using the Beaver Triple protocol.<x*(1-b)> Sharding, followed by local computation by each participant. <z> i =<b*0.1x> i +<x*(1-b)> i

[0073] Elu activation function

[0074] The Elu (Exponential Linear Unit) activation function was first proposed by Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter in their 2015 paper "Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)," and its definition is as follows:

[0075]

[0076] The advantages of the Elu activation function include: the Elu function and its derivative are smooth over the entire domain, which is beneficial for gradient calculation and model optimization; it also has gradients in the negative part, avoiding the zero gradient problem, which helps improve the convergence speed of the network and the learning efficiency of the model; compared with some traditional activation functions, Elu is better at preserving information when dealing with negative inputs, preventing neuron death.

[0077] This invention proposes a Private Elu function, which is an Elu function with privacy protection features. It inherits the design advantages of the Elu function and, based on secure multi-party computation technology, enables computation on secret input data. Furthermore, this method ensures that each participant holds additive slices of the secret data during input and output, so that these slices can be connected to other linear or nonlinear modules in the neural network.

[0078] Private Elu

[0079] When the neural network reaches the Elu function calculation step, participant P i The addition slice holding the function input x <x> i ,Right now <x> 1+ <x> 2+…+ <x> n =x, where x is a secret value for the participants, and the following operations are performed:

[0080] (1) Participant P i Local computing At this time there is Right now, For e x Multiplication by pieces, notation

[0081] (2) All participants P i The protocol of converting multiplication fragmentation to addition fragmentation will be used together to convert [e] x ] i Convert to <e x > i Then each performs linear calculations locally to obtain... <v>=a*(e <x>< / x> -1);

[0082] (3) Using a secure comparison protocol, input <x>Get output

[0083] (4) Finally, using the Beaver Triple protocol twice and addition, we obtain:

[0084] <z> ＝ * <v> +(1- * <x>

[0085] Celu activation function

[0086] The CELU (Continuously Differentiable Exponential Linear Units) activation function is often used in hidden layers of neural networks to enhance the network's expressive power by introducing nonlinear transformations. It was first proposed by Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter in their 2015 paper, "Fast Accurate Deep Network Learning by Exponential Linear Units (ELUs)". CELU is a variant of ELU, but relatively smoother due to its exponential function property on the negative half-axis. Its formula is as follows:

[0087]

[0088] The Celu activation function's advantage over other activation functions lies in its smoothness and favorable properties, making it more stable during training. Furthermore, Celu's characteristics help alleviate the vanishing gradient problem and are particularly effective for networks with a large number of negative inputs. In practice, Celu can improve the model's generalization ability, reduce overfitting, and improve convergence speed during training.

[0089] This invention proposes a Private Celu function, a privacy-preserving Celu function that inherits the design advantages of the Celu function and, based on secure multi-party computation technology, enables computation on secret input data. Furthermore, this method ensures that each participant holds additive slices of the secret data during input and output, allowing for the connection of these slices to other linear or nonlinear modules in the neural network.

[0090] Private Celu

[0091] When the neural network reaches the CeLU function calculation step, participant P i The addition slice holding the function input x <x> i ,Right now <x> 1+ <x> 2+…+ <x> n =x, where x is a secret value for the participants, and the following operations are performed:

[0092] (1) Participant P i Local computing in It is a constant; P i Continue calculation At this time, there are Therefore, in fact, for Multiplication by pieces, notation

[0093] (2) All participants use the multiplication-to-addition partitioning protocol to perform calculations. <y>=[u], that is, at this time

[0094] (3) Then calculate <z> ＝a*( <y>-1), and will <z>As input, the secure comparison protocol is invoked to obtain...

[0095] (4) Then, the Beaver Triple protocol is used to obtain

[0096] (5) The calculation steps for max{0, x} are the same as those for the Private ReLU function, i.e., the participant P i get <z> i ,make <z> 1+ <z> 2+…+ <z> n =max{0, x}

[0097] (6) Participant P i The final result is obtained by summing the fragments of the calculation results obtained in steps (4) and (5) locally, i.e., at this point we have Hardsigmoid activation function

[0098]

[0099] The characteristic of this activation function is that it outputs 0 when the input is less than -3, 1 when the input is greater than 3, and when the input is between -3 and 3, it outputs according to the linear relationship slope*x+offset. Here, slope is the coefficient (slope), x is the input value, and offset is the bias term (constant term).

[0100] The Hardsigmoid function is an approximate and computationally simple activation function commonly used in neural networks. It was first introduced and used in the AlexNet model by Krizhevsky et al. in 2013. The Hardsigmoid function is computationally simple, primarily based on linear operations and threshold truncation, making it suitable for resource-constrained environments such as embedded systems, mobile devices, and edge computing. Furthermore, due to its simplicity, the Hardsigmoid function is generally more numerically stable and less prone to gradient explosion or vanishing gradient problems.

[0101] This invention proposes a Private Hardsigmoid function, which is a privacy-preserving Hardsigmoid function that inherits the design advantages of the Hardsigmoid function. Furthermore, it utilizes secure multi-party computation technology to perform computation on secret input data. In addition, this method ensures that each participant holds an additive slice of the secret data during both input and output, allowing it to be connected to other linear or nonlinear modules in the neural network.

[0102] Private Hardsigmoid

[0103] When the neural network reaches the Hardsigmoid function calculation step, participant P... i The addition slice holding the function input x <x> i ,Right now <x> 1+ <x> 2+…+ <x> n =x, where x is a secret value for the participants, and the following operations are performed:

[0104] (1) Calculations by all parties <y> ＝ <x>+3 slices and <z> ＝ <x>-3 slices;

[0105] (2) Using a secure comparison protocol, input <y>Get output Fragments 1， 2，...， n When y≤0, that is, when x≤-3, b=1; conversely, when x>-3, b=0.

[0106] (3) Use the security comparison protocol again to input... <z>Get output <c>Fragments <c> 1， <c> 2，...， <c> n When z ≤ 0, that is, when x ≤ 3, c = 1; conversely, when x > 3, c = 0.

[0107] (4) Obtain using the Beaver Triple protocol< / c> < / c> < / c> < / c> < / z> < / y> < / x> < / z> < / x> < / y> < / x> < / x> < / x> < / x> < / z> < / z> < / z> < / z> < / z> < / y> < / z> < / y> < / x> < / x> < / x> < / x> < / x> < / v> < / z> < / x> < / v> < / x> < / x> < / x> < / x> < / z> < / x> < / x> < / x> < / x> < / x> < / y> < / z> < / z> < / z> < / z> ＝(1- )* <c>, at this time, there is: when -3 < x ≤ 3, a = 1, and in other cases, a = 0;

[0108] (5) Obtain by using one local quantity multiplication, one local addition, one Beaver Triple protocol, and another local addition <z> ＝ <c>+(slope* <x> +offset)*< / x> < / c> < / z> < / c> .

[0109] Hardtanh activation function

[0110] The Hardtanh activation function, first proposed by Yann LeCun in 2015, is a simple activation function defined as follows:

[0111]

[0112] The Hardtanh function is a simple and intuitive activation function that requires no additional parameters and is easy to implement and compute. Furthermore, it has an output range over non-linear regions, allowing for effective control over the activation output range, which helps neural network models learn complex data patterns. Moreover, since Hardtanh is a hard-saturating activation function, it can be used to clip gradients, helping to prevent gradient explosion and contributing to network stability and convergence.

[0113] This invention proposes a Private Hardtanh function, which is a privacy-preserving Hardtanh function that inherits the design advantages of the Hardtanh function. Furthermore, it utilizes secure multi-party computation technology to perform computation on secret input data. In addition, this method ensures that each participant holds additive slices of the secret data during input and output, allowing them to be connected to other linear or nonlinear modules in the neural network.

[0114] Private Hardtanh

[0115] When the neural network reaches the Hardtanh function calculation step, participant P... i The addition slice holding the function input x <x> i ,Right now <x> 1+ <x> 2+…+ <x> n =x, where x is a secret value for the participants, and the following operations are performed:

[0116] (1) Calculations by all parties <y> ＝ <x>-max partitioning and <z>=min- <x>Fragmentation;

[0117] (2) Using a security comparison protocol, input <y>Get output Fragments 1， 2，...， n ; At this time, when y < 0, that is, when x < max, b = 1, and conversely when x ≥ max, b = 0;

[0118] (3) Use the secure comparison protocol again, input <z>Get output <c>Fragments <c> 1， <c> 2，...， <c> n When z < 0, i.e., when x > min, c = 1; conversely, when x ≤ min, c = 0.

[0119] (4) Obtain using the Beaver Triple protocol< / c> < / c> < / c> < / c> < / z> < / y> < / x> < / z> < / x> < / y> < / x> < / x> < / x> < / x> ＝ * <c>At this point, we have: when min < x < max, a = 1; otherwise, a = 0.

[0120] (5) Calculate using basic protocols such as Beaver Triple. <z>=max*(1- )+min*(1- <c> )+ <x> *< / x> < / c> < / z> < / c> .

[0121] SiLU activation function

[0122] The SiLU (Sigmoid Linear Unit) activation function, also known as the Swish function, was first proposed by Belgian researchers Dan Hendrycks and Kevin Gimpel in a 2016 paper titled "Gaussian Error Linear Units (GELUs)," and is defined as follows:

[0123] SiLU(x) = x * sigmoid(x)

[0124] The sigmoid function is a sigmoid function (logistic function) defined as follows:

[0125]

[0126] The SiLU activation function, as a novel continuously differentiable nonlinear activation function, combines the characteristics of the sigmoid and linear functions. It possesses excellent smoothness properties, with its gradient being continuous relative to the input, avoiding the vanishing gradient problem on the negative half-axis seen with the ReLU function. This is beneficial for the optimization and training of neural networks. Furthermore, the SiLU activation function is parameterized, allowing its shape to be controlled by adjusting parameters, which helps adapt to different types of data and task characteristics.

[0127] This invention proposes a Private SiLU function, which is a privacy-preserving SiLU function that inherits the design advantages of the SiLU function. Furthermore, based on secure multi-party computation (SMC) technology, it achieves computation on secret input data. In addition, this method ensures that each participant holds an additive slice of the secret data during input and output, so that it can be connected to other linear or nonlinear modules in the neural network. The concept of using secure multi-party computation to implement a privacy-preserving SiLU function is mentioned in the patent "A Privacy-Preserving Inference Method and System for Diffusion Model Sampling" proposed by Chen Xiaojun et al., but the core idea of ​​the method used by Chen Xiaojun et al. is to use Chebyshev polynomial fitting of the exponential function e. x Furthermore, the SiLU function is decomposed into a combination of linear operations, and the computation is directly completed using the Secure Multi-Party Computation Protocol. This approximate computation method results in a loss of accuracy, while the method designed in this invention provides precise computation without any loss of accuracy during execution.

[0128] Private SiLU

[0129] When the neural network reaches the SiLU function calculation step, participant P... i The addition slice holding the function input x <x> i ,Right now <x> 1+ <x> 2+…+ <x> n =x, where x is a secret value for the participants, and the following operations are performed:

[0130] (1) Call with <x> i For the input Private Sigmoid protocol, participant P i get< / x> < / x> < / x> < / x> < / x> i ,make 1+ 2+…+ n = sigmoid(x)

[0131] (2) Using the Beaver Triple protocol, find<x*a> Slicing, i.e., participant P i get <z> i ,make <z> 1+ <z> 2+…+ <z> n = x * sigmoid(x)

[0132] The Private Sigmoid protocol mentioned in step (1) was proposed in 2020 by Qi Feng, Debiao He, Zhe Liu, Huaqun Wang, and Kim-Kwang Raymond Choo in their paper "SecureNLP: A System for Multi-Party Privacy-Preserving Natural Language Processing". Its input is the additive slice of x held by each participant, i.e., participant P. i hold <x> i ,satisfy <x> 1+ <x> 2+…+ <x> n =x, and the specific steps are as follows:

[0133] (1) Each participant generates a random number, which is used as a multiplication partition of the random number ρ, i.e., participant P i Holding [ρ] i This satisfies [ρ]1*[ρ]2*…*[ρ] n =ρ, and then use the multiplication-based fragmentation protocol to convert to addition-based fragmentation to calculate <ξ> = [ρ], that is, at this time ξ = <ξ>1 + <ξ>2 + ... + <ξ> n =ρ

[0134] (2) Participant P i Local computing Then all participants will convert [k] by calling the multiplication-addition conversion protocol. i Convert to <φ> i That is, we get e x *ρ addition piecewise <φ> i

[0135] (3) Each participating party P i Local calculation<χ> i =<φ> i -<ξ> i And obtain χ through the reconstruction algorithm, at which point χ=φ-ξ=e x *ρ-ρ

[0136] (4) Each participating party P i Local computing The output result is <y> i

[0137] GeLU activation function

[0138] The GeLU (Gaussian Error Linear Units) activation function was published in 2016 by Belgian researchers Dan Hendrycks and Kevin Gimpel in their paper "Gaussian Error Linear Units (GELUs)". It is defined as follows:

[0139]

[0140] The error function erf is a special mathematical function that calculates the integral value of the normal distribution curve from negative infinity to x, i.e., the probability that the variable falls within a certain range. Its form is: If we use approximate calculations, we have:

[0141]

[0142] The GeLU activation function is a smooth function, avoiding the gradient saturation problem of functions like ReLU on large or small input values, which is beneficial for the stability of backpropagation and gradient descent. As a non-linear activation function, GeLU enables neural networks to learn complex patterns and features, and is widely used in various neural network architectures. GeLU has been widely applied in various deep learning models, especially in the field of Natural Language Processing (NLP), for example, in models such as BERT and GPT-2, where the GeLU activation function is used.

[0143] This invention proposes a Private GeLU function, which is a privacy-preserving GeLU function that inherits the design advantages of the GeLU function and, based on secure multi-party computation (SMC) technology, performs computation on secret input data. Furthermore, this method ensures that each participant holds an additive slice of the secret data during both input and output, allowing it to be connected to other linear or nonlinear modules in the neural network. The patent "A Privacy-Preserving Entity Recognition Tool Based on Secure Multi-Party Computation Technology" proposed by Li Mu et al. touches upon a secure multi-party computation method for the GeLU function, but differs from the scheme proposed in this invention in two ways: first, it sets the approximation of the GELU function to x*sigmoid(1.702x), unlike this invention; second, it uses the Newton-Raphson iterative method to optimize the polynomial approximation of the reciprocal, while this invention does not use approximation during computation, reducing accuracy loss.

[0144] Private GeLU

[0145] When using approximate calculations, when the neural network reaches the GeLU function calculation step, participant P... i The addition slice holding the function input x <x> i ,Right now <x> 1+ <x> 2+…+ <x> n =x, where x is a secret value for the participants, and the following operations are performed:

[0146] (1) Using the Beaver Triple protocol, find<x*x> Slicing, i.e., participant P i get <x 2 > i ,make <x 2 >1+ <x 2 >2+…+ <x 2 > n =x*x=x 2

[0147] (2) Using the Beaver Triple protocol again, find <x 2 *x>slice, i.e., participant P i get <x 3 > i ,make <x 3 >1+ <x 3 >2+…+ <x 3 > n =x 2 *x=x 3

[0148] (3) Afterwards, each participating party P i Local computation

[0149] (4) Call the Private Tanh protocol participant P i get i ,make:

[0150] 1+ 2+…+ n =Fish < / x> < / x> < / x> < / x> < / y> < / x> < / x> < / x> < / x> < / z> < / z> < / z> < / z> i

[0151] (5) Afterwards, each participating party P i Local computation <c> i ＝0.5*( i +1)

[0152] (6) Finally, using the Beaver Triple protocol, we can find...<x*c> Slicing, i.e., participant P i get <z> i ,make:

[0153]

[0154] The Private Tanh protocol mentioned in step (4) is consistent with the aforementioned Private Sigmoid protocol, both of which were proposed in the paper "SecureNLP: A System for Multi-Party Privacy-Preserving NaturalLanguage Processing". Its input is the additive fragment of x held by each participant, i.e., participant P. i hold <x> i ,satisfy <x> 1+ <x> 2+…+ <x> n =x, and the specific steps are as follows:

[0155] (1) Each participant generates a random number, which is used as a multiplication partition of the random number ρ, i.e., participant P i Holding [ρ] i This satisfies [ρ]1*[ρ]2*...*[ρ] n =ρ, and then use the multiplication-based fragmentation protocol to convert to addition-based fragmentation to calculate <ξ> = [ρ], that is, at this time ξ = <ξ>1 + <ξ>2 + ... + <ξ> n =ρ

[0156] (2) Participant P i Local computing Then all participants will convert [k] by calling the multiplication-addition conversion protocol. i Convert to <φ> i That is, we get e 2x *ρ addition piecewise <φ> i

[0157] (3) Each participating party P i Local calculation<χ> i =<φ> i +<ξ> i And obtain χ through the reconstruction algorithm, at which point χ=φ+ξ=e x *ρ+ρ

[0158] (4) Each participating party P i Local computing The output result is <y> i

[0159] Mish activation function

[0160] The Mish activation function was first published by Indonesian researcher Diganta Misra in his 2019 paper "Mish: ASelf Regularized Non-Monotonic Neural Activation Function," and it is defined as follows:

[0161] Mish(x) = x * tanh(ln(1+e)) x ))

[0162] The tanh(x) function is the hyperbolic tangent function, defined as follows:

[0163]

[0164] The Mish activation function combines smoothness and non-monotonicity, which makes it superior to traditional activation functions such as ReLU in some cases. At the same time, the Mish function has a certain degree of regularization, which helps to improve the generalization ability of the model and reduce the risk of overfitting.

[0165] This invention proposes a Private Mish function, which is a privacy-preserving Mish function that inherits the design advantages of the traditional Mish function. Furthermore, it utilizes secure multi-party computation (MPC) technology to compute secret input data. In addition, this method ensures that each participant holds an additive slice of the secret data during both input and output, allowing for its connection to other linear or nonlinear modules in the neural network. Similar to the SiLU function, the patent "A Privacy-Preserving Inference Method and System for Diffusion Model Sampling" proposed by Chen Xiaojun et al. uses Chebyshev polynomial fitting of the exponential function e. x The method can also be used to calculate the Mish function, but this approximate calculation method will cause a loss of accuracy. The method designed in this invention is an accurate calculation without loss of accuracy.

[0166] Private Mish

[0167] When the neural network reaches the Mish function calculation step, participant P... i The addition slice holding the function input x <x> i ,Right now <x> 1+ <x> 2+…+ <x> n =x, where x is a secret value for the participants, and the following operations are performed:

[0168] (1) Each participant generates a random number, which is used as a multiplication partition of the random number ρ, i.e., participant P i Holding [ρ] i This satisfies [ρ]1*[ρ]2*...*[ρ] n =ρ, and then use the multiplication-based fragmentation protocol to convert to addition-based fragmentation to calculate <ξ> = [ρ], that is, at this time ξ = <ξ>1 + <ξ>2 + ... + <ξ> n =ρ

[0169] (2) Participant P i Local computing Then all participants will convert [k] by calling the multiplication-addition conversion protocol. i Convert to <φ> i That is, we get e x *ρ addition piecewise <φ> i

[0170] (3) Each participating party P i Local calculation<χ> i =<φ> i +<ξ> i And obtain χ through the reconstruction algorithm, at which point χ=φ+ξ=e x *ρ+ρ, and each party can directly calculate ln(χ).

[0171] (4) Participant P i Calculate ln([ρ] locally) i ), at this time there is remember <y> i = ln([ρ] i ), then <y> i For participant P i The additive slices of ln(ρ) held by each participant, and the additive slices of ln(χ)-ln(ρ) that each participant can calculate, are denoted as... <v> i ,at this time

[0172] (5) with <v> i As input, call the Private Tanh protocol participant P i get i ,make 1+ 2+…+ n =Fish <v> i

[0173] (6) Finally, using the Beaver Triple protocol, we can find...<x*u> Slicing, i.e., participant P i get <z> i ,make <z> 1+ <z> 2+…+ <z> n = x * u = x * tanh(ln(1 + e x ))

[0174] Like< / z> < / z> < / z> < / z> < / v> < / v> < / v> < / y> < / y> < / x> < / x> < / x> < / x> < / y> < / x> < / x> < / x> < / x> < / z> < / c> Figure 1 As shown, the activation function described in this invention is applied to the inference phase of a neural network, enabling the data holder to complete inference tasks (e.g., classification or prediction) while ensuring that the data held is not visible to the model holder (and other untrusted parties).

[0175] Specifically, the construction and use of a neural network correspond to the training phase and the inference phase, respectively. The training phase is the process by which the neural network continuously adjusts its weights based on the input data and corresponding labels, which is the process of generating a model. The inference phase is when the neural network model has been trained (the weight parameters are fixed), and the neural network processes the input data (for example, if a picture of a cat is input, the neural network will classify the picture as a cat), which is when the model starts working to perform tasks such as regression, prediction, and classification.

[0176] A trained neural network model has the following characteristics: Figure 2 The structure shown includes an input layer, an output layer, and hidden layers (also called implicit layers). The hidden layers further include linear layers and non-linear activation function layers. During the inference phase, the parameters in the neural network model, such as matrices W and B in the linear layers, have been determined during the training phase. By multiplying the input data of each linear layer by the weight matrix W of each layer and adding the bias term B, the linear layers can recombine the features of the input data and map them to a new representation space of different dimensions. More complex classification and regression tasks can then be performed in this space. Simultaneously, matrix operations can be parallelized to accelerate the computation process of the neural network, enabling it to process large-scale datasets more quickly. The non-linear layers contain one or more types of activation functions to enhance the network's expressive power (fitting ability) and increase its stability and convergence speed. These are denoted as the F function, and specific choices include, for example, ReLU and Leaky. Functions such as ReLU, ELU, CeLU, Hardsigmoid, Hardtanh, SiLU, GeLU, and Mish—the choice of which activation function(s) to use in the model is determined before training and remains determined during inference. Furthermore, the computation performed on the function F, where data is input as a vector, refers to using function F to calculate each dimension of the input vector, i.e., Z = F(Y) refers to {z1, z2, ..., z...}. n }={F(y1), F(y2),…,F(y n The number and structure of linear and nonlinear layers in the hidden layers are customized according to the complexity and characteristics of the specific problem. Each layer receives input from the previous layer and passes the output to the next layer until the last layer has completed its computation.

[0177] Taking the image classification task of handwritten digit recognition as an example, this task classifies ten categories of digits from 0 to 9. That is, given an image of a handwritten digit, the task is to identify the digit within that image. During the inference phase, the data holder possesses an image, which is a 28 (rows) × 28 (columns) matrix. This matrix is ​​flattened into a 28 × 28 = 784 rows and 1 column vector, and then input into the input layer of the neural network. After a series of calculations in hidden layers, the results are presented as a vector to the output layer. The output layer is the last layer in the neural network, and its design is tailored to the specific task. The calculation steps match the output dimension and requirements of the task. For example, in the handwritten digit recognition task, the output layer has a dimension of 10. Each dimension corresponds to a digit category, and its output value represents the probability of that category, which is calculated by a specific activation function.

[0178] See Figure 3 In this invention, the model holder first determines the neural network structure, including the number of layers, the dimension of each layer, and the type of activation function used in the non-linear layers. Then, the neural network is trained using training data to obtain the parameters W and B of each layer. After that, all W and B are published to the data holder.

[0179] After the reasoning phase begins, the data holder possesses an image (still using the handwritten digit recognition task as an example). They split the image into n addition fragments and send them to the n participants P1, P2, ..., Pn. n For each participant P i The data holders' image slices are arranged in the same dimension as the input vector and are received by the input layer, ready to be passed to the hidden layers of the neural network. For example, for n=2, the images held by the data holders are the following matrix (simplified to a 3 (rows) × 3 (columns) matrix for ease of description): After flattening it, it takes the form X = [0.444 0.571 0.165 0.67 0.828 0.543 0.592 0.094 0.068]. Then, the data holder splits X into two fragments:

[0180] <x> 1＝[0.142 0.288 0.025 0.754 0.752 0.001 0.763 0.826 0.258]

[0181] <x>2 = [0.302 0.283 0.14 -0.084 0.075 0.542 -0.171 -0.733 -0.19]

[0182] And split <x> 1、 <x>2. The data is sent to participants P1 and P2 respectively. At this point, fragmentation occurs. <x>1 and <x>The sum of the data at each corresponding position in 2 equals the data at the corresponding position in X, while participants P1 and P2 can only see a fragment that resembles a string of random numbers. <x>1 or <x>2. The original data X was not leaked.

[0183] Next, the computation of the linear layers in the hidden layers begins. In this step, participants P1 and P2 complete the computation independently without communication or interaction. Specifically, P1 computes... <y> 1＝W* <x>Calculate 1+B1, P2 <y> 2＝W* <x>2 + B2, where B1 and B2 satisfy B1 + B2 = B, are pre-calculated by the data holder (e.g., B1 = B, B2 = 0) and distributed to P1 and P2. <y>1 and <y>2 is still in vector form (but because it is multiplied by the weight matrix w), <y>1 or <y>The dimension of 2 may be different <x>1 or <x>(dimension 2), and due to the linear property of matrix operations, we have <y> 1+ <y> 2＝W* <x>1+B1+W* <x>+B2=W*X+B, let Y=W*X+R, that is, at this time, participants P1 and P2 respectively hold the addition fragments of Y, and the sum Y obtained by their addition is consistent with the intermediate calculation result of the inference stage of a normal neural network without privacy protection strategy. However, in this invention, each participant only holds the addition fragments in the form of random numbers during the calculation process, and the original data (intermediate result) Y is not leaked.

[0184] Afterwards, the shards held by participants P1 and P2 <y>1 and <y>2. Calculation of the nonlinear layers within the hidden layers. Specifically, for a neural network without privacy protection strategies, the purpose of the nonlinear layers is to calculate Z = F(Y), where F is a nonlinear activation function, and the calculation of Z applies F to each coordinate position, i.e., Z = {z1, z2, ..., z...} n }={F(y1), F(y2),…,F(y n In this invention, the purpose of this step is that after calculation and interaction, participants P1 and P2 respectively hold... <z>1 and <z>2. Satisfy <z> 1+ <z>2 = Z. Specifically, each nonlinear activation function F needs to be replaced with the privacy-preserving activation function implementation scheme proposed in this invention. That is, at the positions in the original neural network where ReLU, Leaky ReLU, Elu, Celu, Hardsigmoid, Hardtanh, SiLU, GeLU, and Mish appear, participants P1 and P2 should apply the PrivateReLU, Private Leaky ReLU, Private Elu, Private Celu, Private Hardsigmoid, Private Hardtanh, Private SiLU, Private GeLU, and Private Mish functions as described in this invention to the input slices. <y>1 and <y>Each element in step 2 is calculated. The calculation steps are shown in the following example (using only representative Leaky ReLU and Elu as examples):

[0185] Example 1:

[0186] The calculation is performed by participants P1 and P2, using Leaky ReLU function calculation with confidential input data. The following steps are performed according to the workflow:

[0187] (1) At the beginning of the system, participant P1 has <x>1 = -1.5, participant P2 has <x>2 = 0.5, at this time x = <x> 1+ <x>2 = -1, and x is unknown for participants P1 and P2.

[0188] (2) Participant P1 calculates locally Participant P2 calculates locally At this time there is e x =[e x ]1*[e x ]2=u=0.367874.

[0189] (3) Using the multiplication fragmentation to convert to the addition fragmentation protocol (see Example 3), participant P1 will use the multiplication fragmentation [e x ]1 Convert to addition fragmentation <e x >1 = 0.0315094, participant P2 will perform multiplication partitioning [e x ]2 Convert to additive partitioning:

[0190] <e x >2=0.336349

[0191] Then each participant completes the linear computation locally, and participant P1 obtains... Participant P2 received:

[0192]

[0193] At this time there is <v> ＝α*(e <x>< / x> -1)＝ <v> 1+ <v>2 = -1.26428.

[0194] (4) Using the security comparison protocol, since x > 0 at this time, b = 0, and participant P1 now possesses... 1 = 389129223699150112, participant P2 currently possesses 2 = -389129223699150111, which satisfies the condition. 1+ 2 = 1.

[0195] (5) Finally, by using the Beaver Triple protocol twice, the final result obtained by participant P1 is:

[0196] <z> 1＝ 1* <v> 1+(1- 1)* <x>1 = 112507553679661.137237548828125

[0197] Participant P2 received:

[0198] <z> 2＝ 2* <v> 2+(1- 2)* <x>2 = -112507553679662.401519775390625

[0199] The result satisfies z = <z> 1+ <z>2 = -1.2642822265625, and the initial x = -1 can also be obtained by Elu function operation to get z ≈ -1.264.

[0200] Example 2:

[0201] The calculation is performed by participants P1 and P2, with confidential input data, to calculate the Elu function when a=2. The following steps are performed according to the workflow:

[0202] (1) At the beginning of the system, participant P1 has <x>1 = -1.5, participant P2 has <x>2 = 0.5, at this time x = <x> 1+ <x>2 = -1, and x is unknown for participants P1 and P2.

[0203] (2) Participant P1 calculates locally Participant P2 calculates locally At this time there is e x =[e x ]1*[e x ]2=u=0.367874.

[0204] (3) Using the multiplication fragmentation to convert to the addition fragmentation protocol (see Example 3), participant P1 will use the multiplication fragmentation [e x ]1 Convert to addition partitioning <e x >1 = 0.0315094, participant P2 will perform multiplication partitioning [e x ]2 Convert to addition fragmentation <e x >2 = 0.336349, and then each participant performs a linear calculation locally. Participant P1 obtains:

[0205]

[0206] Participant P2 received:

[0207] At this point: <v>=a*(e <x>< / x> -1)= <v> 1+ <v>2 = -1.26428.

[0208] (4) Using the security comparison protocol, since x > 0 at this time, b = 0, and participant P1 now possesses... 1 = 389129223699150112, participant P2 currently possesses 2 = -389129223699150111, which satisfies the condition. 1+ 2 = 1.

[0209] (5) Finally, by using the Beaver Triple protocol twice, the final result obtained by participant P1 is:

[0210] <z> 1＝ 1* <v> 1+(1- 1)* <x>1 = 112507553679661.137237548828125

[0211] Participant P2 received:

[0212] <z> 2＝ 2* <v> 2+(1- 2)* <x>2 = -112507553679662.401519775390625

[0213] The result satisfies z = <z> 1+ <z>2 = -1.2642822265625, and the initial x = -1 can also be obtained by Elu function operation to get z ≈ -1.264.

[0214] Example 3:

[0215] This example demonstrates the execution steps and results of the "Element Multiplication Conversion Protocol in Addition Shared State", which is used in step (3) of Example 2, with participants P1 and P2 as the computers.

[0216] (1) At the beginning of the system, participant P1 has [m]1 = 56.53496887348917 and participant P2 has [m]2 = 1.7688167516953. [m]1 and [m]2 are both multiplicative fragments of K, that is, at this time [m]1*[m]2 = K = 100, and the value of K is not public.

[0217] (2) After the conversion protocol operation is completed, participant P1 possesses <m>1 = 14.201186127109288, participant P2 owns <m>2 = 85.79881387289072, where <m>1 and <m>Both 2 are addition partitions of K, that is, at this time we have <m> 1+ <m>2 = K = 100.

[0218] After multiple hidden layer operations, let the data held by participants P1 and P2 be respectively... <z>1 and <z>2. This data is input into the output layer of the neural network for computation. The principles of linear and nonlinear operations involved in the computation are the same as those in the hidden layer. This means that two participants, P1 and P2, hold the computation results of this layer in the form of additive slice vectors. Afterwards, participants P1 and P2 send their computation results back to the data holder, who adds them together to obtain the required neural network inference output. At this point, only the data holder knows the computation result of the task; neither the participants nor the model holder can see the actual value of the computation result.

[0219] This invention has the following two application scenarios:

[0220] (1) Sensitive Information Protection: Using neural networks typically requires inputting data in plaintext. However, this data often contains sensitive personal information such as medical records and financial information, as well as internal company information such as financial information and source code. Traditional data security methods, such as data anonymization or masking, sacrifice some data accuracy, resulting in ineffective data utilization and loss of usability. In this case, the data holder can secretly share the data, splitting it into additive fragments and storing each fragment on different participants, including devices, systems, organizations, cloud servers, cloud storage devices, and distributed nodes. Then, using a privacy-preserving neural network model, secure multi-party computation allows these participants to collaboratively perform neural network calculations without disclosing their respective data fragments, thus achieving data privacy protection while completing the computational task.

[0221] (2) Privacy-preserving computation: Traditional machine learning techniques require aggregating data from distributed data sources (e.g., edge devices, different institutions, etc.) and centralizing them in a data center or machine. The privacy-preserving neural network computation involved in this invention can facilitate the model's utilization of distributed data sources without centralizing or disclosing the original data. Specifically, each distributed data source is regarded as a participant, and the data it holds is a secret shard. The neural network structure and parameters required for the computation task are published to each participant. Participants complete the neural network computation task through local computation and collaborative computation. The combined data shards of the computation results are the result required for the task.

[0222] The method in this invention enables parties to share data and collaborate on calculations using a model without worrying about leaking sensitive information contained within the complete data, achieving "usable but invisible" data. Specific application scenarios include: in the financial sector, the model described in this invention can be used for data sharing and protection during cross-organizational collaboration, solving the data silo problem and helping banks, insurance companies, and other institutions conduct risk assessments and credit approvals while protecting user privacy; in the medical field, the model described in this invention can be used for the analysis and processing of patient data, protecting patient privacy and conducting research such as disease prediction and drug development without disclosing patient personal information; in the government service sector, the model described in this invention can be used for big data analysis, providing public services while protecting personal privacy, and collaboratively scheduling multi-party data sharing without leaking original data.

[0223] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.< / z> < / z> < / m> < / m> < / m> < / m> < / m> < / m> < / z> < / z> < / x> < / v> < / z> < / x> < / v> < / z> < / v> < / v> < / v> < / x> < / x> < / x> < / x> < / z> < / z> < / x> < / v> < / z> < / x> < / v> < / z> < / v> < / v> < / v> < / x> < / x> < / x> < / x> < / y> < / y> < / z> < / z> < / z> < / z> < / y> < / y> < / x> < / x> < / y> < / y> < / x> < / x> < / y> < / y> < / y> < / y> < / x> < / y> < / x> < / y> < / x> < / x> < / x> < / x> < / x> < / x> < / x> < / x> < / x> < / x> < / x> < / x> < / x> < / x> < / v> < / z> < / x> < / v> < / x> < / x> < / x> < / x> < / z> < / x> < / x> < / x> < / x> < / x> < / y> < / x>

Claims

1. A novel privacy activation function design method based on secure multi-party computation, characterized in that, include: Analyze the activation functions in neural networks to determine their mathematical operation steps and algebraic properties; Based on the mathematical operation steps and algebraic properties, select the corresponding basic secure multi-party computation component; The activation function is modified by selecting a basic secure multi-party computation component, so that the modified activation function can adapt to the computation framework required by secure multi-party computation and realize the computation of secret input data. The modified activation function maintains additive slices of secret data held by each participant during input and output, so that the modified activation function can connect to other linear or nonlinear modules in the neural network. The activation function is one or more of LeakyReLU, Elu, Celu, Hardsigmoid, Hardtanh, SiLU, GeLU, and Mish, and the corresponding modified activation functions are Private Leaky ReLU, Private Elu, Private Celu, Private Hardsigmoid, Private Hardtanh, Private SiLU, Private GeLU, and Private Mish, respectively. Specifically, the steps to implement the Private Elu function for the Elu activation function are as follows: When the neural network reaches the Elu function calculation step, participant P i The addition slice holding the function input x <x> i ,Right now <x> 1+ <x> 2+…+ <x> n =x, where x is a secret value for the participants, and the following operations are performed:< / x> < / x> < / x> < / x> Participant P i Local computing get For e x Multiplication by pieces, notation All participants P i The protocol of converting multiplication fragmentation to addition fragmentation will be used together to convert [e] x ] i Convert to <e x > i Then each participant completes a linear computation locally, obtaining... <v>=a*(e <x>< / x> -1);< / v> Using a secure comparison protocol, input <x>Get output ； < / x> Finally, using the Beaver Triple protocol twice and addition, we obtain <z> = * <v> + (1 - ) * <x>.

2. The method according to claim 1, characterized in that, The basic secure multi-party computation component includes one or more of the following: a representation of secret shared values, an element-wise multiplication protocol in an addition-shared state, an addition fragmentation conversion protocol in a multiplication-shared state, and a secure comparison protocol.

3. The method according to claim 1, characterized in that, The specific steps for implementing the PrivateLeaky ReLU function for the Leaky ReLU activation function are as follows: When the neural network reaches the Leaky ReLU function calculation step, participant P... i The addition slice holding the function input x <x> i ,Right now <x> 1+ <x> 2+…+ <x> n =x, where x is a secret value for the participants, and the following operations are performed:< / x> < / x> < / x> < / x> Using a secure comparison protocol, input <x>Get output At this time, each participant holds Fragments 1， 2，...， n ； < / x> The Beaver Triple protocol was used twice to calculate <0.1x*b> and <0.1x*b> respectively.<x*(1-b)> Sharding, followed by local computation by each participant. <z> i =<b*0.1x> i +<x*<1-b> i 。< / z> 4. The method according to claim 1, characterized in that, The specific steps for implementing the Private SiLU function for the SiLU activation function are as follows: When the neural network reaches the SiLU function calculation step, participant P... i The addition slice holding the function input x <x> i ,Right now <x> 1+ <x> 2+…+ <x> n =x, where x is a secret value for the participants, and the following operations are performed:< / x> < / x> < / x> < / x> Call with <x> i For the input Private Sigmoid protocol, participant P i get i ,make 1+ 2+…+ n =sigmoid(x); < / x> Using the Beaver Triple protocol, find<x*a> Slicing, i.e., participant P i get <z> i ,make <z> 1+ <z> 2+…+ <z> n =x*sigmoid(x)。< / z> < / z> < / z> < / z> 5. The method according to claim 4, characterized in that, The specific execution steps of the Private Sigmoid protocol include: Each participant generates a random number, which is then used as a multiplicative partition of the random number ρ, i.e., participant P. i Holding [ρ] i This satisfies [ρ]1*[ρ]2*…*[ρ] n =ρ, and then use the multiplication-based fragmentation protocol to convert to addition-based fragmentation to calculate <ξ> = [ρ], that is, at this time ξ = <ξ>1 + <ξ>2 + ... + <ξ> n =ρ; Participant P i Local computing Then all participants will convert [k] by calling the multiplication-addition conversion protocol. i Convert to <φ> i That is, we get e x *ρ addition piecewise <φ> i ; Each participating party P i Local calculation<χ> i =<φ> i -<ξ> i And obtain χ through the reconstruction algorithm, at which point χ=φ-ξ=e x *ρ-ρ; Each participating party P i Local computing The output result is <y> i 。< / y> 6. The method according to claim 1, characterized in that, When performing privacy activation function calculations, participants include one or more of the following: cloud servers, cloud storage devices, and distributed nodes.

7. A privacy-preserving neural network computing system, comprising a data holder, a model holder, and multiple participating parties, characterized in that: The data holder is responsible for secretly sharing the data, splitting it into multiple additive shards, and sending the shards to each participating party. The model holder is responsible for determining the neural network structure, training the neural network, obtaining the parameters of each layer, and publishing the parameters to the data holder. Multiple parties are involved in receiving data fragments sent by the data holder, performing inference computation of the neural network based on the novel privacy activation function design method according to any one of claims 1-6, including linear layer computation and nonlinear layer computation, and finally sending the computation results back to the data holder.

Images