A secure multi-party computation-based large model privacy protection inference method and system

Abstract

The present application belongs to the cross field of information security and computer application technology, and relates to a large model privacy protection reasoning method and system based on secure multi-party computation. The method comprises: obtaining a prompt required for reasoning in a secret sharing form input by a client; obtaining a large model weight parameter input in a secret sharing form by a server; performing privacy protection calculation on each operation in the large model and executing model reasoning; sending a shared value of the model reasoning result to the client, and reconstructing the reasoning result based on the shared value by the client. The present application uses homomorphic encryption, vector oblivious linear evaluation and other cryptography technologies and segmented polynomial optimal approximation to construct a secure reasoning framework of a large Transformer model, can perform fast and accurate large model collaborative reasoning, and provides semi-honest security for the client and the model owner, that is, both parties need to accurately comply with the agreement and the privacy of each party will not be leaked.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of information security and computer application technology, and particularly relates to a privacy-preserving inference method and system for large models based on secure multi-party computation. Specifically, it utilizes secure multi-party computation techniques such as homomorphic encryption (HE) and vector oblivious linear evaluation (VOLE) to achieve data privacy protection for both the client and the model provider, as well as efficient and accurate collaborative inference. Background Technology

[0002] Entering the 21st century, with the rapid development of technologies such as the Internet, cloud computing, big data, and artificial intelligence, society as a whole is becoming increasingly digitalized, networked, and intelligent. Large-scale model inference services, which are attracting increasing attention from industry and academia, are better able to meet the needs of today's society due to their digital, networked, and intelligent characteristics. Currently, large-scale model inference has very promising application prospects in fields such as military, finance, the Internet of Things, cloud computing, artificial intelligence, communications, insurance, and healthcare.

[0003] Transformer-based models have demonstrated powerful computational capabilities when utilizing attention mechanisms for representation, leading to significant success in both Natural Language Processing (NLP) and Computer Vision (CV). For example, GPT and the proposed BERT model in NLP, and ViT and Swin-Transformer models in CV, have achieved state-of-the-art performance in their respective domains. In particular, the ChatGPT model's outstanding emergent capabilities in text translation, content generation, and knowledge-based question answering have spurred a surge in online inference services for large models.

[0004] However, current large-scale model inference systems suffer from serious privacy leaks. On the user side, clients need to provide their private inputs to the model owner, potentially harming their own interests. On the model owner side, various attacks launched by malicious clients could leak the model weights and parameters trained with significant computational resources and data to malicious clients. One approach to this problem is to use secure multi-party computation (MPC) and homomorphic encryption (HE) to protect the confidentiality of user data and model parameters during inference. However, directly using MPC to evaluate the large number of nonlinear layers in the model incurs significant communication overhead, and HE also has high computational costs in this regard. Therefore, the key to this invention is how to design an efficient and secure inference protocol for large Transformer models by combining multiple cryptographic primitives. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a privacy-preserving inference method and system for large models based on secure multi-party computation.

[0006] The technical solution adopted in this invention is as follows:

[0007] A privacy-preserving inference method for large models based on secure multi-party computation includes the following steps:

[0008] Obtain the clues required for reasoning input by the client in a secret sharing format;

[0009] Obtain the large model weight parameters that are secretly shared by the server;

[0010] The publicly available large model network structure is obtained, and privacy-preserving calculations are performed on each operational layer in the large model, and model inference is performed. The server and client respectively obtain the shared value of the inference result.

[0011] The server sends the shared values ​​of the inference results to the client, and the client reconstructs the inference results based on the complete shared values.

[0012] Furthermore, the client can only obtain the model inference results but not the model parameter information, and the server cannot obtain the client's private prompts and model inference results.

[0013] Furthermore, the large model is based on the Transformer architecture; the privacy-preserving computation for each operational layer in the large model includes:

[0014] Linear layer operations: All linear layer operations are reduced to matrix multiplication, using the shared secrets of the client and server as input, and running the matrix-safe multiplication protocol. And obtain the output of the current linear layer;

[0015] Non-linear layer operations: Non-linear operations include Softmax, GeLU, and LayerNorm, which are shared by both the client and server through addition. <x>As input, call the function to compute the protocol and output the addition sharing. <y>=P Func ( <x>), where Func∈(Softmax,GeLU,LayerNorm); the function computation protocol includes Softmax's secure computation protocol Π Softmax LayerNorm's secure computation protocol Π LayerNorm and GeLU's secure computing protocol Π GeLU .

[0016] A privacy-preserving inference system for large models based on secure multi-party computation, comprising:

[0017] The input module is used to obtain the prompts required for reasoning input by the client in a secret sharing manner, and to obtain the large model weight parameters and the publicly available large model network structure input by the server in a secret sharing manner.

[0018] The privacy-preserving computation and inference module is used to perform privacy-preserving computations and model inferences on each operational layer in the large model. The server and client respectively obtain the shared value of the inference results.

[0019] The output module is used by the server to send the shared values ​​of the inference results to the client, and the client reconstructs the inference results based on the complete shared values.

[0020] The beneficial effects of this invention are as follows:

[0021] Large Transformer Models (LLMs) have achieved great success in various tasks in natural language processing and computer vision. Meanwhile, secure multi-party computation protects the data privacy of both users and servers in model inference service scenarios. However, existing solutions exhibit high latency and large communication overhead when inference with LLMs, which limits the further implementation of privacy-preserving inference services for large models. This invention utilizes cryptographic techniques such as homomorphic encryption and vector skeptical linear evaluation, along with piecewise polynomial optimal approximation, to construct a secure inference framework for large Transformer models. This framework can perform fast and accurate collaborative inference for large models and provides semi-honest security for both the client and the model owner, meaning that both parties must accurately adhere to the protocol without any leakage of their privacy. Attached Figure Description

[0022] Figure 1 This is a functional description diagram of the underlying protocols invoked by the privacy-preserving inference system of this invention.

[0023] Figure 2 This is a diagram of the Transformer privacy-preserving inference system architecture of the present invention.

[0024] Figure 3 Example diagrams for encoding mapping and ciphertext compression. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0026] This invention utilizes cryptographic techniques such as vector skeptical linear evaluation, homomorphic encryption, and secret sharing, along with high-performance underlying sub-protocols from previous work, to construct a privacy-preserving inference framework for large models. Before introducing these techniques, a unified explanation of the notation is provided.

[0027] 1. Symbol Explanation

[0028] Indicates from set A uniformly random sample r is generated. If x is true, {x} returns 1; otherwise, it returns 0. Where L = 2 l Furthermore, is a bit width. [N] represents the set {0, 1, ..., N-1}, where These represent rounding down, rounding up, and rounding to the nearest integer, respectively. For x >> f means arithmetic right shift of x by f bits. Column vectors and matrices are represented by lowercase bold letters respectively. and uppercase bold letters express.

[0029] polynomials It means, and Representing a polynomial The j-th term (i.e., X) j The coefficient of ). For N (usually a power of 2) and make Represents the polynomial quotient cycle Where X represents a polynomial variable. Given two polynomials... Due to X N ≡-1mod(X N +1), therefore polynomial product The coefficient can be represented as follows:

[0030]

[0031] 2. Secret sharing

[0032] This invention uses a two-way addition secret-sharing scheme: for values It is split into two random shared Make When L=2 Arithmetic sharing on <x> L This becomes a Boolean shared property and is denoted as <x> B This scheme can perfectly hide x, and each party can obtain the linear operation by local operation.

[0033] 3. Vector Oblivious Linear Evaluation

[0034] Two parties can obtain the linear operation by invoking protocol Π VOLE So that P0 party (server) gets and P1 party (client) gets scalar and And Where represents the n-dimensional vector space with the number field represents uniform random sampling. In order to make the VOLE protocol run on the ring , this paper uses the protocol instance proposed by Baum et al. and calls it in a black box way.

[0035] 4. Additive homomorphic encryption

[0036] The additive homomorphic encryption scheme can perform any linear homomorphic operation on the ciphertext. That is, given the ciphertext of the message m, the ciphertext of L(m) can be calculated without decryption, where L is a linear function. This paper adopts the BFV (Brakerski-Fan-Vercauteren) scheme, and uses the homomorphic automorphism (HA) technology to compress the ciphertext. The specific scheme consists of public parameters pp={N,σ,q,p} and the following operations:

[0037] 1) Key generation: the security parameter is 1 λ , the private key is The public key is Where the polynomial is sampled from the ciphertext space The polynomial The coefficients of the polynomial are uniformly randomly sampled from the discrete Gaussian distribution with a standard deviation of σ

[0038] 2) Encryption of plaintext: given the plaintext of the message Its ciphertext is Where RLWE represents the encryption of the plaintext based on the lattice problem. The coefficients of the polynomial are sampled from the distribution The coefficients

[0039] 3) Ciphertext decryption: given the RLWE ciphertext Decrypted by the private key sk to ​

[0040] 4) Linear operations: Suppose there are two plaintext polynomials The RLWE ciphertexts are CT0 and CT1 (corresponding to the same private key sk), and there is a plaintext polynomial. Then RLWE ciphertext Decrypted into in This represents the homomorphic multiplication of ciphertext and plaintext. This indicates homomorphic addition between ciphertexts.

[0041] 5) Homomorphism: Given a plaintext message RLWE ciphertext And if the odd number o ≤ 2N-1, then the operation Decryption Auto indicates that the ciphertext is subjected to a homomorphic automorphism operation.

[0042] 5. Sub-protocols in the prior art

[0043] The secure computation of nonlinear functions in this invention primarily utilizes existing low-level protocols, such as the multiplexer protocol. MUX Security Comparison Protocol Π LT Maximum value protocol Π max Interception Protocol Square root reciprocal protocol H rSqrt Multiplication Agreement and cross-term multiplication protocol Π CT .exist Figure 1 The document provides functional descriptions of these protocols and replaces the IKNP (Ishai-Kilian-Nissim-Petrank) type OTs invoked by these protocols with VOLE type OTs. OT stands for Oblivious Transfer.

[0044] The following section details the privacy-preserving inference method for large models based on secure multi-party computation in this invention.

[0045] 1. Transformer model structure and inference scenarios

[0046] The GPT model only has a decoding layer, while the BERT model has a decoder-encoder structure. Since the decoder and encoder structures are largely similar, this invention designs protocols for the sub-layers within the encoder. For example... Figure 2 As shown on the right, the Transformer layer consists of sub-layers such as multi-head self-attention mechanism, feedforward network and LayerNorm (layer regularization), and also includes input embedding and residual connection operations.

[0047] Multi-head attention (fMHA): The multi-head self-attention mechanism is implemented by parallelizing a single attention layer. Before computing the attention layer, the input token needs to be processed. Perform linear projection to obtain Therefore, the expression for the attention layer is:

[0048]

[0049] in, These represent the query, key, and value matrices, respectively, where d is the publicly disclosed feature dimension. For client-side private input, These are the private weights of the model owner.

[0050] Feedforward Network (FFN): An FFN consists of two linear layers and one GeLU activation layer, specifically expressed as follows:

[0051]

[0052] The server holds private parameters. The client holds private input

[0053] Layer regularization: for vectors held by the client First calculate its component mean. and variance The model owner holds private hyperparameters (γ, β), so the expression for layer regularization is:

[0054]

[0055] Furthermore, input embedding is needed to transform the client's private input into a continuous feature vector representation, i.e. in The private weight held by the server.

[0056] 2. Construction of a Privacy-Preserving Inference Scheme for Large-Scale Transformer Models

[0057] like Figure 2 As shown, this invention names the proposed large-scale Transformer privacy-preserving inference framework CryptFormer. This inference framework consists of four protocols: one matrix-secure multiplication protocol. and three sets of nonlinear layer computation protocols (Π) GeLU , Π Softmax , Π LayerNorm The specific structure of these protocols is as follows:

[0058] (1) Matrix-safe multiplication protocol:

[0059] To design a low-traffic matrix multiplication protocol using homomorphic encryption and its ciphertext compression techniques, we first need to construct the following two natural mapping functions.

[0060] Make

[0061]

[0062] and and The coefficients in all other positions are 0, so the matrix product... Can be obtained by polynomial product Let's give the following: for i∈[m], j∈[k], there is a mapping... Make exist Figure 3 The text provides a simple example to illustrate these two mappings. It can be seen that the resulting polynomial... It contains a large number of coefficients that are irrelevant to the matrix multiplication result. Therefore, to eliminate the redundant protocol communication caused by these irrelevant coefficients, this invention utilizes homomorphic automorphism technology to eliminate them, which is reflected in… Figure 3 Polynomial after compression Since directly using homomorphic automorphism can only eliminate coefficients that are not multiples of 2r (where r is a positive integer), the required interval (n) of the coefficients in the final polynomial product must be a power of 2. However, this can be satisfied by the preprocessing step of the input matrix in the Strassen matrix multiplication algorithm. Therefore, a safe matrix multiplication protocol (Algorithm 1) for two small square matrices (powers of 2) can be designed as the recursive exit protocol for the general large matrix multiplication protocol (Algorithm 2). Algorithms 1 and 2 are shown in Tables 1 and 2, respectively, where P0 represents the server and P1 represents the client.

[0063] Table 1

[0064]

[0065] Table 2

[0066]

[0067] (2) Softmax's secure computation protocol:

[0068] The core operation in the attention layer is the Softrmax function, which is used for vectors. The expression for the Softrmax function is:

[0069]

[0070] The key lies in the efficient and safe calculation of negative exponents, which is achieved using a fast approximation formula for negative exponents and range clipping techniques:

[0071]

[0072] To determine the range of x, the comparison protocol Π can be used. LT .for Use Π max The protocol is used for calculation. Divide by 2 t The calculation can be performed through the truncation protocol. This is how it is achieved. Finally, the t-th squared protocol Π can be called. square We can then calculate 2^t powers. This leads to the safe Softmax protocol in Algorithm 3, as shown in Table 3.

[0073] Table 3

[0074]

[0075] (3) LayerNorm's secure computation protocol:

[0076] For vectors The expression for the LayerNorm function is: in Furthermore, γ and β are hyperparameters obtained by the model owner through prior training. It can be seen that LayerNorm requires squaring, multiplication, and inverse square root operations. For MPC, the most challenging operation is... Because the computational costs of square roots and reciprocal square roots are similar, while the cost of division is significantly greater than that of multiplication, the reciprocal square root protocol Π can be used. rsqrt and OT-based multiplication protocols To calculate σ -1 / 2 ·(x[i]-μ). Complete computation protocol Π LayerNorm As shown in Algorithm 4, see Table 4.

[0077] Table 4

[0078]

[0079] (4) Secure computation protocol for GeLU functions:

[0080] The original expression for the GeLU function is: in Considering the asymptotic properties of the GeLU function on both sides of the x-axis and its non-monotonicity in the middle interval, a piecewise polynomial optimal approximation of GeLU in the function space is performed, the specific expression of which is:

[0081]

[0082] in The expression is:

[0083]

[0084] The error of the above best square approximation is Therefore, a secure GeLU protocol Π as shown in Algorithm 5 can be constructed. GeLU See Table 5.

[0085] Table 5

[0086]

[0087] The privacy-preserving inference framework for large-scale Transformer models in this invention is based on a variety of efficient and secure multi-party computation tools, such as homomorphic encryption and vector skeptical linear evaluation, to achieve data privacy protection for both the client and the model provider, as well as efficient and accurate two-party collaborative inference. Combined with... Figure 2 This section introduces the specific implementation of the framework:

[0088] 1) Both the client and the model owner (i.e., the server) first obtain the public parameters of the cryptographic primitives required by the framework;

[0089] 2) The client inputs its private inputs for inference into the privacy-preserving inference system CryptFormer in a secret sharing manner;

[0090] 3) The model owner discloses the network structure of their large Transformer model and inputs the model weight parameters into CryptFormer in a secret, shared manner;

[0091] 4) CryptFormer's privacy protection calculations for each operation in the Transformer model are as follows:

[0092] i) Linear layer operations: All linear layer operations can be reduced to matrix multiplication. CryptFormer takes the secrets shared by the client and the model owner as input (i.e., ... Operating Protocol And obtain the output of the current linear layer.

[0093]

[0094] ii) Nonlinear layer operations: These nonlinear operations include (Softmax, GeLU, LayerNorm), which are shared by addition. For input, call the function computation protocol (Π) in CryptFormer. Softmax , Π GeLU , Π LayerNorm And output the addition sharing Where Func∈(Softmax,GeLU,LayerNorm).

[0095] 5) After both parties have invoked the CryptFormer inference system, only the client receives the desired model inference result, and the client is unaware of any information regarding the model parameters. The server, on the other hand, is unaware of both the client's private input and the model's inference output.

[0096] Another embodiment of the present invention provides a privacy-preserving inference system for large models based on secure multi-party computation, comprising:

[0097] The input module is used to obtain the prompts required for reasoning input by the client in a secret sharing manner, and to obtain the large model weight parameters and the publicly available large model network structure input by the server in a secret sharing manner.

[0098] The privacy-preserving computation and inference module is used to perform privacy-preserving computations and model inferences on each operational layer in the large model. The server and client respectively obtain the shared value of the inference results.

[0099] The output module is used by the server to send the shared values ​​of the inference results to the client, and the client reconstructs the inference results based on the complete shared values.

[0100] The above division of modules is merely illustrative. In practical applications, the functions described above can be assigned to different functional modules as needed to complete all or part of the functions described in the aforementioned method. The specific working process of each module can be found in the corresponding process in the aforementioned method embodiments, and will not be repeated here.

[0101] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0102] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0103] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention should be determined by the claims.< / x> < / x> < / x> < / y> < / x>

Claims

1. A privacy-preserving inference method for large models based on secure multi-party computation, characterized in that, Includes the following steps: Obtain the clues required for reasoning input by the client in a secret sharing format; Obtain the large model weight parameters that are secretly shared by the server; The publicly available large model network structure is obtained and privacy-preserving computations are performed on each operational layer to execute model inference. The server and client respectively obtain the shared value of the inference result. The server sends the shared values ​​of the inference results to the client, and the client reconstructs the inference results based on the complete shared values; The large model is based on the Transformer architecture; The privacy-preserving calculations performed on each of the operational layers include: Linear layer operations: All linear layer operations are reduced to matrix multiplication, using the shared secrets of the client and server as input, and running the matrix-safe multiplication protocol. And obtain the output of the current linear layer; Non-linear layer operations: Non-linear operations include Softmax, GeLU, and LayerNorm, which are shared by both the client and server through addition. As input, call the function to compute the protocol and output the addition sharing. ,in The function computation protocol includes Softmax's secure computation protocol. LayerNorm's secure computation protocol Secure computing protocol with GeLU ; The matrix-secure multiplication protocol Secure multiplication protocol using special matrices Recursive Secure Multiplication Protocol for General Matrices Recursive export agreement; The secure multiplication protocol of the special matrix Includes the following steps, wherein Indicates server, Indicates the client: enter: Each holding a matrix , in , It is a positive integer; Output: and Get each , in ; 1) Each will have a matrix sum matrix Encode them as polynomials respectively ; 2) polynomial Encryption And send to ; 3) Uniform random sampling of a polynomial , then decoded into ; 4) Upon receiving from ciphertext back, Calculate the ciphertext locally ,in This is homomorphic subtraction; 5) make This step is to perform scaling correction on the target position coefficient; 6) For The value ranges from 0 to Execute step 7); 7) Local computation This step is to eliminate position coefficients that are irrelevant to the result; 8) Obtain the compressed ciphertext polynomial Send it to ; 9) Output As a shared ; 10) Decryption calculation and output ; The recursive secure multiplication protocol for general matrices ,include: enter: Each holding a matrix , Let be the order of the polynomial in the BFV scheme; Output: Get Addition sharing; (1) Each will Fill with zeros to expand to the same size to obtain ; (2) If If the first step is to execute steps (3) and (4), then execute steps (5) to (8); (3) Calling Protocol Get ; (4) According to the expansion method in step (1), each of them... Crop and return ; (5) , Each of them divides its filling matrix as follows: (6) Recursive call To obtain intermediate quantity Addition sharing: (7) Each can obtain it through local computation. Addition sharing: (8) Each according to step (4) Perform cropping and return .

2. The method according to claim 1, characterized in that, The client can only obtain the model inference results but not the model parameter information, and the server cannot obtain the client's private prompts and model inference results.

3. The method according to claim 1, characterized in that, The Softmax secure computing protocol include: enter: Hold separately , in ; Output: Get each , in ; 1) Call the maximum value protocol To calculate: ; 2) Local calculations ; 3) Invoke Comparison Protocol To calculate: ; 4) Invoke the truncation protocol To calculate: ; 5) For The value ranges from arrive Execute step 6): 6) Call calculate: ; 7) Local computing and call Calculated ; 8) Calling the multiplication protocol To obtain ; 9) Calling the Multiplexer protocol Calculate and output .

4. The method according to claim 3, characterized in that, The LayerNorm secure computing protocol include: enter: Hold separately , in ; Output: Get each , in ; 1) Each local calculation: , in ; 2) Call Protocol to obtain ; 3) Each local calculation yielded: ; 4) Calling the square root reciprocal protocol To calculate: ; 5) Calling Protocol Get ; 6) Calling Protocol To calculate .

5. The method according to claim 4, characterized in that, The secure computing protocol of GeLU include: enter: Hold separately , in ; Output: Get each , in ; 1) Invoke Comparison Protocol To calculate: ; 2) Local computing ; 3) Call Square Protocol To calculate and call To obtain ; 4) Based on Calculate using piecewise polynomial optimal approximation on the GeLU function space ; 5) Calling Protocol To calculate and output: .

6. A privacy-preserving inference system for large models based on secure multi-party computation, employing the method described in any one of claims 1 to 5, characterized in that, include: The input module is used to obtain the prompts required for reasoning input by the client in a secret sharing manner, and to obtain the large model weight parameters and the publicly available large model network structure input by the server in a secret sharing manner. The privacy-preserving computation and inference module is used to perform privacy-preserving computations and model inferences on each operational layer in the large model. The server and client respectively obtain the shared value of the inference results. The output module is used by the server to send the shared values ​​of the inference results to the client, and the client reconstructs the inference results based on the complete shared values.

7. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 5.

Images