A function secret sharing-based transformer secure inference method
By optimizing the secure computation protocols for linear and nonlinear layers of Transformer secure inference, the problems of excessive communication overhead and key size in existing schemes are solved, achieving more efficient Transformer secure inference and improving service capacity and response time.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2026-03-20
- Publication Date
- 2026-07-14
AI Technical Summary
Existing Transformer secure inference schemes based on function secret sharing suffer from excessively long response times, communication overhead, and large FSS key sizes in the offline phase, resulting in low service capacity.
A Transformer-based secure inference method based on function secret sharing is designed. By optimizing the secure computation protocols of the linear and nonlinear layers, including CPMT, SignExt, range lookup table construction, and secure range lookup table protocol, the online communication overhead and FSS key size are reduced.
Compared to the optimal method, it reduces online communication overhead by 3 times and FSS key size by 4 times, increases service capacity, and reduces response time over wide area networks.
Smart Images
Figure CN122394796A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of deep learning and data security, specifically to a Transformer secure inference method based on function secret sharing. This method can be applied to the protection of privacy information during the inference process of Transformer neural networks, such as inference services for large models like BERT, ViT, and GPT. It ensures that during Transformer neural network inference, the client will not disclose its private data to the server, and the server will not disclose its model parameters to the client. Background Technology
[0002] With the development of artificial intelligence, large models based on Transformer have achieved remarkable results in natural language processing and computer vision. Recently, ChatGPT demonstrated the successful application of emerging Transformer neural network inference services, driving a series of large model inference services built on Transformer. Online inference services typically deploy trained models to the cloud. The security requirements here are to ensure the privacy of client data and server-side model parameters, enabling secure inference. Simultaneously, to ensure the availability of online inference services, low response time and high service capacity are also necessary. Specifically, low response time refers to the need for users to receive inference results as quickly as possible after submitting input; high service capacity refers to the number of inferences that can be completed per unit.
[0003] Existing work has proposed several two-party Transformer secure inference frameworks, such as NEXUS, Bolt, BumbleBee, SHAFT, and Sigma. These works use homomorphic encryption (HE), oblivious transfer (OT), secret sharing (SS), and function secret sharing (FSS) techniques to design secure computation protocols for each layer of the Transformer, thereby achieving secure Transformer inference. Works represented by Bolt, BumbleBee, and BLB design computationally and communication-efficient secure matrix multiplication protocols based on HE, and use OT and other techniques to design secure computation protocols for nonlinear functions. Although these have relatively low communication overhead, the large number of homomorphic operations introduces significant computational overhead, resulting in long online runtimes, high response times, and low service capacity. SS-based secure inference schemes, represented by SHAFT, introduce significant communication overhead and a large number of communication rounds when securely computing nonlinear functions, leading to long runtimes in wide area networks. FSS-based secure inference schemes, exemplified by Sigma, use a preprocessing model to shift most of the computational and communication overhead to an offline stage independent of the input, thus achieving an efficient online stage.
[0004] While FSS-based secure inference schemes offer lower response times, they still suffer from several drawbacks: First, the current best FSS-based scheme, Sigma, has a high number of communication rounds, leading to a sharp increase in response time over wide area networks (WANs). When network latency increases from 40 milliseconds to 200 milliseconds, Sigma's online inference time increases from 1.48 minutes to 4.04 minutes. Second, FSS-based schemes typically incur significant offline overhead, resulting in limited service capacity. The Sigma secure inference scheme requires generating a 16.69GB FSS key and transmitting it to both the client and server for a single inference attempt, resulting in a total inference time of approximately 12 minutes and a daily service capacity of only 120 attempts. Therefore, reducing the online communication overhead and the size of the offline FSS key in FSS-based secure inference schemes is crucial for improving the efficiency of Transformer secure inference. Summary of the Invention
[0005] This invention addresses some shortcomings of existing FSS-based Transformer secure inference schemes by proposing an FSS-based Transformer secure inference method. Compared to the best existing methods, this method reduces online communication overhead and FSS key size by 3 and 4 times respectively, improves service capacity, and reduces response time over wide area networks. This invention can be used to protect privacy information in Transformer-based inference services, such as inference services for large models like BERT, ViT, and GPT. During Transformer secure inference, the client will not disclose its private data to the server, and the server will not disclose its model parameters to the client. Attached Figure Description
[0006] Figure 1 This is the overall design diagram of the Transformer secure reasoning based on function secret sharing designed in this invention.
[0007] A Transformer-based secure reasoning method based on function secret sharing, the overall structure of which is shown in the attached figure. Figure 1 As shown, the entire network can be divided into linear and nonlinear layers. This invention designs efficient secure computation protocols for both linear and nonlinear layers. Specifically, it includes the following three parts: a secure computation protocol for linear layers, a secure computation protocol for bounded single-variable nonlinear functions, and a secure computation protocol for nonlinear layers. The secure computation protocol for linear layers includes the Ciphertext-Plaintext Multiplication with Truncation (CPMT) protocol and the SignedExtension (SignExt) protocol; the secure computation protocol for bounded single-variable nonlinear functions includes an interval lookup table construction method and a secure interval lookup table protocol; the secure computation protocol for nonlinear layers includes the Softmax secure computation protocol, the GeLU secure computation protocol, and the LayerNorm secure computation protocol.
[0008] This invention includes three parties, using This indicates a client using the inference service, using This indicates the server that holds the model parameters. The trusted processing party can generate FSS keys offline to accelerate computation in the online phase. Each of the three participants holds the same pseudorandom generator (PRG). All real-valued data is converted into signed fixed-point numbers and mapped onto the ring, i.e., for a given precision... ,data The signed fixed-point number is represented as The complement of the two's complement is mapped to Above. All privacy data that needs protection. Represented in the form of a mask, i.e., a mask Record the participants hold In the ring The sharing value on ,satisfy When the secure computing protocol described in this invention is defined with input and output as single elements, it can also support matrix-form input, which can be regarded as executing the secure computing protocol in parallel for each element in the matrix.
[0009] Part 1, the following section describes the linear layer secure computation protocol.
[0010] Linear layer secure computation protocols include the CPMT protocol and the SignExt protocol. In the CPMT protocol, the server has weights. , Holding the input mask value , Holding input mask Each participating party invokes the CPMT protocol. The multiplication is calculated and truncated to obtain the mask value of the multiplication result. In the SignExt protocol, Holding the input mask value , Holding a mask ,satisfy Each participating party invokes the SignExt protocol. ,Will from Extended to Finally, the mask value of the expanded result is obtained. .
[0011] The proposed CPMT protocol Including offline phase execution Protocol and online phase execution protocol:
[0012] Executed in the offline phase The protocol is shown in line af:
[0013] a. and Use PRG to generate the same random numbers. and .
[0014] b. and Use PRG to generate the same random numbers. and .
[0015] c. calculate Generate random numbers ,calculate ,Will Send to .
[0016] d. Will Send to .
[0017] e. Will Send to .
[0018] f. calculate .
[0019] Executed in the online phase The protocol is shown in line ad:
[0020] a. calculate .
[0021] b. calculate ,in This indicates a right shift operation.
[0022] c. Will Send to .
[0023] d. Output mask .
[0024] The proposed SignExt protocol Including offline phase execution Protocol and online phase execution protocol:
[0025] Executed in the offline phase The protocol is shown in line ac:
[0026] a. calculate ,in Indicates will from Bit extension A sign bit is added to the higher-order bits.
[0027] b. calculate ,in express The highest bit.
[0028] c. right and exist Share above: First, randomly generate Next, calculate Finally Send to ,Will Send to .
[0029] Executed in the online phase The protocol is shown in line ae:
[0030] a. calculate .
[0031] b. calculate
[0032] c. Will Send to
[0033] d. calculate and will Send to
[0034] e. Output mask .
[0035] Part Two, the following section describes the secure computation protocol for bounded single-variable nonlinear functions.
[0036] The secure computation protocol for bounded single-variable nonlinear functions includes two steps: first, constructing an interval lookup table. This represents a bounded, single-variable, nonlinear function, and is implemented using a safe interval lookup table protocol. This enables safe computation of bounded single-variable nonlinear functions.
[0037] For monotonic bounded single-variable nonlinear functions Let its output range in fixed-point representation be... The input range is The proposed interval lookup table construction method first divides the input into multiple non-uniform intervals, and sets the left endpoint function value of each interval as the lookup table value for that interval, thus constructing the interval lookup table as shown below:
[0038]
[0039] The function of range lookup table is to For a given input ,return Belong The corresponding result The proposed range lookup table construction includes parameters. ,when When the value is large, the number of items in the interval lookup table is small, and the subsequent computational cost is less, but it has a larger approximation error; when While it achieves the smallest approximation error, it leads to significant computational overhead. The proposed interval lookup table construction method can support both scenarios with high efficiency and those with high accuracy requirements.
[0040] For a non-monotonic bounded single-variable nonlinear function, it can be divided into several monotonic intervals, an interval lookup table can be constructed for each interval, and the interval lookup tables of each interval can be concatenated to obtain the interval lookup table of any bounded single-variable nonlinear function.
[0041] The proposed safe interval lookup table protocol as follows:
[0042] The function of the secure range lookup table protocol is to ,enter mask Output Belong The corresponding result mask The required conditions include a secure Multiple Interval Containment (MIC) protocol. and safe multiplication protocol The former's function is ,enter mask and interval sequence Returns a sequence Secret sharing value The latter's function is: ,enter and The shared value is returned. mask The protocol flow is shown in line ae.
[0043] a. Regarding , Call ,enter , respectively obtained .
[0044] b. For , calculate .
[0045] c. For , calculate .
[0046] d. For , Call ,enter Output .
[0047] e. For , Output .
[0048] Part Three, the following section describes the secure computation protocol for nonlinear functions.
[0049] The proposed nonlinear function secure computation protocol includes the Softmax function secure computation protocol. GeLU Function Secure Computation Protocol and LayerNorm function secure computation protocol .
[0050] The proposed Softmax function secure computation protocol takes a vector as input. mask format Output mask ,in The single-variable nonlinear functions that need to be called include the exponential function and the reciprocal function. The input range of the exponential function is... The output range is It satisfies the bounded property; the input range of the reciprocal function is... The output range is This satisfies the boundedness characteristic. Therefore, the safe computation protocol for bounded single-variable nonlinear functions proposed in this invention can achieve safe computation of exponential and reciprocal functions. Required conditions include a safe maximum value protocol. and safe multiplication protocol The former has an input length of . vector mask Output vector maximum value mask The protocol flow is shown in line ag.
[0051] a. Regarding , Call ,enter ,get .
[0052] b. For , calculate .
[0053] c. For , Secure computation protocol for calling exponential functions ,enter Output .
[0054] d. For , calculate .
[0055] e. For , Secure computation protocol for calling reciprocal functions ,enter Output .
[0056] f. For , Invoking the safe multiplication protocol ,enter and ,get .
[0057] g. For , Output .
[0058] The proposed GeLU function secure computation protocol input is mask format Output mask Since GeLU is not a bounded function, it needs to be determined through... To achieve secure computation, where For bounded functions, the secure computation protocol for bounded single-variable nonlinear functions proposed in this invention can be used to achieve [the desired result]. Secure computation. Required conditions include a secure ReLU protocol. ,enter mask Output mask The protocol flow is shown in line ae.
[0059] a. Regarding , Call ,enter Output .
[0060] b. For , calculate .
[0061] c. For , Call Function-Safe Computation Protocol ,enter Output .
[0062] d. For , calculate .
[0063] e. For , Output .
[0064] The proposed LayerNorm function secure computation protocol takes a vector as input. mask format Output mask ,in , This represents the element-wise multiplication of two vectors. First, an equivalent transformation of LayerNorm is performed to eliminate the division involved in calculating the mean and variance. The transformed reciprocal square root function is then a bounded function. The safe computation protocol for bounded single-variable nonlinear functions proposed in this invention enables the safe computation of the reciprocal square root function. Finally, a linear function is executed to obtain the output of LayerNorm. Required conditions include the CPMT protocol. SignExt protocol Safe multiplication agreement Security Square Protocol The function of the Security Square Protocol is as follows: ,enter The shared value is returned. mask .
[0065] The proposed LayerNorm equivalent transformation is: ,in , , It is a pre-set constant.
[0066] The proposed LayerNorm protocol flow is shown in line ag:
[0067] a. Regarding , calculate .
[0068] b. For , Call ,enter Output Summing the vectors yields .
[0069] c. For , Calling the Secure Reciprocal Square Root Protocol ,enter Output .
[0070] d. For , Call ,enter and ,get .
[0071] e. For , Call ,enter Output .
[0072] f. For , Call ,enter and Output .
[0073] g. For , Output .
Claims
1. A Transformer-based secure reasoning method based on function secret sharing, characterized in that: Linear layer secure computation protocols include the proposed Ciphertext-Plaintext Multiplication with Truncation (CPMT) protocol and Signed Extension (SignExt) protocol; A secure computation protocol for bounded univariate nonlinear functions, including the proposed Interval Look-up Table (ILUT) construction method and secure interval look-up table protocol; Nonlinear function secure computation protocols, including the proposed Softmax secure computation protocol and LayerNorm secure computation protocol; The proposed CPMT protocol requires both the client and server to hold input... mask Furthermore, the server holds the model parameters during the offline phase. The server calculates the result during the offline phase. irrelevant It can be computed locally during the online phase. This yields the mask of the truncated multiplication result. The proposed SignExt protocol allows both the client and server to hold input... mask Both parties obtained correlation randomness during the offline phase. and They can be computed together during the online phase. This yields the mask for the sign extension result. ,in Indicates the highest bit; A secure computation protocol for bounded univariate nonlinear functions is characterized by the following method for constructing an interval lookup table: First, the bounded univariate nonlinear function is divided into multiple monotonically bounded univariate nonlinear functions; second, for each monotonically bounded function, the input is divided into multiple non-uniform intervals, and the function value at the left endpoint of each interval is taken as the lookup table entry for that interval; finally, the interval lookup tables of each monotonically bounded function are concatenated to obtain the interval lookup table for the bounded univariate nonlinear function; the proposed secure interval lookup table protocol first uses a multiple interval inclusion protocol to obtain a selection vector, and then performs an inner product between the selection vector and the lookup table. The Softmax function safe computation protocol utilizes the proposed bounded univariate nonlinear function safe computation protocol to perform safe computation of its exponential and reciprocal functions; the proposed GeLU function safe computation protocol first transforms it. Then, the proposed safe computation protocol for bounded single-variable nonlinear functions is used to complete the task. Safe computation; The proposed LayerNorm function secure computation protocol first performs an equivalent transformation on LayerNorm: ,in , , The proposed bounded single-variable nonlinear function safe computation protocol is used to perform safe computation of the reciprocal square root function, which is a pre-set constant.