Induction and conversion method of encrypted deep neural network model

Abstract

The invention provides an inference method for encrypting a deep neural network, comprising the following steps performed by an arithmetic device: encoding a message according to a quantization parameter to generate a plaintext, encrypting the plaintext according to a private key to generate a ciphertext, sending the ciphertext to a deep neural network model to generate a ciphertext result, the ciphertext result is decrypted according to the private key to generate a plaintext result, and the plaintext result is decoded according to the quantization parameter to generate an inference result.

Description

Technical Field

[0001] The present invention relates to deep neural networks and homomorphic encryption, and in particular to an inference and conversion method for an encrypted deep neural network model. Background Art

[0002] With the recent advances in artificial intelligence and deep learning (DL), deep neural network (DNN) models have attracted widespread attention due to their remarkable success in many fields such as robotics, natural language processing, healthcare, and computer vision. On the other hand, machine learning as a service (MLaaS) provides a cloud-based platform to integrate and deploy DNN models. However, the adoption of MLaaS brings concerns about data security and privacy. Data owners are concerned about the security and privacy of their data on MLaaS platforms, while model owners are worried about the leakage or adversarial attacks of their DL models. In fact, some companies have banned commercial data from being placed on public MLaaS platforms to prevent potential data leakage. Finding a balance between practicality and security in MLaaS is a research topic that urgently needs to be addressed.

[0003] Fully homomorphic encryption (FHE) is a groundbreaking concept that allows computation to be performed directly on encrypted data. The combination of FHE and MLaaS to create encrypted MLaaS can advance the development of artificial intelligence. In the encrypted MLaaS process, users encrypt their private data and send it to the cloud server, which performs homomorphic computations on the encrypted data for ML model inference and then sends the encrypted results back to the user, who can then decrypt the results and obtain the MLaaS results. This approach of using FHE can ensure semantic security and effectively prevent the cloud server from obtaining any information entered by the user.

[0004] The main bottleneck in integrating FHE with DNN models is the high cost of cryptographic computation. Specifically, computations that cannot be replaced or approximated by polynomial computations are the main obstacle. Computational operations in DNNs can generally be divided into linear and nonlinear operations. Linear operations including convolutional layers and fully connected layers have been widely studied in the context of FHE and have relatively low computational costs. However, nonlinear operations, such as activation functions such as Rectified Linear Unit (ReLU) or sigmoid functions, pose significant challenges in FHE. This is true for methods that use CKKS encryption schemes to approximate activation functions by polynomials, and other methods that use FHEW / TFHE schemes to bootstrap through functional restarts. Despite significant progress in the development of FHE in recent years, homomorphic computation of DNNs still requires a lot of computing time and resources, making FHE-based DNN models impractical for practical applications.

[0005] Although FHE-based DNN encrypted computing has important potential, current methods are still far from practical applications. How to effectively utilize FHE schemes to achieve low computational cost and high accuracy DNNs remains an unresolved challenge. Summary of the invention

[0006] In view of this, the present invention proposes an inference and conversion method of an encrypted deep neural network.

[0007] According to one embodiment of the present invention, an inference method of an encrypted deep neural network includes executing with a computing device, encoding a message according to a quantization parameter to generate a plaintext, encrypting the plaintext according to a private key to generate a ciphertext, sending the ciphertext to a deep neural network model to generate a ciphertext result, decrypting the ciphertext result according to the private key to generate a plaintext result, and decoding the plaintext result according to the quantization parameter to generate an inference result.

[0008] According to one embodiment of the present invention, a method for converting an encrypted deep neural network model includes: inputting a training data set to an integer model to extract multiple intermediate calculation results of multiple layers, and adding multiple noises to multiple outputs of the multiple layers according to the numerical range of the multiple intermediate calculation results; retraining the integer model with the multiple noises added to generate a floating-point fine-tuning model with the training data set; quantizing the model based on the floating-point fine-tuning model to generate an integer fine-tuning model; and inputting the training data set to the integer fine-tuning model to extract multiple quantization parameters, and the multiple quantization parameters are used to control the output values ​​of multiple excitation functions of the integer fine-tuning model.

[0009] The above description of the content of the present invention and the following description of the implementation modes are used to demonstrate and explain the spirit and principle of the present invention, and to provide a further explanation of the scope of the patent application of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a schematic diagram of an inference system of an encrypted inference neural network according to an embodiment of the present invention.

[0011] Figure 2 is a schematic diagram of a deep neural network model and quantization parameters according to an embodiment of the present invention.

[0012] Figure 3 is a flow chart of generating quantization parameters according to an embodiment of the present invention.

[0013] Figure 4 is a flow chart of the internal operation of an activation function according to an embodiment of the present invention.

[0014] Figure 5 It is a flowchart of a method for converting an encrypted deep neural network model proposed according to an embodiment of the present invention.

[0015] Figure 6 It is a flowchart of a method for converting an encrypted deep neural network model proposed according to an embodiment of the present invention.

[0016] Figure 7 It is a schematic diagram of the relationship between the experimentally obtained quantization parameter value and the error percentage (noise level).

[0017] Component number description

[0018] 10. First computing device

[0019] 20 Second computing device

[0020] S1~S5 Procedure

[0021] Steps T1 to T3

[0022] Steps S31 to S33

[0023] Steps S331 to S333

[0024] U1~U8 Steps DETAILED DESCRIPTION

[0025] The detailed features and characteristics of the present invention are described in detail in the following embodiments, and the content is sufficient to enable any person skilled in the relevant art to understand the technical content of the present invention and implement it accordingly, and according to the content disclosed in this specification, the scope of the patent application and the drawings, any person skilled in the relevant art can easily understand the relevant concepts and characteristics of the present invention. The following examples are used to further illustrate the viewpoints of the present invention in detail, but are not intended to limit the scope of the present invention in any way.

[0026] Figure 1Schematic diagram of an inference system of an encrypted inference neural network according to an embodiment of the present invention. The inference system comprises a first computing device 10 of a data owner and a second computing device 20 of a model owner, and the two computing devices are communicatively connected to each other.

[0027] In one embodiment, the first computing device 10 and the second computing device 20 may be implemented using one or more of the following examples: a personal computer, a network server, a microcontroller (MCU), an application processor (AP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system-on-a-chip (SOC), a deep learning accelerator, or any electronic device with similar functions. The present invention does not limit the hardware types of the first computing device 10 and the second computing device 20.

[0028] Figure 2 is a flow chart of an inference method for an encrypted deep neural network according to an embodiment of the present invention. The inference method is applicable to Figure 1 The inference system shown.

[0029] In step S1 , the first computing device 10 encodes information according to a quantization parameter to generate a plain text.

[0030] m = Encode (Scale0, fp) = |fp × Scale0|, (Formula 1);

[0031] In one embodiment, the plaintext is generated as in the above method 1. The present invention adopts a learn with error (LWE) encryption system. Since LWE only supports encryption and decryption of integers, integer quantization parameters are used. The encoding function Encode(·) converts floating point type messages Converts plain text to integer type The modulus P represents the size of the plaintext message space. The method of generating the quantization parameter will be described in detail later.

[0032] In step S2, the first computing device 10 encrypts the message and the private key to generate a ciphertext.

[0033] s = KeyGen(·), (Formula 2);

[0034] In one embodiment, the private key is generated in the same manner as in the second embodiment. The KeyGen function of LWE is used to generate the public key (a, b) and the private key in is a random matrix, b=as+e mod Q, is the error vector and Q is the modulus of the ciphertext.

[0035] ct P / Q =Encrypt(s, P, Q, m), (Formula 3);

[0036] In one embodiment, the ciphertext is generated as in method 3 above. The input of the encryption function Encrypt(·) is the plaintext in integer form. And the private key s, the output is LWE ciphertext

[0037] In step S3, the first computing device 10 sends the ciphertext to the second computing device 20, and the deep neural network model running on the second computing device 20 performs encrypted inference to generate a ciphertext result, which is then transmitted back to the first computing device 10. The deep neural network model uses fully homomorphic encryption (FHE) technology and is a FHEW-variant deep neural network encrypted inference architecture. Details of this model are described in detail later.

[0038] In step S4 , the first computing device 10 receives the encrypted result and decrypts it according to the private key to generate a plain text result.

[0039] m=Decrypt(s,P,Q,ct P / Q ), (Formula 4);

[0040] In one embodiment, the plaintext result is generated as in the fourth method above, and the input of the decryption function Decrypt(·) is the LWE ciphertext ct P / Q and private key s, output the plaintext result in integer form

[0041] In step S5 , the first operation device 10 decodes the plaintext result according to the quantization parameter to generate an inference result.

[0042]

[0043] In one embodiment, the inference result is generated as in method 5 above. The input of the decoding function Decode(·) is the plaintext result in integer form. And the quantization parameters The output is the inferred result of floating point type

[0044] Before encoding the message according to the quantization parameter to generate plain text in step S1, the quantization parameter must be configured first. The following describes how the quantization parameter is generated:

[0045] Figure 2 FIG. 1 is a schematic diagram of a deep neural network model (using plain text) and quantization parameters according to an embodiment of the present invention. Figure 2 As shown, this model includes n+1 layers Layer1, Layer2, ..., Layer n+1 Except for the last layer n+1 Only execute Dot n+1 Operation, other layers i All by Dot i and Act i Two operations, where i = 1...n. Dot i Used to calculate the model weight W i and input data X i-1 The inner product (dot product). i For the Rectified Linear Unit (ReLU), use the quantization parameter Scale i Scaling output range. The following formulas 6 to 11 are Figure 2 Mathematical definition of the model M shown.

[0046]

[0047]

[0048]

[0049]

[0050]

[0051]

[0052] Figure 3 is a flow chart of generating quantization parameters according to an embodiment of the present invention, and Table 1 is an algorithm for generating quantization parameters. Figure 3 It includes step T1, encoding the message according to the default value of the quantization parameter to generate the encoding result (corresponding to lines 02, 06, and 13 of the algorithm); step T2, calculating the inner product result according to the encoding result and the weight of the deep neural network model (corresponding to lines 03, 09, and 16 of the algorithm); and step T3, generating the quantization parameter according to the size of the plaintext message space and the inner product result (corresponding to lines 04, 10, and 17 of the algorithm).

[0053] Table 1. Algorithm for generating quantization parameters.

[0054]

[0055]

[0056] Next, the details of step S3, where the deep neural network model performs encrypted inference to generate ciphertext results, are described:

[0057] The deep neural network model includes multiple neurons. When the FHEW-variant deep neural network model proposed in the present invention performs inference, the internal operation process of each neuron is as follows: Figure 4 As shown in Table 2. Figure 4 is a flowchart of the ciphertext generated by the deep neural network model according to an embodiment of the present invention, and Table 2 corresponds to Figure 4 The algorithm of the process.

[0058] Table 2. Internal operation algorithm of FHEW-variant neuron.

[0059]

[0060] like Figure 4 As shown: Step S31, the second computing device 20 calculates the encrypted inner product result of the ciphertext and the model weight, corresponding to line 01 of the algorithm. The input of the DotProduct function is the k-dimensional plaintext array W k (representing model weights) and k-dimensional ciphertext array The output is the encrypted inner product of the two.

[0061] In one embodiment, the DotProduct function includes two steps, scalar multiplication and homomorphic encryption addition, as shown in Formula 12 and Formula 13 below, respectively.

[0062]

[0063]

[0064] Scalar multiplication ScaleMul based on the ciphertext array Contains multiple encrypted values and model weight W k Contains multiple scalar values ​​w i Compute multiple scalar products Where i=1,...,k.

[0065] Homomorphic encryption addition HEAdd is based on the multiple scalar products Perform homomorphic encrypted addition to produce an encrypted inner product result.

[0066] Step S32, the second operation device 20 reduces the first modulus of the encrypted inner product result to a second modulus to generate an analog-to-digital conversion result, corresponding to line 02 of the algorithm. The input of the ModSwitch function is the first modulus Q in The encrypted inner product result of The output is the analog-to-digital conversion result ct with the second modulus q P / q .

[0067] Step S33, input the analog-to-digital conversion result to the activation function to generate a ciphertext result, corresponding to line 03 of the algorithm. The present invention proposes an activation function ActFunBoot with functional bootstrapping to support encryption calculation of any nonlinear function. In one embodiment, the nonlinear function is a rectified linear unit (ReLU). Figure 5 is a flowchart of the internal operation of ActFunBoot according to an embodiment of the present invention. Table 3 corresponds to Figure 5 The algorithm.

[0068]

[0069] Step S331, based on the analog-to-digital conversion result ct P / q The message space size P and the encrypted inner product result The first modulus Q in Calculate the first ratio α, corresponding to line 01 of the algorithm.

[0070] Step S332, based on the first ratio α and the analog-to-digital conversion result ct P / q The second ratio α′ is calculated based on the message space size P and the quantization parameter Scale, corresponding to the second line of the algorithm.

[0071] Step S333, based on the second ratio α′, the lookup table f ACT And the analog-to-digital conversion result ct P / q Execute the function evaluation algorithm EvalFunc to generate a ciphertext result. In one embodiment, the lookup table f ACT It is used to map an input ciphertext to an output ciphertext to implement a nonlinear function such as ReLU, and the second ratio α′ is used to set the slope of ReLU.

[0072] The function evaluation algorithm EvalFunct(·) can eliminate the noise of the input ciphertext when performing function evaluation. Its input includes the ciphertext Extract function f in , and represents the message space The integer α, the output is via f in Estimated denoised ciphertext As shown in method 14 below.

[0073]

[0074] For the implementation of EvalFunct(·), please refer to the following literature: Z. Liu, D. Micciancio, and Y. Polyakov, "Large-precision homomorphic sign evaluation using FHEW / TFHEbootstrapping," in ASIACRYPT 2022, Part II, ser. LNCS, S. Agrawal and D. Lin, Eds., vol. 13792. Springer, Heidelberg, Dec. 2022, pp. 130-160.

[0075] When the existing FHEW technology performs a bootstrapping operation, the modulus of the ciphertext increases from q to Q, and Q is much larger than q, and then the ModSwitch function is used to reduce the modulus back to q, which limits the size of the message space. This limitation makes the existing FHEW technology unable to perform encryption inferences beyond Boolean operations. In order to modify FHEW to make it suitable for low-bit integer calculation requirements, the present invention modifies the original FHEW gate operation and develops a new FHEW-variant deep neural network neuron architecture, such as Figure 4 As shown in Table 2. Neurons include two operations: linear calculation (such as convolution or fully connected layer) and nonlinear activation function. Encrypted linear calculation is achieved by performing inner product on ciphertext and model weights, and a larger modulus Q is used to perform linear calculation on ciphertext, thereby accelerating the calculation of deep neural networks. Subsequently, the linear operation ciphertext (encrypted inner product result) is converted from a larger first modulus Q to a smaller second modulus q through modulus conversion. Then, by looking up the table f ACT Mapping is performed and a bootstrap operation is performed on the ciphertext to reduce noise.

[0076] Next, it is explained how to establish the FHEW-variant deep neural network model proposed in the present invention starting from the training data set. Figure 6 It is a flowchart of a method for converting an encrypted deep neural network model proposed according to an embodiment of the present invention.

[0077] In step U1 and step U2, the second computing device 20 trains a teacher model according to the training data set, and then performs knowledge distillation according to the teacher model to generate a student model.

[0078] The application of homomorphic encryption may incur considerable computational overhead, which requires a lot of computing resources and time. Therefore, techniques for compressing or simplifying deep neural network models must be used to facilitate execution in an encrypted environment. Knowledge distillation is a technique for converting knowledge from a large and complex model (called a teacher model) into a smaller and simpler model (called a student model). This technology can improve the performance of the student model while reducing its computational requirements. Due to its simplicity and flexibility, in one embodiment, the knowledge-distillation-pytorch library is used to implement it. This library supports various types of distillation methods, such as mean square error and cross entropy distillation, as well as different types of temperature scaling.

[0079] In step U3 , the second operation device 20 performs model quantization according to the student model to generate an integer model.

[0080] FHEW technology can only use integers for encryption and calculation, so the deep neural network model needs to be converted from floating point format to integer format. Post-Training Quantization (PTQ) is a technology that converts the weights of a trained deep neural network model into low-precision integers (e.g., 8-bit integers). PTQ is usually performed after the deep neural network model training is completed. The quantization process converts the original floating point weights and activation values ​​into integers.

[0081] The following steps U4 to U8 are the noise-aware fine-tuning method proposed by the present invention, so as to enhance the resistance of deep neural networks to noise. This method is used to reduce the adverse effects of noise on the performance of deep neural networks, thereby improving their accuracy and reliability. The existing method is to use Gaussian noise injection to fine-tune the trained neural network model to prevent overfitting. Using noise injection to fine-tune the model can smooth the decision boundary and ultimately avoid the overfitting problem. The present invention assumes that a smooth decision boundary can improve the robustness of the model to noise during the calculation process. By assuming that the errors generated by FHEW technology are some type of noise injection, the model can be fine-tuned to achieve tolerance to errors.

[0082] Step U4, input the training data set to the integer model to extract multiple intermediate operation results of multiple layers. Step U5, add multiple noises to the multiple outputs of the multiple layers according to the numerical range of the multiple intermediate operation results. Step U6, retrain the integer model with the multiple noises added using the training data set to generate a floating-point fine-tuning model. Step U7, perform model quantization based on the floating-point fine-tuning model to generate an integer fine-tuning model. Step U8, input the training data set to the integer fine-tuning model to extract multiple quantization parameters, which are used to control the output values ​​of multiple excitation functions of the integer fine-tuning model.

[0083] Specifically, consider a neural network with L layers, and the weight of each layer is W k , and ReLU is used as the activation function. When the input data x of the training data set is input into the deep neural network, the output of the activation function of the kth layer is h k , as shown in method fifteen below.

[0084]

[0085] Where k = 1, ..., (L-1); the value range of the intermediate operation result of the kth layer is [-R k , R k ].

[0086] In the addition of uniform distribution ∈ k (uniform distribution) and the range is [-R k , R k ] After the noise reaches each layer, the output of the kth layer becomes as shown in the following formula sixteen.

[0087]

[0088] Note that the uniform distribution R k R is determined by selecting a suitable quantization parameter value with the minimum noise level. k . Figure 7 : is a diagram showing the relationship between the experimentally obtained quantization parameter value and the error percentage (noise level). The selected quantization parameter value can make the noise added to each layer similar to the noise distribution generated by ActFunBoot. In one embodiment, a quantization parameter value corresponding to the minimum error percentage is selected. Then, R is calculated using the following method 17: k .

[0089]

[0090] Among them, e kis the percentage of errors generated by ActFunBoot in layer k. Then, we can use R k Defining the noise added to each layer during fine-tuning and retraining the weights using the deep neural network model with added noise can enhance the robustness of the model.

[0091] In summary, the neurons of the deep neural network model proposed in the present invention consist of an encrypted inner product calculation and a cost-effective integer-based functional bootstrap activation function (ActFunBoot). ActFunBoot supports encrypted calculations of arbitrary nonlinear functions and fast scalar division of ciphertext. ActFunBoot allows adjustment of quantization parameters to control the scale and noise level of the ciphertext output of the neurons of the deep neural network model.

[0092] The present invention proposes a conversion method for an encrypted deep neural network model to solve the numerical overflow and noise problems generated during the encryption calculation process. The method of the present invention utilizes fine-tuning of the encrypted deep neural network and estimation of the ActFunBoot quantization parameters to achieve a balance between the precision limitations in the design choices and the computational accuracy required to implement the FHEW-variant deep neural network. The proposed fine-tuning process involves the following three optimization aspects: 1. Through model distillation and model quantization, the pre-trained network is converted into a smaller quantized network to meet the acceleration requirements of FHEW-variant operations; 2. The quantization parameters of the FHEW-variant deep neural network neurons are estimated, which can effectively prevent numerical overflow and error accumulation; 3. The noise-aware fine-tuning method proposed in the present invention can optimize the model weights for the FHEW-variant deep neural network, thereby further improving the overall accuracy of the deep neural network inference.

[0093] Table 4 below compares the method of the present invention with the existing FHE method (Q. Lou and L. Jiang, "SHE: A fast and accurate deep neural network for encrypted data," in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, HM Willach, H. Larochelle, A. Beygelzimer, F. d'Alch′e-Buc, E B Fox, and R. Garnett, Eds., 2019, pp. 10 035-10 043).

[0094]

[0095] The data in Table 4 is the encrypted inference of a 6-layer deep neural network for classifying a 32×32×3 CIFAR-10 image, where the running time is in seconds. It can be seen from Table 4 that the method of the present invention is superior to the existing encrypted inference method in terms of efficiency. The total execution time of the present invention only takes 642 seconds, while achieving a test accuracy of 80%. In addition, the memory used during runtime is less than 20GB. It is worth noting that the running time of the present invention is significantly reduced compared to SHE.

[0096] Although the present invention is disclosed as above with the aforementioned embodiments, it is not intended to limit the present invention. Without departing from the spirit and scope of the present invention, all changes and modifications are within the scope of patent protection of the present invention. Please refer to the attached patent application for the scope of protection defined by the present invention.

Claims

1. An inference method for an encrypted deep neural network model, characterized in that: The method comprises executing, by a computing device: Encoding a message according to a quantization parameter to generate a plaintext; Encrypting the plaintext according to a private key to generate a ciphertext; Sending the ciphertext to a deep neural network model to generate a ciphertext result; Decrypting the ciphertext result according to the private key to generate a plaintext result; as well as The plaintext result is decoded according to the quantization parameter to generate an inference result.

2. The inference method of the encrypted deep neural network model according to claim 1, characterized in that: Before encoding the message according to the quantization parameter to generate the plaintext, the method further includes: encoding the message according to a default value of the quantization parameter to generate an encoding result; Calculating an inner product result according to the encoding result and a weight of the deep neural network model; and The quantization parameter is generated according to the message space size of the plaintext and the inner product result.

3. The inference method of the encrypted deep neural network model according to claim 1, characterized in that: Sending the ciphertext to the deep neural network model to generate the ciphertext result includes: Calculate an encrypted inner product result of the ciphertext and a model weight; reducing a first modulus of the encrypted inner product result to a second modulus to generate an analog-to-digital conversion result; and The analog-to-digital conversion result is input into an excitation function to generate the ciphertext result.

4. The inference method of the encrypted deep neural network model according to claim 3, characterized in that: Calculating the encrypted inner product result of the ciphertext and the model weight includes: Calculating a plurality of scalar products according to a plurality of encrypted values ​​included in the ciphertext and a plurality of scalar values ​​included in the model weight; and Homomorphic encryption addition is performed according to the multiple scalar products to generate the encrypted inner product result.

5. The inference method of the encrypted deep neural network model according to claim 3, characterized in that: Inputting the analog-to-digital conversion result into the excitation function to generate the ciphertext result includes: Calculating a first ratio of the message space size of the analog-to-digital conversion result to the first modulus of the encrypted inner product result; Calculating a second ratio according to the first ratio, the information space size of the analog-to-digital conversion result, and the quantization parameter; and A function estimation algorithm is executed according to the second ratio, a lookup table and the analog-to-digital conversion result to generate the ciphertext result.

6. The inference method of the encrypted deep neural network model according to claim 5, characterized in that: The lookup table is used to implement a linear rectification unit, and the second ratio is used to set a slope of the linear rectification unit.

7. A method for converting an encrypted deep neural network model, characterized in that: include: Inputting a training data set into an integer model to retrieve a plurality of intermediate operation results of a plurality of layers; Adding multiple noises to the multiple outputs of the multiple layers respectively according to the numerical range of the multiple intermediate operation results; Retraining the integer model after adding the multiple noises with the training data set to generate a floating point fine-tuning model; Performing model quantization according to the floating-point fine-tuning model to generate an integer fine-tuning model; as well as The training data set is input to the integer fine-tuning model to extract a plurality of quantization parameters, where the plurality of quantization parameters are used to control output values ​​of a plurality of activation functions of the integer fine-tuning model.

8. The method for converting an encrypted deep neural network model according to claim 6, characterized in that: Also includes: Before inputting the training data set into the integer model to retrieve the plurality of intermediate operation results, training a teacher model according to the training data set; Performing knowledge distillation based on the teacher model to generate a student model; and The model is quantized according to the student model to generate the integer model.

Images