An operator fitting based deep neural network device for quantum state conversion

Abstract

The application realizes a deep neural network secret state conversion device based on operator fitting through the method in the network security field. It includes an input unit, a fitted additive attention mechanism and an output unit. The fitted additive attention mechanism includes three implementation steps: step one, generating a secret state model based on the additive attention algorithm fitting the original attention calculation method; step two, designing a new attention matrix to obtain a high-performance secret state model; step three, using homomorphic encryption request data to realize efficient inference in the secret state model and output encrypted data. The method provided by the application can optimize the neural network model structure and reduce resource consumption; the model training of the secret state neural network reduces the performance loss to the minimum; a unified process of neural network secret state conversion is formed, and the effect of a unified calling process of using a secret state model to obtain an inference result of homomorphic encryption request data is provided.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a dense-state transformation device for deep neural networks based on operator fitting. Background Technology

[0002] With the further development of artificial intelligence, neural network models are being applied to an increasing number of privacy-sensitive fields, such as financial institutions, government departments, and healthcare, making data privacy and security a critical issue for the large-scale application of AI. In traditional AI dialogue services, users typically need to transmit raw, sensitive data to the cloud-based AI service to obtain the corresponding plaintext inference results. In this model, the user's raw input and output data can be directly obtained by the server, significantly compromising user data privacy and security, while also limiting the effectiveness of advanced AI technologies in critical and sensitive areas.

[0003] To address the privacy and security issues of neural networks, technologies such as differential privacy, secure multi-party computation, and homomorphic encryption have been introduced into the field of deep learning. However, differential privacy and secure multi-party computation aim to protect the data security of machine learning model builders and are not suitable for scenarios providing AI cloud services. Homomorphic encryption, on the other hand, is a non-interactive, fully encrypted computation scheme that can largely mitigate the risk of exposing raw data and intermediate neural network computation results in scenarios where "models are deployed in the cloud, and clients request inferences from the cloud."

[0004] The CKKS homomorphic encryption algorithm supports both addition and multiplication operators and floating-point arithmetic, making it a widely used homomorphic encryption scheme in machine learning. However, the exponentially increasing parameter size and numerous matrix multiplications in modern neural network models, such as the Transformer skeleton model, limit the application scope of homomorphic encryption. Specifically, as the depth of the neural network increases, the multiplication circuit for a single neural network inference becomes very deep. This leads to: 1. Rapid consumption of the space for the ciphertext scaling factor, resulting in reduced computational accuracy; 2. The use of bootstrapping to recalculate the computational depth to prevent noise in the ciphertext from exceeding a threshold. Bootstrapping operations are extremely expensive, and when the multiplication circuit depth is very large, frequent bootstrapping operations are needed to support a single ciphertext computation, significantly increasing the computational overhead.

[0005] Therefore, limiting the depth of multiplication calculations in neural networks, enabling neural network models to initially achieve encrypted computation capabilities, and further reducing the risk of privacy exposure, has become an important requirement for the deployment and development of future artificial intelligence models.

[0006] As mentioned above, homomorphic encryption schemes can protect the privacy of user request data in cloud-based artificial intelligence services. However, the multiplication depth in the current Transformer architecture's neural networks increases the cost of homomorphic encryption schemes, limiting the application potential of ciphertext inference. Summary of the Invention

[0007] To this end, the present invention first proposes a deep neural network dense-state transformation device based on operator fitting, including an input unit that inputs the artificial intelligence dialogue data to be encrypted, converts the natural language text of the input dialogue content into a floating-point number type, performs homomorphic encryption, and then uses an additive attention mechanism of input fitting to transform the traditional Transformer model into a dense-state model, as well as an output unit.

[0008] The additive attention mechanism described in the fitting transforms the traditional Transformer model into a dense-state model, and includes three implementation steps:

[0009] Step 1: Fit the original attention calculation method based on the additive attention algorithm. On the basis of the original Transformer skeleton model, use the fitted additive attention mechanism to transform the traditional Transformer model into a dense model and generate a dense model based on the additive attention mechanism.

[0010] Step 2: Design a new attention matrix and use a layer-by-layer distillation method to transfer the parameters of the original model to the new dense neural network. In the distillation of the attention layer, the attention score output loss, softmax output loss and attention layer output loss are measured respectively to obtain a high-performance dense model.

[0011] Step 3: Use homomorphically encrypted request data to perform efficient inference in the encrypted model and output the encrypted output text.

[0012] The specific implementation method of the additive attention mechanism for fitting is as follows: similarity is measured using L1 distance, as shown in the following formula:

[0013] L1(x,y)=|x1-y1|+|x2-y2|+...+|x n -y n |

[0014] The similarity score matrix S satisfies S i，j =L1(qi, k j ).

[0015] The HardTanh function limits the output to [-1, 1]. The HardTanh function formula is as follows:

[0016]

[0017] Additive attention is calculated as follows: Attention add = softmax(HT(S)).

[0018] The attention matrix is ​​constructed as follows: A layer-by-layer distillation method is used. In the distillation of each attention layer, the HENet network learns and fits the attention matrix from the original network. The teacher model is the attention layer of the Transformer skeleton model, and the student model is an additive attention layer. In each distillation layer, the loss can be defined as two parts: Q and V output losses, softmax output loss, and Att output loss. The softmax output loss and Att output loss are consistent with traditional knowledge distillation losses and are used to measure the similarity between the key and query / value matrices in the two models. The softmax output loss can be expressed as:

[0019]

[0020] in, These represent the softmax outputs of the teacher network's original attention mechanism and the student network's additive attention mechanism, respectively.

[0021] Subsequently, to measure the output loss of matrix V, the final result of the attention calculation is used as the loss, expressed as:

[0022] L att =MSE(AS i AT i )

[0023] Where AS represents the attention output of the student network, and AT represents the attention function output of the teacher network;

[0024] Finally, an additional learning loss is set for the attention score, and two kernel transformations are introduced to map the output distributions of the two networks to the same distribution. The two kernel transformations are as follows:

[0025]

[0026]

[0027] After transformations using the Laplace kernel and Gaussian kernel respectively, the QK matrix calculation results are output as follows:

[0028]

[0029]

[0030] Therefore, the distillation loss for attention score is:

[0031]

[0032] Finally, in the distillation of the attention layer, the loss is:

[0033] Loss = L softmnax +L att +L score .

[0034] The technical effect to be achieved by this invention is as follows: This technical solution is based on the CKKS homomorphic encryption scheme and the Transformer skeleton model. It uses a fitted additive attention mechanism to transform the traditional Transformer model into a dense model. The dense model parameters are trained through a hierarchical distillation method, achieving low resource consumption and low performance loss in dense model inference. This reduces the resource expenditure problem in artificial intelligence dialogue. The specific technical effect is as follows:

[0035] 1. Optimize the neural network model structure to construct a neural network structure suitable for homomorphic encryption inference. The encrypted model reduces a large number of matrix multiplication operations between variables, and the space consumption of the CKKS ciphertext scaling factor during model inference is greatly reduced, improving computational accuracy. Secondly, the frequency of using the extremely expensive bootstrapping is reduced, resulting in a significant reduction in ciphertext computation overhead. Reducing the depth of multiplication computation mitigates the resource consumption problem under homomorphic computation in the neural network model.

[0036] 2. A distillation technique for the attention layer is designed to transfer knowledge from the original model to a new dense-state model. Specifically, this invention employs targeted distillation loss designs for the Q, K, and V matrices of the attention layer, effectively balancing performance loss while preserving the characteristics of the additive attention layer. The dense-state neural network model is trained using a hierarchical distillation method, minimizing performance loss.

[0037] 3. By combining the replacement of the dense-state model with the training of its parameters, a unified process for dense-state transformation of neural networks is formed, and a unified calling process is provided for obtaining inference results using the dense-state model from homomorphic encrypted request data. It has strong scalability; the dense-state model transformation device designed in this invention can be applied to all deep neural networks with attention mechanisms based on K and V matrix calculations, including mainstream models such as Transformer and BERT. Attached Figure Description

[0038] Figure 1 Design framework for dense state transitions in deep neural networks; Detailed Implementation

[0039] The following are preferred embodiments of the present invention, which are described in conjunction with the accompanying drawings. However, the present invention is not limited to these embodiments.

[0040] This invention proposes a deep neural network dense-state transformation device based on operator fitting. It is applied to the "next word prediction service" in artificial intelligence services. For example, if a user types "today is a good" on the keyboard, the intelligent service predicts the next word as "day" and displays it on the user's phone keyboard for selection. In this case, if the user's keyboard input involves personal privacy, it is desirable for the data uploaded to the cloud server to be encrypted and protected. Similarly, in chatbots (similar to ChatGPT), where a user can ask questions like "My height is X, my weight is X, do I need to lose weight?", the cloud service will answer "no" for the user's reference. If the user's question involves personal privacy, it is undesirable to directly expose personal information to the cloud service provider; therefore, the user's question needs to be encrypted. The solution is based on the CKKS homomorphic encryption scheme and the Transformer skeleton model. It uses a fitted additive attention mechanism to transform the traditional Transformer model into a dense-state model. The dense-state model parameters are trained using a hierarchical distillation method to achieve low-resource consumption and low-performance loss inference of the dense-state model. Finally, the encrypted text is output, achieving the design purpose and application goals.

[0041] This method includes the following parts and processes:

[0042] Step 1: Fit the original attention calculation method based on the additive attention algorithm, and generate a dense state model based on the additive attention mechanism on the basis of the original Transformer skeleton model.

[0043] Step 2: Transfer the parameters of the original model to the new dense neural network using a layer-by-layer distillation method. In the distillation of the attention layer, measure the attention score output loss, softmax output loss, and attention layer output loss respectively to obtain a high-performance dense model.

[0044] Step 3: Request data using homomorphic encryption can be efficiently inferred in the encrypted model.

[0045] The overall framework is as follows: A fitted additive attention mechanism is used to transform the traditional Transformer model into a dense-state model. The parameters of the dense-state model are trained using a layered distillation method to achieve low-resource consumption and low-performance loss inference, thus fulfilling the design objectives and application goals. First, the original attention calculation method is fitted based on the additive attention algorithm, generating a dense-state model based on the additive attention mechanism on the original skeleton model. Since the additive attention mechanism changes the calculation method of the original attention matrix, parameters cannot be directly transferred from the original model. Therefore, this invention uses a layer-by-layer distillation method to transfer the parameters of the original model to the new dense-state neural network. In the distillation of the attention layers, the attention score output loss, softmax output loss, and attention layer output loss are measured separately to obtain a high-performance dense-state model. Finally, CKKS homomorphically encrypted request data can be used to efficiently perform inference tasks within the dense-state model.

[0046] The additive attention mechanism for fitting:

[0047] This invention fits the original attention calculation method with an additive attention algorithm, generating a dense-state model based on the additive attention mechanism on the original Transformer skeleton model. The original attention mechanism relies on a large number of multiplications, which significantly increases the overhead of ciphertext computation. Therefore, this invention improves the original dot-product attention mechanism by proposing an additive attention mechanism, which greatly reduces the number of multiplication operations.

[0048] The original formula for calculating dot product attention is as follows:

[0049]

[0050] attention(Q,K)=softmax(score(Q,K))

[0051] The similarity between the query matrix (Query) and the key matrix (Key) is measured using the dot product as a score. This score is then passed through a softmax function to obtain the final attention weights, which are then divided by... This is to prevent gradient vanishing. This invention uses L1 distance instead of the dot product to measure similarity. L1 distance is the sum of the absolute values ​​of the coordinate differences between two points, and does not involve multiplication.

[0052] The L1 distance formula is as follows:

[0053] L1(x,y)=|x1-y1|+|x2-y2|+...+|x n -y n |

[0054] The final similarity score matrix S satisfies S i，j =L1(qi k j ).

[0055] Meanwhile, to prevent gradient vanishing and gradient exploding, this invention uses the HardTanh function to limit the output to [-1, 1]. The HardTanh function formula is as follows:

[0056]

[0057] In summary, the calculation method for additive attention is as follows:

[0058] Attention add =softmax(HT(S))

[0059] Attention add The calculation formula will be used in self-attention layers and multi-head attention layers, significantly reducing the number of multiplication operations.

[0060] The specific steps for fitting the attention matrix parameters are as follows:

[0061] This invention employs an additive attention matrix distillation method, enabling the student attention network to learn information from the teacher attention network. By using a layer-by-layer distillation approach, in the distillation of attention layers, we aim to have the HENet network learn and fit the attention matrix in the original network. Here, the teacher model is the attention layer of a Transformer skeleton model (e.g., a pre-trained model like Bert-base), and the student model is an additive attention layer. In each distillation layer, the loss can be defined as two parts: Q and V output losses, softmax output loss, and Att output loss. The softmax output loss and Att output loss are consistent with traditional knowledge distillation losses, used to measure the similarity between the key and query / value matrices outputs in the two models. The softmax output loss can be expressed as:

[0062]

[0063] in, These represent the softmax outputs of the teacher network's original attention mechanism and the student network's additive attention mechanism, respectively.

[0064] Subsequently, to measure the output loss of matrix V, the final result of the attention calculation is used as the loss, expressed as:

[0065] L att =MSE(AS i AT i )

[0066] Where AS represents the attention output of the student network, and AT represents the attention function output of the teacher network.

[0067] Finally, to better learn the Q and K matrix parameters of the teacher network, it is desirable to set an additional learning loss for the attention score. However, due to the significant difference in the calculation method of the attention score, the student network cannot directly learn the parameter distribution and output distribution of the teacher network. Therefore, it is necessary to introduce two kernel transformations to map the output distributions of the two networks to the same distribution. The two kernel transformations are as follows:

[0068]

[0069]

[0070] After transformations using the Laplace kernel and Gaussian kernel respectively, the QK matrix calculation results are output as follows:

[0071]

[0072]

[0073] Therefore, the distillation loss for attention score is:

[0074]

[0075] Finally, in the distillation of the attention layer, the loss is:

[0076] Loss = L softmax +L att +L score .

Claims

1. A deep neural network dense-state transformation device based on operator fitting, characterized in that: The system includes an input unit that takes in AI dialogue data that needs to be encrypted, converts the natural language text of the input dialogue content into floating-point numbers, performs homomorphic encryption, uses an additive attention mechanism to fit the input to transform the traditional Transformer model into a dense model, and an output unit. The additive attention mechanism described in the fitting transforms the traditional Transformer model into a dense-state model, and includes three implementation steps: Step 1: Fit the original attention calculation method based on the additive attention algorithm. On the basis of the original Transformer skeleton model, use the fitted additive attention mechanism to transform the traditional Transformer model into a dense state model and generate a dense state model based on the additive attention mechanism. Step 2: Design a new attention matrix and use a layer-by-layer distillation method to transfer the parameters of the original model to the new dense neural network. In the distillation of the attention layer, the attention score output loss, softmax output loss and attention layer output loss are measured respectively to obtain a high-performance dense model. Step 3: Use homomorphically encrypted request data to perform efficient inference in the encrypted model and output the encrypted output text; The specific implementation method of the additive attention mechanism for fitting is as follows: similarity is measured using L1 distance, as shown in the following formula: Similarity score matrix satisfy ; The HardTanh function limits the output to [-1, 1]. The HardTanh function formula is as follows: The additive attention calculation method is as follows: .

2. The deep neural network dense state transformation device based on operator fitting as described in claim 1, characterized in that: The attention matrix is ​​constructed as follows: using a layer-by-layer distillation method, the HENet network learns and fits the attention matrix from the original network during the distillation of each attention layer. The teacher model is the attention layer of the Transformer skeleton model, and the student model is an additive attention layer. In each distillation process, the loss can be defined as two parts, namely… Output loss, Output loss and Output loss, where Output loss and The output loss is consistent with the traditional knowledge distillation loss, used to measure the similarity between the outputs of the key and query / value matrices in the two models. The output loss can be expressed as: in, , These represent the softmax outputs of the teacher network's original attention mechanism and the student network's additive attention mechanism, respectively. Subsequently, in order to measure the matrix The output loss, taking the final result of the attention calculation as the loss, is expressed as: Where AS represents the attention output of the student network, and AT represents the attention function output of the teacher network; Finally, an additional learning loss is set for the attention score, and two kernel transformations are introduced to map the output distributions of the two networks to the same distribution. The two kernel transformations are as follows: After undergoing Laplace kernel and Gaussian kernel transformations respectively, The matrix calculation results are output as follows: Therefore, the distillation loss for attention score is: Finally, in the distillation of the attention layer, the loss is: 。

Images