Method and device for performing model reasoning by using dense state neural network

Abstract

The embodiment of the invention provides a method and a device for performing model reasoning by using a dense state neural network. The dense state neural network comprises a plurality of linear layers. The method comprises: receiving ciphertext data and key data of a user, the key data being a part of data of a conversion key in a target format, and the conversion key in the target format comprising a plurality of extended ciphertexts; then, ciphertext data are input into the encrypted neural network for encrypted processing, the encrypted processing comprises the step that in any target linear layer in a plurality of linear layers, a first data part corresponding to a first ciphertext grade t in a conversion key is used for carrying out ciphertext rotation operation, and the first data part only comprises part of data of a single extended ciphertext.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of cryptography, and more particularly, to a method and apparatus for performing model reasoning using a dense neural network. Background Art

[0002] In today's highly digitalized context, protecting data security and maintaining personal privacy have become issues that cannot be ignored, which has given rise to an emerging technology field - privacy computing. The implementation of this technology can perform data analysis and processing in an encrypted state, ensuring the confidentiality and security of sensitive data during use. Fully homomorphic encryption technology (FHE) is an important technical means in the field of privacy computing. It can complete high-level computing tasks of data in a state where the data is encrypted as ciphertext, ensuring accurate calculations while protecting the privacy of the data. With the continuous in-depth application in cloud computing, secure data sharing, and multi-faceted information collaboration platforms, homomorphic encryption is gradually building a bridge that connects the needs of efficient data utilization and strict data protection, leading the field of secure computing to a new stage of development.

[0003] Dense neural network reasoning is a very important application scenario of fully homomorphic encryption (FHE). In this application scenario, after the input data of the network is encrypted into FHE ciphertext state by the private key of the data holder, each subsequent layer of network operation (convolution, pooling, activation, full connection, etc.) and the final network reasoning result output are all in ciphertext state. After the output ciphertext is returned to the data holder, the reasoning result can be obtained through FHE decryption operation, and the reasoning calculation party of the network model cannot obtain any information of the reasoning data throughout the process. Dense neural network reasoning has great uses in various industries. For example, in the field of medical health, hospitals can use dense neural networks to predict diseases or recommend treatment plans without disclosing patients' personal information; in financial services, financial institutions can safely analyze encrypted financial records and conduct credit assessments.

[0004] Therefore, we hope to have an improved solution to enhance the reasoning performance of dense neural networks. Summary of the invention

[0005] One or more embodiments of this specification describe a solution for performing dense reasoning using a dense neural network, which improves the performance of dense reasoning by using a conversion key with a smaller amount of data through ciphertext level control.

[0006] According to a first aspect, a method for model reasoning using a dense neural network is provided, wherein the dense neural network includes a plurality of linear layers, and the method includes:

[0007] Receiving ciphertext data and key data of a user, wherein the key data is a part of the conversion key in the target format, wherein the conversion key in the target format includes a plurality of extended ciphertexts;

[0008] The ciphertext data is input into the dense neural network for dense processing, wherein the dense processing includes, at any target linear layer among the several linear layers, performing a ciphertext rotation operation using a first data part corresponding to a first ciphertext level t in the conversion key, wherein the first data part includes only partial data of a single extended ciphertext.

[0009] In one embodiment, the ciphertext single term in the ciphertext data includes L basic polynomials corresponding to L basic moduli, and the ciphertext single term of the extended ciphertext includes L first polynomials corresponding to the L basic moduli, and K second polynomials corresponding to K auxiliary moduli; the partial data of the single extended ciphertext is Lt first polynomials and K second polynomials; and the value of Lt is less than a preset threshold.

[0010] Furthermore, in one example, the target linear layer is a convolutional layer, and its operation consumes 2 levels of ciphertext levels; the first ciphertext level t is set so that Lt is greater than or equal to 3.

[0011] In another example, the operation of the target linear layer consumes 1 ciphertext level; the first ciphertext level t is set such that Lt is greater than or equal to 2.

[0012] According to one implementation, the cipher state processing includes, after the target linear layer, performing a bootstrapping operation, wherein the bootstrapping operation uses a second data portion corresponding to a second cipher text level t in the conversion key to perform cipher text level reduction processing.

[0013] In one embodiment, the key data transmitted by the user only includes the first data portion used by each linear layer and the second data portion used by the bootstrapping operation.

[0014] In a specific example, the number K of auxiliary modules is set to be equal to the preset threshold.

[0015] According to one embodiment, before performing the ciphertext rotation operation, in response to the ciphertext level input to the target linear layer being less than a first ciphertext level t, a ciphertext level raising operation is performed to raise the ciphertext level to the first ciphertext level t.

[0016] According to one embodiment, before the dense state processing, it also includes: determining the ciphertext level consumption of each network layer in the dense state neural network; according to the ciphertext level consumption, under the target constraint, determining the input ciphertext level of each network layer and the position of the bootstrap operation, so that the input ciphertext level of each linear layer is higher than the preset level threshold, and the target constraint includes that the output ciphertext level of each network layer is not greater than the highest ciphertext level.

[0017] According to a second aspect, a device for performing model reasoning using a dense neural network is provided, wherein the dense neural network includes a plurality of linear layers, and the device includes:

[0018] A receiving unit configured to receive ciphertext data and key data of a user, wherein the key data is a portion of data of a conversion key in a target format, wherein the conversion key in the target format includes a plurality of extended ciphertexts;

[0019] The encrypted processing unit is configured to input the ciphertext data into the encrypted neural network for encrypted processing, wherein the encrypted processing includes, at any target linear layer among the plurality of linear layers, performing a ciphertext rotation operation using a first data portion corresponding to a first ciphertext level t in the conversion key, wherein the first data portion includes only partial data of a single extended ciphertext.

[0020] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect.

[0021] According to a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein an executable code is stored in the memory, and when the processor executes the executable code, the method of the first aspect is implemented.

[0022] In the embodiments of this specification, by designing the ciphertext level of each network layer of the dense neural network, the amount of data effectively used in the Evk data can be reduced, the amount of Evk data can be compressed, and the performance of the dense neural network reasoning can be improved. Furthermore, the number of auxiliary moduli can be reduced to further reduce the amount of Evk data and further reduce the transmission and storage costs of Evk data. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0024] Figure 1 Shows the main structure of a typical neural network;

[0025] Figure 2 Show the process of performing vector inner product under homomorphic operation;

[0026] Figure 3A The ciphertext data format in a specific example is shown;

[0027] Figure 3B The data format of the conversion key Evk in a specific example is shown;

[0028] Figure 4A The process of BV decomposition at ciphertext level 0 is shown;

[0029] Figure 4B The process of BV decomposition at ciphertext level 8 is shown;

[0030] Figure 4C The process of BV decomposition at ciphertext level 9 is shown;

[0031] Figure 4D The process of BV decomposition at ciphertext level 26 is shown;

[0032] Figure 5A The process of expanding the inner product of the ciphertext single term and Evk at ciphertext level 0 is shown;

[0033] Figure 5B The process of expanding the inner product of ciphertext single term and Evk at ciphertext level 8 is shown;

[0034] Figure 5C The process of expanding the inner product of ciphertext single term and Evk at ciphertext level 9 is shown;

[0035] Figure 5D The process of expanding the inner product of ciphertext single term and Evk at ciphertext level 26 is shown;

[0036] Figure 6 Shows the maximum level at which each part of Evk will be used under a given parameter instance;

[0037] Figure 7 The process steps for determining the density level of a dense neural network in one embodiment are shown;

[0038] Figure 8 An example of setting the ciphertext level of a dense neural network in one embodiment is shown;

[0039] Fig. 9 A schematic diagram showing a flow chart of model processing using a dense neural network in one embodiment;

[0040] Fig.10A schematic diagram of a device for performing model processing using a dense neural network in one embodiment is shown. DETAILED DESCRIPTION

[0041] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0042] Fully Homomorphic Encryption (FHE) is an encryption technology with homomorphic operation properties, which allows calculations (such as addition, subtraction, multiplication, etc.) to be performed directly on encrypted data. After the calculation is completed, the result is still encrypted, and only those with the correct key can decrypt and see the original data and calculation results. This provides a powerful privacy protection method for data processing.

[0043] Encrypted neural network reasoning is to perform reasoning calculations on encrypted data using fully homomorphic encryption or other encryption schemes. This means that both the input / output data and the calculation process remain encrypted, ensuring data privacy in machine learning / deep learning applications, which is of great significance in fields such as medicine and finance that have strict requirements on data privacy.

[0044] Generally, the neural network structure will alternately contain linear operation layers (such as convolutional layers, fully connected layers, etc.) and activation layers (such as Relu, Sigmoid, etc.). Figure 1 The main structure of a typical neural network is shown, which alternately includes convolutional layers and Relu layers. The classic neural network Resnet20 adopts such a main network structure. It can be seen that activation layers such as Relu layers generally perform calculations independently for each node in the layer, while linear layer calculations such as convolutional layers involve calculations between nodes in the layer. Typically, homomorphic operations generally use dedicated acceleration hardware and SIMD (single instruction multiple data) methods for parallel computing operations, and under the SIMD characteristics, homomorphic calculations (such as addition, subtraction, and multiplication) can only occur between components at the same position of different vectors. Therefore, linear layer operations involving calculations between nodes in the layer require ciphertext rotation operations to complete the calculations.

[0045] Specifically, the most basic operation in the linear layer is the vector inner product. For example, the operation in the convolution layer is to continuously perform the inner product of the parameter vector of the convolution kernel with the feature vector of the corresponding area of ​​the convolution kernel in the feature map to generate the output feature value of the corresponding position in the feature map. The basic operation in the fully connected layer is also to perform the inner product of each parameter vector in the parameter matrix on the feature vector. Figure 2The process of performing vector inner product under homomorphic operation is shown. As shown in the figure, the dense state vector Ma is stored in the first vector register, and the dense state vector Mb is stored in the second vector register. The elements at corresponding positions in the two vector registers can be densely multiplied by SIMD to obtain the dense state vector Mc. The dense state vector here is shown as 16 dimensions. However, the operation under the SIMD mode does not support the operation between elements at different positions in the same vector register. Therefore, it is impossible to directly sum the values ​​of each element of the dense state vector Mc. To this end, the dense state vector Mc can be rotated once, and the rotation step is half of the vector dimension, so that the first half of the vector is moved to the second half. In the case of a 16-dimensional vector, the elements with indexes 0-7 are moved to positions 8-15. Then, the elements at corresponding positions in the Mc vector before rotation and the Mc vector after rotation are densely added by SIMD to obtain the dense state vector Mc2. At this time, each element in the dense state vector Mc2 is the homomorphic sum of two elements in the original Mc vector. Next, the dense state vector Mc2 is rotated again, with a rotation step of 1 / 4 of the vector dimension, and then the Mc2 before and after the rotation is added to obtain the dense state vector Mc4. By performing rotation operations with different step lengths and performing vector addition multiple times, a result vector can be obtained, in which each element is equal to the sum of each element in the original Mc vector, thereby realizing the vector inner product.

[0046] The ciphertext rotation operation needs to be completed using the evaluation key (Evaluation Key, Evk). More specifically, after the ciphertext is rotated under homomorphic operation, the key form corresponding to the ciphertext will change. In order to continue the homomorphic operation, the ciphertext needs to be converted from the changed key back to the original key through key switching. The key switching process depends on the use of the evaluation key Evk.

[0047] Since the linear layer calculation of the dense neural network frequently performs ciphertext rotation operations at different step sizes, the data holder is required to provide the conversion keys Evk at all the required rotation step sizes, and the use of these Evk is almost everywhere in all the linear operation layers of the network. In an implementation environment with limited storage space capabilities (such as GPU, FPGA, ASIC, etc.), the storage and access of a large number of conversion keys Evk is a difficult problem. A large amount of Evk data usually cannot be loaded into on-chip storage at one time, and the acceleration hardware needs to frequently access the main memory for loading. This also has an adverse impact on the efficiency of dense reasoning.

[0048] To this end, in the scheme of this specification, after fully studying the changing rules of ciphertext levels and the use of Evk in dense neural network reasoning, the inventors proposed to make targeted settings for the ciphertext level of the linear layer to compress and reduce the data volume of the required conversion key Evk, so as to reduce the storage, memory access, transmission volume, etc. of Evk used in the operation of the dense network, thereby improving the performance of dense neural network reasoning.

[0049] The following first describes the data format changes of the ciphertext and conversion key Evk in the dense neural network inference.

[0050] The CKKS (Cheon-Kim-Kim-Song) scheme is a specific scheme for fully homomorphic encryption for numerical computing. Compared with other FHE schemes, CKKS is particularly suitable for processing real or complex operations with limited precision requirements. Therefore, it is very practical in applications that require floating-point operations such as machine learning and statistical analysis. It is one of the most commonly used FHE schemes in dense neural network reasoning. The following description takes the CKKS scheme as an example. However, it should be understood that in other similar FHE implementation schemes, the format and implementation principle of the ciphertext and conversion key Evk are similar to those of CKKS, so the conclusions drawn are equally applicable to other FHE schemes with similar implementation principles.

[0051] Specifically, in the CKKS scheme, the data format of the ciphertext C includes two polynomial groups, each of which is called a ciphertext monomial, and its length is L, that is, it contains L N-point polynomials (hereinafter referred to as polynomials). Each polynomial corresponds to a homomorphic modulus q_i, and the number of polynomials L contained in the two polynomial groups is generally the same.

[0052] The concept of ciphertext level is involved in ciphertext operations. In fully homomorphic schemes such as CKKS, ciphertext has a controllable amount of noise. Homomorphic operations will cumulatively increase the noise of the ciphertext. When the noise accumulates to a certain level, the ciphertext cannot be correctly decrypted. The ciphertext level can be understood as reflecting the noise level of the ciphertext. Generally, the initial ciphertext is level 0. After the ciphertext undergoes a homomorphic multiplication, the level of the resulting ciphertext is the input ciphertext level plus one.

[0053] For the above ciphertext items, the number of polynomials will decrease as the ciphertext level increases. For every increase in the ciphertext level, the number of polynomials will decrease by one, that is, the polynomial with the largest subscript in the homomorphic modulus q_i will be removed. Specifically, each ciphertext item in the ciphertext of level 0 contains L polynomials, and the corresponding homomorphic moduli are q_0, q_1, ..., q_{L-1}, and the number of polynomials contained in each ciphertext item in the ciphertext of level 1 is L-1, and the corresponding homomorphic moduli are q_0, q_1, ..., q_{L-2}, and so on. Therefore, after the FHE parameter set is selected and the length L of the specific ciphertext item is determined, the maximum noise limit of the ciphertext is determined, the highest level of the ciphertext supported is also determined, and the ciphertext multiplication depth supported by the scheme is also determined. The data size of the ciphertext decreases as the ciphertext level increases.

[0054] Figure 3A The ciphertext data format in a specific example is shown. In this example, the ciphertext single item length L = 27. Figure 3A As shown, the ciphertext C contains two ciphertext items, the polynomials of which are shown as c_A and c_B respectively. In the initial state corresponding to level 0, each ciphertext item contains L=27 polynomials, corresponding to 27 moduli q0 to q26. In level 1, the last modulus q26 and its polynomial are removed from each ciphertext item, so that 26 polynomials are included. For every increase of 1 in the level, the number of polynomials decreases by 1. Until the highest ciphertext level 26, each ciphertext item has only one polynomial corresponding to q0.

[0055] The data format of the conversion key Evk can be understood as a combination of multiple extended ciphertexts, where the extended ciphertext has a format similar to the aforementioned ciphertext C and is also composed of two ciphertext terms. However, the modulus corresponding to the polynomial in each ciphertext term in the extended ciphertext includes the GHS auxiliary modulus p_j in addition to the aforementioned homomorphic modulus q_i (hereinafter, this part of the modulus is referred to as the basic modulus). The total number of auxiliary moduli in the extended ciphertext is denoted as K, and the corresponding auxiliary moduli are p_0, p_1,…, p_{K-1}. The number of extended ciphertexts in the conversion key Evk of level 0 is denoted as dnum, and the number of polynomials contained in each extended ciphertext term is L+K, and the corresponding moduli are q_0, q_1,…, q_{L-1}, p_0, p_1,…, p_{K-1}.

[0056] In addition, the conversion key Evk also involves a concept called BV modulus segmentation. Specifically, the dnum extended ciphertexts in Evk will correspond to a certain number of homomorphic modulus segments, and the size of the modulus segment is recorded as W, which satisfies the relationship: dnum*W≥L but (dnum-1)*W <L。

[0057] In general, the BV modulus segment corresponding to the 0th extended ciphertext of Evk is q_0, q_1, …, q_{W-1}, the BV modulus segment corresponding to the 1st extended ciphertext is q_{W}, q_{W+1}, …, q_{2*W-1}, and so on. The BV modulus segment corresponding to the dnum-1th extended ciphertext is q_{(dnum-1}*W}, q_{(dnum-1}*W+1}, …, q_{L-1}.

[0058] Figure 3B The data format of the conversion key Evk is shown in a specific example. Figure 3A The example corresponds to that of , where L = 27. In addition, in the example, the Evk-specific parameters are set as follows: K = 8, dnum = 3, W = 9. Figure 3B As shown, the conversion key Evk includes dnum=3 extended ciphertexts, each of which includes 2 extended ciphertext items indicated by subscripts A and B, and each of which includes L+K polynomials, and the corresponding moduli are L=27 basic moduli composed of q0-q26, and K=8 auxiliary moduli composed of p0-p7. In addition, in the conversion key Evk, the BV modulus segment corresponding to the 0th extended ciphertext is q0-q8, the BV modulus segment corresponding to the 1st extended ciphertext is q9-q17, and the BV modulus segment corresponding to the 2nd extended ciphertext is q18 to q26.

[0059] Unless otherwise specified in the following content of this article, the parameter settings for ciphertext and conversion keys are the same as above.

[0060] The following describes the data flow process in the classic key switching algorithm. Since this solution focuses on the size of data and storage and transmission, the specific detailed algorithm steps will be omitted, and only the input and output data content will be introduced. Taking the classic BV-GHS key switching algorithm as an example, the key switching process can generally be divided into three steps: BV decomposition of ciphertext items, inner product of extended ciphertext items and Evk, and extended ciphertext reduction. Since extended ciphertext reduction does not involve the use of conversion key Evk and data format transformation, its operation process is not described here, and only the BV decomposition and inner product process are focused on.

[0061] 1) BV decomposition of ciphertext single item

[0062] In the BV decomposition stage of the ciphertext single term, the parameters involved in the operation include the ciphertext level t and the current number of BV segments u. The input of the operation is a ciphertext single term Cin of level t, which contains Lt polynomials with moduli q_0, q_1, …, q_{L-1-t} respectively. The Lt polynomials correspond to u BV segments. The 0th BV modulus segment is q_0, q_1, …, q_{W-1}, the 1st BV modulus segment is q_{W}, q_{W+1}, …, q_{2*W-1}, …, and the u-1th BV modulus segment is q_{(u-1}*W}, q_{(u-1}*W+1}, …, q_{Lt-1}.

[0063] The algorithm process of BV decomposition is as follows: for the input ciphertext single term Cin, extract the polynomial corresponding to the 0th BV modulus segment in Cin, perform the Bconv algorithm (the algorithm process is omitted here), and obtain CE_0; extract the polynomial corresponding to the 1st BV modulus segment in Cin, perform the Bconv algorithm, and obtain CE_1; and so on, until all u generated extended ciphertext single terms are obtained.

[0064] FIG. 4A to FIG. 4D The process of BV decomposition at different levels is shown. This example still uses the parameter settings of L=27, K=8, and W=9.

[0065] like Figure 4A As shown, at ciphertext level 0, the number of BV segments is 3. Extract the polynomial of the 0th BV modulus segment q0-q8 from the ciphertext single term Cin (shown in red), perform the Bconv algorithm, and obtain CE_0 (shown in red). Extract the polynomial of the 1st BV modulus segment q9-q17 from the ciphertext single term Cin (shown in blue), perform the Bconv algorithm, and obtain CE_1 (shown in blue). Then extract the polynomial of the 2nd BV modulus segment q18-q26 from the ciphertext single term Cin (shown in green), perform the Bconv algorithm, and obtain CE_2 (shown in green).

[0066] Figure 4B The BV decomposition of the ciphertext level 8 is shown. Figure 3A The principle shown is that when the ciphertext level t = 8, the length of the ciphertext single term Cin is reduced to Lt = 19, corresponding to the basic modulus q0 to q18. That is, compared with the original ciphertext single term, the polynomial of the basic modulus q19-q26 is reduced. At this time, the number of BV segments is still 3, but different from Figure 4A The second BV modulus segment only contains one polynomial corresponding to q18. Accordingly, the generated extended ciphertext single terms CE_0 to CE_2 all lack the terms corresponding to the basic modulus q19-q26.

[0067] Figure 4CThe BV decomposition of ciphertext level 9 is shown. Compared to Figure 4B It can be seen that as the ciphertext level increases further, the length of the ciphertext item Cin is further reduced to 18, corresponding to the basic modulus q0 to q17. At this time, the number of BV segments is reduced to 2. The process of BV decomposition only includes, based on the polynomial of the 0th BV modulus segment q0-q8 (shown in red), CE_0 (shown in red); and based on the polynomial of the 1st BV modulus segment q9-q17 (shown in blue), CE_1 (shown in blue). At this time, the generated extended ciphertext items only include CE_0 and CE_1, and both of them are missing the items corresponding to the basic modulus q18-q26.

[0068] As the ciphertext level increases further, it is conceivable that the length of the ciphertext item Cin further decreases. It can be understood that when the ciphertext level t=18, the number of BV segments decreases to 1, and the corresponding generated extended ciphertext item only includes CE_0.

[0069] Figure 4D The BV decomposition of the ciphertext level 26 is shown. At this highest level, the ciphertext single term Cin contains only one polynomial corresponding to q0, and the number of BV segments is 1. The generated extended ciphertext single term only includes CE_0, and the terms corresponding to the basic modulus q1-q26 are all missing.

[0070] 2) Inner product of extended ciphertext single item and Evk

[0071] In the inner product stage of the extended ciphertext single term and Evk, the parameters involved in the operation include the ciphertext level t and the current number of BV segments u. The input of the operation is the conversion key Evk (for example Figure 3B As shown), as well as u extended ciphertext items CE_0, CE_1, …, CE_{u-1} of level t generated by BV decomposition of the ciphertext item, which is the output of the previous step.

[0072] The algorithm process of this stage includes multiplying the extended ciphertext single term CE_0 with the 0th extended ciphertext of Evk to obtain CE_pro_0, where the multiplication operation only occurs between two polynomials with the same modulus, and the redundant polynomials in Evk will not participate in the operation. Multiply the extended ciphertext single term CE_1 with the 1st extended ciphertext of Evk to obtain CE_pro_1; and so on; then add u extended ciphertext single terms CE_pro_0, CE_pro_1, ..., CE_pro_{u-1} of level t together to obtain CE_innpro as the inner product result ciphertext.

[0073] FIG. 5A to FIG. 5D The process of expanding the inner product of the ciphertext single term and Evk at different levels is shown. This example still uses the parameter settings of L=27, K=8, and W=9, and is respectively FIG. 4A to FIG. 4D The ciphertext level corresponds to .

[0074] like Figure 5A As shown, at ciphertext level 0, the number of BV segments is 3, and three generated extended ciphertext items CE0, CE1 and CE2 are obtained through BV decomposition. On the other hand, the conversion key Evk has Figure 3B The form shown includes dnum=3 extended ciphertexts, each of which includes two ciphertext items marked by subscripts A and B respectively, and each ciphertext item includes L+K=35 polynomials.

[0075] Figure 5A The lower part shows the generation process of a ciphertext item in the inner product result ciphertext CE_innpro. Specifically, CE0 (shown in red) is bitwise multiplied with the A-group ciphertext item of the 0th extended ciphertext in Evk (the first column in the red extended ciphertext in Evk), CE1 (shown in blue) is bitwise multiplied with the A-group ciphertext item of the 1st extended ciphertext in Evk (the first column in the blue extended ciphertext in Evk), CE2 (shown in green) is bitwise multiplied with the A-group ciphertext item of the 2nd extended ciphertext in Evk (the first column in the green extended ciphertext in Evk), and the results of the bitwise multiplication of the red, blue and green parts are added to obtain the A-group ciphertext item in the inner product result ciphertext CE_innpro. Similarly, by multiplying CEi with the B-group ciphertext items of the i-th extended ciphertext respectively and adding the results, the B-group ciphertext items in the inner product result ciphertext CE_innpro can be obtained.

[0076] Figure 5B The inner product diagram when the ciphertext level t is 8 is shown. As shown in the figure, at this ciphertext level, the number of BV segments is still 3, and the generated extended ciphertext items CE_0 to CE_2 are missing the items corresponding to the basic modulus q19-q26. Due to the missing items, the polynomials corresponding to the modulus in Evk will not be used during the inner product process, and this part of the data is shown in black. Correspondingly, in the generated inner product result ciphertext CE_innpro, the ciphertext items also lack the items corresponding to the basic modulus q19-q26.

[0077] Figure 5C The inner product diagram of the ciphertext level t is 9. Figure 4CIt can be seen that when the ciphertext level increases to 9, the number of BV segments is reduced to 2, and the extended ciphertext items generated by BV decomposition only include CE_0 and CE_1, and the items corresponding to the basic modulus q18-q26 are missing. When the inner product with Evk is performed based on such extended ciphertext items CE_0 and CE_1, due to the missing CE_2, the second extended ciphertext in Evk will not be used as a whole; and in the 0th and 1st extended ciphertexts, the polynomial corresponding to the basic modulus q18-q26 will not be used. The data that will not be used is shown in black.

[0078] As the ciphertext level increases further, it can be imagined that the number of ciphertext items generated by BV decomposition will further decrease, and their length will further decrease. It can be understood that when the ciphertext level t=18, the number of BV segments is reduced to 1, and the corresponding extended ciphertext items generated only include CE_0. Correspondingly, in the inner product process with Evk, only some polynomials in the 0th extended ciphertext of Evk will be used.

[0079] Figure 5D The inner product diagram of the ciphertext level 26 is shown. At this highest level, the generated extended ciphertext single item only includes CE_0, and the items corresponding to the basic modulus q1-q26 are missing. Correspondingly, in the process of inner product with Evk, the first and second extended ciphertexts in Evk will not be used as a whole, and in the 0th extended ciphertext, the polynomial corresponding to the basic modulus q1-q26 will not be used. The data that will not be used is shown in black.

[0080] right FIG. 5A to FIG. 5D A comparative analysis shows that in the key switching process of the ciphertext of level t, the polynomial in Evk actually used (i.e. FIG. 5A to FIG. 5D The entire non-black part of the ciphertext (the non-black part of the ciphertext) constitutes the effective use Evk data at level t. Obviously, the higher the level of the input ciphertext item for the key switching operation, the smaller the amount of effective use Evk data. Figure 6 It indicates the maximum level to which each part of Evk will be used under a given parameter instance. The maximum level T corresponding to a polynomial P means that the key switching process of ciphertexts exceeding level T does not need to use the polynomial P.

[0081] In the CKKS scheme, a key switch is required when performing a ciphertext rotation operation on the ciphertext, and the Evk used for ciphertext rotation under different rotation steps is different. Therefore, when a dense neural network requires a large number of different step sizes for rotation operations, the number of Evk required to execute the dense network will also be very large. Taking the commonly used Resnet20 network as an example, the number of polynomial points N is generally 65536, and the single-point data size Wordsize is 8 bytes. The data size corresponding to each level 0 Evk can be expressed as Wordsize*N*2*(L+K)*dnum. When N=65536, Wordsize is 8 bytes, and the ciphertext parameters are set to L=27, K=8, and dnum=3 as mentioned above, an Evk size of 8Bytes*65536*2*(27+8)*3=105MB can be obtained. The current CKKS implementation of Resnet20 generally uses about 300 different rotation step sizes. If all Evk are stored in the level 0 format, the corresponding Evk data volume is about 30GB.

[0082] The excessive amount of data in the conversion key Evk will affect the execution efficiency of dense neural network reasoning in two aspects. On the one hand, users need to upload Evk data to the server where the dense neural network is deployed so that it can perform dense reasoning. Excessive Evk data will affect the communication efficiency between users and the server. In addition, excessive Evk data will also affect the computing efficiency of the server. Because the server usually uses high-computing performance acceleration hardware such as GPU, FPGA, ASIC, etc. to perform dense reasoning, and the storage space of high-speed storage units in these acceleration hardware is limited. After exceeding the storage limit, real-time movement and scheduling of data will be required, resulting in performance bottlenecks.

[0083] Based on the above analysis and research, it is proposed in the embodiments of this specification that by designing the dense operation level of the dense neural network, the amount of effectively used Evk data can be reduced, and the performance of dense neural network reasoning can be improved by compressing the Evk data volume.

[0084] To this end, first Figure 7 As shown, determine the density level of the dense neural network. Figure 7 As shown, first in step 71, the ciphertext level consumption of each network layer in the dense neural network is determined.

[0085] As mentioned above, the neural network structure will alternately include linear operation layers (such as convolutional layers, fully connected layers, etc.) and activation layers (such as Relu, Sigmoid, etc.). The linear layer usually consumes the depth of 2 levels of multiplication operations, and therefore consumes 2 levels of ciphertext. Take the convolutional layer as an example. In the convolutional layer, the inner product of the parameter vector of the convolution kernel and the feature vector needs to be performed. The inner product process is as follows: Figure 2 As shown. Obviously, the positional multiplication of the parameter vector and the feature vector will consume one multiplication depth. In addition, the position of the convolution kernel will change continuously, and multiple rotation operations will often produce unnecessary or erroneous garbage points in the next round. In order to eliminate these garbage points, the intermediate result is generally multiplied by a mask vector, in which the positions of the unnecessary garbage points are set to zero. Therefore, the removal of garbage points will also consume one multiplication depth. Generally, most linear layer operations will consume 2 levels of multiplication depth. The settings of individual fully connected layers make the rotation operation process not produce garbage points, and may only consume 1 level of multiplication depth.

[0086] Since homomorphic operations only support linear operations, the operations of the activation layer in dense neural networks are generally implemented through polynomial fitting. In other words, the activation function of the activation layer, such as Relu, Sigmoid, etc., is approximated by polynomial fitting. Common polynomial fitting includes Taylor expansion, etc. Therefore, the ciphertext level consumed by the activation layer depends on the order of the polynomial expansion. For example, if the activation function is expanded by the 4th order Taylor, the corresponding activation layer will consume 4 levels of multiplication depth, that is, 4 ciphertext levels. However, the operation of the activation layer does not require rotation operations, and there is no need to transform the key Evk.

[0087] The dense neural network may also contain other computational layers, such as downsampling, etc. The ciphertext level consumption of each computational layer can be inferred based on the multiplication depth used in the dense operation.

[0088] After determining the consumption of ciphertext level by each network layer, in step 73, according to the ciphertext level consumption, under the target constraint, the input ciphertext level of each network layer and the position of the bootstrap operation are determined so that the input ciphertext level of each linear layer is higher than the first threshold. The target constraint includes that the output ciphertext level of each network layer is not greater than the highest ciphertext level.

[0089] It can be understood that since the operation of the linear layer involves a large number of rotation operations of different step lengths, which in turn requires a large number of conversion keys Evk, the design of the dense neural network mainly focuses on the ciphertext level of the linear layer. Since the higher the ciphertext level, the less effective Evk data is used, one goal of the ciphertext level design is to make the ciphertext level of the linear layer as high as possible. To this end, a threshold can be set so that the input ciphertext level of each linear layer is higher than the threshold. In one embodiment, the threshold can be set so that only a single extended ciphertext is included in the effective use of Evk data. For example, in the aforementioned example of L=27, W=8, when the ciphertext level t is higher than 18, only the 0th extended ciphertext is included in the effective use of Evk data.

[0090] The design of the ciphertext level must also meet the target constraint, that is, the output ciphertext level of each network layer is no greater than the highest ciphertext level. If a network layer operates based on the input ciphertext level and the output ciphertext level will be greater than the highest ciphertext level, a bootstrap operation can be inserted before the network layer to reduce the ciphertext level.

[0091] Figure 8 An example of ciphertext level setting for a dense neural network in one embodiment is shown. In this example, taking the network structure of Resnet20 as an example, a ciphertext level setting at each network layer in a neural network is given. This example still follows the parameter settings of L=27, K=8, dnum=3, and W=9. Under this parameter setting, the highest ciphertext level supported by the dense operation is level 26. In order to reduce the amount of data that effectively uses Evk data, the operation of the convolutional layer is set to a higher ciphertext level. Considering that the convolutional layer consumes 2 ciphertext levels, Figure 8 In the example, the input ciphertext level of the convolutional layer is set to 24. In this way, the ciphertext level output by the convolutional layer is 26, which is the highest level. The highest level ciphertext cannot be further multiplied, so a bootstrap operation can be inserted after the convolutional layer.

[0092] The role of the bootstrap operation is to reduce the noise level of the ciphertext while ensuring that the plaintext content corresponding to the ciphertext remains unchanged. The actual performance is to reduce the ciphertext level. The bootstrap operation usually includes four steps: ModRaise, C2S, HeMod, and S2C. C2S and S2C also involve the rotation operation of the ciphertext, using Evk. However, the number of Evk involved is relatively limited.

[0093] The bootstrap operation can reduce the ciphertext level to varying degrees according to the parameter settings in the operation. Figure 8 In the example, assume that the bootstrapping operation reduces the ciphertext level from 26 to 17.

[0094] Next, the ciphertext of level 17 will be input to the Relu layer. In the implementation of the dense neural network, the essence of the dense Relu operation is to perform a polynomial fit on the Relu function and then perform the calculation. In a typical example, the number of polynomial fits is 119, which consumes 7 multiplication depths. Therefore, the ciphertext level output by the Relu layer will be upgraded to 24 again. The next convolutional layer can also be set to start operating at ciphertext level 24, so that further operations can be performed directly on the output of the Relu layer. Subsequent network layers can be set in a similar way.

[0095] After designing the dense neural network in the above manner and determining the density level of each network layer, such a dense neural network can be used to perform dense model inference. Fig. 9 The following is a flow chart showing the process of using a dense neural network to process a model in one embodiment. It is understood that the method flow can be executed by a computing device deployed with the above-mentioned dense neural network, which can be a device of a cloud service provider, and can be specifically embodied as any computing platform or device cluster with strong computing resources. The plaintext model corresponding to the above-mentioned dense neural network can be obtained by the computing device through self-training, or provided by a third-party model provider, or provided by a user, which is not limited here. After the ciphertext level of the dense neural network is set, it can be executed Fig. 9 The steps process.

[0096] like Fig. 9 As shown, the encrypted model reasoning process may include step 91, in which the user's ciphertext data and key data are received, and the key data is a part of the conversion key Evk in the target format, wherein the conversion key Evk in the target format includes multiple extended ciphertexts.

[0097] Generally, the format parameters of the ciphertext data C and the conversion key Evk, including the aforementioned number of basic moduli L, the number of auxiliary moduli K, the number of extended ciphertexts dnum, and the size of the modulus fragment W, can be determined in advance by negotiation between the user and the computing device, or determined by the computing device after considering the performance of the cryptanalytic operation, and notified to the user's device. Thus, the user's device can homomorphically encrypt the user data plaintext to be calculated (which may involve the user's privacy data) according to the agreed algorithm and format parameters to obtain the ciphertext data C. As mentioned above, the ciphertext items in the ciphertext data C include L basic polynomials corresponding to the L basic moduli. Accordingly, the user can also prepare a conversion key Evk in the target format, which includes dnum extended ciphertexts, and the ciphertext items of the extended ciphertext include L first polynomials corresponding to the L basic moduli, and K second polynomials corresponding to the K auxiliary moduli.

[0098] Since the ciphertext levels of the dense neural network have been determined, the effective use of the Evk data portion used in the dense reasoning can also be determined. In one embodiment, the computing device can send indication information of the effective use of the Evk data portion to the user device, and the indication information includes, for example, the number of the extended ciphertext to be used in the Evk corresponding to the specific rotation step, the number of the polynomial (or modulus) to be used in the extended ciphertext, and so on. In this way, the user device can extract the effective use of the Evk data portion from the full amount of data of the conversion key Evk according to the indication information as the key data for transmission.

[0099] Based on this, in step 93, the ciphertext data C is input into a encrypted neural network for encrypted processing, wherein the encrypted processing includes, at any target linear layer among several linear layers, performing a ciphertext rotation operation using the first data part corresponding to the first ciphertext level t in the conversion key, wherein the first data part includes only partial data of a single extended ciphertext.

[0100] It should be understood that the first ciphertext level t here is the ciphertext level set for the above-mentioned target linear layer. As mentioned above, in order to minimize the amount of data that effectively uses the Evk data, the ciphertext level of each linear layer is set to a higher value, for example, higher than a level threshold. In one embodiment, the level threshold can be set so that only a single extended ciphertext is included in the effectively used Evk data. In this way, when performing a ciphertext rotation operation at the first ciphertext level t, only the first data portion, that is, part of the data of a single extended ciphertext in Evk, needs to be used.

[0101] If the ciphertext level input to the target linear layer is less than the first ciphertext level t, the ciphertext level can be increased to the first ciphertext level t through the ciphertext level increase operation. The ciphertext increase operation can be implemented through simple calculation operations. For example, the ciphertext level can be increased by n levels by performing a multiplication of the current ciphertext with a unit vector n times.

[0102] Further, according to Figure 5A-5D In the example shown, at the ciphertext level t, the part of the data used in the conversion key Evk is the first polynomial corresponding to Lt basic moduli and the second polynomial corresponding to K auxiliary moduli. In order to further reduce the amount of Evk data, given L, the ciphertext level t can be set so that the value of Lt is less than a preset threshold Th, for example, the threshold Th can be set to 4. In this way, at most Th+K polynomials are used in the conversion key Evk.

[0103] As mentioned above, the setting of the ciphertext level t also needs to consider the ciphertext level consumed by the network layer, so that the entire encrypted inference process complies with the constraints of the overall ciphertext level. Therefore, if the target linear layer consumes 2 levels of ciphertext, such as the convolutional layer, then the above ciphertext level t is set so that Lt is greater than or equal to 3. In this way, it can be ensured that the output of the target linear layer does not exceed the highest ciphertext level L-1. If the operation of the target linear layer consumes 1 level of ciphertext, the above ciphertext level t can be set so that Lt is greater than or equal to 2.

[0104] Since the target linear layer is set to a higher ciphertext level, the ciphertext outputted by it may not be able to continue the ciphertext multiplication calculation of the subsequent network layer. At this time, a bootstrap operation can be performed after the target linear layer operation, and the bootstrap operation uses the second data part corresponding to the second ciphertext level in the conversion key to perform ciphertext level reduction processing. As mentioned above, the ciphertext level in the bootstrap operation is generally low, so the second ciphertext level is generally less than the first ciphertext level. Accordingly, the second data part may contain more Evk data.

[0105] Accordingly, in one embodiment, the key data transmitted by the user in step 91 may only include the first data portion used by each linear layer and the second data portion used by the bootstrapping operation.

[0106] Through the above settings and secret execution process, the Evk data that needs to be transmitted and stored can be greatly reduced.

[0107] As mentioned above, in the CKKS implementation of the Resnet20 network, the data volume of a single level 0 Evk key is 105M. The ciphertext rotation in the Resnet20 network model involves a total of 298 different rotation steps; under the premise that each evk is stored in the level 0 format, the total data volume is 298*105MB=30.56GB. Among the above 298 rotation steps, 39 steps will be used in bootstrapping, and the remaining 259 steps are used for the linear operation layer of the network. According to the ciphertext level setting method of the embodiment of this specification, the linear operation layer can be stored in the level 24 format. When the format parameters (L, K, W, etc.) remain unchanged, the size of each evk is 8Bytes*65536*2*(3+8)*1=11MB, so the total evk data size is 39*105MB+259*11MB=6.78GB, which is relatively reduced by (30.56-6.78) / 30.56*100%=77.8% of the evk data volume, which is very significant for both transmission and storage.

[0108] Furthermore, through the study of the algorithm for key switching using Evk, it is found that the role of the auxiliary modulus p_j is to prevent the accumulation of large noise during the key switching process. To achieve this goal, it is generally only necessary to ensure that the sum of the total number of bits of the auxiliary modulus p_j is slightly larger than the sum of the total number of bits of the homomorphic modulus in each segment of the u BV modulus segments during the ciphertext single-term BV decomposition. This condition can be called the modulus bit constraint.

[0109] In the previous example, the parameter configuration of K=8 and W=9 is always maintained. Such parameter configuration can satisfy the above modulus bit constraint at any ciphertext level, because at all levels, the product of all homomorphic moduli q_i in the BV segment is always smaller than the product of auxiliary moduli p_j. Specifically, in a typical parameter setting, q_i is 47 bits and p_j is 58 bits. Obviously, the total sum of auxiliary key bits 58*8 is slightly larger than the total modulus bits 47*9 of the BV segment.

[0110] However, for high-level key switching, there is only one BV segment and some of the moduli are missing. For example, in the aforementioned embodiment, the ciphertext level t is set so that the value of Lt is less than the preset threshold Th, where Lt is the number of moduli remaining in the BV segment. When the number of moduli does not exceed the threshold Th, the number of auxiliary moduli K can be set to the threshold Th, or slightly larger than the threshold, to ensure that the aforementioned modulus bit constraint is met.

[0111] For example, in a specific example, the size of Lt is set not to exceed 3 (that is, Th=3), so the number K of auxiliary moduli p_j can be reduced to 3 in a targeted manner. In this way, the size of each evk of the linear layer is further reduced from 11MB to 8Bytes*65536*2*(3+3)*1=6MB, and the total evk data size is 39*105MB+298*6MB=5.75GB. Compared with the case of storing the entire Evk, the evk data volume is reduced by (30.56-5.75) / 30.56*100%=81.2%. Obviously, as the number of rotation step types in the linear layer operation increases, the compression ratio of the evk data volume will be greater.

[0112] Reviewing the above process, in the embodiments of this specification, by designing the dense operation level of the dense neural network, the amount of data effectively using the Evk data can be reduced, the amount of Evk data can be compressed, and the performance of the dense neural network reasoning can be improved. Furthermore, the number of auxiliary moduli can be reduced to further reduce the amount of Evk data and further reduce the transmission and storage costs of Evk data.

[0113] According to an embodiment of another aspect, a device for performing model reasoning using a dense neural network is provided. Fig.10 FIG. 1 is a schematic diagram showing the structure of a model inference device according to an embodiment, which can be deployed in any device, platform or device cluster with data storage, computing and processing capabilities. Fig.10 As shown, the model reasoning device 100 includes:

[0114] The receiving unit 101 is configured to receive the ciphertext data and key data of the user, wherein the key data is a part of the conversion key in the target format, wherein the conversion key in the target format includes a plurality of extended ciphertexts;

[0115] The encrypted processing unit 103 is configured to input the ciphertext data into the encrypted neural network for encrypted processing, wherein the encrypted processing includes, at any target linear layer among the plurality of linear layers, performing a ciphertext rotation operation using a first data portion corresponding to a first ciphertext level t in the conversion key, wherein the first data portion includes only partial data of a single extended ciphertext.

[0116] The implementation of each unit in the above device can refer to the description of the combined method embodiment. Through the above device, the amount of Evk data used in dense state reasoning can be compressed to improve the performance of dense state reasoning.

[0117] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Fig. 9 The method described.

[0118] According to another embodiment of the present invention, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the Fig. 9 The method described.

[0119] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0120] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for model reasoning using a dense neural network, wherein: The dense neural network includes a plurality of linear layers, and the method includes: Receiving ciphertext data and key data of a user, wherein the key data is a part of the conversion key in the target format, wherein the conversion key in the target format includes a plurality of extended ciphertexts; The ciphertext data is input into the dense neural network for dense processing, wherein the dense processing includes, at any target linear layer among the several linear layers, performing a ciphertext rotation operation using a first data part corresponding to a first ciphertext level t in the conversion key, wherein the first data part includes only partial data of a single extended ciphertext.

2. The method according to claim 1, wherein: The ciphertext monoterm in the ciphertext data includes L basic polynomials corresponding to the L basic moduli, and the ciphertext monoterm of the extended ciphertext includes L first polynomials corresponding to the L basic moduli, and K second polynomials corresponding to the K auxiliary moduli; The partial data of the single extended ciphertext is Lt first polynomials and K second polynomials; and the value of Lt is less than a preset threshold.

3. The method according to claim 2, wherein: The target linear layer is a convolutional layer, and its operation consumes 2 ciphertext levels; the first ciphertext level t is set so that Lt is greater than or equal to 3.

4. The method according to claim 2, wherein: The operation of the target linear layer consumes 1 ciphertext level; the first ciphertext level t is set so that Lt is greater than or equal to 2.

5. The method according to claim 1, wherein: The cipher state processing includes, after the target linear layer, performing a bootstrap operation, wherein the bootstrap operation uses a second data portion corresponding to a second cipher text level t in the conversion key to perform cipher text level reduction processing.

6. The method according to claim 5, wherein: The key data only includes the first data portion used by each linear layer and the second data portion used by the bootstrapping operation.

7. The method according to claim 3, wherein: The number K of auxiliary modules is set to be equal to the preset threshold.

8. The method according to claim 1, wherein: The cipher state processing includes, before performing the ciphertext rotation operation, in response to the ciphertext level input to the target linear layer being less than the first ciphertext level t, performing a ciphertext level raising operation to raise the ciphertext level to the first ciphertext level t.

9. The method according to claim 1, wherein: Before the secret state processing, the method further includes: Determining the ciphertext level consumption of each network layer in the dense neural network; According to the ciphertext level consumption, under the target constraint, the input ciphertext level of each network layer and the position of the bootstrap operation are determined so that the input ciphertext level of each linear layer is higher than a preset level threshold. The target constraint includes that the output ciphertext level of each network layer is not greater than the highest ciphertext level.

10. A device for model reasoning using a dense neural network, wherein: The dense neural network includes a plurality of linear layers, and the device includes: A receiving unit configured to receive ciphertext data and key data of a user, wherein the key data is a portion of data of a conversion key in a target format, wherein the conversion key in the target format includes a plurality of extended ciphertexts; The encrypted processing unit is configured to input the ciphertext data into the encrypted neural network for encrypted processing, wherein the encrypted processing includes, at any target linear layer among the plurality of linear layers, performing a ciphertext rotation operation using a first data portion corresponding to a first ciphertext level t in the conversion key, wherein the first data portion includes only partial data of a single extended ciphertext.

11. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 9.

12. A computing device comprising a memory and a processor, characterized in that: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 12 is implemented.

Images