Ciphertext convolution method, system and device for improving matrix vector multiplication and medium
By flattening the input image into a vector and constructing a weight matrix, calculating and reshaping the output tensor, the problem of low efficiency of ciphertext convolution calculation and homomorphic matrix vector multiplication is solved, and more efficient ciphertext convolution calculation is achieved.
Patent Information
- Application Number
- CN202510259978.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-06
- Publication Date
- 2025-06-24
AI Technical Summary
In the prior art, cryptographic convolution calculation and homomorphic matrix vector multiplication are relatively low.
By flattening the input image into an input vector and arranging it by channel; constructing the weight matrix based on the convolution kernel; calculating the multiplication of the weight matrix and the vector to obtain the output vector; and using the inverse process to reshape the output vector into an output tensor, the convolution operation between the input image and the convolution kernel is realized.
While implementing ciphertext convolution calculation, the efficiency of ciphertext convolution calculation is improved and the calculation time is reduced by about 20%.
Smart Images
Figure CN120196847A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of deep learning, and in particular, to a ciphertext convolution method, system, device, and medium for improving matrix-vector multiplication. Background Art
[0002] With the continuous improvement of privacy protection requirements, fully homomorphic encryption, as a technology that allows direct computation on ciphertexts, has become a core tool in privacy-preserving deep learning. In a ciphertext neural network based on fully homomorphic encryption, ciphertext convolution calculations are often converted into matrix-vector multiplications. However, traditional ciphertext convolution calculations and homomorphic matrix-vector multiplications are less efficient. Summary of the Invention
[0003] The present invention provides a ciphertext convolution method, system, device, and medium for improving matrix-vector multiplication, aiming to effectively solve the technical problem of low efficiency in existing ciphertext convolution calculations and homomorphic matrix-vector multiplications.
[0004] According to a first aspect of the present invention, there is provided a ciphertext convolution method for improving matrix-vector multiplication, including: flattening an input image into an input vector and arranging it channel by channel; constructing a weight matrix based on a convolution kernel; calculating the multiplication of the weight matrix and the vector to obtain an output vector; and using the inverse process of flattening the input image into the input vector to reshape the output vector into an output tensor, where the output tensor is the convolution operation output of the input image and the convolution kernel.
[0005] Further, the size of the input image is , the number of input channels is , the size of the convolution kernel is and the number of output channels is .
[0006] Further, in the convolution, both the padding and the stride are 1, then the size of the output image is , and the total number of pixels of the convolution output is .
[0007] Further, in the step of flattening the input image into a vector, the input image is flattened into a vector according to the row-major principle, and the length of the vector is ; the output tensor has channels and a size of .
[0008] Further, the method for constructing the weight matrix includes: constructing a weight matrix with the number of rows being , and the number of columns being a matrix; initialize the matrix as a zero matrix, and one row of the zero matrix corresponds to a convolution window; obtain an input window related to the convolution window, and place the elements in the convolution window one by one at the corresponding positions of the zero matrix to obtain a weight matrix corresponding to the input window.
[0009] Further, the step of calculating the multiplication of the weight matrix and the vector includes: dividing the weight matrix evenly by rows into sub-matrices, each sub-matrix being s rows, and if the number of rows is insufficient, padding the last sub-matrix with 0s; extracting generalized diagonals from each sub-matrix, sorting the diagonals of each sub-matrix according to the column numbers, and mixing and connecting the diagonals of the sub-matrices with the same column number at intervals, and then outputting s diagonal vectors ; performing times of ciphertext rotation operations on the input ciphertext , and performing copy operations in the order of diagonal mixing connection, so that the slot distribution of each ciphertext matches the composite diagonal after mixing connection, and then outputting a ciphertext vector x ; calculating ; performing rotation and summation operations on , and outputting a ciphertext with the first valid slots being the result of the convolution operation; setting the positions other than the first n slots of the ciphertext to zero. Y ; setting the positions other than the first n slots of the ciphertext to zero.
[0010] According to the second aspect of the present invention, the present invention also provides a ciphertext convolution system for improving matrix-vector multiplication, including: an image flattening module for flattening an input image into an input vector and arranging it channel by channel; a weight matrix construction module for constructing a weight matrix based on a convolution kernel; a matrix-vector multiplication module for calculating the multiplication of the weight matrix and the vector to obtain an output vector; and an operation output module for reshaping the output vector into an output tensor by using the inverse process of flattening the input image into an input vector, and the output tensor is the convolution operation output of the input image and the convolution kernel.
[0011] According to the third aspect of the present invention, the present invention also provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, implementing the ciphertext convolution method for improving matrix-vector multiplication described in any one of the above.
[0012] According to a fourth aspect of the present invention, the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the ciphertext convolution method for improving matrix-vector multiplication described in any one of the above is implemented.
[0013] According to a fifth aspect of the present invention, the present invention further provides a computer program for executing the ciphertext convolution method for improving matrix-vector multiplication described in any one of the above.
[0014] Through one or more of the above embodiments in the present invention, at least the following technical effects can be achieved: In the technical solution disclosed in the present invention, by flattening the input image into an input vector and arranging it channel by channel; constructing a weight matrix based on the convolution kernel; calculating the multiplication of the weight matrix and the vector to obtain an output vector; and using the inverse process of flattening the input image into an input vector to reshape the output vector into an output tensor, where the output tensor is the convolution operation output of the input image and the convolution kernel. Therefore, while implementing ciphertext convolution calculation, the efficiency of ciphertext convolution calculation can be improved. Description of the Drawings
[0015] The following will, with reference to the accompanying drawings, clearly and completely describe the technical solutions in the embodiments of the present invention through a detailed description of the specific embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts fall within the scope of protection of the present invention.
[0016] Figure 1 It is a flowchart of the ciphertext convolution method for improving matrix-vector multiplication provided by an embodiment of the present invention; Figure 2 It is an example of the process of converting convolution into matrix multiplication in the ciphertext convolution method for improving matrix-vector multiplication provided by an embodiment of the present invention; Figure 3 It is a calculation example of the ciphertext convolution method for improving matrix-vector multiplication provided by an embodiment of the present invention; Figure 4 It is a framework diagram of the ciphertext convolution system for improving matrix-vector multiplication provided by an embodiment of the present invention; Figure 5 It is a structural schematic block diagram of an electronic device provided by an embodiment of the present invention. Detailed Embodiments
[0017] The following will, with reference to the accompanying drawings in the embodiments of the present invention, clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts fall within the scope of protection of the present invention.
[0018] In the description of the present invention, it should be noted that unless otherwise clearly specified and defined, the term "and / or" in this article is only a correlation relationship describing associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after without special explanation.
[0019] With the continuous improvement of privacy protection requirements, fully homomorphic encryption, as a technology that allows direct execution of calculations on ciphertexts, has become a core tool in privacy-preserving deep learning. In a ciphertext neural network based on fully homomorphic encryption, ciphertext convolution calculations are often converted into matrix-vector multiplications. However, traditional ciphertext convolution calculations and homomorphic matrix-vector multiplications are less efficient.
[0020] The embodiments of the present application provide a ciphertext convolution method, system, device, and medium for improving matrix-vector multiplication, which can improve the efficiency of ciphertext convolution calculations while implementing ciphertext convolution calculations.
[0021] Figure 1 The following shows the ciphertext convolution method for improving matrix-vector multiplication provided by the embodiments of the present invention, including: S101. Flatten the input image into an input vector and arrange it channel by channel; S102. Construct a weight matrix based on the convolution kernel; S103. Calculate the multiplication of the weight matrix and the vector to obtain an output vector; S104. Use the inverse process of flattening the input image into an input vector to reshape the output vector into an output tensor, and the output tensor is the convolution operation output of the input image and the convolution kernel.
[0022] The ciphertext convolution method for improving matrix-vector multiplication provided by this embodiment can improve the efficiency of ciphertext convolution calculations while implementing ciphertext convolution calculations.
[0023] In step S101, the size of the input image is , the number of input channels is , the size of the convolution kernel is and the number of output channels is , and under the conditions that both the padding and the stride are 1, the size of the output image is , and the total number of pixels of the convolution output is .
[0024] In some embodiments, in the step of flattening the input image into a vector, the input image is flattened into a vector according to the row-major principle, and the length of the vector is ; The output tensor has channels and a size of .
[0025] In some embodiments, the method for constructing the weight matrix includes: Construct a matrix with the number of rows being and the number of columns being ; Initialize the matrix to a zero matrix, and one row of the zero matrix corresponds to a convolution window; Obtain an input window related to the convolution window, and place the elements in the convolution window one by one at the corresponding positions of the zero matrix to obtain a weight matrix corresponding to the input window.
[0026] In some embodiments, the step of calculating the multiplication of the weight matrix and the vector includes: Divide the weight matrix of size uniformly into s sub - matrices by rows, each sub - matrix being rows, and if the number of rows is insufficient, pad the last sub - matrix with 0s; Extract generalized diagonals from each sub - matrix, sort the diagonals of each sub - matrix according to the column numbers, and mix - connect the diagonals of the s sub - matrices with the same column number at an interval of , and then output diagonal vectors ; Perform x times of ciphertext rotation operations on the input ciphertext , and perform a copy operation in the order of diagonal mix - connection so that the slot distribution of each ciphertext matches the composite diagonal after mix - connection, and then output the ciphertext vector ; Calculate ; Perform a rotation and summation operation on , and output the ciphertext n with the first Y valid slots being the result of the convolution operation ; n Set the positions other than the first
[0027] Please refer to Figure 2 , in this embodiment, the process of converting convolution into matrix - vector multiplication is visualized, where the input channels and output channels , the input image size is (3, 3), the convolution kernel size is (3, 3), and the output image size is (3, 3).
[0028] Please refer to Figure 3 , this embodiment also shows a computational example of an optimized matrix-vector multiplication algorithm according to the above-mentioned method for encrypting convolution of matrix-vector multiplication. Among them, the weight matrix A has a size of (4, 4), and the encrypted vector has a size of (4, 1), s and the value is 2.
[0029] To further illustrate the performance of the method for encrypting convolution of matrix-vector multiplication provided in this embodiment, this embodiment also conducts an experimental analysis, which is as follows: All experiments are carried out on the same server. The CPU of the server is Intel(R) Core(TM) i9-10900X, which has 10 cores and 256GB of memory. Encryption is based on the Microsoft SEAL library, and the data is encrypted by the fully homomorphic encryption algorithm CKKS. The encryption parameters set the degree N of the polynomial ring to 8192 and the scaling factor δ to 2 40 . The input image size is set to (5, 5), and the convolution kernel is set to a size of (3, 3).
[0030] A computational experiment is conducted on the traditional method (please refer to the literature "Ebel A, Garimella K, Reagen B. Orion: A Fully Homomorphic Encryption Compiler for Private Deep Neural Network Inference[J]. arXiv preprint arXiv:2311.03470, 2023") and the method for encrypting convolution of matrix-vector multiplication provided in this embodiment. Table 1 shows the comparison of the convolution operation times of the two methods: Table 1 Comparison of the average operation times of the two methods
[0031] As can be seen from Table 1, the method of this embodiment reduces the time by about 20% compared with the traditional method. This is because this embodiment mixes and connects the diagonals to retain the original row order of the weight matrix, so that the final output no longer requires an additional alignment step, eliminating the dependence on the permutation matrix, reducing additional addition, multiplication, and rotation operations, and thus reducing the overall computational overhead. In addition, the directly and tightly arranged output enables more efficient utilization of the previous slot positions during the processing of subsequent layers, further improving the overall inference efficiency.
[0032] Please refer to Figure 4, The embodiment of the present application also provides a ciphertext convolution system for improving matrix-vector multiplication, including: an image flattening module 1, a weight matrix construction module 2, a matrix-vector multiplication module 3, and an operation output module 4; the image flattening module 1 is used to flatten the input image into an input vector and arrange it channel by channel; the weight matrix construction module 2 is used to construct a weight matrix based on the convolution kernel; the matrix-vector multiplication module 3 is used to calculate the multiplication of the weight matrix and the vector to obtain an output vector; the operation output module 4 is used to use the inverse process of flattening the input image into an input vector to reshape the output vector into an output tensor, and the output tensor is the convolution operation output of the input image and the convolution kernel.
[0033] The ciphertext convolution system for improving matrix-vector multiplication provided by this embodiment can improve the efficiency of ciphertext convolution calculation while realizing ciphertext convolution calculation.
[0034] In some embodiments, the size of the input image is , the number of input channels is , the size of the convolution kernel is and the number of output channels is .
[0035] In some embodiments, in the convolution, both the padding and the stride are 1, then the size of the output image is , and the total number of pixels of the convolution output is .
[0036] In some embodiments, in the step of the image flattening module 1 flattening the input image into a vector, the input image is flattened into a vector according to the row-major principle, and the length of this vector is ; the output tensor has channels and a size of .
[0037] In some embodiments, the weight matrix construction module 2 includes: a construction unit for constructing a matrix with rows and columns; an initialization unit for initializing this matrix into a zero matrix, and one row of the zero matrix corresponds to a convolution window; a convolution matrix corresponding unit for obtaining an input window related to the convolution window and placing the elements in the convolution window one by one at the corresponding positions of the zero matrix to obtain a weight matrix corresponding to the input window.
[0038] In some embodiments, the matrix-vector multiplication module 3 includes: a matrix partitioning unit for evenly partitioning the weight matrix with a size of into s sub-matrices by rows, and each sub-matrix is Rows. If the number of rows is insufficient, pad zeros to the last sub-matrix; A generalized diagonal extraction unit for extracting generalized diagonals from each sub-matrix, sorting the diagonals of each sub-matrix by column number, and s mixing and connecting the diagonals of sub-matrices with the same column number at intervals, and then outputting a diagonal vector ; ; A ciphertext vector output unit for performing x a ciphertext rotation operation on the input ciphertext and performing a copying operation in the order of diagonal mixing connection, so that the slot distribution of each ciphertext matches the composite diagonal after mixing connection, and then outputting a ciphertext vector ; A calculation unit for calculating ; A ciphertext output unit for performing a rotation and summation operation on and outputting the first n valid slots as the ciphertext of the convolution operation result Y ; A slot zeroing unit for zeroing other positions except the first n slots of the ciphertext.
[0039] An embodiment of the present application provides an electronic device. Please refer to Figure 5 . The electronic device includes: a memory 601, a processor 602, and a computer program stored on the memory 601 and executable on the processor 602. When the processor 602 executes the computer program, the ciphertext convolution method for improving matrix-vector multiplication described above is implemented.
[0040] Further, the electronic device further includes: at least one input device 603 and at least one output device 604.
[0041] The above-mentioned memory 601, processor 602, input device 603, and output device 604 are connected through a bus 605.
[0042] Among them, the input device 603 can specifically be a camera, a touch panel, a physical button, or a mouse, etc. The output device 604 can specifically be a display screen.
[0043] The memory 601 can be a high-speed random access memory (RAM), or a non-volatile memory, such as a disk memory. The memory 601 is used to store a set of executable program codes, and the processor 602 is coupled to the memory 601.
[0044] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, which can be disposed in the electronic device in the above embodiments. The computer-readable storage medium can be the memory 601 in the foregoing embodiments. A computer program is stored on the computer-readable storage medium, and when the program is executed by the processor 602, it implements the ciphertext convolution method for improving matrix vector multiplication described in the foregoing method embodiments.
[0045] Furthermore, the computer-readable storage medium can also be various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk, or an optical disc.
[0046] In several embodiments provided by the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there can be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or modules can be in electrical, mechanical, or other forms.
[0047] The modules described as separate components may or may not be physically separated. The components displayed as modules may or may not be physical modules, that is, they can be located in one place, or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0048] In addition, in each embodiment of the present invention, the functional modules can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules.
[0049] When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.
[0050] It should be noted that, for the foregoing method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily all essential to the present invention.
[0051] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0052] In summary, although the present invention has been disclosed above with preferred embodiments, the above preferred embodiments are not intended to limit the present invention. Those of ordinary skill in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention is subject to the scope defined by the claims.
Claims
1. A ciphertext convolution method for improving matrix-vector multiplication, characterized in that: include: Flatten the input image into input vectors and arrange them channel by channel; Construct a weight matrix based on the convolution kernel; Calculate the multiplication of the weight matrix and the vector to obtain an output vector; The output vector is reshaped into an output tensor by using the inverse process of flattening the input image into the input vector, and the output tensor is the output of the convolution operation of the input image and the convolution kernel.
2. The ciphertext convolution method for improving matrix-vector multiplication as claimed in claim 1, characterized in that: The size of the input image is , the number of input channels is , the size of the convolution kernel is And the number of output channels is .
3. The ciphertext convolution method for improving matrix-vector multiplication as claimed in claim 2, characterized in that: In the convolution, the padding and stride are both 1, so the size of the output image is , and the total pixels of the convolution output are .
4. The ciphertext convolution method for improving matrix-vector multiplication as claimed in claim 3, characterized in that: In the step of flattening the input image into a vector, the input image is flattened into a vector according to the row-first principle, and the length of the vector is ; The output tensor has channels, size .
5. The ciphertext convolution method for improving matrix-vector multiplication as claimed in claim 3, characterized in that: The method for constructing the weight matrix includes: The number of construction lines is , the number of columns is Matrix of Initialize the matrix to a zero matrix, wherein one row of the zero matrix corresponds to one convolution window; Get the input window related to the convolution window, and place the elements in the convolution window at the corresponding positions of the zero matrix one by one to obtain the weight matrix corresponding to the input window.
6. The ciphertext convolution method for improving matrix-vector multiplication as claimed in claim 1, characterized in that: The step of calculating the multiplication of the weight matrix and the vector comprises: The size is The weight matrix is evenly divided into rows: s sub-matrices, each of which is If the number of rows is insufficient, the last sub-matrix is padded with 0; Extract from each submatrix The generalized diagonal lines of each submatrix Sort the diagonals by column number and separate the diagonals with the same column number. s The submatrix diagonal spacing Mixed connection and output Diagonal vectors ; For input ciphertext x implement The ciphertext is rotated and copied in the order of diagonal mixed connection, so that the slot distribution of each ciphertext matches the composite diagonal after mixed connection, and then the ciphertext vector is output. ; calculate ; right Perform the rotation and sum operation, output the front n The valid slots are the ciphertext of the convolution operation result. Y ; Before the ciphertext n Set all other positions except the 1 slot to zero.
7. A ciphertext convolution system for improved matrix-vector multiplication, characterized in that: include: Image flattening module, used to flatten the input image into input vectors and arrange them channel by channel; A weight matrix construction module is used to construct a weight matrix based on the convolution kernel; A matrix-vector multiplication module, used to calculate the multiplication of the weight matrix and the vector to obtain an output vector; The operation output module is used to reshape the output vector into an output tensor by using the inverse process of flattening the input image into the input vector, and the output tensor is the convolution operation output of the input image and the convolution kernel.
8. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method described in any one of claims 1 to 6 is implemented.
10. A computer program, characterized in that Used to perform the method according to any one of claims 1 to 6.