Efficient convolution in a tile-enabled environment

Abstract

This application relates to efficient convolution in tiled environments. One method includes: receiving an input tensor, the input tensor having a domain consisting of [n1, ..., n]. k The shape is defined by [t1, ...,t], where k is equal to the number of dimensions representing the input tensor; it receives tile tensor metadata, which includes: [t1, ...,t] k The tile tensor shape is defined, along with information indicating the interleaving stride to be applied relative to each dimension of the tile tensor; an output tensor comprising multiple tile tensors is constructed by applying a packing algorithm that maps each element of the input tensor to at least one slot position of one of the multiple tile tensors based on the tile tensor shape and the interleaving stride, wherein the interleaving stride results in a discontinuous mapping of the elements of the input tensor such that each of the tile tensors comprises a subset of the elements of the input tensor that are spaced apart within the input tensor according to the interleaving stride.

Description

Technical Field

[0001] This invention generally relates to the fields of encryption / decryption schemes, algorithms, techniques, methods, computer programs, and systems. Background Technology

[0002] Fully homomorphic encryption (FHE) provides scalable ciphertext that allows operations to be performed on encrypted data without first decrypting it. For example, if d represents some data, then E(d) represents the encryption of d using an FHE scheme. Given E(d) and the scheme's public key, it is possible to compute E(f(d)) for any function f without knowing the decryption key and without learning anything about d. The resulting computation is stored in encrypted form, which, when decrypted, produces the same output as if the operation had been performed on unencrypted data.

[0003] One potential use case for FHE is outsourcing the storage and processing of sensitive data while preserving privacy. For example, outsourcing services could be used to compute classification predictions for medical data (e.g., medical images) while maintaining patient privacy. Using an FHE scheme allows data owners to send their data in encrypted form to a cloud service that hosts a trained classifier. Encryption ensures the data remains confidential because the cloud service will not have access to the private key needed to decrypt the data. The cloud service will then be able to apply the trained classifier to the encrypted data to make encrypted predictions and return the predictions to the data owner in encrypted form.

[0004] However, running large neural networks using only FHE is still considered a computationally expensive task. This obstacle forces users to search for other safer alternatives rather than enjoying the advantages of solutions that rely solely on FHE.

[0005] The examples and related limitations of the relevant art described above are intended to be illustrative rather than exclusive. Further limitations in the relevant art will become apparent to those skilled in the art upon reading the specification and studying the accompanying drawings. Summary of the Invention

[0006] The following embodiments and aspects thereof are described and illustrated in conjunction with systems, tools and methods, which are intended to be exemplary and illustrative, and not to limit the scope.

[0007] In one embodiment, a system is provided, comprising: at least one hardware processor; and a non-transitory computer-readable storage medium having program instructions stored thereon, the program instructions being executable by the at least one hardware processor to: receive an input tensor, wherein the input tensor has a value consisting of [n1, ..., n]. kThe shape is defined by [t1, ..., t], where k is equal to the number of dimensions representing the input tensor; it receives tile tensor metadata, which includes at least: (i) the shape of the input tensor from [t1, ..., t] k (ii) Define the shape of the tile tensor, and (iii) information indicating the interleaving stride to be applied relative to each dimension of the tile tensor; and construct an output tensor comprising a plurality of tile tensors by applying a packing algorithm that maps each element of the input tensor to at least one slot position of one of the plurality of tile tensors based at least in part on the tile tensor shape and the interleaving stride, wherein the interleaving stride results in a discontinuous mapping of the elements of the input tensor such that each of the tile tensors comprises a subset of the elements of the input tensor spaced apart within the input tensor according to the interleaving stride.

[0008] In one embodiment, a computer-implemented method is also provided, comprising: receiving an input tensor, wherein the input tensor has a domain consisting of [n1, ..., n]. k The shape is defined by [t1, ..., t], where k is equal to the number of dimensions representing the input tensor; receive tile tensor metadata, which includes at least: (i) the shape of the input tensor defined by [t1, ..., t]. k (ii) Define the shape of the tile tensor and (iii) information indicating the interleaving stride to be applied relative to each dimension of the tile tensor; construct an output tensor comprising multiple tile tensors by applying a packing algorithm that maps each element of the input tensor to at least one slot position of one of the multiple tile tensors based at least in part on the tile tensor shape and the interleaving stride, wherein the interleaving stride results in a discontinuous mapping of the elements of the input tensor such that each of the tile tensors comprises a subset of the elements of the input tensor that are spaced apart within the input tensor according to the interleaving stride.

[0009] In an embodiment, a computer program product including a non-transitory computer-readable storage medium is also provided. The non-transitory computer-readable storage medium has program instructions implemented therewith, which are executable by at least one hardware processor to: receive an input tensor, wherein the input tensor has a value consisting of [n1, ..., n]. k The shape is defined by [t1, ..., t], where k is equal to the number of dimensions representing the input tensor; receive tile tensor metadata, which includes at least: (i) the shape of the input tensor defined by [t1, ..., t]. k(ii) Define the shape of the tile tensor, and (iii) information indicating the interleaving stride to be applied relative to each dimension of the tile tensor; and construct an output tensor comprising multiple tile tensors by applying a packing algorithm that maps each element of the input tensor to at least one slot position of one of the multiple tile tensors based at least in part on the tile tensor shape and the interleaving stride, wherein the interleaving stride results in a discontinuous mapping of the elements of the input tensor such that each of the tile tensors comprises a subset of the elements of the input tensor that are spaced apart within the input tensor according to the interleaving stride.

[0010] In some embodiments, the program instructions may also be executed to store, and the method further includes storing the output tensor and tile tensor metadata.

[0011] In some embodiments, program instructions may also be executed to unpack, and the method further includes unpacking the input tensor from the stored output tensor based on the tile tensor metadata.

[0012] In some embodiments, the tile tensor metadata also includes a copy parameter, and wherein the packing algorithm is configured to perform a copy of the elements of the input tensor based on the copy parameter, such that each element of the input tensor is mapped to a plurality of slot positions along one of the dimensions of the tile tensor.

[0013] In some embodiments, the program instructions may also be executed, and the method further includes: receiving a filter associated with a convolution computation on an input tensor; computing the convolution by applying a multiplication operator that multiplies the filter element-wise relative to each tile tensor in the output tensor; applying a summation algorithm to sum the results of the multiplication; and outputting the result of applying the summation algorithm as the result of the convolution.

[0014] In some embodiments, convolution is part of neural network inference.

[0015] In some embodiments, tensor tiles are homomorphic encrypted ciphertext.

[0016] In addition to the exemplary aspects and embodiments described above, other aspects and embodiments will become apparent from the accompanying drawings and from the following detailed description. Attached Figure Description

[0017] Exemplary embodiments are shown in the accompanying drawings with reference to the figures. The dimensions of the parts and features shown in the figures are generally chosen for ease of presentation and clarity, and are not necessarily drawn to scale. The drawings are listed below.

[0018] Figure 1A This is a block diagram of an exemplary system for a data packaging structure for a fully homomorphic encryption (FHE) scheme according to some embodiments of the present disclosure;

[0019] Figure 1B A schematic diagram of a packaging optimizer according to some embodiments of the present disclosure is presented;

[0020] Figure 1C This is a flowchart of functional steps in an example method according to some embodiments of the present invention;

[0021] Figure 2A This is a diagram of the basic convolutional neural network architecture.

[0022] Figure 2B The sliding window (or filter or kernel) function is shown.

[0023] Figures 3A-3D The tile tensor packing operation according to some embodiments of this disclosure is illustrated;

[0024] Figure 4A The interleaving tiling process according to some embodiments of the present disclosure is illustrated;

[0025] Figure 4B and Figure 4C The following is illustrated using two 8-parallel kernel evaluations of a 2×2 filter or kernel performed on a matrix, according to some embodiments of the present disclosure;

[0026] Figures 5A-5C Interleaving tiling operations according to some embodiments of the present disclosure are shown;

[0027] Figures 6A-6D This is another illustration of the interlacing tiling process of this disclosure according to some embodiments thereof; and

[0028] Figure 7 This is a diagram illustrating the experimental results of the CryptoNet implementation using tile tensors. Detailed Implementation

[0029] This paper discloses a technique for data packing in the context of a fully homomorphic encryption (FHE) scheme, implemented in methods, systems, and computer program products. This technique allows for efficient high-level tensor manipulation operations, such as convolution, on encrypted data while reducing computational overhead.

[0030] As background, the FHE scheme is an encryption scheme that allows evaluation of any circuit (especially any function) over encrypted data. The FHE scheme takes a vector M[s] as input data and returns (ENC) ciphertext. FHE also generates a secret encryption key / public encryption key pair associated with the encrypted input data. The created ciphertext has the number s of slots determined during key generation. The generated private encryption key can be used to decrypt (DEC) the ciphertext to return an s-dimensional vector, where M = Dec(Enc(M)). Functions for addition, multiplication, and rotation are then defined as follows:

[0031] Dec(Add(Enc(M), Enc(M′)))=M+M′

[0032] Dec(Mul(Enc(M), Enc(M′)))=M*M′

[0033] Dec(Rot(Enc(M),n))(i)=M((i+n)mods)

[0034] Some FHE schemes, such as CKKS (see J. Cheon et al., “Homomorphic Encryption for Approximate Numbers,” in Proceedings of Advances in Cryptology - ASIACRYPT 2017. Springer Cham, 112017, pp. 409-437), operate on ciphertext in a homomorphic SIMD manner. This means that a single ciphertext encrypts a fixed-size vector, and the homomorphic operation on the ciphertext is performed element-wise on the plaintext vector. To take advantage of SIMD features, more than one input element must be packed and encrypted in each ciphertext. Packing methods can significantly affect latency (i.e., the time to perform computations), throughput (i.e., the number of computations performed per unit of time), communication costs, and memory requirements. However, deciding which packing to use is difficult and requires expertise, and a more efficient packing may not be a simple one. Furthermore, different packing schemes offer different trade-offs in terms of optimization objectives. Finding the optimal packing becomes more difficult as the size of the FHE code increases. For example, finding the best package for large neural network inference algorithms is a challenging task because the input is often a four- or five-dimensional tensor, and the computation involves long sequences of operations such as matrix multiplication and convolution.

[0035] Therefore, in some embodiments, this disclosure provides an FHE data packing technique that provides high-level tensor manipulation operations, such as convolution.

[0036] In some implementations, this technique uses a data packing structure called a 'tile tensor'. A tile tensor allows a user to store tensors of arbitrary shape and size. Tile tensors automatically pack tensor data into a fixed-size vector set, as required in an FHE environment, using various configurable options. In some embodiments, the tile tensors of this disclosure also provide a set of operators that manipulate tensors in their packed form. In some embodiments, the tile tensors of this disclosure support the use of operators on encrypted data, wherein the operators are implemented using a generic algorithm that can work with any packing arrangement even when the internally chosen packing arrangement is unknown. Therefore, this disclosure provides a packing-agnostic programming framework that allows a user to focus on designing algorithms that are inferred against the data, rather than against data packing decisions.

[0037] In some embodiments, this disclosure also provides a packing optimizer that operates in conjunction with the tile tensor data structure of this disclosure. In some embodiments, given user requirements and preferences, this optimizer searches for the optimal configuration of the tile tensors. In some embodiments, the optimizer estimates the time and memory required to run a given function for each option and returns a configuration that optimizes a given objective (whether latency, throughput, or memory). In some embodiments, this optimizer can be used to improve latency in small networks, adapt to various batch sizes, and scale to much larger networks.

[0038] In some embodiments, this disclosure provides solutions for convolution computation, a popular building block in the fields of neural networks and machine learning models.

[0039] In some embodiments, this disclosure is particularly useful in the context of image processing. For example, in some embodiments, this disclosure supports convolutions on an infinite image size (e.g., total number of pixels) and reduces the required number of rotations to the order of filterSize*sqrt(#imageSize-#slots). In some embodiments, the invention can process images using convolutions of multiple channels, kernels, or filters, and batch processing of multiple images, and the parameters have been easily tuned to optimize for specific scenarios.

[0040] In some embodiments, this disclosure provides a packing scheme using interleaved tiling, wherein adjacent input tensor values ​​are stored in the same slot across different tiles. This simplifies computation over the region covered by the filter when one wishes to compute the convolution of a small filter onto a large matrix, since all values ​​are in different tiles but within the same slot.

[0041] In some embodiments, this disclosure interprets the ciphertext as T-sized 'tiles'. x Multiply by T yThe matrix. If the input tensor has dimension I. x Multiply by I y Then it is necessary Multiply The entire image is stored using a grid of tiles. Adjacent or close input tensor values ​​within the same region can be stored, such that they map to the same slots in different tiles. Specifically, an image pixel at (i, j) can be mapped to the tile at position (i%Ex, j%Ey), and mapped to the tile at position... The groove at the location.

[0042] When computing the convolution of a small filter onto a large matrix, all computations over the region covered by the filter occur within the same slot. When computations need to cross regions (i.e., blend different slot positions), the tiles can be rotated so that all values ​​are aligned again within the same slot position. The exact number of rotations can be calculated using the following method. Given an image size I... x I y Having size F x F y Filter and tile size T x T y The number of rotations will be Therefore, the number of rotations required is the square root of the image size, rather than being on the same order of magnitude as the image size. This method, combined with a tile tensor data structure, can be extended with additional dimensions, thus allowing for more efficient processing of multiple images (batch processing), multiple image channels, and multiple filters. For example, sub-tile tensors can be used, where C is the number of channels and B is the number of batches, where the sub-tile tensor contains tensors [C, *, B], where * denotes the replication dimension. For a certain tile dimension [t1, t2, t3], the corresponding sub-tile tensor shape will be... And the full tile tensor (including all pixels) will be The '~' indicates the interleaving dimension. The filter pixels are sub-tile tensors containing the tensor [C, F, *], where F represents the filter size. Multiplying and summing along dimension C yields the tensor [*, F, B]. The full-tile tensor shape for all filter pixels is... Therefore, even if the image does fit within the same ciphertext, it does not fit within a single ciphertext for all channels and batches, and this disclosure is able to avoid duplication and reduce rotation.

[0043] Hardware and software environment

[0044] Figure 1A This is a block diagram of an exemplary system 100 for a data packaging structure for a fully homomorphic encryption (FHE) scheme according to some embodiments of the present disclosure.

[0045] System 100 may include one or more hardware processors 102, random access memory (RAM) 104, and one or more non-transitory computer-readable storage devices 106. For example... Figure 1A As shown, system 100 is an embodiment of a hardware and software environment used in conjunction with various embodiments of the present invention.

[0046] One or more storage devices 106 may have program instructions and / or components stored thereon configured to operate one or more hardware processors 102. The program instructions may include one or more software modules, such as a data packing module 108 and / or an optimizer module 110. The software components may include an operating system with various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between various hardware and software components. System 100 can operate by loading instructions into RAM 104 when the instructions of the data packing module 108 and / or the optimizer module 110 are executed by one or more processors 102.

[0047] Figure 1B This is a block diagram of optimizer module 110. In some embodiments, optimizer module 110 is responsible for finding the most efficient packing arrangement for a given computation, as well as the optimal configuration of the underlying FHE library, for example, by combining neural network operations applied to FHE data.

[0048] In some embodiments, the optimizer module 110 of this disclosure receives the model architecture of the neural network to be applied as input. The optimizer module 110 automatically converts it into an FHE computation with optimal packing and optimal FHE library configuration. Users can further define packing constraints, such as the required security level or maximum memory usage, and select optimization objectives, whether for optimizing CPU time, latency, or throughput, or for optimizing memory usage.

[0049] In some embodiments, optimizer module 110 selects from different possible configurations of the FHE library and different packaging techniques that support certain operators. In some embodiments, optimizer module 110 also selects the tile shape, i.e., the values ​​of t1, t2, ..., in the tile tensor shape. For example, consider an FHE scheme configured to have 16,384 slots in each ciphertext. Assuming the desired convolution operator uses five-dimensional tiles, the number of possible tuples t1, ..., t5 makes Π i t i =16,384 is

[0050] In some embodiments, the optimizer module 110 includes three main units: a configuration generator 110a, a cost estimator 110b, and a simulator 110c. In some embodiments, the optimization process can begin by a user providing a file (e.g., a JSON file) containing details of the model architecture to be applied at runtime. The configuration generator 110a generates a list of all possible configurations, including packaging details and FHE configuration details applicable to the architecture. The simulator unit 110b tests each such configuration and outputs one or more of the following data for each configuration: computation time for different stages, including cryptographic model and input samples, running inference and decryption results; throughput; memory usage of the cryptographic model; inputs; and outputs. The optimizer module 110 transmits this data to the cost estimator 110c for evaluation. Finally, the optimizer module 110 returns the configuration option that produces the optimal cost to the user along with the simulation output configuration file.

[0051] In some embodiments, the configuration generator unit 110a of the optimizer module 110 receives the model architecture to be applied at runtime and generates all applicable configurations for it. The generator unit 110a then creates multiple complete configurations by exploring all possible tile shapes. The generator unit 110a uses one of two strategies to explore possible tile shapes. The first strategy involves a brute-force approach over all valid options for the tile shapes. Since there may be many of these, the second strategy uses a "steepest ascent" local search algorithm. The local search begins with balanced tile shapes where the number of slots in each dimension is of the same order of magnitude. This is a heuristic designed to avoid evaluating tile shapes that might be computationally expensive at the start of the search. Thus, all neighboring tile shapes of the current shape can be iteratively evaluated, where the best-improving neighbor is selected whenever one exists. In some embodiments, two tile shapes can be considered neighbors, where one shape can be obtained from the other shape by multiplying or dividing the size of one of its dimensions by two. Based on the cost received from the cost evaluator, one tile shape can be considered superior to the other. Using local search algorithms greatly accelerates the search process and often leads to a global optimum.

[0052] In some embodiments, simulator unit 110b receives the model architecture to be applied at runtime and configuration options from configuration generator 110a as input. At this stage, the configuration can be evaluated by running it on encrypted input under FHE. To reduce computational costs, simulator 110b can use pre-computed benchmarks, such as CPU time per HE operation and memory consumption per tile (i.e., memory consumption of a single ciphertext). Simulator 110b then uses these benchmarks to evaluate the model to be applied to model tile tensor objects. These model tile tensors contain only metadata and collect performance statistics. Using this method, simulator 110b can simulate inference operations several orders of magnitude faster than when running the full model on encrypted data.

[0053] In some embodiments, the cost evaluation unit 110c evaluates the output data of the simulator 110b by considering constraints and optimization objectives, which may be user-provided. After testing all possible configurations, one or more of the highest-scoring configurations are sent back as output.

[0054] As described herein, System 100 is merely an exemplary embodiment of the invention and may actually be implemented in hardware only, software only, or a combination of both. System 100 may have more or fewer components and modules than shown, may combine two or more components, or may have different configurations or arrangements of components. System 100 may include any additional components that enable it to function as an operable computer system, such as a motherboard, data bus, power supply, network interface card, display, input devices (e.g., keyboard, pointing device, touch-sensitive display), etc. (not shown). Furthermore, the components of System 100 may be co-located or distributed, or the system may be configured to run as one or more cloud computing “instances,” “containers,” “virtual machines,” or other types of encapsulated software applications, as known in the art.

[0055] The programs described herein are based on their identification in specific embodiments of the invention for the applications in which they are implemented. However, it should be understood that any particular procedural terminology used herein is for convenience only, and therefore the invention should not be limited to use only in any particular application identified and / or implied by such terminology.

[0056] Various embodiments of the present invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0057] like Figure 1A As shown, system 100 is an environment in which example methods according to some embodiments of the present invention can be executed.

[0058] Figure 1C This is a flowchart of functional steps in an example method according to some embodiments of the present invention. The flowchart describes a method for interleaving and packing tile tensors from input tensors. Reference will now be made to the process in the following paragraphs. Figures 1A-1C This paper discusses the method and associated software used for packing tile tensors.

[0059] Processing begins at step 120, where system 100 receives data with sizes n1, n2...n k The data in dataset A1 consists of k-dimensional input tensors. A tensor is a multidimensional array with k dimensions and a shape [n1, n2, ... nn]. k For example, if the input tensor is a matrix with 4 rows and 5 columns, then k = 2, n1 = 4, n2 = 5. If the input tensor is a vector of length 7, then k = 1, n1 = 7. If the input tensor is a 3D array with dimensions 10 x 20 x 30, then k = 3, n1 = 10, n2 = 20, n3 = 30.

[0060] In step 122, system 100 also receives tile tensor metadata including the tile tensor shape and interlacing stride. The tile tensor shape is the shape of the dimensions of the input tensor A1 (i.e., k, n1, n2, ... n). k ) and packaging details t1, t2, ..., t k Composition. The tile tensor data structure also includes copies of the tile tensor shape.

[0061] The required number of tile tensors can be calculated as follows. Let Where ceil represents rounding up to the nearest integer above. The required number of tiles is e1*e2*...*e k .

[0062] In step 124, the system 100 instructs the packing module 108 to apply a packing algorithm that maps each element of the input tensor to at least one slot location in the tile tensor in a discontinuous manner based on the tile tensor shape and the interleaving stride. In some embodiments, the interleaving stride results in a discontinuous mapping of the elements of the input tensor, such that each tile tensor includes a subset of the elements of the input tensor that are spaced apart within the input tensor according to the interleaving stride.

[0063] In step 126, the instructions of system 100 cause packaging module 108 to construct an output tensor that includes multiple tile tensors.

[0064] In step 128, the system 100 is instructed to store the output tensor and tile tensor metadata in, for example, storage device 106.

[0065] In step 130, the system 100 is instructed to unpack the input tensor from the stored output tensor based on the tile tensor metadata.

[0066] Basic operations of tensors

[0067] As used in this article, the term 'tensor' is synonymous with multidimensional array, as this is common in the field of AI.

[0068] A k-dimensional tensor can be defined by [n1, n2, ..., nn] k ] indicates that 0 < n i It is the size of the i-th dimension. For example, the shape of a 5×6 matrix M is [5, 6]. For a tensor R[n1, ..., n], ... k ], can be expressed as R(j1, j2, ..., j k () is used to refer to a specific element, where 0 ≤ j i <n i .

[0069] Matrix multiplication should be represented without the multiplication symbol in this paper; for example, M1M2 represents the product of M1 and M2. The transpose of matrix M can be represented as M... T Furthermore, labels (e.g., M′, M″) can be used to represent different objects. The operations M1+M2 and M1*M2 refer to element-wise addition and multiplication, respectively.

[0070] If for i≤k, m i =n i , or n i =1, or m i =1, then tensor A[n1, ..., n] k ] and B[m1,...,m k They are called compatible shapes. Their mutually expanding shapes are [max{n] i m i}] i≤k .

[0071] When tensor A has more dimensions than tensor B, their dimensions can be matched by expanding B with a dimension of size 1. This results in equal tensors up to their transpose. For example, tensors V[b] and V[b, 1] represent column vectors, while V[1, b] = V T This represents a row vector.

[0072] The broadcast operation takes two tensors with compatible but different shapes and expands each of them into a shape that expands upon the other. For each i = 1, ..., k, the tensor A[n1, ..., n] k and tensor shape s = [m1, ..., m k ], where n i ∈{1,m i The operation C = broadcast(A, s) copies the content m of A along the r-th dimension. r Next, for each r = 1, ..., k and n r =1<m r The output tensor C has shape s.

[0073] Tensors A[3,4,1] and B[1,4,5] have compatible shapes. Their mutual expansion shape is s = [3,4,5], and broadcast(A,s) has the same shape s as broadcast(B,s).

[0074] By first using broadcasting to expand two tensors with compatible shapes A and B into their mutually expanded shapes, and then performing the relevant element-wise operations, element-wise operations such as addition (A+B) and multiplication (A*B) of the two tensors can be performed. Therefore, for tensor A[n1, ..., n]... k The operation B = sum(A, t) sums the elements of A along the t-th dimension, and the resulting tensor B has shape [n1, ..., n]. t-1 ,1,...,n k ],and

[0075]

[0076] For all j i <n i For i∈{1, 2, ..., k}\{t},

[0077] Broadcasting and summation can be used to perform common algebraic operators. For two matrices M1[a, b], M2[b, c] and column vector V[b, 1], the expression M1V = sum(M1*V) can be used. T 2) Perform matrix-vector multiplication, where M1 and V T It has compatible shapes, and their mutually extending shapes are [a, b]. It can be represented using M1M2 = sum(M... 1′ *M 2′ 2) to perform matrix-matrix multiplication, where M 1′ =M1[a, b, 1] and M 2′ =M2[1,b,c].

[0078] Tile tensor

[0079] In some embodiments, a 'tile tensor' is a data structure containing an outer tensor as data and a tile tensor shape as metadata. An 'outer tensor' is a tensor in which each element is a tile.

[0080] In some embodiments, the tile tensors of this disclosure are data structures that pack tensors into fixed-size blocks as required by the FHE and allow manipulation of them in a manner similar to (i.e., unencrypted) regular tensors. In other words, performing operations on the tile tensors of this disclosure can be considered equivalent to (or “homomorphic”, as in mathematical terms) directly on the input tensors.

[0081] As used in this document, a "tensor" is any multidimensional array of numbers. A special case of a tensor is when the array is one-dimensional; in this case, the array is often referred to as a vector. In the case of a two-dimensional vector, it is often referred to as a matrix. It will be understood in the art that the numbers constituting a tensor can be integers, floating-point numbers, or complex numbers. However, in some tensors, the elements of the vector may not take the form of numbers; instead, these elements may take other forms, such as characters, strings, or other objects. In the specific examples discussed in this document, it will generally be assumed that tensors contain elements of numerical form unless otherwise indicated.

[0082] It is known that tensors can be packed into tiles, and furthermore, mathematical operators can be performed on these tensors while they are in packed form. As used herein, a "tile" is any contiguous block of fixed-size data storage in a data storage device (e.g., a volatile memory-type data storage device) capable of holding n numbers, where n is a fixed size determined by the system configuration. In this way, all tiles will have the same size of n numbers.

[0083] Those skilled in the art will understand that mathematical operations can be performed on the elements of a tile (e.g., the numbers that make up the tile). Such mathematical operations are typically performed element-wise on a corresponding number(s) of corresponding tiles undergoing the mathematical operation. For example, if tile T1 has the numbers (x1, x2, ... x...),... n And tile T2 has numbers (y1, y2, ... y2). n If T1+T2 contains (x1+y1, x2+y2, ..., x...), then T1+T2 is a subset of (x1+y1, x2+y2, ..., x...). n +y n T1*T2 is a set of tiles containing (x1*y1, x2*y2, ..., x...). Similarly, T1*T2 is a set of tiles containing (x1*y1, x2*y2, ..., x...). n *y nThe tiles are arranged in a grid. The tiles can also be rotated by any offset r, meaning that the slot for each element r is moved to the left (or right, if r is negative), and the first element rotates back to the end. For example, if T1 = (1, 2, 3, 4, 5, 6, 7, 8), then T_1 rotated by 2 is (3, 4, 5, 6, 7, 8, 1, 2), and rotated by -2 is (7, 8, 1, 2, 3, 4, 5, 6).

[0084] The process of creating a tile tensor from an input tensor is called packing. The input to this process is the tensor itself and the tile tensor shape, for example, metadata indicating the packing scheme. The output is a tile tensor data structure containing a collection of tiles filled with data copied from the tensor and arranged within the tiles according to the tile tensor shape. The tile tensor data structure also includes a copy of the tile tensor shape. The tensor can be retrieved from the tile tensor using a process called "unpacking." The tile tensor shape is used to identify how the tensor's elements are arranged within the tiles and is copied back into the tensor. Given one or more tile tensors, operators can be applied to the tile tensors. These operators modify both the content of the tiles and the tile tensor shape of the tile tensor.

[0085] In some embodiments, the 'outer tensor' as used herein is a k-dimensional tensor, wherein each of its elements is itself a k-dimensional tensor and has the same shape. These inner tensors are called 'tiles' or 'tile tensors' and have a 'tile shape', while the outer tensor has an 'outer shape'. The slot in E is defined by E(a1, ..., a...). k (b1, ..., b) k ) identifier, where a i It is the external index of the tile, and b i It is an internal index within the tile.

[0086] The k-dimensional “tile tensor shape” includes the outer shape [e1, ..., e] k ], Tile shape [t1, ..., t k ], original shape [n1, ..., n k Replication count [r1, ..., r] k ], interleaved Boolean indicator [l1, ..., l k ] and unknown Boolean pointers [u1, ..., u k ].Require

[0087] Given a tile tensor shape S, an outer tensor E, and outer indices (a1, ..., a2), ... k and internal indexes (b1, ..., b) kIf a specific slot in E is specified, then that slot is associated with a logical index (c1, ..., c) relative to S. k The following calculation is performed in association with i = 1, ..., k: if the interleaving indicator l i If true, then c i =b i e i +a i Otherwise c i =a i t i +b i .

[0088] If the outer shape matches the tile shape, then the tile tensor shape S is valid for the outer tensor E, and there exists a tensor T[n1, ..., n]. k ], such that for T1 = broadcast(T, [n1r1, ..., n2r2]), for those with internal, external, and logical indices a i b i c i All slots, making E(a1, ..., a) k (b1, ..., b) k ) = T1(c1, ..., c k ) holds true. For all other slots of E, if These slots are then set to zero. T is the packed tensor.

[0089] The tile tensor is a pair (E, S), where E is the outer tensor and S is the tile tensor shape to which it is valid.

[0090] Given a tile tensor T A = (E, S), the operator unpack(E) yields T. A The packaged tensor.

[0091] Given a tensor A and a tile tensor shape S whose original shape matches the shape of A, the packing operator pack(A, S) yields a tile tensor T. A = (E, S), such that A is T A The packaged tensor.

[0092] The shape of a tile tensor can be specified using a special annotation involving a list of symbols. Each element in the list specifies details for one dimension. Specify the original shape and tile shape along this dimension, as well as r i =1, l i =u i =false. Further specify the replication count and n i =1, and Specify n i =1, r i =t i . Specify l i =true, and Specify e other than the default values ​​mentioned above. i The value of . For any of the above options, the "?" symbol above the line indicates u. i =true.

[0093] Tile tensor data structure

[0094] In some embodiments of this disclosure, the basic tiling process may include obtaining the tensor A[n1, n2, ..., n]. k Furthermore, the tensor is decomposed into blocks or tiles of equal size, each block or tile having a shape [t1, t2, ..., t]. k ].

[0095] Figure 3A The tile tensor packing operation is illustrated. The input tensor 302 (e.g., a one-dimensional vector containing eight values ​​00-07) can be reinterpreted into a multidimensional array 304, e.g., [2, 4].

[0096] In some embodiments, a tensor E[e1, e2, ..., e] can then be constructed. k ], which can be called an 'external tensor', such that each element of E is a tile, and Therefore, for 0≤a i <e i T = E(a1, a2, ..., a k Let be a specific tile in E, and for 0 ≤ b i < t i , T(b1, b2, ..., b k ) is a specific groove within this tile. The original tensor A(c1, c2, ..., c) is... k The elements will be mapped to the tile index. and the index b within the tile i =c i modt i All other slots in E that are not mapped to any element of A will be set to 0. Figure 3B Three examples are shown using an 8-slot tile tensor with shape [2, 4] to tile the matrix M[5, 6]. It can be seen that the outer tensor has the shape [3, 2] and the tile shape is [2, 4]. Figure 3CThe tile tiling matrix M[9,10] with shape [2,4] is shown.

[0097] The shape of a tile tensor can be represented using the following notation. For example, It specifies the use of shapes [t1, ..., t] k The tiles are packaged and laid flat, having the shape [n1, ..., n]. k The shape of the tile tensor of the input tensor. In this notation, if t i =1, then it can be omitted. For example, It can be written as

[0098] You can use the tensor A[n1, ..., n] to be packed. k The `pack` operation, which determines the desired tile tensor shape, is used to create the tile tensor. The `pack` operator uses the tiling process described above to compute the outer tensor and stores the tile tensor shape along its edges to form the complete tile tensor T. A The unpack operation can be used to retrieve A: A = unpack(T) A Similar to regular tensors, the tile tensor T... A It can be represented together with its shape:

[0099] In some embodiments, tile tensors can be manipulated by operators by changing the contents of the multidimensional array of tiles, along with the accompanying packing details and other metadata. This is done in such a way that the operators are equivalent to applying them directly to the tensors they contain within the packing. For example, if tensor A1 is packed within tile tensor TA1, and tensor A2 is packed within tile tensor TA1, software instructions can apply an "addition" operator to TA1 and TA2 to obtain a new tile tensor TA3. Unpacking TA3 will result in tensor A3, which is equal to A1 + A2.

[0100] copy

[0101] For certain calculations, it is useful to replicate tensor data multiple times within the tile slots. The shape of the tile tensor is determined by using... The symbol is used to indicate this. It means n i =1, but each element of the original tensor is copied along the i-th dimension t. i Next. When considering tensor A[n1, ..., n] k Pack and n i =1, and using the specified When the tile tensor shape is defined, the packaging operation executes broadcast(A, [n1, ..., t).i , ..., n k The tensor is then unpacked and tiled. The unpacking process shrinks the tensor back to its original size. Copying can be ignored, or their average can be taken; this is useful if the data is stored on noisy storage media, such as in approximate FHE schemes.

[0102] Unknown value

[0103] When a tensor is packed into a tile tensor, unused slots are filled with zeros, as shown in the figure (e.g.) Figure 3B As shown in the diagram (in Chinese). When operators are applied to a tile tensor, unused slots may be filled with arbitrary values. Although these unused slots are ignored when unpacking the tile tensor, the presence of arbitrary values ​​in these slots can still affect the effectiveness or performance of applying additional operators. To reflect this state, the tile tensor shape includes an additional flag for each dimension, denoted by the symbol "?", indicating the presence of unknown values.

[0104] Figure 3D This demonstrates how tensor V[5,1] can be packed into tile tensors using different tile tensor shapes:

[0105] - Panel A uses a shape The tile tensor.

[0106] Panel B shows a tiling with replication, where the packing process calculates V' = broadcast(V, [5, 4]) and uses a shape... Tiles.

[0107] - Panel C uses a shape The tile tensor. The "?" in the second dimension indicates that any unknown value may be encountered if the contents of the packing tensor exceed its valid range along that dimension. However, V = unpack(T V This still holds true because these unused slots are ignored.

[0108] Operators

[0109] The tile tensor operator is a homomorphic operation between a tile tensor and the packing tensors it contains. For two tile tensors T... A and T B , and the binary operator ⊙, unpack(T A ⊙ T B ) = unpack(T A )⊙unpack(T B This holds true. The unary operator is defined similarly.

[0110] Binary element-wise operators are implemented by applying the operator tile-by-tile to the outer tensor, and the shape of the tile tensor is updated to reflect the shape of the result. If the input has the same shape, then the result will also have the same shape, for example, in... Figure 3C In Can be packaged with the same matrix Multiply, and thus obtain Where R = M * N. Like regular tensors, tile tensor shapes do not need to be identical, but must be compatible. Compatible tile tensor shapes have the same number of dimensions, and for each dimension specification, they are identical, or one is... The other one is Intuitively, if a tensor is already broadcast inside a tile, it can be further broadcast by copying the tile itself to match any size. For example, for Panel B, T M″ *T V′ It can be calculated, and thus obtained T M″ +T V′ It can also be calculated, but this yields... That is, there are unknown values ​​in the unused slots along the second dimension. This is because in that dimension, in T... V′ The values ​​are filled with duplicates, and this occurs after they are added to fill the unused slots of the result. Calculate T. M″ *T V It is illegal because their shapes are incompatible.

[0111] The sum operator is also defined homomorphically: unpack(sum(T) A ,i))= sum(unpack(T A It works by summing the outer tensor along the i-th dimension, and then by summing within each tile along the i-th dimension. In the FHE environment, the latter summing requires a rotation and summation algorithm. Typically, the sum operator reduces the i-th dimension, and the resulting tile tensor shape becomes... However, there are some useful special cases. If t i =1, then it is reduced to Or simplify to 1. When i is such that t i When the smallest i is greater than 1, the dimension is reduced to That is, the summation result is copied. This is due to the property of the rotation and summation algorithm. This is a useful property because sometimes this copying is needed to ensure compatibility with another tile tensor. For example, suppose T A It has shape The tile tensor. Then, sum(T)A 1) Having shape sum(T A 2) Has shape and sum(T) A 3) Has shape

[0112] The three other operators do not change the packed tensor, only the shape of the outer tensor and the tile tensor. `clear(T)` is achieved by multiplying with a mask containing 1s for all used slots. A The ) operator clears unknown values; that is, it removes the "?" from the shape of the tile tensor. For example, rep(T A The i) operator assumes that the i-th dimension is And copy it using a rotation and summation algorithm. flatten(T A The operator (i, j) flattens dimensions i to j, assuming they are all copied. This is easily done by only changing the metadata, for example, turn out

[0113] higher-level operators

[0114] Various algebraic operations can be performed on tile tensors by using element-wise operators and summation.

[0115] In some embodiments, given a matrix M[a, b] and a vector V[b], V can be reformatted to V[1, b] for compatibility, and for some selected tile shapes [t1, t2], the two tensors are packed together as and The tile tensor. These terms can then be multiplied using the following formula:

[0116]

[0117] Formula (1) holds for any values ​​of a, b, t1, t2. This is because T M and T V The tile tensor shape is compatible, and therefore, due to homomorphism, this computes R[a, 1] = sum(M[a, b] * V[1, b], 2), which produces the correct result as described above.

[0118] The second option is to first transpose M and V and then pack them into a tile tensor. and In the middle. Now, they can be multiplied as follows:

[0119]

[0120] This uses the same inference as before to calculate the correct result. The advantage here is that, due to the properties of the sum operator, the result... It is copied along the first dimension. Therefore, it prepares T in the above formula (1). V This allows for the execution of two consecutive matrix-vector multiplications without any processing in between. The `rep(clean(T)` function can be used. R ), 2) Process the output of the above formula (1) to be suitable as the input of the following formula (5).

[0121] The above inference can be easily extended to the following matrix-matrix multiplication. Given matrices M1[a, b] and M2[b, c], their product can be computed using either of the following two formulas, where, in the second formula, M1 is transposed before being packed. As mentioned earlier, the result of the second formula is suitable as input to the first formula.

[0122]

[0123]

[0124] By packing the matrix into a tile tensor and The following formula is used to calculate the product of the four matrices M1[100, 90], M2[90, 80], M3[80, 70], and M4[70, 60].

[0125]

[0126]

[0127] Intertwined and flat

[0128] In some embodiments, this disclosure provides a tiling process in which tiles are not covered in a sequential, continuous manner over a region of the tensor, but rather unfolded over the tensor using equal steps.

[0129] Another option for tiling is indicated by the symbol "~" in the shape of the tile tensor. This symbol indicates that the tiles do not cover consecutive blocks of the tensor, but are unfolded with equal strides. If the dimensions are interleaved, the original tensor A(c1, c2, ..., c...) is... k The element of ) will be mapped to the tile index a i = c i mode i and the index inside the tile. (where e) i It is the size of the external tensor.

[0130] Figure 4A The interleaving tiling process of this disclosure is illustrated. It can be seen that the input tensor 402 is a matrix M[6, 6]. In some embodiments, the input matrix 402 may be, for example, an image, where each element in the matrix may be, for example, an image pixel.

[0131] The input tensor 402 is packed with tile tensors 404a-404f of shape [2, 4], denoted as However, performing the packing in a non-contiguous manner ensures that tensor tiles 404a-404f do not cover every contiguous region of matrix 402. Therefore, for example, tile tensor 404a could include matrix 492 elements 00, 02, 04, 30, 32, 34.

[0132] Figure 4B and Figure 4C Two exemplary 8-parallel kernel evaluations are shown using a 2×2 filter or kernel executed on matrix 402. The dashed rectangles illustrate the convolution operator of the [2,2] filter or kernel 403 on the M[6,6] matrix 402. The tile tensors 404a-404f illustrate the same operation using tile tensor representations and component-wise summation of the tiles. Figure 4B The first convolution iteration is shown, ignoring empty slots. Figure 4C The left circumferential rotation 1 is shown.

[0133] The input tensor 402 is packed with tile tensors 404a-404f of shape [2, 4], denoted as However, performing the packing in a non-contiguous manner ensures that tensor tiles 404a-404f do not cover every contiguous region of matrix 402. Therefore, for example, tile tensor 404a could include matrix 492 elements 00, 02, 04, 30, 32, 34.

[0134] Figure 4B and Figure 4C Two exemplary 8-parallel kernel evaluations are shown using a 2×2 filter or kernel executed on matrix 402. The dashed rectangles illustrate the convolution operator of the [2,2] filter or kernel 403 on the M[6,6] matrix 402. The tile tensors 404a-404f illustrate the same operation using tile tensor representations and component-wise summation of the tiles. Figure 4B The first convolution iteration is shown, ignoring empty slots. Figure 4C The left circumferential rotation 1 is shown.

[0135] In some embodiments, interlacing tiling can be specified individually for each dimension. For example, in In the middle, only the second dimension interweaves. Moreover, although for a basic tiling, It is valid, but for interlaced tiling, ei Having a larger value is sometimes useful. In this case, a flag can be used: Explicitly represent this value.

[0136] Figures 5A-5C The interlacing tiling operation is shown. Figure 5A In this context, tensor tiles of size [2, 4] can be used to tile a matrix 502 of size M[9, 10]. Figure 5A In this context, the first tile tensor 504a covers the region of matrix 502 indicated by the dashed rectangle; however, it does not cover this region continuously. Instead, the first tile tensor 504a includes matrix elements indicated by solid rectangles. Figure 5B and Figure 5C The tile tensors 504b and 504c, which are expanded on matrix 502 using equal steps, are shown respectively.

[0137] Figures 6A-6D The interleaving and tiling process of this disclosure is also illustrated. It can be seen that the input tensor 602a is a region covering a 3×5 element matrix 602. The input tensor 602b can be packed into tile tensors 604, 606, and 608. Figure 6B In this process, input tensor 602a is interleaved and packed into 15 tile tensors 604a-604o. Elements of region 602a are packed into the first slot of tile tensors 604a-604o. In some embodiments, the same region may be packed into all slots of the tile tensor in a repeated manner. Figure 6C In other embodiments, each tile tensor slot can pack different regions of matrix 602. Figure 6D ).

[0138] Convolution using tile tensors

[0139] As described above, this disclosure provides a data packaging scheme that allows for efficient high-level tensor manipulation operations, such as convolution, on encrypted data while reducing computational overhead.

[0140] Convolution can be described as a sliding window function applied to a matrix. Figure 2A This is a diagram of the basic convolutional neural network architecture. Figure 2B The sliding window (or filter or kernel) function is shown.

[0141] The input to a convolutional layer is typically an image tensor (e.g., Figure 2A and Figure 2B The image tensor 202 in the image is represented as I[w I h I [c, b]) and filter tensor F[w F h F [c, f](for example, Figure 2B The filter 204 in the image has the following shape parameters: width w I w F Height h I h F And the number of image channels c (e.g., 3 for an RGB image). In some cases, convolution can be computed for a batch of b images and for f filters. Informally, the convolution operator moves each filter in F as a sliding window over the elements of I, starting at position (0, 0) and using a stride δ. w and δ h When the filter perfectly fits the input, calculate the inner product between the elements of the filter and the corresponding elements of I. For example, in Figure 2B In the input matrix 202, the input matrix 202 is a [5, 5] matrix representing the image, where each matrix element corresponds to a pixel (e.g., for a grayscale image, a value between 0 and 255). The sliding window 204 can be referred to as a kernel, filter, or feature detector, and is a 3×3 filter. The filter 204 slides across the matrix 202 with equal strides of one pixel, multiplying its filter values ​​element-wise with the values ​​of the matrix 202, and then summing them to output the convolutional feature 206. To obtain the complete convolution, this process is performed on each element by sliding the filter 204 across the entire matrix 202.

[0142] Let I[w I h I [c,b] and F[w] F h F [I, F] represents the two input tensors of the convolution operator for the image and the filter, respectively. The result of the operation O = conv2d(I, F) is the tensor O[w O h O [f, b], where δ w And δh are strides, and

[0143]

[0144] In the case of degradation, equation (5) can be simplified to

[0145]

[0146] In FHE settings, it is sometimes useful to transform convolution operations into matrix-matrix multiplications by preprocessing the input before encrypting it. One such approach is image-to-column, which works as follows for the case c = b = 1. Given an image I[w I h I ] and f filters F[w F h FThe operators I′ and F′ = im2col(I, F) calculate matrix I′[w], f]. O h O w F h F ], where each row preserves the effective window position in I and flattens it to the content of the row vector, and F′[w F h F ,f] contains each filter of F flattened into a column vector. Here, tensor O′[w O h O , f]=I′F′ is the flattened version of the convolution result O[w O h O f] = conv2d(I, F).

[0147] In some embodiments, this disclosure provides a variant I″, F″ = im2col′(I, F), which computes I″[w] by copying each row of I′ consecutively f times. O h O f, w F h F ], and by using matrix F′ T Connect w O h O Next, calculate F″[w O h O f, w F h F Tensor O″[w O h O f, 1] = sum(I″*F″, 2) contains the convolution result O[w O h O The advantage of this variant is that the output is fully flattened into a column vector, which is useful in cases where flattening is expensive (e.g., in FHE). The disadvantage of this variant is that it is impossible to perform two consecutive convolution operators without expensive preprocessing in between.

[0148] In some embodiments, this disclosure provides a novel method for computing convolutions on FHE data. In some embodiments, this method provides greater efficiency when the input is a large image and allows for efficient computation of consecutive convolutional layers in FHE-only systems.

[0149] Convolution using interwoven tile tensors

[0150] In some embodiments, this disclosure provides for efficiently computing convolutions on FHE data using interleaved tile tensors as detailed above.

[0151] Return to reference Figures 4A-4C , used as Six different tile tensors 404a-404f are used to pack the matrix M[6,6]402. Here, the tile shape is [2,4] and the outer tensor shape (i.e., the total number and arrangement of the tile tensors used to pack the matrix 402) is [3,2]. Each tile contains 2×4 submatrices, but they are not contiguous; instead, they consist of a set of matrix elements evenly spaced within the matrix. For example, as in Figure 4B As can be seen, elements 00, 01, 10, 11, 20, and 21 are all mapped to the same slot in the different tile tensors 404a-404f. In other words, the entire region of matrix 402 indicated by the dashed rectangle 403 is mapped to the top-left slot in each tile tensor 404a-404f.

[0152] Interleaving and packing allow for a more efficient implementation of Equation 6 above, both in terms of runtime and storage. Intuitively, SIMD is used to compute multiple elements of the output in a single operation. Filters are simply packed as... That is, it has w F h F Each image tile contains a filter value across all slots. This allows each image tile to be multiplied by each filter value.

[0153] For example, Figure 4B The computation of the convolution output is shown when the [2,2] filter 403, indicated by the dashed rectangle, is placed at the top left position of matrix 402. The SIMD properties of the computation are also calculated in other regions of the output. The result is a single tile, where each slot contains the convolution result of the corresponding region, such that the tile is packed with the same interleaving packing scheme as the input tile.

[0154] Figure 4C A more complex example is given below. Here, filter 403 is placed one pixel to the right. Therefore, filter 403 needs to be multiplied by elements appearing in different regions, i.e., they map to slots with different indices. In this case, the tiles need to be rotated appropriately. For example, with the filter positioned so that its top-left corner is at pixel (01), the convolution is computed using the (0,0) slots of tiles 404d and 404e, and the (0,1) slots of tiles 404a and 404b. Accordingly, tiles 404a and 404b are therefore rotated to also move the desired value to slot (0,0).

[0155] When using this packing method, the total cost of convolution is summarized in the following lemma:

[0156] Let s be the number of slots in the ciphertext. Then, given the input image I[w] I h I ] and filter F[w F h F ], package I as And the filter is packaged as O(w) can be used I h I w F h F / s) multiplications and The convolution is computed using rotations. The input encoding is in O(w) time. I h I / s) ciphertext.

[0157] Proof: Multiplication. To compute the convolution, the input tensor w... I h I Each of the elements must be matched with the filter's w. F h F Multiply each element in the array, excluding edge cases that do not change the asymptotic behavior. Since each multiplication multiplies by s slots, it only requires O(w) time. I h I w F h F / s) multiplications.

[0158] Rotation. The recall output has a size (w) I -w F +1)(h I -h F +1). The k-th slot of different ciphertexts is mapped to an indexed slot. and The elements of I. Therefore, the analysis is directed towards and The cost of computing the convolution is sufficient because computing other elements of the output incurs no cost due to the SIMD features. Therefore, when or When needed, a selection is required. This totals for

[0159] Storage. Since O(s) slots are used for each ciphertext, the input can be encoded as O(w). I h I / s) ciphertext.

[0160] The output of convolution is a tile tensor. Unknown values ​​are introduced by filter locations that extend beyond the image, such as... Figure 4C As shown. Further note the tile tensor T. I external dimensions and In T O They remain the same, and they can be greater than the tensor O[w]. O h O The actual required size. Therefore, TO A more precise description of its shape However, this can actually be ignored.

[0161] Processing stride, batch processing, and multiple channels and filters.

[0162] In some embodiments, this convolution algorithm can be extended to process multiple channels (e.g., images with multiple color channels, such as RGB images with red, green and blue channels), multiple filters, and batch processing (e.g., processing multiple images together).

[0163] Make the received input data an image I[w] I h I A tensor of [c, b], where c is the number of channels and b is the batch size. Then, the input is packed into... and filter F[w F h F [c, f] are packaged together as Where f is the number of filters, And Πt i =s.

[0164] Calculate the convolution in a similar manner to the description above, using T I The tiles and T F Multiply the appropriate number of tiles. The result is a shape of... The tile tensor. Using The rotations are summed along the channel (i.e., the third) dimension to obtain

[0165] For larger strides, >1 (correspondingly >1), t1 = 1 (correspondingly t2 = 1) or (correspondingly) Then, the implementation simply skips the ciphertext in each row and each column.

[0166] Convolutional sequences

[0167] In some embodiments, this disclosure provides a sequence for implementing multiple convolutional layers. This is common in neural networks. One advantage of the tile tensor method is that the output of one convolutional layer can be easily adapted to the input of the next convolutional layer.

[0168] In some embodiments, the input can be a batch tensor I[w] I h I [c, b] and convolutional layer sequences, where the l-th layer has a filter tensor For the first layer, c 1 =c, and for subsequent layers l>1, c l=f l-1 As mentioned earlier, input tensors can be packaged into... For odd-numbered layers, Filter tensors can be packed as Then the output is For even-numbered layers Filters can be packaged as:

[0169] As can be seen, the shape of the layer output does not match the shape of the input to the subsequent layer. Therefore, in some embodiments, the present invention provides to adjust the output of odd-numbered layers to fit the next even-numbered layer. To this end, in some embodiments, this disclosure provides to clear the unknowns by multiplying by a mask and then copying the channel dimension. The result is a tile tensor of this shape: It matches the input format of the next layer because f l =c l+1 To make the output of even-numbered layers suitable for the next odd-numbered layer, this invention similarly clears and replicates along the filter dimension.

[0170] It can be noted that changing the order of dimensions leads to a small improvement. The improvement is because summation on the first dimension is ultimately a copy on that dimension. Therefore, setting the channel dimension to the first dimension saves the copying step when preparing input for even-numbered layers. In some embodiments, scavenging can also be skipped because unknown values ​​along the image width and height dimensions do not affect the result. Alternatively, when preparing input for odd-numbered layers, the filter dimension can be set to the first and the subsequent copying step can be skipped.

[0171] When t1 = t2 = t3 = t5 = 1, the above method simplifies to a simpler method known by various names, such as SIMD packing. In this case, each element in the tensors of the image and filters is stored in a separate ciphertext, and slots are used only for batch processing. In some embodiments, a reduction of matrix multiplication as described above can also be used; however, this may only be applicable to neural networks with a single convolutional layer.

[0172] Experimental results

[0173] The inventors conducted experiments using two benchmark neural network models:

[0174] - CryptoNets (see Ran Gilad-Bachrach et al., Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In International Conference on Machine Learning, pp. 201-210, 2016.). This network has convolutional layers followed by two fully connected layers.

[0175] - AlexNet AlexNet (See, Alex Krizhevsky et al., Imagenet classification with deep convolutional neural networks. Neural Information Processing Systems, 25, 012012.)

[0176] The model was trained on the MNIST (see Yann LeCun et al., The MNIST database of handwritten digits. 10:34, 1998) and COVIDx CT-2A (see Hayden Gunraj et al., Covidnet ct-2: Enhanced deep neural networks for detection of covid-19 from chest CT images through bigger, more diverse learning. arXiv preprint arXiv: 2101.07433, 2021) datasets, respectively.

[0177] This paper reports results on performing model inference using these model weights in both encrypted and unencrypted forms. AlexNet is used to demonstrate the capabilities of this disclosure, and CryptoNets are used to demonstrate the impact of different packaging on computational performance and memory usage.

[0178] The experiments were run on a computer with 44 cores (88 threads) and 750GB of memory, running an Intel Xeon CPU E5-2699 v4@2.20GHz. The experiments were implemented using CKKS SEAL with 128-bit security as the target, and all reported results are the average of at least 10 runs.

[0179] CryptoNets Results

[0180] CryptoNets models are models that use shapes The implementation uses a tile tensor, where b is the batch size. In practice, only the result that minimizes the total latency by filling all ciphertext slots (8192 in this case) for the case t3 = b is reported. For convolutional layers, the naive SIMD method is used when b equals the number of plaintext slots and t1 = t2 = 1. Otherwise, a variant of the im2col operator is used (see above). These methods work better than this novel convolution operator when the image is small and the network has only one convolutional layer.

[0181] Figure 7 This is a diagram illustrating how CryptoNets uses tile tensors. For simplicity, only the component-wise square activation layers are indicated by the sq() function, as they maintain the shape of the tile tensors. The equations on the right represent the underlying tensor operations. The input tensors are I, F, and B. c , W1, B1, W2, B2, where I′, F′=im2col′(I, F). Figure 7 The tile tensor flow in this implementation is illustrated. Here, the inputs I and F are the image and filter matrices, respectively, and I′, F′ = im2col′(I, F). Furthermore, B... c W1, W2, B1, and B2 are the training biases of the convolutional layer, and W1, W2, B1, and B2 are the training weights and bias information of the FC layer.

[0182] Table 1 reports the latency and memory usage for performing model inference with different tile shapes when t3 = b = 1. For simplicity, t1 is considered only at extreme points (e.g., t1 = 1, 8192) or the t1 value leading to the optimal solution, along with some additional samples. Optimal latency and memory usage are achieved for t1 = 32, which allows for packing tensors I, F, W1 using a minimum number of tiles.

[0183] Table 2 reports the latency, amortized latency, and memory usage used to perform model inference with different t3 = b values. For each such value, only the t1 and t2 values ​​that lead to the optimal solution are reported. Unlike the case where b = 1, here each choice of t3 results in different trade-offs between performance metrics. For example, increasing t3 increases latency and memory consumption, but decreases the amortized latency per sample. Encryption and decryption times also increase with t3, except in the case where t3 = 8192 uses the naive SIMD convolution operator.

[0184] <![CDATA[t1]]> <![CDATA[t2]]> <![CDATA[t3]]> Delay (seconds) Encryption + Decryption (seconds) Memory (GB) 1 8192 1 0.86 0.04 1.58 8 1024 1 0.56 0.04 0.76 32 256 1 0.56 0.04 0.73 64 128 1 0.57 0.04 0.77 128 64 1 0.61 0.04 0.94 256 32 1 0.68 0.05 1.37 1024 8 1 1.93 0.14 3.17 8192 1 1 11.10 0.80 14.81

[0185] Table 1: Model inference with different tile shapes [t1, t2, t3] when t3 = b = 1. The reported values ​​are inference latency, encryption and decryption times, and peak memory usage.

[0186]

[0187]

[0188] Table 2: Model inference with different tile shapes [t1, t2, t3], reporting only the optimal t1 and t2 choices for different ranges of t3 = b values. The reported values ​​are: inference latency, amortized latency (latency / b), encryption and decryption times, and peak memory usage.

[0189] AlexNet benchmark

[0190] For the AlexNet benchmark, a variant of the AlexNet network was used, consisting of 5 convolutional layers, 3 fully connected layers, 7 ReLU activations, 3 batch normalization layers, and 3 max pooling layers. A CKKS-compatible variant of AlexNet was created by replacing the ReLU and max pooling components with scaled square activations and average pooling, along with some other changes. It was trained and tested on the COVIDx CT-2A dataset. The COVIDx CT-2A dataset is an open-access benchmark containing CT images of three classes of chest CT images: normal, pneumonia, and COVID-19 cases. Experiments used a subset of 10,000 images per class for training, 1,000 images per class for validation, and a total of 201 images for testing, with 67 random samples from each class. The images were resized to 224*224*3 to fit the input size expected by AlexNet.

[0191] These biases are packaged into 5D tile tensors with compatible shapes, allowing us to add them to the convolutional output. Fully connected layers are processed using matrix-matrix multiplication techniques (see above). The inputs to these layers are 5D tile tensors. It originates from the convolutional layer. Therefore, the first fully connected layer is also packaged in 5 dimensions: Its output It is copied along dimensions 2 to 4, and then flattened using the flatten operator. It can proceed normally from that result.

[0192] The accuracy of running regular AlexNet and HE-friendly AlexNet on a plaintext test set was measured using PyTorch (PyTorch library 1.5.1, see https: / / pytorch.org). The results were 0.861 and 0.806, respectively. No additional accuracy degradation was observed when running HE-friendly AlexNet on encrypted data using this framework.

[0193] Table 3 reports the time and memory consumption of the subsequent experiment using four configurations on a representative set of 30 samples. The configurations involve unencrypted model weights (PT) and encrypted model weights (CT) optimized for low latency (TP) or high throughput (TP). For these configurations, the inference results were also compared to those obtained by running HE-friendly AlexNet on PyTorch on a plaintext test set by calculating the root mean square error (RMSE). These were always less than 4e-3.

[0194] Configuration Delay (seconds) Amortized delay (seconds) Encryption + Decryption (seconds) Memory (GB) PT-Latency 181.9 181.9 5.3 123.8 PT-TP 720.8 90.1 5.4 568.1 CT-Latency 358.1 358.1 5.4 223.4 CT-TP 1130.4 282.6 5.6 688.8

[0195] Table 3: AlexNet executed in this framework with different configurations.

[0196] Optimizer accuracy

[0197] Optimizer module 110 (see Figure 1A , Figure 1B The simulator 110b estimates time and memory usage on a single CPU thread for a given configuration option. For this, it relies on pre-benchmarked metrics for different FHE operations. To evaluate the accuracy of these estimates, experiments were conducted on HE-friendly AlexNet using a cryptographic model. Four configuration options were selected that achieved the lowest estimated latency when using local search, and the inference time and encryption time of the input were compared with the model's output versus the actual run on the encrypted data. Table 4 summarizes the results. It can be observed that the simulator provides relatively accurate time estimates for all four configurations. The average estimated time deviations for inference, model encryption, and batch input encryption are -15.8%, -11.9%, and -7.2%, respectively. Note that the simulated memory matches the measured memory for all configurations; therefore, this data is not included in Table 4.

[0198]

[0199]

[0200] Table 4: Configurations formatted as [Tile Shape - Convolution Mode]: [16, 8, 8, 16, 1] - CWHFB, [8, 8, 8, 32, 1] - CWHFB, [16, 8, 8, 16, 1] - FWHCB, [32, 8, 8, 8, 1] - FWHCB (Conf i ) i＝1..4 The simulation time estimate. The acronyms CWHFB and FWHCB indicate the dimensional order in the tile tensor. The deviation between the estimated time and the actual time is reported in parentheses.

[0201] Additional comparisons

[0202] Tile tensor capture is a simple method where each element of the input matrix is ​​placed in a separate ciphertext as a special case. Table 2 reports the results of this method in the last row.

[0203] Two additional special cases of the matrix-vector multiplication algorithm are described in [Crockett 2020] (see Eric Crockett. A low-depth homomorphic circuit for logistic regression model training. Cryptology ePrint Archive, Report 2020 / 1483, 2020.). These are equivalent to Equations (1) and (2) above. Furthermore, [Crockett 2020] shows an extension to matrix-matrix multiplication by extracting columns from a second matrix and applying matrix-vector multiplication to each column. This extraction of columns requires multiplication by a mask and increases the multiplication depth. Using this tile tensor method, a natural extension to matrix-matrix multiplication can be obtained that does not require increasing the multiplication depth.

[0204] Different families of techniques are based on diagonalization. A fundamental method for matrix-vector multiplication is described in [Halevi 2014] (see Shai Halevi and Victor Shoup. Algorithms in helib. In Juan A. Garay and Rosario Gennaro, editors, Advances in Cryptology - CRYPTO 2014 - 34th Annual Cryptology Conference, Santa Barbara, CA, USA, August 17-21, 2014, Proceedings, Part I, volume 8616 of Lecture Notes in Computer Science, pp. 554-571. Springer, 2014. doi: 10.1007 / 978-3-662-44371-2_31.). For a ciphertext with n slots, an n×n matrix is ​​preprocessed to form a new matrix, where each row is the diagonal of the original matrix. Multiplication with vectors can then be performed using n rotations, multiplications, and additions. The performance of this method depends on the tile shape. For example, for tiles with approximately the same shape... For square tiles, the cost of matrix-vector multiplication is n multiplications. n rotations. (In this case, the matrix is ​​decomposed into n tiles; each needs to be multiplied by a vector tile. Summation reduces the shape of the outer tensor to n.) And use (This involves rotating the tiles to sum over each of the remaining tiles).

[0205] Several improvements to the diagonalization technique have been proposed, which, under certain conditions and by utilizing specific properties of the HE scheme of Helib [Halevi 2014], reduce the number of required rotations to [missing information]. This method makes no special assumptions. Exploring such properties and combining them with tile tensor data structures is left for future work.

[0206] In [Ciaoqian 2018] (see Xiaoqian Jiang et al., Secure outsourced matrix computation and application to neural networks. In Proceedings of the 2018 ACMSIGSAC Conference on Computer and Communications Security, CCS'18, pp. 1209-1222, New York, NY, USA, 2018. Association for Computing Machinery.), a diagonalized matrix-matrix multiplication method is described. For multiplication with n vectors, it reduces the number of rotations to O(n) instead of O(n^2). 2 However, this comes at the cost of increasing the multiplication depth by performing two plaintext multiplications. This is a significant drawback in non-customer-assisted FHEs, as the performance of the circuit is typically proportional to the square of its depth, and the depth is sometimes finite for practical considerations.

[0207] convolution

[0208] Convolutional layers are fundamental building blocks in neural networks. Previous work has proposed optimizations for small inputs: GAZELLE (see, Chiraag Juvekar et al., GAZELLE: A low latency framework for secure neural network inference. In 27th USENIX Security Symposium (USENIX Security 18), pp. 1651-1669, Baltimore, MD, August 2018. USENIX Association.) considers 28×28 grayscale images. GALA (see, Qiao Zhang et al., Gala: Greedy computation for linear algebra inprivacy-preserved neural networks. arXiv preprint arXiv: 2105.01827, 2021.) considers 16×16 images. HEAR (see Miran Kim et al., HEAR: human action recognition via neural networks on homomorphically encrypted data. CoRR, abs / 2104.09164, 2021.) considers a 3D tensor input of size 32×15×2.

[0209] Instead, the inventors considered a 224×224 RGB image. For such a large input, methods using Gazelle, GALA, and / or HEAR are inefficient because they pack the cn channels of the input into a single ciphertext. They then utilize SIMD features to operate on all cn channels. For example, GALA and GAZELLE require a total of A rotation and multiplication operation, where f, c, w I h I Defined above. In HEAR, consider a convolutional sequence. Then, a preprocessing step is needed between the two convolutional steps. The preprocessing step is computed, and the convolution takes... There are rotation and multiplication operations. For an image of size 244×244=50,176, at most one channel can fit into a ciphertext with 65,536 slots, i.e., c n=1. Using ciphertext with fewer slots or larger images leads to performance degradation because the data for a single channel is scattered across several ciphertexts. Previous work did not explain how to scale up to efficiently support this situation. Simply put, more slots can be simulated using several ciphertexts. This adds a factor to the runtime that is proportional to the image size, i.e., O(w I h I ).

[0210] In this convolution method, the number of rotations for larger images is increased by a factor. And the number of multiplications increases This is superior to previous work for large images. However, for multiple channels, filters, and samples in a batch, the runtime of this method increases by a factor. And the additional cost required to sum the channels within the tile Rotate. By selecting the tile shape t i The value of can be optimized for a given computational size.

[0211] Convolutional layer sequence

[0212] In GAZELLE and GALA, optimization is performed on individual convolutional layers. While this is important, deep networks consist of long sequences of convolutional networks with different sizes and numbers of filters. For example, AlexNet has five consecutive convolutional layers of different sizes.

[0213] To support more layers, previous work assumed (in the client-aided approach) a non-FHE step after each layer, such as a scrambling circuit or another MPC protocol. The non-FHE step performs the activation function and places the input for the next layer into the correct packet. Transforming the packet using only an FHE system is expensive. In HEAR, a full FHE scheme is considered. However, they require a preprocessing step, which requires O(w) I h I cb) multiplications and One rotation.

[0214] Conversely, this packaging method only requires a preprocessing step before even-numbered layers. In that case, it needs... A rotation; here, w I h I c and b refer to the image dimension, number of channels, and batch size in the input of this layer.

[0215] Neural network inference

[0216] This method is a companion to other end-to-end neural network inference schemes, such as ngraph-HE2 (see Fabian Boemer et al., NGraph-HE2: A High-Throughput Framework for Neural Network Inference on Encrypted Data. In Proceedings of the 7 th ACM Workshop on Encrypted Computing & Applied Homomorphic Cryptography, WAHC'19, pages 45-56, New York, NY, USA, 2019. Association for Computing Machinery.) and TenSEAL (see Ayoub Benaisa et al., TenSEAL: A Library for Encrypted TensorOperations Using Homomorphic Encryption. arXiv, 2021.)

[0217] Table 5 reports the comparison results. TenSEAL uses diagonalization for matrix multiplication and im2col for convolution, assuming a single image as input. Furthermore, TenSEAL assumes unencrypted model weights. Therefore, TenSEAL is a peer of this framework when optimizing for batch size 1 with unencrypted model weights (PT), and results for encrypted model weights (CT) are also shown for completeness. nGraph-HE2 also focuses on unencrypted models. It uses SIMD packing, and it is a special case of this framework when optimizing for the maximum possible batch size.

[0218] frame Delay (seconds) Amortized delay (seconds) TenSeal(b=1) 3.55 3.55 Present Method - PT (b = 1) 0.48 0.48 Present Method-CT (b=1) 0.56 0.56 nGraph-HE2(b=8192) 11.93 0.00146 Present Method-TP(b=8192) 13.52 0.00165 Present Method-CT (b=8192) 41.32 0.00504

[0219] Table 5: Comparison of CryptoNets benchmarks with current tile tensor frameworks and other freely available online NN compilers. b=1 and b=8192 are set for the top and bottom lines, respectively.

[0220] These results highlight the efficiency and versatility of this framework. Aiming for optimal latency, this framework offers at least seven times the speedup compared to nGraph-HE2 and TenSEAL. Furthermore, it can adapt to variable batch sizes. When targeting optimal throughput, nGraph-HE2 is slightly faster than this framework. This can be explained by the fact that this library currently focuses on optimizing the packing scheme, which in this case is the same as the packing scheme used by nGraph-HE2. Therefore, both libraries perform the exact same set of homomorphic operations, but nGraph-HE2 also provides optimizations for pipelined underlying FHE instructions (e.g., through lazy rescaling). It can be emphasized that the power difference in using different packing schemes is more pronounced for large networks involving a series of operations and is generally not reflected in smaller networks such as CryptoNets.

[0221] Another framework not included in the above comparative experiments is the CHET compiler (see Roshan Dathathri et al., Chet: An optimizing compiler for fully-homomorphic neural-network inferencing. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2019, pp. 142-156, New York, NY, USA, 2019. Association for Computing Machinery.), which performs inference on encrypted data in unencrypted networks. They reported a 2.5-second latency using 16 threads on a similarly sized (albeit less accurate) MNIST neural network classifier. They employ a similar approach using an abstract data structure (CipherTensor) combined with automatic optimization. CipherTensors are currently considered less flexible than tile tensors. They consist of a small, fixed set of implemented layouts, each with its own algorithmic kernel, while tile tensors utilize a single set of general algorithms to provide a wider variety of options. Furthermore, it has not been proven that CipherTensors provide a simple way to trade throughput and control memory consumption with latency, as is possible in tile tensors by controlling the batch dimension. Finally, CipherTensors require rotations to copy the input data, while using tile tensors avoids some of these copies.

[0222] The EVA compiler (see Roshan Dathathri et al., Eva: An encrypted vector arithmetic language and compiler for efficient homomorphic computation. In Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2020, pp. 546-561, New York, NY, USA, 2020. Association for Computing Machinery.) is built on top of CHET. They report an improved performance of 0.6 seconds using 56 threads on the same network, along with various packing-independent optimizations; these optimizations are beyond the scope of this paper. For a more precise CryptoNets architecture, a state-of-the-art result of 0.48 seconds was achieved. The inventors believe that even better results can be obtained by combining this packing optimization with EVA's optimizations (e.g., eliminating rescaling operations to reduce the total main chain length).

[0223] The LoLa network (see Alon Brutzkus et al., Low latency privacy preserving inference. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 812-821, Long Beach, California, USA, 09-15 Jun 2019. PMLR.) also reports results for CryptoNets architectures. They achieve a latency of 2.2 seconds using 8 threads. The LoLa network uses 150 ciphertext-ciphertext multiplications, 279 rotations, and 399 additions for a single prediction. (These numbers are inferred from the detailed description of LoLa in this invention.) This method requires 32 multiplications, 89 rotations, and 113 additions. These differences roughly explain the observed latency results.

[0224] The present invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the present invention.

[0225] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices on which instructions are recorded, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires. Rather, computer-readable storage media are non-transient (i.e., non-volatile) media.

[0226] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0227] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Java, Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by personalizing the electronic circuitry using state information from the computer-readable program instructions in order to perform aspects of the invention. In some embodiments, electronic circuitry, including, for example, an application-specific integrated circuit (ASIC), may have computer-readable program instructions incorporated at the time of manufacture, such that the ASIC is configured to execute these instructions without programming.

[0228] This document describes aspects of the invention with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0229] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create parts for implementing the functions / actions specified in one or more boxes of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other devices to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0230] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions that execute on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0231] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing one or more specified logical functions. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action or performs a combination of dedicated hardware and computer instructions.

[0232] In the specification and claims, when describing numerical values, the terms “approximately,” “substantially,” and their forms each refer to a deviation of up to 20% (i.e., ± 20%) from the value. Similarly, when such terms describe a range of numerical values, they imply a wider range of up to 20%—10% above and below the defined range.

[0233] In this specification, any given range of numbers should be considered as having specifically disclosed all possible subranges and individual numbers within that range, such that each such subrange and individual number constitutes an embodiment of the invention. This applies regardless of the width of the range. For example, a description of an integer range from 1 to 6 should be considered as having specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, etc., and individual numbers within that range, such as 1, 4, and 6. Similarly, a description of a decimal range (e.g., from 0.6 to 1.1) should be considered as having specifically disclosed subranges such as from 0.6 to 0.9, from 0.7 to 1.1, from 0.9 to 1, from 0.8 to 0.9, from 0.6 to 1.1, from 1 to 1.1, etc., and individual numbers within that range, such as 0.7, 1, and 1.1.

[0234] Various embodiments of the present invention have been presented for illustrative purposes but are not intended to be exhaustive or limited to explicit description. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0235] In the specification and claims of this application, the words “comprising,” “including,” and “having,” and each of their forms, are not necessarily limited to members of the list that these words may be associated with.

[0236] In the event of any inconsistency between this specification and any document referenced or otherwise relied upon, the intent is for this specification to take control.

Claims

1. A computer-implemented method, comprising: Receive an input tensor, wherein the input tensor has a composition derived from... The defined shape, where, It is equal to the number of dimensions that characterize the input tensor; Receive tile tensor metadata, which includes at least: (i) data from... (ii) Defined tile tensor shape, and (ii) information indicating the interlacing stride to be applied relative to each dimension of the tile tensor; An output tensor comprising multiple tile tensors is constructed by applying a packing algorithm, which maps each element of the input tensor to at least one slot position of one of the multiple tile tensors based at least in part on the shape of the tile tensors and the interlacing stride. The interleaving step results in a discontinuous mapping of the elements of the input tensor, such that each tile tensor in the tile tensor includes a subset of the elements of the input tensor spaced apart within the input tensor according to the interleaving step. The tile tensor is a homomorphic encrypted ciphertext.

2. The computer-implemented method of claim 1 further includes storing the output tensor and the tile tensor metadata.

3. The computer-implemented method as described in claim 1, further comprising: The input tensor is unpacked from the stored output tensor based on the tile tensor metadata.

4. The computer-implemented method as described in claim 1, wherein, The tile tensor metadata also includes a copy parameter, and the packing algorithm is configured to perform a copy of the elements of the input tensor based on the copy parameter, such that each element of the input tensor is mapped to a plurality of slot positions along one of the dimensions of the tile tensor.

5. The computer-implemented method as described in claim 1, further comprising: Receive the filter associated with the convolution calculation on the input tensor; The convolution is computed by applying a multiplication operator, which multiplies the filter element-wise with each tile tensor in the output tensor. A summation algorithm is applied to sum the results of the multiplication; and The result of the application of the summation algorithm is output as the result of the convolution.

6. The computer-implemented method as described in claim 5, wherein, The convolution is part of the neural network inference.

7. A system comprising: At least one hardware processor; as well as Program instructions that, when executed by the at least one hardware processor, perform the method as described in any one of claims 1-6.

8. A computer program product comprising program instructions executable by at least one hardware processor to cause the processor to perform the method as described in any one of claims 1-6.

Images