Optimizing computer programs for table lookup operations.

Abstract

The present invention relates to a computer-implemented method (800) for optimizing a computer program for an execution environment that supports table lookup operations. The computer program is represented as a computation graph, with each node of the computation graph representing a respective operation. For each node, it is determined whether the respective node can be implemented by a table lookup operation. For a node that can be implemented by a table lookup operation, it is determined that one or more further nodes of the computation graph can be fused into the table lookup operation. A transformed representation of the computer program is output, in which the node and the one or more further nodes are fused into a single table lookup operation.

Description

[Technical Field]

[0001] The present invention relates to a computer-implemented method, and corresponding system, for optimizing a computer program for an execution environment that supports table lookup operations.The present invention further relates to a computer-readable medium. [Background technology]

[0002] Homomorphic encryption allows a party to run a computer program, e.g., a circuit evaluation, on encrypted data without the party being able to decrypt it. For example, input data and computation results may be received and returned in encrypted form. Intermediate data of the computation, e.g., internal state, may also be in encrypted form. Although the output of the computer program is returned in encrypted form, when decrypted, the output is expected to be the same as, or very close to, the result when the operation is performed on unencrypted data. Homomorphic encryption can be used for privacy-preserving outsourced storage and computation. This allows data to be encrypted and outsourced to a cloud environment for processing and / or storage, all while remaining encrypted. In particular, homomorphic encryption techniques exist that can, at least in principle, be used to compute any function on encrypted data. Such techniques are called "fully homomorphic encryption" (FHE) techniques.

[0003] Homomorphic encryption techniques typically provide an execution environment that supports a relatively limited set of operations. More complex operations can then be implemented in terms of this set of supported operations. Supported operations typically include addition and subtraction of integer ciphertexts; and multiplication of integer ciphertexts with known scalars. Homomorphic encryption techniques can also support, for example, multiplication of encrypted integer ciphertexts.

[0004] The execution environment provided by various homomorphic encryption techniques also supports table lookup operations. In particular, the table lookup operation can be a programmable bootstrapping (PBS) operation. Such an operation can not only implement the table lookup operation but also reduce the noise present in the ciphertext used to a certain level. Homomorphic encryption techniques that support the PBS operation are also called TFHE-like homomorphic encryption schemes. Such schemes are described in I. Chillotti et al., "Programmable bootstrapping enables efficient homomorphic inference of deep neural networks," Cyber ​​Security Cryptography and Machine Learning (CSCML 2021), Lecture Notes in Computer Science, Vol. 12716, pp. 1-19, Springer, 2021 (incorporated herein by reference). In general, TFHE-like homomorphic encryption schemes can be based on lattice encryption, such as LWE encryption or NTRU encryption.

[0005] Although the computational efficiency of homomorphic encryption techniques has improved significantly in recent years, significant overhead remains, especially when applying these techniques to relatively large amounts of data and / or relatively complex calculations. In particular, the overhead of performing table lookups remains significant compared to other operations such as homomorphic addition or scalar multiplication. Therefore, computer programs need to be optimized to be efficiently executed by homomorphic encryption execution environments that support table lookup operations.

[0006] Table lookup operations are also used in various other execution environments. For example, in hardware, lookup tables can be used to calculate arbitrary functions. Such hardware lookup tables can be implemented, for example, by slices of RAM (random access memory) or ROM (read-only memory). Lookup tables can be useful to speed up calculations, especially those applied to large amounts of data, such as in neural network accelerator circuits for training or evaluating neural networks. For example, the activation function of a neural network can be implemented as a lookup table. Even in this setting, it is important to optimize computer programs for efficient execution. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] I. Chillotti et al., “Programmable bootstrapping enables efficient homomorphic inference of deep neural networks,” Cyber ​​Security Cryptography and Machine Learning (CSCML 2021), Lecture Notes in Computer Science, volume 12716, pp. 1-19, Springer, 2021. [Non-patent document 2] L. Ducas et al., “FHEW: bootstrapping homomorphic encryption in less than a second,” proceedings EUROCRYPT 2015 [Non-patent document 3] C. Bonte et al., “FINAL: Faster FHE instantiated with NTRU and LWE,” https: / / eprint.iacr.org / 2022 / 074 [Non-patent document 4] K Kluczniak, “NTRU-ν-um: Secure Fully Homomorphic Encryption from NTRU with Small Modulus”, https: / / eprint.iacr.org / 2022 / 089 [Non-Patent Document 5] B. Jacob et al., “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” doi:10.1109 / CVPR.2018.00286 Summary of the Invention [Problem to be solved by the invention]

[0008] In particular, it is desirable to optimize computer programs for execution environments that support table lookup operations by minimizing the number of table lookup operations required to execute the computer program in the execution environment. [Means for solving the problem]

[0009] According to a first aspect of the present invention, there is provided a computer-implemented method and corresponding system for optimizing a computer program, as defined respectively by claims 1 and 14. According to a further aspect of the present invention, there is provided a computer-readable medium as defined by claim 15.

[0010] Various embodiments relate to optimizing computer programs for execution environments that support table lookup operations. To this end, a representation of the computer program may be captured as a computation graph. Each node of the computation graph may represent a respective operation. The computation graph is often a tensor computation graph, where the nodes represent operations applied to one or more input tensors and resulting in output tensors corresponding to the edges of the graph; however, computation graphs that represent only scalars are also possible.

[0011] The representation may be transformed such that multiple nodes of the computation graph are implemented as a single table lookup operation (where, in the case of a tensor computation graph, a table lookup operation may represent an element-wise application of a table lookup). Implementing multiple nodes together as a single table lookup operation may be referred to as "fusing" the nodes (whether the fusion is done explicitly, for example, by merging nodes of the computation graph, or by outputting a set of instructions for the execution environment based on determining that the nodes can be implemented as a single table lookup operation). The optimization may output a transformed representation of the computer program to which the fusion has been applied.

[0012] While it is known per se to manually annotate (single-input, single-output) operations of a computer program as fusible and then fuse these operations into a single table lookup operation, e.g., fusing operations that represent respective univariate functions applied successively to the same input, the inventors have conceived of improving such fusion. For this purpose, a computation graph can be obtained. The computation graph can, in principle, represent any computation that can be performed by an execution environment. Given this graph, it can be automatically determined for each node thereof whether the node can be implemented by a table lookup operation. This determination can be made not only based on the node itself, but also based at least on the node's incident edges in the computation graph. In particular, this determination can be based on whether the output of the node can be expressed as a univariate (element-wise) function of the output of another node, even if the node has multiple inputs.

[0013] For a node determined to be implementable by a table lookup operation, it may be determined that one or more further nodes in the computation graph (e.g., nodes between the node itself and other nodes) can be fused with the table lookup operation, and in the output transformed representation, these further nodes may be fused with the node.

[0014] In particular, the computational graph of, for example, a general neural network, can be transformed into a form that fuses as many univariate operations as possible, and a mechanism can be provided to compute the resulting operations using lookup tables. For example, univariate operations can include, but are not limited to, quantization, normalization, and activation functions.

[0015] In general, by determining in an automatic way whether a node can be implemented by a table lookup operation, e.g., based on the operation itself, it is possible to find fusible nodes that may not be found using manual annotations alone. For example, a node representing an operation applied (element-wise) to multiple (tensor) inputs may be implementable by a table lookup operation, depending on whether those inputs themselves depend (element-wise) on a common previous input. Thus, whether a node is implementable by table lookup may depend on what other nodes the node is connected to. Determining this manually may be infeasible, especially for large-scale computations. Also, while certain heuristics are known to fuse nodes, such heuristics, by their nature, only work in certain settings and may miss fusion opportunities.

[0016] Thus, by automatically determining whether a node can be implemented by a table lookup operation and then fusing such a node with other nodes into a single table lookup operation, the efficiency of a computer program in a lookup table-based execution environment can be improved, particularly by reducing the number of table lookup operations performed to execute the computer program. (As explained elsewhere, this does not mean that all nodes determined to be implementable by a table lookup table will actually be implemented in this way; e.g., a scalar multiplication that is not preceded or followed by other nodes implementable by table lookup may be implemented natively, e.g., using the native operations of homomorphic encryption.)

[0017] This is particularly beneficial in some settings. One such setting is when the table lookup operations are hardware lookup tables, for example in hardware accelerators. For example, hardware accelerators with low bandwidth registers, such as up to 16 bits or even up to 8 bits. Here, computing a group of univariate operations using a table lookup can be faster than performing a series of arithmetic operations that compute the same result. Furthermore, the resulting transformed representation is faster than performing successive table lookups, one for each univariate operation.

[0018] Furthermore, when applied to tensor computation graphs that perform predictions for quantized neural networks, the provided techniques can eliminate the need for iterative requantization of inputs for each individual univariate operation when several operations are fused. Thus, the method can improve the accuracy of the final result by eliminating the need to quantize univariate operations individually, as quantization incurs a loss of accuracy due to the approximations made.

[0019] The provided techniques also enable the design of simpler hardware accelerators, such as specialized neural network hardware accelerators for quantized neural networks, avoiding the inclusion of more complex circuitry for normalization, quantization, and activation functions.

[0020] Another setting in which the provided techniques are particularly useful is that of homomorphic encryption, e.g., fully homomorphic encryption (FHE). In this setting, encrypted values ​​may be stored as ciphertexts composed of one or more numeric values ​​(e.g., LWE or NTRU encryption). The size of such ciphertexts, and the computational efficiency of encrypted computation operations defined on them, may depend on the bit-width of the input to the table lookup operator, meaning that it is useful to compute at low bit-widths, e.g., up to 8 bits or up to 12 bits.

[0021] In this situation, by applying the provided technique, a transformed representation of a computer program optimized for execution by homomorphic encryption can be obtained as follows. First, the number of lookup table operations can be minimized. In fact, the computational cost of lookup table operations is relatively high, and having redundant LUTs increases the computational cost. Furthermore, homomorphic encryption typically operates on encrypted integer values ​​and therefore typically uses quantization. By avoiding individually encoding the quantizer, activation, and normalization layers into the TLU, the associated quantization error caused by each LUT individually introducing quantization error is avoided. Furthermore, in general, applying a TLU operation can add small errors or noise to the ciphertext, which means that the result of a TLU can be off by 1 with a small probability. Therefore, chaining multiple TLUs increases the probability of error. The provided technique can reduce the number of TLUs used, thereby also reducing the possibility of error.

[0022] Each node in the computation graph may represent a respective operation. Optimizations may be made so that no operation in the first set of operations is implemented by a table lookup operation. When the computation graph is defined on tensors, this may be true, for example, with respect to operations that are applied to one or more input tensors rather than element-wise, such as convolutions. Such operations may be implemented in terms of the native operations of the execution environment (in this case, the operations themselves, in the sense of the function of the operations in the computation graph, are not implemented by a table lookup, although pre-processing and / or post-processing, e.g., (de-)quantization, may still use table lookups; this is described in more detail elsewhere herein).

[0023] Nodes representing operations from the second set of operations may be implemented as lookup tables depending on the computation graph, e.g., their incoming and / or outgoing edges. For these nodes, a native implementation of the operation by, e.g., the execution environment may be available, but the node may still be fused with other nodes into a table lookup if it makes the overall computation more efficient. This may be the case, e.g., for addition or scalar multiplication. Nodes representing operations from the third set of operations may always be implemented by table lookup operations (e.g., activation functions), e.g., because a native implementation of the operation in the execution environment is not available.

[0024] Optionally, the optimization may include a substitution operation, where if it is determined that a node can be implemented by a table lookup operation, the node is replaced by a first subgraph representing the first implementation; otherwise, the node is replaced by a second subgraph representing the second implementation. The substitution operation may be performed after it is determined which nodes can be implemented by a table lookup operation, but before it is determined that such nodes can be fused with additional nodes. In particular, nodes that can be fused may include one or more nodes from the second subgraph of another node, for example, a node corresponding to a dequantization operation of a previous node, or a quantization node of a next node. In this way, as discussed in more detail elsewhere herein, optimization may be combined with a transformation that converts a high-level computation graph into a lower-level computation graph.

[0025] Optionally, the second implementation used by the permutation operation may be configured to quantize the input (e.g., a floating-point input) to an integer; apply an integer operation corresponding to the node to obtain an integer output; and dequantize the integer output (e.g., to a floating-point). For example, the input graph of the optimizer may allow various types of input for a particular operation (e.g., supporting both integer and floating-point inputs), where the permutation operation may result in a graph in which each operation only allows input of the respective type (e.g., addition only supports integers, and the permutation operation includes quantization if the previous output is not an integer). Depending on the input and / or output supported by the node, nodes corresponding to quantization and / or dequantization may be included in the sub-graph.

[0026] Optionally, for a node with multiple inputs (e.g., multiple tensor inputs if the computation graph is a tensor computation graph), it may be determined that the node is implemented by a table lookup operation based on determining that the multiple inputs have a common ancestor in the graph. For example, it may be determined that the (element-wise) addition of two (tensor) inputs that themselves represent an (element-wise) function applied to a common preceding edge can itself be represented as an element-wise function applied to a common preceding edge, and therefore can be implemented by a table lookup operation. Because such a determination is not feasible to make manually, especially for large computations, and is based, in particular, on the structure of the computation graph, this is a particular example where it is beneficial to automatically determine whether a node can be implemented as a table lookup.

[0027] A computation graph may include one or more reshaping nodes. Such reshaping nodes may represent operations that rearrange elements of one or more input tensors without changing their values, such as transposing, reshaping, or flattening. In such cases, the computation graph may be reordered to merge at least one operation preceding the reshaping node with at least one operation following the reshaping node. For example, swapping a reshaping node with its preceding or succeeding node may result in two adjacent nodes representing element-wise operations that can be implemented by lookup tables and then merged into a single table lookup. As described herein, when the optimization includes a permutation operation, reordering is preferably performed after the permutation operation, but in principle could also be applied before the permutation operation.

[0028] Optionally, the transformed representation of the computer program does not include table lookup operations applied to the output of further table lookup operations. Instead, such subsequent table lookups may be fused, resulting in a more efficient implementation of the computer program. Optionally, the transformed representation of the computer program does not include reshaping operations applied to the output of table lookup operations. By swapping TLUs and reshaping whenever reshaping is applied to the output of a TLU, it may be ensured that no TLUs have only reshaping among them; in other words, such TLUs are grouped together in the computation graph. This makes it possible to fuse TLUs, for example, by selecting them as connected subcomponents of the graph.

[0029] Optionally, the computer program may represent the evaluation of machine-learnable models such as artificial neural networks, generalized linear models, decision trees, or ensemble models. Such types of computations can be well described by (tensor) computational graphs, can be relatively complex, and can be applied repeatedly to a variety of data, making it important to optimize computer programs for their efficient evaluation.

[0030] In general, the input computation graph may be defined in a format that is independent of a particular execution environment and / or using a set of supported operations that are independent of a particular execution environment. For example, the input computation graph may be a high-level graph defined, for example, in the ONNX (Open Neural Network Exchange) format. The transformed representation may be a low-level representation specific to the execution environment being optimized, for example, an executable format executable by the execution environment, or may be a computation graph that uses a limited set of nodes and / or edges supported by the execution environment. For example, a permutation operation may be used to implement a high-level operation in terms of a low-level operation. For example, an optimizer may be used in a cryptographic computation system to enable a machine learning model or other type of computation specified according to a high-level representation to be efficiently executed using homomorphic encryption.

[0031] Those skilled in the art will appreciate that two or more of the above-described embodiments, implementations and / or optional aspects of the invention may be combined in any way deemed useful.

[0032] Modifications and variations of any system and / or any computer-readable medium corresponding to the described modifications and variations of the corresponding computer-implemented methods can be made by one skilled in the art based on this specification.

[0033] These and other aspects of the invention will be apparent from and will be further elucidated with reference to the embodiments described by way of example in the following description and with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0034] [Figure 1] FIG. 1 illustrates a compiler system. [Figure 2a] FIG. 1 illustrates an example of an encryption computing system. [Figure 2b] FIG. 1 illustrates an example of a device that supports a hardware lookup table. [Figure 3a] FIG. 10 is a diagram illustrating a detailed example of a computation graph. [Figure 3b] FIG. 10 is a diagram illustrating a detailed example of a computation graph. [Figure 3c] FIG. 10 is a diagram illustrating a detailed example of a computation graph. [Figure 3d] FIG. 10 is a diagram illustrating a detailed example of a computation graph. [Figure 4] FIG. 1 illustrates an example of a method for optimizing a computer program. [Figure 5a] FIG. 10 is a diagram illustrating a detailed example of a transformed computation graph. [Figure 5b] FIG. 10 is a diagram illustrating a detailed example of a transformed computation graph. [Figure 5c] FIG. 10 is a diagram illustrating a detailed example of a transformed computation graph. [Figure 6a] FIG. 10 is a diagram illustrating a detailed example of reordering a computation graph. [Figure 6b] FIG. 10 is a diagram illustrating a detailed example of reordering a computation graph. [Figure 7] FIG. 1 illustrates an example of a method for optimizing a computer program. [Figure 8] FIG. 1 illustrates a computer-implemented method. [Figure 9] FIG. 1 illustrates a computer-readable medium containing data. DETAILED DESCRIPTION OF THE INVENTION

[0035] It should be noted that the figures are merely schematic and are not drawn to scale, in which elements corresponding to elements already described may be provided with the same reference numerals.

[0036] Figure 1 shows a compiler system 100. System 100 may be for optimizing a computer program for an execution environment that supports table lookup operations, such as for homomorphic encryption computations as illustrated in Figure 2a, or for a hardware implementation as illustrated in Figure 2b.

[0037] The system 100 may include a data interface 120 for accessing data 040 that represents a computer program as a computation graph. Each node in the computation graph may represent a respective operation. For example, the input computation graph may include at least 50 nodes, at least 250 nodes, or at least 10,000 nodes.

[0038] For example, as also shown in FIG. 1 , the data interface 120 may be configured with a data storage interface 120 that can access data 040 from a data storage 021. For example, the data storage interface 120 may be a memory interface or a persistent storage interface, such as a hard disk or SSD interface, or a personal, local, or wide area network interface, such as a Bluetooth, Zigbee, or Wi-Fi interface, or an Ethernet or fiber optic interface. The data storage 021 may be internal data storage of the system 100, such as a hard drive or SSD, or may be external data storage, such as a network-accessible data storage. In some embodiments, each piece of data may be accessed from a different data storage, for example, via a different subsystem of the data storage interface 120. Each subsystem may be of the type described above for the data storage interface 120.

[0039] System 100 may further include a processor subsystem 140 that, during operation of system 100, may be configured to determine, for each node of the computation graph, whether the respective node can be implemented by a table lookup operation. Processor subsystem 140 may be further configured to determine, for a node that can be implemented by a table lookup operation, that one or more additional nodes of the computation graph can be fused into the table lookup operation. Processor subsystem 140 may be further configured to output a transformed representation of the computer program in which the node and the one or more additional nodes are fused into a single table lookup operation.

[0040] System 100 may further include an output interface for outputting the transformed representation of the computer program. In this example, the output interface is constituted by a communication interface 180, through which the transformed representation 126 may be output to another system, such as a cryptographic device that performs encrypted calculations according to the output representation. Communication interface 180 may internally communicate with processor subsystem 140 via data communication 125. Communication interface 180 may be configured to communicate directly with other systems, for example, using a USB, IEEE 1394, or similar interface. As shown in this figure, communication interface 180 may also communicate over a computer network 1010, such as a wireless personal area network, the Internet, an intranet, a LAN, a WLAN, or the like. For example, communication interface 180 may include connectors appropriate for the computer network, such as a wireless connector, an Ethernet connector, a Wi-Fi, a 4G or 4G antenna, a ZigBee chip, or the like. Communication interface 180 may also be an internal communication interface, such as a bus, an API, a storage interface, or the like.

[0041] However, alternative ways of implementing the output interface are possible. For example, the output interface may be constituted by a data interface 120, which in these embodiments is an input / output ("IO") interface through which the determined transformed representation may be stored in data storage 021. In some embodiments, the output interface may be separate from data storage interface 120, but may generally be of the type described above for data storage interface 120.

[0042] In general, each system described herein, including but not limited to system 100 of FIG. 1, may be embodied as or within a single device or apparatus, such as a workstation or server. The device may be an embedded device. The device or apparatus may include one or more microprocessors executing appropriate software. For example, the processor subsystem of each system may be embodied by a single central processing unit (CPU), but may also be embodied by a combination or system of such CPUs and / or other types of processing units. Software may be downloaded and / or stored in corresponding memory, e.g., volatile memory such as RAM or non-volatile memory such as Flash. Alternatively, the processor subsystem of each system may be implemented in the device or apparatus in the form of programmable logic, e.g., a field programmable gate array (FPGA). Generally, each functional unit of each system may be implemented in the form of a circuit. Each system may be implemented in a distributed manner, including various devices or apparatuses, e.g., distributed local servers and cloud-based servers.

[0043] 2a shows a detailed, but non-limiting, example of an encrypted computation system 199. The system 199 is configured to perform encrypted computations using homomorphic encryption, for example fully homomorphic encryption.

[0044] The cryptographic computing system 199 may include a cryptosystem 112 configured to perform encrypted computations using homomorphic encryption. The cryptographic computing system 199 may further include a compiler system 111 configured to determine a representation of a computer program optimized for the execution environment provided by homomorphic encryption. The compiler system 111 may be based, for example, on system 100 of FIG. 1. The cryptographic computing system 199 may further include a data provider device 113. Each of systems 111, 112, 113 in this example may generally be based on the hardware configuration of device 100 of FIG. 1a and may include, for example, a processor system, data, and / or communication interfaces, as in FIG. 1.

[0045] System 112 may be configured to receive encrypted data items from data provider 113. At least one or more data items may be received in encrypted form. One or more additional data items may be received in plain format. System 112 may be configured to receive key material for performing encrypted calculations, such as bootstrapping and / or key-switching keys, from, for example, a data provider system or a key generation system.

[0046] System 112 may perform encrypted computations on received data items, and possibly also on stored data items. Interestingly, this device allows computations to be performed on encrypted data without decrypting the data, e.g., without converting the encrypted data items to data in plain format. System 112 may perform encrypted computations according to the transformed representation determined by compiler system 111.

[0047] Compiler system 111 may be combined with data provider system 113, in which case data provider system 113 may provide a transformed representation of a computer program along with data items, for example, for executing the computer program. Alternatively, compiler system 111 may be combined with or invoked by cryptosystem 112 to perform optimizations, in which case data provider system 113 may provide a computation graph to which optimizations are applied along with data items, for example, for executing the computer program. For example, the computation graph may be in a format defined independently of execution by encrypted computations, such as the ONNX format, and may not even be constructed for use with homomorphic encryption. Nevertheless, the optimizations may enable efficient execution of computer programs using homomorphic encryption, particularly reducing the number of table lookup operations.

[0048] Although not shown in this figure, cryptographic computation system 199 may comprise multiple cryptosystems, e.g., two, three, four, or more. Encrypted computations may be distributed among multiple cryptosystems. Cryptosystems may typically exchange encrypted intermediate computation results with each other. Each cryptographic device may be implemented as a cryptosystem 112 and may perform encrypted operations as described herein.

[0049] In general, various types of homomorphic encryption can be used that support encrypted table lookup operations. Often, homomorphic encryption involves integer and / or polynomial plaintexts defined modulo a particular modulus (modular integers are sometimes referred to in the literature as correspondences).

number

[0050] Aside from supporting encrypted table lookup operations, homomorphic encryption typically supports a set of additional operations that do not involve table lookups. Such operations are sometimes referred to as homomorphic encryption's native operations and may include, for example, addition operations and / or encrypted scalar multiplications. Such operations are typically more computationally efficient than table lookups. Therefore, where possible, implementing these operations natively may generally be preferred over table lookups. Interestingly, however, in some cases, these operations may still be fused into preceding and / or subsequent lookup table operations as described herein, leading to improved overall efficiency.

[0051] In particular, the computation may use homomorphic lattice encryption, such as LWE encryption and / or NTRU encryption. For such encryption, methods for implementing various operations, including encrypted table lookup operations, are known per se. For example, the homomorphic encryption may be a "TFHE-like" homomorphic encryption, in the sense that the table lookup operation is implemented as a so-called programmable bootstrapping operation. The operation may perform a table lookup on the input ciphertext based on a blind rotation. The blind rotation may perform decryption of the input ciphertext with the exponent of a monomial. By multiplying the encrypted monomial by a polynomial representing the lookup table, an encrypted polynomial having the output of the lookup table as a constant coefficient can be obtained. For example, the polynomial coefficients may be extracted, or the encrypted polynomial may be further processed. Such TFHE-like programmable bootstrapping is known per se for LWE encryption, e.g., from I. Chillotti et al., "Programmable bootstrapping enables efficient homomorphic inference of deep neural networks," and L. Ducas et al., "FHEW: bootstrapping homomorphic encryption in less than a second," proceedings EUROCRYPT 2015; and for NTRU encryption, e.g., from C. Bonte et al., "FINAL: Faster FHE instantiated with NTRU and LWE," https: / / eprint.iacr.org / 2022 / 074 and K Kluczniak, "NTRU-ν-um: Secure Fully Homomorphic Encryption from NTRU with Small Modulus," https: / / eprint.iacr.org / 2022 / 089.

[0052] Programmable bootstrapping (PBS) may take a ciphertext as input and output a ciphertext of the same message, or a function of that message (in other words, a lookup table applied to that message), with noise that is independent of the input. PBS may include blind rotations that evaluate homomorphic decryption of the input ciphertext at the exponent of a polynomial. In particular, programmable bootstrapping may take as inputs a ciphertext that encrypts a message m; a bootstrapping key; and an encryption of a lookup table L. Programmable bootstrapping may output an encryption with a constant level of noise that encrypts the message L[m]. Blind rotations are also known as accumulator updates.

[0053] For example, the cryptographic computing system 199 may be configured to train a machine learning model, e.g., an image classifier, e.g., a medical model, without the cryptographic computing device having access to the plain data items. A neural network may be trained. For example, backpropagation may be performed on the input data. The resulting model parameters may be returned to the entity that owns the decryption key. This allows multiple providers of medical data to pool their data by sending it to a cloud provider. The cloud provider then returns the model parameters without accessing the plain data. The encryption key may be the same as the decryption key.

[0054] After the model is trained, the cryptographic computation system 199 can be used to provide the model for use with, for example, medical data. This can be done using plain model parameters or encrypted model parameters. In both cases, encrypted data, e.g., encrypted input, intermediate, and output data, is used. Using plain model parameters is typically much more efficient. In either case, the effect of the system is that computations such as image classification, e.g., medical image classification, are performed without the computer having knowledge of the plain data items. For example, a mammogram may be evaluated for cancer, but the image is not plain in the cryptographic system 112, and the results of the cancer assessment are not known to the cryptographic system 112 or a federation of such devices. From a privacy perspective, computing a plain model on encrypted privacy-sensitive data may be acceptable, but computing on plain privacy-sensitive data may not be acceptable.

[0055] 2b shows a detailed, but non-limiting, example of a device 200 that supports lookup tables, where the lookup tables are implemented in hardware.

[0056] In particular, the diagram shows a processor subsystem 240 in which a hardware lookup table (LUT) 241 is implemented. The processor subsystem 240 may be similar to that described for the processor subsystem 140 of FIG. 1. In particular, the processor subsystem 240 may be implemented in programmable logic, for example as a field programmable gate array (FPGA). The processor subsystem 240 may also be implemented, in whole or in part, as a so-called application specific integrated circuit (ASIC), e.g., an integrated circuit (IC) customized for their specific application. For example, the circuit may be implemented in CMOS using a hardware description language such as Verilog or VHDL.

[0057] As shown in this figure, the lookup table LUT may access an input IN, 231 and determine an output OUT, 233 based on accessing data 232 that stores the lookup table in a memory location corresponding to the input IN. Here, the data 231-233 are shown as being stored in a separate memory 240, which may correspond, for example, to storage 021 in FIG. 1a. For example, the lookup table may be implemented through a slice of RAM (random access memory) or ROM (read only memory). The data 231-233 may also be accessed by the lookup table LUT in various ways, for example, as values ​​in registers or over a bus.

[0058] As shown in this diagram, the output OUT may correspond directly to the data stored in the memory location indicated by the input IN. However, alternative implementations of hardware lookup tables are possible. For example, instead of an output, the data 232 may represent parameters of a function that can be calculated relatively efficiently. For example, according to known techniques, the most significant bits of the input value IN may be used to look up two values, which may then be interpolated using the least significant bits of the input IN to obtain the output OUT.

[0059] As a particular example, device 200 may be used to perform inference or training of a neural network. For example, processor subsystem 240 may be a neural network accelerator, e.g., specialized circuitry optimized for computing the operations performed by a neural network.

[0060] 3a-3d show detailed examples of computational graphs that can be optimized using the techniques described herein. These examples illustrate the application of the present invention to a computer program representing the evaluation of a machine-learnable model, in this example an artificial neural network.

[0061] In particular, the computational graph of Figure 3a shows the evaluation of neurons of a neural network. As known per se, neural networks can be used as statistical models that learn using training data to detect or generate patterns. Such neural networks can comprise a plurality of neurons, for example at least 1000 or at least 10,000 neurons. A neuron can perform a weighted combination 312 of a plurality of inputs 310 according to respective weights 311 and apply an activation function 313 to the result. The inputs 310 of a neuron can be, for example, raw data or the activation of a previous neuron. In this example, a neuron is assigned weights and a bias b k (This figure shows the combination of m-1 inputs with bias w k0 =b k (Note that nodes / edges relating to such constant values ​​are typically not considered part of the computation graph being processed here.)

[0062] The activation function 313 may be, for example, a univariate function with a single variable input (but possibly using a constant). In practice, the activation function can determine whether a neuron is activated, e.g., may or may not have a high output value 314. Thus, for example, using simpler mathematical operations, one can effectively determine whether a neuron's input to the network is important in the prediction process. For example, the activation function can be a Rectified Linear Unit, a Sigmoid, or an Exponential Linear Unit.

[0063] Mathematically, the computation of a neuron in a neural network can be expressed as follows: y=φ(Σ i w i x i +b)=φ(s) The dot product 312 between the weight 311 and the input 310 may also be called the neuron accumulator s.

[0064] In a neural network, neurons may be arranged in successive groups called layers. For example, a neural network may have at least 10 layers or at least 100 layers. For example, a neural network may have one or more convolutional layers and / or one or more densely connected layers (also called linear layers). Regardless of the layer type, the computation is performed by a neuron accumulator. s=Σ i w i x i +b where w i , 311 is the learned weight, x i , 310 is the input, and b is the bias value. For example, for a convolutional layer, i=(c,y,x) can be a tuple of iterators over the number of channels and the height and width of the input tensor 310.

[0065] In addition to the weights 312 and activation functions 313, the neural network values ​​may be subjected to various other operations, such as normalization, in particular batch normalization; scaling; and / or non-linear operations, which are applied, for example, to the output 314 and then used as input in the next neural network layer.

[0066] The example in Figure 3a shows a scalar computation graph, where edges 310, 314 may represent scalar values ​​and nodes 312, 313 may represent operations applied to the scalar values. However, in many other cases, a computation graph may be a tensor computation graph. In such cases, the nodes of the graph may represent operations applied to one or more input tensors, resulting in output tensors corresponding to the node's incoming and outgoing edges, respectively.

[0067] In general, a tensor X is a set of values ​​indexed by a set of m indices

number

[0068] A tensor computation graph may represent a sequence of operations performed on data stored in tensors. Tensors are a generalization of numerical matrices to three or more dimensions. In various embodiments, directed acyclic tensor computation graphs may be considered. In general, a tensor computation graph may represent an ordered set of operations performed on tensors. Operations may have inputs computed by other operations or may be inputs to an entire computer program. Operations may also include constants. For purposes of this specification, input to an operation refers to a non-constant input.

[0069] For example, the tensor computation graphs described herein may represent the evaluation of a neural network, but as is known per se, tensor computation graphs can also be used for other types of computation, for example in stream processing systems or databases.

[0070] In general, computation graphs can represent computations at various levels of abstraction. For example, an activation function may be represented in a computation graph as a single node in the graph, which may then be implemented in code as a sequence of basic tensor operations or as a sequence of more basic tensor computation graph operations such as "Add" and "Subtract." The provided techniques can in principle be applied to graphs at various levels of abstraction, as further discussed with respect to FIGS. 4 and 7. As a specific example, the provided techniques can be applied to graphs in the ONNX (Open Neural Network Exchange) format, which is a format for tensor computation graphs that use a given set of operators.

[0071] For example, Figures 3b-3d show tensor computation graphs representing computer programs that may be used as part of neural network evaluation. These figures provide visual representations of tensor computation graphs specified in the ONNX format. As shown in these figures, the nodes of a tensor computation graph may represent basic operations such as addition (Add) and subtraction (Sub); division (Div); multiplication (Mul); convolution (Conv); or arithmetic operations such as matrix multiplication (MatMul). In a computation graph, non-constant inputs of an operation are represented as incident edges, and constant inputs are built into the operation nodes themselves.

[0072] In particular, Figures 3b and 3c show computation graphs representing quantized computations. Quantized computations may be used in neural network evaluation. Neural networks typically represent values ​​such as inputs, activations, and / or weights as floating-point values. However, in various cases, neural network operations may be applied to integer values. This may be the case, for example, for computational performance reasons or to evaluate neural networks in encrypted form.

[0073] Quantization calculations may convert floating-point values ​​to integers. Floating-point numbers may have fine resolution for representing real numbers with a limited number of bits. Furthermore, the resolution of a floating-point number may be inversely proportional to the absolute value of the number. When quantizing neural networks to use integers to improve computational performance, the number of bits of these integers may be low, for example, up to 12 bits, up to 8 bits, or up to 6 bits. Various quantization techniques exist that aim to perform quantization with minimal quantization error.

[0074] For example, a subgraph of the computation graph may represent uniform quantization, in which discrete values ​​may be assigned at equal intervals on the real axis. This is illustrated by Figure 3b, where uniform quantization is applied element-wise to an input tensor 320, resulting in an output tensor 321, which is input to a matrix multiplication operation 322, e.g., of a linear layer of a neural network. In particular, the computations performed in the illustrated computation may be expressed mathematically as follows:

number

number

number

[0075] The illustrated uniform quantization can be undone by performing an inverse quantization calculation (not shown), thereby obtaining quantized floating-point values ​​that approximate the initial raw values. Mathematically, inverse uniform quantization can be expressed as follows: x f’ =(x q -zp x)*s f

[0076] Figure 3c shows the computation graph of a non-uniform quantization computation. In non-uniform quantization, the intervals that map to the integers do not all have the same size. Mathematically, non-uniform quantization is the computation

number

number

number

[0077] In the illustrated example, non-uniform quantization is applied to a tensor 330 of n=2-bit integers. In this example, the output of the quantization 331 is input to a Convolution (Conv) operation 332, which corresponds, for example, to a convolution layer of a neural network. i The value corresponds to the value at the subnode in the graph.

[0078] Figure 3d shows the computation graph of the normalization computation. This example shows batch normalization in the 2D case. Mathematically, this computation can be expressed as:

number

[0079] In this example, batch normalization is performed on the output of a Matrix Multiplication (MatMul) layer 341. The Div operation 343 in this example represents the division of the centered input x-μ by the standard deviation 342, where σ = Var 0.5 In this example, standard deviation 342 is shown as an edge representing a variable input to division 342, but more typically it may be a constant that is built into division 342 and therefore may not be represented as an edge in the computation graph.

[0080] Instead of or in addition to batch normalization, the computer programs described herein may also include tensor normalization (e.g., using a single μ and σ per tensor), layer normalization, scaling to 0-1, etc.

[0081] 4 shows a detailed, but non-limiting, example of optimizing a computer program. In this example, the optimization involves implementation operations (also called permutation operations) that transform a high-level computational graph into a low-level computational graph. In this example, it is determined whether a node in the high-level graph can be implemented by a table lookup; the implementation operations are based on this determination; and the nodes in the low-level graph are fused into a single table lookup operation.

[0082] In particular, this figure shows a computation graph CG1, 411, representing a computer program to be optimized, e.g., as discussed with respect to Figures 3a-3d. Each node of the computation graph represents a respective operation. In particular, the computation graph CG1 may be a tensor computation graph, where each operation is applied to one or more input tensors and results in an output tensor. For example, the computation graph CG1 may be represented in ONNX format, e.g., as a directed acyclic graph of tensor operations from the ONNX reference (available at https: / / github.com / onnx / onnx / blob / 55d3b80f3ea1261f3f74f2e35844c204de85835a / docs / Operators.md, which is incorporated herein by reference).

[0083] A variety of operations may be supported through optimization. In particular, the set of global operations may include at least an addition operation, a multiplication by a constant, and a multiplication by two non-constants. The set of global operations may also include at least one non-linear element-wise operation, such as application of an activation function or a numerical comparison. In general, several sets of operations may be distinguished that may occur in the tensor computation graph CG1. Nodes representing operations from the first set may never be implemented by a table lookup operation; nodes representing operations from the second set may be implemented by a table lookup operation depending on the current computation graph; and nodes representing operations from the third set may always be implemented by a table lookup operation. In particular, based on the type of operation and the input of the operation, a node may be identified as being of a particular type. Note that, as discussed in more detail below, a node representing a particular operation (e.g., addition) may be of different types depending on its context in the graph, e.g., depending on its input. In particular, the following types may be defined:

[0084] The first type of operation is a mixing operation (MIX). A mixing operation may be defined as an operation that modifies the values ​​of its input tensors but is not applied element-wise to its inputs. In particular, a mixing operation may combine different values ​​from a single input tensor of the operation, optionally by including an additional constant tensor. For example, the operation may be a convolution or a neuron accumulator operation (e.g., used to implement a densely coupled neural network layer), such as matrix multiplication with a constant tensor. Alternatively, a mixing operation may combine values ​​from two different graph input tensors, or values ​​from tensors produced by different MIX operations. For example, the operation may be the addition or concatenation of two graph inputs, or may be an arithmetic or concatenation operation applied to a tensor that is the output of two MIX operations. In various embodiments, a node determined to represent a mixing operation is not implemented by a table lookup operation (although table lookup operations may be used to pre- and post-process the inputs and outputs of the operation, for example, by quantization). For example, a given set of operations, such as convolution and dot product, may be determined to be of type MIX without regard to other nodes in the computation graph and therefore cannot be implemented by table lookup.

[0085] The second type of operation is a univariate operation (UNI). A univariate function may be applied element-wise to its input tensor and return an output tensor of the same size (but possibly with a different value type). For example, the operation may be an activation, normalization, quantization, or an arithmetic operation with a scalar constant. Note that a univariate computation applied to a tensor is univariate with respect to the tensor cells (scalars). In particular, different functions may be applied to different cells within a tensor. A node determined to be a univariate operation may be implemented by a table lookup operation. In particular, as shown below, a node may be determined to be of type UNI despite having multiple incident edges in the computation graph.

[0086] Interestingly, there may be a set of operations for which a node can be determined to be a MIX or UNI type operation, depending at least on the input. This set of operations may include, for example, addition (Add) and / or multiplication (Mul). For example, when the input type of such an operation is UNI, the operation itself may also be determined to be UNI depending on the operation. For example, an Add operation may be UNI if one of the inputs is produced by a univariate operation and the other input is a constant, or if the inputs are produced by a UNI operation acting on the same single input, etc. For example, an Add operation may be MIX if the inputs are produced by a UNI operation on different inputs. As a result, operations from this set may or may not be implemented by table lookup, depending on other nodes in the graph. In particular, an operation may be determined to be a MIX or UNI operation depending on the preceding operation; for example, addition x+1 may be determined to be a UNI operation if the preceding operation is a UNI operation, e.g., sigmoid(x), and may be determined to be a MIX operation if the preceding operation is a MIX operation, e.g., conv(x,w). In some embodiments, for simplicity, it is determined that addition of constants is always a UNI operation.

[0087] For various univariate operations, it may be the case that a native implementation is not available in the execution environment and they are always determined to be of UNI type and implemented accordingly by a lookup table. For example, this may be the case for non-linear functions such as activation functions or sine functions, depending on the execution environment.

[0088] A third type of operation is a reshape operation (SHAPE). Such an operation may rearrange the elements of one or more input tensors, for example, by reordering the values ​​or changing tensor metadata such as shape. The values ​​in the tensor may remain unchanged. Examples of such operations include transpose, reshape, flatten, etc. Such operations are also typically not implemented using lookup tables because they cannot affect the values.

[0089] An identification operation Ident, 420, may be applied to the computation graph CG1 to determine, for each node in the computation graph CG1, whether the respective node can be implemented by a table lookup operation, particularly by determining whether the node is of type UNI as described above. This may result in an updated computation graph CG2, 412, in which each node is flagged as either implementable by table lookup or not. Such flags are also called execution modes or CanFuse flags. Nodes that are implementable by table lookup may be said to have a TLU (table lookup) execution mode.

[0090] Generally speaking, the identification Ident may process the tensor operation graph CG1 and determine the execution mode for each of its operations by: (a) iterating over the operation graph CG1, which can generally be done in any order, but as many times as necessary; (b) evaluating certain compatibility conditions for each operation to determine whether a TLU execution mode can be used for the operation; and (c) assigning an execution mode to each operation of the graph, which is indicated in the figure by marking some nodes of the identification result CG2 with diagonal stripes, indicating that they have a TLU execution mode.

[0091] Interestingly, identification may include that for nodes of graph CG1 that have multiple inputs, it may be implemented by a table lookup operation. That is, multiple inputs may be determined to have a common ancestor, from which the output of the node may be calculated as an element-wise function. This allows this node to be merged with preceding and possibly succeeding nodes into a single table lookup, leading to an overall more efficient execution in the execution environment.

[0092] As a detailed example, an example algorithm for determining which operations can be fused is now provided. That is, for each operation at index i in the operation graph, the algorithm may set CanFuse[i] to the corresponding execution mode: TLU (e.g., a UNI-type operation) or as a primitive operation (e.g., a SHAPE or MIX-type operation). As shown, for one or more nodes, the nodes may be determined to be of MIX or SHAPE type (e.g., a convolution operation or a flattening operation) regardless of their position in the graph, or in other words, may have a preset operation type. For one or more other nodes, the operation type (or, in this example, the resulting CanFuse flag) may be determined based on their position in the graph.

[0093] [Table 1]

[0094] Following the identification operation Ident, an implementation operation Impl, 430, may be applied to the computation graph CG2. The implementation operation may replace a node of the computation graph CG2 by a first subgraph representing a first implementation if the node is flagged as implementable by a table lookup operation; otherwise, it may replace the node by a second subgraph representing a second, different implementation, resulting in an updated computation graph CG3. In that sense, the high-level computation graph CG2 may be transformed into a lower-level computation graph CG3.

[0095] In particular, the high-level computation graph CG2 may include one or more operations that are not directly executable by the execution environment; for example, operations on floating-point values. The implementation Impl may transform the high-level computation graph into a lower-level computation graph CG3 that may include only basic operations, e.g., operations that are directly executable by the execution environment; for example, such basic operations may be lookup table matching operations (on integers and / or floating-point numbers) and primitive operations on integers (however, such operations may still be implemented by multiple primitive operations of the execution environment; for example, matrix multiplication, convolution, etc. may be performed by performing multiple multiplications of integers).

[0096] In general, each available implementation of an operation may be a subprogram represented as a subgraph that can permute the nodes of the operation. At least one of these subprograms may be used to construct a TLU. An implementation may, for example, represent a sequence of elementary operations (such as addition and subtraction).

[0097] In particular, the first implementation may be a floating-point subprogram. Such a floating-point subprogram may be represented by a single node with floating-point input and output tensors. Algorithmically, the first implementation may be represented as follows:

[0098] [Table 2]

[0099] The second implementation may represent a sequence of floating-point and / or integer subprograms. In particular, the implementation may be configured to quantize inputs to integers (if necessary); apply integer operations corresponding to the nodes to obtain integer outputs; and dequantize the integer outputs (if necessary). For example, the arithmetic operations

number

[0100] [Table 3]

[0101] More generally, the operations of computation graph CG2 can be implemented in the transformed computation graph CG3 as follows:

[0102] [Table 4]

[0103] In particular, the second implementation may perform integer arithmetic corresponding to the operation to be implemented, e.g., matrix multiplication may be performed on the quantized values, as discussed in B. Jacob et al., “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” doi:10.1109 / CVPR.2018.00286 (incorporated herein by reference).

[0104] In particular, as shown in this figure, in the transformed computation graph CG3, nodes identified as fusible (Ident) may be replaced by corresponding floating-point operation nodes (illustrated as the same as the original nodes in the computation graph CG2), while nodes identified as not fusible (Ident) may be replaced by corresponding subgraphs of their integer implementations (illustrated as the resulting striped nodes).

[0105] After the implementation operation Impl, a fusion decision operation DFuse, 450 may be applied, which determines, for nodes implementable by a table lookup operation, that one or more further nodes of the computation graph CG3 can be fused to this table lookup operation. This is shown in the figure as computation graph CG4, 414, where it is determined that node 415 can be fused to a single table lookup operation. Thus, the figure shows an explicit identification step Ident, where nodes are identified as fusible or not fusible; and a fusion decision step DFuse, where such nodes are grouped into a subset of nodes to be fused, e.g., set 415.

[0106] In particular, the fusion decision step DFuse may identify connected components of the computation graph CG3 that share the common property of being implementable by a table lookup operation, e.g., can be described as the application of a univariate function to the output of a preceding node. Interestingly, at this stage, it may be sufficient to identify connected components that satisfy this property. At this point, it may no longer be necessary to check whether they have a common ancestor.

[0107] As shown in this figure, it is possible that the fusion decision step DFuse includes nodes in the fusion step that do not correspond to fusible operations of the computation graph CG2. In particular, they may result from the implementation of non-fusible nodes. For example, as shown in this figure, quantization operations resulting from subsequent non-fusible nodes of the computation graph CG2 may be fused with fusible operations of the computation graph CG2 into a single lookup table.

[0108] Interestingly, the fusion decision step DFuse can be performed in such a way that no table lookup operation is applied to the output of a further table lookup operation, in which respect the number of table lookups performed can be minimized.

[0109] When it is determined that nodes can be fused, the output write operation Write, 460 can be performed, and a transformed representation of a computer program in which node 415 is fused into a single table lookup operation can be output. The transformed representation can take various forms. This can be explicitly stored as a computational graph, but can also be stored as a set of instructions INSTR, 415 that implicitly define the computational graph for the target execution environment. The set of instructions can be in a form of instructions portable across multiple machine architectures or in a form of code executable on a specific machine architecture. As is known per se, the write operation Write can include constructing a lookup table by applying a univariate function represented by the nodes to be fused to a set of possible (e.g., integer) inputs to obtain the corresponding (e.g., integer) outputs. Such a write operation is known, for example, from Zama's Concrete-Numpy available at https: / / github.com / zama-ai / concrete-numpy.

[0110] The computational graph explicitly or implicitly defined by the write operation Write can be in a specific canonical form. The computational graph can be an integer computational graph and can utilize table lookup operations. The table lookup (TLU) operation is applied element-wise to a single input tensor and can be defined as follows:

Number

number

number

[0111] The canonical form is sometimes called the Form TLU-Shape canonical form. This canonical form may refer to an integer (typically tensor) computational operation graph that has the following properties: 1. The operation graph contains only MIX, SHAPE, and TLU type operations. 2. TLU operations have integer inputs and outputs, e.g.

number

[0112] In particular, this diagram shows a write operation, Write, 460, producing a set of instructions, INSTR, 415, for a target execution environment based on the computation graph, CG4. In this set of instructions, INSTR, the nodes 415 are fused into a single table lookup operation; in other words, a single table lookup operation is used to implement the set of nodes. The set of instructions can take various forms, such as assembly; a set of instructions executable by a homomorphic computation engine; a circuit layout for a hardware circuit; programming language code for a compiled or interpreted programming language; etc.

[0113] Optionally, the optimization may include an ordering operation Ord, 440. As shown in this figure, the ordering operation Ord may be applied to a low-level computation graph CG3 obtained from the implementation operation Impl. It is also possible to apply the operation to the high-level computation graph CG2 before performing the implementation operation Impl (but typically after identifying Ident which nodes can be implemented by table lookup operations); however, applying the ordering operation to the low-level computation graph has the advantage of being able to identify more cases in which reordering can be applied.

[0114] An ordering operation Ord may operate on a reshape (SHAPE) node. Such a node may represent an operation that rearranges the elements of one or more input tensors without changing their values, such as a transpose of a flatten. The ordering operation may rearrange a computation graph, e.g., graph CG3, to fuse at least one operation preceding the reshape node with at least one operation following the reshape node. Interestingly, such a reorganization may not affect the result of the computation represented by the computation graph CG3. However, the reorganization may have the effect that the preceding and succeeding nodes in the graph are directly connected to each other, and therefore the fusion decision step DFuse may determine that these nodes can be fused into a single table lookup operation (possibly involving further nodes).

[0115] In particular, after the ordering operation Ord, the computation graph CG3 may have the property that no reshaping operation is applied to the output of the table lookup operation. This may be achieved by moving the reshaping operation after the table lookup operation. In particular, a sequence of nodes in the graph that includes one or more sequences of a mixing operation, then a table lookup, then a reshaping, then optionally a table lookup, may be replaced by a mixing operation, a respective reshaping, and a respective table lookup. The respective table lookups may then be fused into a single table lookup operation (possibly together with additional subsequent table lookups) by a fusion decision step DFuse.

[0116] In particular, the ordering operation Ord may identify the pattern MIX-(TLU-SHAPE)+{-TLU}, where + indicates at least one repetition and {x} indicates an optional operation. This operation may replace such a pattern with a new pattern of type SHAPE+-TLU+{-TLU}, allowing adjacent TLU operations to be fused by the DFuse operation.

[0117] In particular, the computation graph explicitly or implicitly defined by a write operation Write may be in a canonical form, referred to herein as the TLU-only canonical form. A computation graph in this canonical form may be an integer (typically tensor) computation graph that has the following properties in addition to the properties of the TLU shape described elsewhere: 1. There are no TLU operations between a MIX operation and a SHAPE operation, or between two SHAPE operations; 2. TLU operations cannot perform identity mapping between inputs and outputs. An example implementation of the ordering operation Ord is as follows:

[0118] [Table 5]

[0119] 5a-5c show detailed examples of transformed computation graphs. The illustrated transformed computation graphs may correspond to the output of the optimizations described herein.

[0120] These transformed computational graphs show the lookup tables used to implement quantization (in FIGS. 5a and 5b) and normalization (in FIG. 5b). Quantization and normalization, but also application of activation functions, are examples of operations in neural networks that operate on individual activations, e.g., individual neuron outputs. These functions are therefore univariate with respect to activations and can be implemented accordingly by lookup tables. FIGS. 5a-5c show computational graphs determined according to the techniques described herein, where these operations are implemented by a single lookup table. In particular, the subgraphs are in TLU-SHAPE canonical form, as also discussed with respect to FIG. 4.

[0121] In particular, Figure 5a shows a transformed representation of quantization, in this case uniform quantization. This output can be obtained by applying the techniques provided herein to the computation graph of Figure 3b. This diagram shows that the portion of the computation graph of Figure 3b that computes edge 321 from edge 320 is implemented as a single lookup table in Figure 5a. Note that the graph of Figure 3b includes several nodes with multiple inputs, in particular two "Where" nodes; interestingly, these nodes are nevertheless implemented by lookup table 523 in Figure 5a, since they can be determined to have a common ancestor (in particular, a previous "Add" or "Where" node).

[0122] Similarly, Figure 5b shows a transformed representation of non-uniform quantization resulting from applying the provided techniques to the computation graph of Figure 3c. In this example, the portion of the computation graph of Figure 3c that computes edge 331 from edge 330 is implemented as a lookup table 533. Again, various nodes that themselves have multiple inputs are included in the lookup table; this is determined to be possible because the nodes share a common ancestor, i.e., they are all computable as univariate functions of the output 330 of the multiplication operation.

[0123] Finally, Fig. 5c shows a transformed representation of a normalization operation, in this case a batch normalization operation, that can be obtained by applying the provided techniques to the computation graph of Fig. 3d (edge ​​342 is incorporated as a constant input into Div node 343, as also discussed with respect to Fig. 3d). In this case, the batch normalization of Fig. 3d applied to the output of matrix multiplication 341, resulting in output edge 344, is implemented as a single lookup table 545.

[0124] 6a and 6b show a detailed example of reordering a computation graph. This example illustrates how a subgraph of a computation graph can be reordered to facilitate the merging of table lookup operations. In particular, reordering can be applied to a reshaping node. Such a node can rearrange the elements of an input tensor without changing their values. If a table lookup operation precedes and follows such a node (or a sequence of multiple such nodes), by moving the reshaping node, the table lookup nodes can become directly connected in the subgraph and can be merged (e.g., in a subsequent operation) into a single table lookup operation.

[0125] 6a, 6b show an input computation graph and an output computation graph to which a "Merge", eg, a reordering followed by fusion, has been applied.

[0126] In particular, FIG. 6a illustrates a computation graph representing the respective application of a convolution CONV, 601; a table lookup TLU, 602; a reshape RESHAPE, 603; a further table lookup TLU, 604; and a matrix multiplication MATMUL, 605 to a tensor. For simplicity, the further operands of the convolution 601 and matrix multiplication 605 are not shown in this example. In this example, the table lookups 602, 604 precede and follow the reshape operation 603, respectively. Reordering may include moving the reshape operation 603 before the table lookups 602, 604. Adjacent table lookups may then be merged into a single table lookup 609. It is also possible to move the reshape operations after the table lookups 602, 604, thereby obtaining a computation graph in which the single table lookup 609 precedes the reshape 603. In either case, by iteratively moving the reshaping before or after the table lookups, a sequence of table lookups to be merged can be obtained. Operations 601, 605 that are not applied element-wise to the input and do not represent reshaping cannot be affected by the transformation.

[0127] Similarly, Figure 6b shows a computation graph in which a sequence of operations is applied to a tensor: a convolution CONV, 611; a first table lookup TLU, 612; a reshape RESHAPE, 613; a second table lookup TLU, 614; a reshape operation in the form of a transpose TRANSPOSE, 615; a third table lookup TLU, 616; and a matrix multiplication MATMUL, 617 (for simplicity, further operands of operations 611, 617 are not shown). In this case, the described transformation may be applied twice: once to swap the order of table lookup 612 and reshape 613; and once to swap the order of transpose and table lookups 612, 614 (which were adjacent due to the previous swap). After these swaps, table lookups 612, 614, 616 are adjacent and can therefore be fused into a single table lookup 619. As above, it is also possible to swap the transpose operation 615 with the table lookup 616 and the reshape operation 613 with the table loop 614, 616, thereby obtaining a transformed graph in which the fused table lookup 619 comes before the reshape 613 and the transpose 615. The non-element-wise and / or non-reshape operations 611, 617 remain unchanged.

[0128] It is noted that in both example subgraphs resulting from the proposed reordering, there is no reshaping operation applied to the output of the table lookup operation. Otherwise, the reshaping operation and the table lookup may be swapped. In particular, the resulting subgraphs are in TLU-Only canonical form.

[0129] 7 shows a detailed, but non-limiting, example of computer program optimization. This figure does not show the implementation operations that transform a high-level computation graph, such as that of FIG. 4, into a lower-level computation graph; for example, such operations may have been previously applied in the techniques described with respect to this figure. The identification, Ident, 420, of whether each node can be implemented by a table lookup operation, followed by the determination, DFuse, 450, that the nodes can be fused, may be performed on the computation graph without any intervening implementation operations.

[0130] In particular, shown in this figure is a computation graph CG1, 711. This computation graph may be a low-level computation graph, meaning that each node may represent a respective operation that translates directly to implementation in the execution environment. In particular, computation graph CG1 may include low-level operations that implement quantization and / or cast operations as needed. For example, if the execution environment supports integer and table lookup operations, e.g., along with homomorphic encryption, the computation graph may include nodes representing integer operations supported by the execution environment (e.g., integer addition and subtraction, scalar multiplication, matrix multiplication by a constant matrix, convolution by a constant matrix, etc.) and nodes representing operations that can be implemented by table lookup operations (e.g., floating-point operations such as activation functions, quantization, or portions thereof). The low-level computation graph CG1, sometimes referred to as a basic op graph, may be the operation graph of Zama's Concrete-Numpy, available at https: / / github.com / zama-ai / concrete-numpy.

[0131] In particular, as also discussed with respect to FIG. 4, the computation graph CG1 may include nodes of several different types. In particular, as also discussed with respect to FIG. 4, the computation graph may include nodes of mixed type (MIX); univariate function type (UNI); reshaping type (SHAPE); and / or table lookup operation type (TLU). As also discussed with respect to FIG. 4, for a given set of operations, their types may depend on other nodes in the graph, and accordingly, the operations may or may not be implemented as table lookups. This set of operations may include at least addition and multiplication. Another set of operations may or may not always be implemented as table lookups.

[0132] An identification operation Ident, 720, is shown that can determine, for each node in computation graph CG1, whether the respective node can be implemented by a table lookup operation, e.g., whether the node is of mixed type MIX. The identification operation Ident can be configured to determine that a node with multiple inputs can be implemented by a table lookup operation based on determining that the multiple inputs have a common ancestor node, in particular such that the node's output can be written as an element-wise function of the outputs of the common ancestor node. In particular, the figure shows computation graph CG2 obtained by annotating the diagonally striped nodes in computation graph CG1 as being implementable by a table lookup operation based on determining that their multiple inputs have a common ancestor, in the form of horizontally striped nodes. Note that whether this is true for nodes with multiple inputs depends on the structure of the graph. Accordingly, the identification operation Ident can determine whether a node can be implemented as a lookup table not only based on the node itself but also on other parts of the subgraph.

[0133] Following the identification operation Ident, a fusion determination operation DFuse, 750, can be used to determine that for a node determined to be implementable by a table lookup operation, one or more further nodes of the computation graph can be fused to the table lookup operation. This is illustrated in the figure by computation graph CG3, which has been annotated with fusible subgraphs. A fusible subgraph in this case is a subgraph that starts with an outgoing edge of a common ancestor node and ends with the node itself.

[0134] In this case, as shown in this figure, the identified subgraph may be replaced in the computation graph CG3 by a node representing a single table lookup operation, which may be a TLU type node, as also discussed in Figure 4. After this, the optimization may iteratively apply the identification Ident and the fusion decision DFuse, for example, until no more nodes can be fused.

[0135] After fusing the nodes, a write operation Write may be performed, as also discussed with respect to Figure 4. As discussed, the write operation Write may output a transformed representation of the computer program in which the nodes are fused and implemented by a single table lookup, as known per se.

[0136] This figure also shows an ordering operation Ord that can reorder the computation graph CG2 to fuse at least one operation preceding the reshape node with at least one operation following the reshape node, as also discussed with respect to Figure 4. This operation can be implemented as discussed with respect to Figure 4.

[0137] In particular, the following algorithm provides a detailed example of how to implement the identification operation Ident and the fusion decision operation DFuse. This example uses a generalized pattern detection technique to find nodes that can be implemented as table lookup operations and then fused. In particular, in the described algorithm, UnwantedProperty can be configured to determine whether a given operation is a MIX or SHAPE operation. ConformsToFusingRules can be configured to determine whether there are multiple TLU operations in a subgraph; whether operations in a subgraph produce tensors with shapes corresponding to the subgraph input tensors; and / or whether operations in a subgraph use constants with the same shape as the input tensors.

[0138] In particular, in the following algorithm, the identification operation Ident may be implemented by the functions FindClosestAncestorsWithoutUnwantedProperty and FindLowestFusingPointCandidate, and the fusion decision operation DFuse may be represented by the function CreateFusableSubgraphFromTo. Accordingly, if an operation in the operation graph belongs to a subgraph found by the algorithm FindSubgraphToFuse, it may be implemented in TLU mode, otherwise it may be implemented in native mode.

[0139] [Table 6] TIFF0007811304000021.tif238170TIFF0007811304000022.tif245167TIFF0007811304000023.tif197168

[0140] 8 shows a block diagram of a computer-implemented method 800 for optimizing a computer program for an execution environment that supports table lookup operations. Method 800 may correspond to the operations of system 100 of FIG. 1. However, this is not a limitation, in that method 800 may be performed using another system, apparatus, or device.

[0141] The method 800 may include accessing 810 a representation of the computer program as a computation graph in an operation titled "ACCESS COMPUTATION GRAPH." In the computation graph, each node may represent a respective operation. The method 800 may include, for each node of the computation graph, determining 820 whether the respective node can be implemented by a table lookup operation in an operation titled "DETERMINE TLU IMPLEMENTATION." The method 800 may include, for a node that can be implemented by a table lookup operation, determining 830 that one or more additional nodes of the computation graph can be fused to the table lookup operation in an operation titled "DETERMINE FUSING." The method 800 may include outputting 840 a transformed representation of the computer program in an operation titled "OUTPUT." In the transformed representation, the node and one or more additional nodes may be fused into a single table lookup operation.

[0142] In particular, method 800 may be a compiler method. In particular, method 800 may receive as input source code in a compiled or interpreted programming language (e.g., Python, C++, etc.), compile the source code to determine a computation graph, and output a transformed representation in a representation executable by an execution environment, e.g., a homomorphic executable file for an FHE computation engine.

[0143] In general, it should be understood that the operations of method 800 of FIG. 8 may be performed in any suitable order, e.g., sequentially, simultaneously, or a combination thereof, where applicable, as dictated by, for example, input / output relationships, necessitating a particular order.

[0144] This method may be implemented on a computer as a computer-implemented method, as dedicated hardware, or as a combination of both. As also shown in FIG. 9 , computer-related instructions, e.g., executable code, may be stored on a computer-readable medium 900, e.g., in the form of a series of machine-readable physical marks 910 and / or as a series of elements having different electrical, e.g., magnetic, or optical, properties or values. The medium 900 may be transitory or non-transitory. Examples of computer-readable media include memory devices, optical storage devices, integrated circuits, servers, online software, etc. FIG. 9 illustrates an optical disk 1100. Alternatively, the computer-readable medium 900 may include data 910 representing a transformed representation of a computer program determined according to the methods described herein.

[0145] The following numbered clauses include non-limiting examples of what is contemplated:

[0146] Clause 1. A computer-implemented method for optimizing a computer program for an execution environment that supports table lookup operations, comprising: accessing a representation of a computer program as a computation graph, each node of the computation graph representing a respective operation; For each node of the computation graph, determining whether the respective node can be implemented by a table lookup operation; For a node that can be implemented by a table lookup operation, determining that one or more further nodes of the computation graph can be fused to the table lookup operation; outputting a transformed representation of the computer program, wherein the node and one or more further nodes are fused into a single table lookup operation; A computer-implemented method comprising:

[0147] Clause 2. The method of clause 1, wherein the execution environment is a homomorphically encrypted computation that supports at least an encrypted addition operation, an encrypted scalar multiplication operation, and an encrypted table lookup operation.

[0148] Clause 3. The method of clause 1, wherein the table lookup operation is a hardware lookup table.

[0149] Clause 4. The method of any of clauses 1 to 3, wherein one or more operations from a first set of operations are never implemented by a table lookup operation, one or more operations from a second set of operations are implemented by a table lookup operation according to the computation graph, and one or more operations from a third set of operations are always implemented by a table lookup operation.

[0150] Clause 5. The method of clause 4, wherein the second set of types includes at least an addition operation.

[0151] Clause 6. The method of clause 4 or 5, comprising, if it is determined that the node can be implemented by a table lookup operation, replacing the node by a first subgraph representing the first implementation; otherwise, replacing the node by a second subgraph representing the second implementation.

[0152] Clause 7. The method of clause 6, wherein the second implementation is configured to quantize the input to an integer; apply an integer operation corresponding to the node to obtain an integer output; and dequantize the integer output.

[0153] Clause 8. The method of any of clauses 1 to 7, comprising determining that a node having multiple inputs can be implemented by a table lookup operation based on determining that the multiple inputs have a common ancestor.

[0154] Clause 9. The method of any of clauses 1 to 8, wherein the operation is applied to one or more input tensors, resulting in an output tensor, and the table lookup operation is applied element-wise to a single input tensor.

[0155] Clause 10. The method of clause 9, comprising obtaining a reshape node representing an operation that rearranges elements of one or more input tensors without changing their values, and reordering the computation graph to fuse at least one operation preceding the reshape node with at least one operation following the reshape node.

[0156] Clause 11. A method according to any of clauses 1 to 10, wherein the transformed representation of the computer program does not include a table look-up operation applied to the output of a further table look-up operation.

[0157] Clause 12. The method of clause 11, wherein the transformed representation of the computer program does not include a reshaping operation applied to the output of a table lookup operation, the reshaping operation rearranging one or more inputs without changing their values.

[0158] Clause 13. The method of any of clauses 1 to 12, wherein the computer program represents the evaluation of a machine-learnable model, such as an artificial neural network, a generalized linear model, a decision tree, or an ensemble model.

[0159] Clause 14. A compiler system (100) for optimizing a computer program for an execution environment that supports table lookup operations, comprising: a data interface (120) for accessing a representation of a computer program as a computation graph, each node of the computation graph representing a respective operation; A processor subsystem (140), the processor subsystem (140) comprising: For each node in the computation graph, determine whether the respective node can be implemented by a table lookup operation; For a node that can be implemented by a table lookup operation, determining that one or more further nodes of the computation graph can be fused to the table lookup operation; outputting a transformed representation of the computer program in which the node and one or more further nodes are fused into a single table lookup operation; A compiler system (100) configured as follows.

[0160] Clause 15. A temporary or non-transitory computer-readable medium (900), instructions which, when executed by a processor system, cause the processor system to perform the computer-implemented method of any one of clauses 1 to 13; and / or a transformed representation of a computer program determined in accordance with the computer-implemented method of any one of clauses 1 to 13. A transitory or non-transitory computer-readable medium (900) containing data (910) representing:

[0161] Whether or not indicated as non-limiting, no examples, embodiments, or optional features should be understood to limit the claimed invention.

[0162] The above-described embodiments illustrate the present invention without limiting it, and it should be noted that those skilled in the art can design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the scope of the claim. The use of the verb "comprise" and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The use of phrases such as "at least one of," when following a list or group of elements, refers to the selection of all elements from the list or group, or any subset of elements. For example, the phrase "at least one of A, B, and C" should be understood to include A only, B only, C only, both A and B, both A and C, both B and C, or all of A, B, and C. The present invention can be implemented by means of hardware comprising several distinct elements and by means of a suitably programmed computer. In a device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. 1. A computer-implemented method (800) for optimizing a computer program for an execution environment that supports table lookup operations, comprising: accessing 810 a representation of a computer program as a computation graph, each node of the computation graph representing a respective operation; determining 820, for each node of the computation graph, whether the respective node can be implemented by a table lookup operation, including determining that a node having multiple inputs can be implemented by a table lookup operation based on determining that the multiple inputs have a common ancestor; if it is determined that the node can be implemented by the table lookup operation, replacing the node by a first subgraph representing the first implementation, otherwise replacing the node by a second subgraph representing the second implementation; For a node that can be implemented by a table lookup operation, determining (830) that one or more further nodes of the computation graph can be fused to the table lookup operation, and determining that one or more further nodes from a second subgraph of another node can be fused to the table lookup operation; outputting (840) a transformed representation of the computer program, wherein the node and the one or more further nodes are fused into a single table lookup operation; The method (800) includes:

2. 10. The method (800) of claim 1, wherein the execution environment is a homomorphically encrypted computation that supports at least encrypted addition operations, encrypted scalar multiplication operations, and encrypted table lookup operations.

3. The method (800) of claim 1, wherein the table lookup operation is implemented with a hardware lookup table.

4. 2. The method (800) of claim 1, wherein one or more operations from a first set of operations are never implemented by a table lookup operation, one or more operations from a second set of operations are implemented by a table lookup operation according to the computation graph, and one or more operations from a third set of operations are always implemented by a table lookup operation.

5. The method (800) of claim 4, wherein the second set of types includes at least an addition operation.

6. 2. The method of claim 1, comprising iterating through the computation graph and determining, for each operation, whether a table lookup operation can be used for the operation.

7. 10. The method (800) of claim 1, wherein the second implementation is configured to quantize an input to an integer, apply an integer operation corresponding to the node to obtain an integer output, and dequantize the integer output.

8. 10. The method of claim 1, wherein an operation is applied to one or more input tensors resulting in an output tensor, and wherein a table lookup operation is applied element-wise to a single input tensor.

9. 10. The method of claim 1, comprising obtaining a reshape node representing an operation that rearranges elements of one or more input tensors without changing their values, and reordering the computation graph to fuse at least one operation preceding the reshape node with at least one operation following the reshape node.

10. 8. The method (800) of claim 1, wherein the transformed representation of the computer program does not include a table lookup operation applied to the output of a further table lookup operation.

11. 10. The method (800) of claim 1, wherein the transformed representation of the computer program does not include a reshaping operation applied to an output of a table lookup operation, the reshaping operation rearranging one or more inputs without changing their values.

12. The method (800) of claim 1 , wherein the computer program represents an evaluation of a machine-learnable model.

13. A compiler system (100) for optimizing a computer program for an execution environment that supports table lookup operations, comprising: a data interface (120) for accessing a representation of a computer program as a computation graph, each node of the computation graph representing a respective operation; and a processor subsystem (140), wherein the processor subsystem (140) for each node of the computation graph, determining whether the respective node can be implemented by a table lookup operation, and determining that a node having multiple inputs can be implemented by a table lookup operation based on determining that the multiple inputs have a common ancestor; if it is determined that the node can be implemented by the table lookup operation, replacing the node by a first subgraph representing the first implementation, otherwise replacing the node by a second subgraph representing the second implementation; For a node that can be implemented by a table lookup operation, determining that one or more further nodes of the computation graph can be fused to the table lookup operation, and determining that one or more further nodes from a second subgraph of another node can be fused to the table lookup operation; outputting a transformed representation of the computer program in which the node and one or more further nodes are fused into a single table lookup operation. A compiler system (100) configured as follows.

14. Instructions which, when executed by a processor system, cause the processor system to perform the computer-implemented method of any one of claims 1 to 12. A non-transitory computer-readable medium (900) comprising:

Images