Optimizing computer programs for table lookup operations

Abstract

The invention relates to a computer-implemented method (800) of a computer program for optimizing an execution environment for supporting table lookup operations. The computer program is represented as a computational graph, wherein respective nodes of the computational graph represent respective operations. For a respective node, determining whether the respective node may be implemented by a table lookup operation. For nodes that may be implemented by a table lookup operation, determining that one or more additional nodes of the computational graph may be fused into the table lookup operation. A transformed representation of the computer program is output, wherein the node and the one or more further nodes are fused into a single table lookup operation.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method for optimizing a computer program for an execution environment supporting table lookup operations, and a corresponding system. The present invention also relates to a computer-readable medium. Background Art

[0002] Homomorphic cryptography allows a party to execute a computer program (e.g., circuit evaluation) on encrypted data without the party being able to decrypt it. For example, input data and computational results can be received and returned in encrypted form. Intermediate data (e.g., the internal state of a computation) can also be in encrypted form. Even if the output of a computer program is returned in encrypted form, the output is expected to be the same or very close when decrypted, as if the operation was performed on unencrypted data. Homomorphic encryption can be used to outsource storage and computation to preserve privacy. This allows data to be encrypted and outsourced to a cloud environment for processing and / or storage while always being encrypted. Specifically, there are homomorphic cryptography techniques that can, at least in principle, be used to compute any function on encrypted data. This technique is called "fully homomorphic encryption" (FHE) technology.

[0003] Homomorphic encryption techniques typically provide an execution environment that supports a relatively limited set of operations. More complex operations can then be implemented in terms of this set of supported operations. Supported operations typically include addition and subtraction of integer ciphertexts; and multiplication of integer ciphertexts by known scalars. For example, homomorphic encryption techniques can also support multiplication of encrypted integer ciphertexts.

[0004] The execution environment provided by various homomorphic encryption technologies also supports table lookup operations. Specifically, the table lookup operation can be a programmable bootstrap (PBS) operation. Such an operation can not only implement a table lookup operation, but also restore the noise present in the ciphertext used to a fixed level. Homomorphic encryption technology that supports PBS operations is also called TFHE-like homomorphic encryption schemes. This scheme is described in "Programmable bootstrapping enables efficient homomorphic inference of deep neural networks" by I. Chillotti et al., Cyber ​​Security Cryptography and Machine Learning (CSCML 2021), Lecture Notes in Computer Science, Vol. 12716, pp. 1-19, Springer, 2021 (incorporated herein by reference). In general, TFHE-like homomorphic encryption schemes can be based on lattice encryption, such as LWE or NTRU encryption.

[0005] Although the computational efficiency of homomorphic encryption techniques has greatly improved over the past few years, there is still a significant overhead, especially when the technique is applied to relatively large amounts of data and / or relatively complex computations. In particular, the overhead of performing a table lookup is still significant compared to other operations such as homomorphic addition or scalar multiplication. Therefore, there is a need to optimize computer programs so that they can be efficiently executed by a homomorphic encryption execution environment that supports table lookup operations.

[0006] Table lookup operations are also used in various other execution environments. For example, in hardware, a lookup table can be used to calculate an arbitrary function. Such a hardware lookup table can be implemented, for example, by a piece of RAM (random access memory) or ROM (read only memory). Lookup tables can be used to accelerate calculations applied to particularly large amounts of data, such as in neural network accelerator circuits for training or evaluating neural networks. For example, the activation function of a neural network can be implemented as a lookup table. Also in this case, it is important to optimize the computer program to improve execution efficiency. Summary of the invention

[0007] It is desirable to optimize a computer program for an execution environment that supports table lookup operations, in particular by minimizing the number of table lookup operations required to execute the computer program in the execution environment.

[0008] According to a first aspect of the invention, there is provided a method of optimizing a computer implementation of a computer program and a corresponding system, as defined in claims 1 and 14, respectively. According to another aspect of the invention, there is provided a computer readable medium as defined in claim 15.

[0009] Various embodiments relate to the optimization of a computer program for an execution environment that supports table lookup operations. To this end, a computer program can be obtained as a representation of a computational graph. The corresponding nodes of the computational graph can represent corresponding operations. The computational graph is in many cases a tensor computational graph, in which the nodes represent operations applied to one or more input tensors and produce output tensors, corresponding to the edges of the graph; however, a computational graph that only represents scalars is also possible.

[0010] The representation may be transformed in such a way that multiple nodes of the computation graph are implemented as a single table lookup operation (where, in the case of a tensor computation graph, the table lookup operation may represent an element-by-element application of the table lookup). Jointly implementing multiple nodes as a single table lookup operation may be referred to as "fusing" the nodes (whether the fusion is performed explicitly by, for example, merging the nodes of the computation graph, or by outputting a set of instructions for an execution environment based on a determination that the nodes are to be implemented as a single table lookup operation). The optimization may output a transformed representation of the computer program to which the fusion has been applied.

[0011] Although it is known per se to manually annotate (single-input, single-output) operations of a computer program as fusible and then fuse these operations into a single table lookup operation, e.g., to fuse operations representing corresponding univariate functions that are applied consecutively to the same input, the inventors envision improving such fusion. To this end, a computational graph may be obtained. A computational graph may in principle represent any computation that can be performed by an execution environment. In view of the graph, it may be automatically determined for its corresponding node whether the node can be implemented by a table lookup operation. This determination may be based not only on the node itself, but also on at least the incoming edges of the node in the computational graph. In particular, the determination may be based on determining whether the output of a node can be expressed as a univariate (element-by-element) function of the output of another node, e.g., even if the node has multiple inputs.

[0012] For a node that has been determined to be implementable by a table lookup operation, it may be determined that one or more additional nodes in the computation graph may be fused into the table lookup operation; for example, nodes between other nodes and the node itself. These additional nodes may be fused with the node in the output transformed representation.

[0013] Specifically, a mechanism can be provided that can transform a computational graph (e.g., a computational graph of a general neural network) into a form that incorporates single-variable operations as much as possible, and calculates the resulting operations using a lookup table. For example, single-variable operations can include, but are not limited to, quantization, normalization, and activation functions.

[0014] In general, by determining in an automated fashion whether a node can be implemented via a table lookup operation, it is possible to find fusible nodes that might not be found using only manual annotations, e.g. based on the operation itself. For example, a node that represents an operation that applies (element-wise) to multiple (tensor) inputs may be implementable via a table lookup operation, depending on whether the inputs themselves depend (element-wise) on common previous inputs. Thus, whether a node is implementable via a table lookup may depend on the other nodes to which it is connected. Determining this manually is infeasible, especially for large computations. Furthermore, while certain heuristics are known to fuse nodes, such heuristics are inherently only effective in specific cases and may miss fusion opportunities.

[0015] Thus, by automatically determining whether a node is implementable via a table lookup operation and then fusing such a node with other nodes into a single table lookup operation, the efficiency of computer programs in a lookup table based execution environment may be improved, in particular by reducing the number of table lookup operations performed when executing the computer program (as illustrated elsewhere, this does not mean that all nodes determined to be implementable via a table lookup are actually implementable in that manner, e.g., scalar multiplications that are not preceded or followed by other nodes implementable via table lookups may be implementable natively, e.g., as native operations using homomorphic encryption).

[0016] This is particularly beneficial in several settings. One such setting is where the table lookup operation is a hardware lookup table, such as in a hardware accelerator. For example, a hardware accelerator with low bandwidth registers (e.g., low bandwidth registers of at most 16 bits or even at most 8 bits). Here, using a table lookup to compute a group of single variable operations can be faster than performing a series of arithmetic operations that compute the same result. Furthermore, the resulting transformed representation is faster than performing successive table lookups (once for each single variable operation).

[0017] Furthermore, when applied to tensor computation graphs that perform quantized neural network predictions, the provided technique can avoid the need to repeatedly requantize the inputs to each individual univariate operation if several such operations are fused. Thus, the approach eliminates the need to individually quantize univariate operations and can improve the accuracy of the final result since quantization incurs a loss in accuracy due to the approximations performed.

[0018] The presented techniques also allow the design of simpler hardware accelerators, such as dedicated neural network hardware accelerators for quantized neural networks, thereby avoiding the inclusion of more complex circuits for normalization, quantization, and activation functions.

[0019] Another setting in which the provided techniques are particularly beneficial is the setting of homomorphic encryption, such as fully homomorphic encryption (FHE). In this setting, encrypted values ​​can be stored as ciphertexts consisting of one or more numeric values, such as LWE or NTRU encryption. The size of such ciphertexts and the computational efficiency of cryptographic operations defined on them can depend on the bit width of the input to the table lookup operator, which means that it is beneficial to use lower bit widths (e.g., bit widths of up to 8 bits or up to 12 bits) for calculations.

[0020] In this setting, by applying the provided technique, a transformed representation of a computer program can be obtained, which is optimized for execution by homomorphic encryption as follows. First, the number of lookup table operations can be minimized. In fact, the computational cost of lookup table operations is relatively high, and having redundant LUTs increases the computational cost. In addition, homomorphic encryption usually operates on encrypted integer values, and therefore quantization is usually used. By avoiding encoding the quantizer, activation, and standard layers into the TLU separately, the correlated quantization errors caused by the quantization errors caused by each LUT alone are avoided. Moreover, in general, applying TLU operations can add small errors or noise to the ciphertext, which means that the result of the TLU may deviate from 1 with a small probability. Therefore, linking multiple TLUs increases the probability of errors. The provided technique can reduce the number of TLUs used, thereby also reducing the probability of errors.

[0021] The corresponding nodes of the computation graph may represent corresponding operations. The optimization may be performed in such a way that an operation in the first set of operations is never implemented by a table lookup operation. When the computation graph is defined on tensors, this may, for example, be the case for operations (e.g., convolutions) that are applied to one or more input tensors in a non-element-wise manner. Such operations may be implemented based on native operations of the execution environment. (In this case, even if the operation itself is not implemented by a table lookup in the functional sense of the operation in the computation graph, it may still be the case that pre-processing and / or post-processing (e.g., (de)quantization) uses table lookups. This is explained in more detail elsewhere in this specification.)

[0022] Nodes representing operations from the second group of operations may be implemented as lookup tables depending on the computation graph (e.g., depending on their incoming and / or outgoing edges). For example, for these nodes, the execution environment's native implementation of the operation may be available, but these nodes may still be fused into table lookups with other nodes if this can make the overall computation more efficient. This is the case, for example, for addition or scalar multiplication. Nodes representing operations from the third group of operations may always be implemented by table lookup operations, for example, because no native implementation of the operation is available in the execution environment, such as activation functions.

[0023] Optionally, the optimization may involve a substitution operation, wherein if it is determined that a node can be implemented by a table lookup operation, the node is replaced by a first subgraph representing a first implementation; and in other cases the node is replaced by a second subgraph representing a second implementation. The substitution operation may be performed after determining which nodes can be implemented by a table lookup operation but before determining that such nodes can be fused with other nodes. Specifically, the nodes to be fused may include one or more nodes from a second subgraph of another node, for example, a node corresponding to a dequantization operation of a previous node or a quantization node of a next node. In this way, the optimization may be combined with a transformation that converts a high-level computational graph into a low-level computational graph, as discussed in more detail elsewhere in this specification.

[0024] Optionally, the second implementation used in place of the operation can be configured to quantize the input (e.g., floating point input) to an integer; apply the integer operation corresponding to the node to obtain an integer output; and dequantize the integer output (e.g., dequantize to floating point). For example, the input graph of the optimizer may allow different types of inputs for a particular operation (e.g., supporting both integer inputs and floating point inputs), where the replacement operation may cause the corresponding operation to only allow inputs of the corresponding type (e.g., only supporting addition of integers, if the previous output is not an integer, the replacement operation includes quantization). Depending on the inputs and / or outputs supported by the node, nodes corresponding to quantization and / or dequantization may be included in the subgraph.

[0025] Optionally, for a node with multiple inputs (e.g., multiple tensor inputs, where the computation graph is a tensor computation graph), based on determining that the multiple inputs have a common prototype in the graph, it can be determined that the node is implemented by a table lookup operation. For example, the (element-by-element) addition of two (tensor) inputs that itself represents an (element-by-element) function applied to a common predecessor edge can itself be represented as an element-by-element function applied to a common predecessor edge, and can therefore be determined to be implementable by a table lookup operation. Since such a determination cannot be made manually, particularly for large computations, and particularly based on the structure of the computation graph, this is a particular embodiment where it is beneficial to automatically determine whether a node can be implemented as a table lookup.

[0026] The computational graph may include one or more reshape nodes. Such a reshape node may represent an operation that reorganizes the elements of one or more input tensors without changing the values ​​of the elements of one or more input tensors, such as transposition, reshape, flattening, etc. In this case, the computational graph may be reordered to fuse at least one operation before the reshape node with at least one operation after the reshape node. For example, exchanging a reshape node with its previous or next node may result in two adjacent nodes representing element-by-element operations that can be implemented by a lookup table and can then be fused into a single table lookup. When the optimization involves a substitution operation as described herein, the reordering is preferably performed after the substitution operation, but in principle may also be applied before the substitution operation.

[0027] Optionally, the transformed representation of the computer program does not include a table lookup operation applied to the output of another table lookup operation. Instead, such subsequent table lookups can be fused, resulting in a more efficient implementation of the computer program. Optionally, the transformed representation of the computer program does not include a reshaping operation applied to the output of a table lookup operation. By swapping TLUs and reshaping whenever a reshaping is applied to the output of a TLU, it can be ensured that there are no TLUs that only have reshaping between them, in other words, such TLUs are grouped together in the computational graph. This allows the TLUs to be fused, for example by selecting them as connected subcomponents of a graph.

[0028] Alternatively, the computer program may represent the evaluation of a machine-learnable model (such as an artificial neural network, a generalized linear model, a decision tree, or an ensemble model.) This type of computation can be well described by a (tensor) computation graph, can be relatively complex, and can be repeatedly applied to different data, making it important to optimize computer programs for their efficient evaluation.

[0029] Typically, the input computational graph can be defined in a format and / or using a set of supporting operations that are independent of a specific execution environment. For example, the input computational graph can be a high-level graph defined, for example, in the ONNX (Open Neural Network Exchange) format. The transformed representation can be a low-level representation specific to the optimized execution environment, for example, it can be an executable format that can be executed by the execution environment, or it can be a computational graph using a set of restricted nodes and / or edges supported by the execution environment. For example, substitution operations can be used to implement high-level operations based on low-level operations. For example, the optimizer can be used in an encrypted computing system to allow machine learning models or other types of computations specified according to high-level representations to be efficiently executed using homomorphic encryption.

[0030] It will be appreciated by those skilled in the art that two or more of the above-mentioned embodiments, implementations, and / or alternative aspects of the invention may be combined in any way deemed useful.

[0031] Those skilled in the art may make modifications and variations to any system and / or any computer-readable medium based on the present description, which modifications and variations correspond to modifications and variations of the corresponding computer-implemented methods described. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] These and other aspects of the invention will be further apparent from and elucidated with reference to the embodiments described by way of example in the following description and with reference to the accompanying drawings, in which:

[0033] Figure 1 A compiler system is shown;

[0034] Figure 2a An embodiment of an encryption computing system is shown;

[0035] Figure 2b An embodiment of a device supporting a hardware lookup table is shown;

[0036] Figure 3a to Figure 3d A detailed embodiment of a computation graph is shown;

[0037] Figure 4 An embodiment showing how to optimize a computer program;

[0038] Figure 5a to Figure 5c A detailed embodiment of a computational graph of a transformation is shown;

[0039] Figure 6a to Figure 6b A detailed embodiment of reordering a computation graph is shown;

[0040] Figure 7 An embodiment showing how to optimize a computer program;

[0041] Figure 8 A computer-implemented method is shown;

[0042] Fig. 9 A computer readable medium including data is shown.

[0043] It should be noted that these figures are diagrammatic only and are not drawn to scale.In the figures, elements corresponding to elements already described may have the same reference numerals. DETAILED DESCRIPTION

[0044] Figure 1 A compiler system 100 is shown. The system 100 may be used to optimize an execution environment for supporting table lookup operations (e.g., for example, Figure 2a The homomorphic encryption computation exemplified in Figure 2b A computer program implemented by the hardware illustrated in FIG.

[0045] The system 100 may include a data interface 120 for accessing data 040 representing a computer program as a computational graph. The corresponding nodes of the computational graph may represent corresponding operations. For example, the input computational graph may include at least 50, at least 250, or at least 10,000 nodes.

[0046] For example, the same Figure 1 As illustrated, the data interface 120 may be composed of a data storage interface 120 that can access data 040 from a data storage 021. For example, the data storage interface 120 may be a memory interface or a persistent storage interface, such as a hard disk or SSD interface, or may be a personal area network interface, a local area network interface, or a wide area network interface (such as a Bluetooth, Zigbee or Wi-Fi interface, or an Ethernet or fiber optic interface). The data storage 021 may be an internal data storage of the system 100 (such as a hard disk drive or SSD), or may be an external data storage (such as a network accessible data storage). In some embodiments, the corresponding data may be accessed from different data storages (e.g., via different subsystems of the data storage interface 120). Each subsystem may be of the type described above for the data storage interface 120.

[0047] The system 100 may also include a processor subsystem 140, which may be configured to determine, for a corresponding node of the computation graph, whether the corresponding node can be implemented by a table lookup operation during operation of the system 100. The processor subsystem 140 may also be configured to determine, for a node that can be implemented by a table lookup operation, that one or more additional nodes of the computation graph can be fused into the table lookup operation. The processor subsystem 140 may also be configured to output a transformed representation of the computer program, wherein the node and the one or more additional nodes are fused into a single table lookup operation.

[0048] The system 100 may also include an output interface for outputting a transformed representation of a computer program. In this embodiment, the output interface is composed of a communication interface 180, via which the transformed representation 126 can be output to another system, for example, an encryption device that performs encryption calculations according to the output representation. The communication interface 180 can communicate internally with the processor subsystem 140 via data communication 125. The communication interface 180 can be arranged to communicate directly with other systems, for example, using USB, IEEE 1394 or similar interfaces. As illustrated in the figure, the communication interface 180 can also communicate through a computer network 010 (e.g., a wireless personal area network, the Internet, an intranet, a LAN, a WLAN, etc.). For example, the communication interface 180 may include a connector, such as a wireless connector, an Ethernet connector, a Wi-Fi, a 4G or 4G antenna, a ZigBee chip, etc., to suit a computer network. The communication interface 180 may also be an internal communication interface, such as a bus, an API, a storage interface, etc.

[0049] However, other methods of implementing the output interface are possible. For example, the output interface may be comprised of a data interface 120, which in these embodiments is an input / output ('IO') interface, via which the transformed representation determined may be stored in a data store 021. In some embodiments, the output interface may be separate from the data storage interface 120, but may generally be of the type described above for the data storage interface 120.

[0050] Generally speaking, each system described in this specification (including but not limited to Figure 1 The system 100 of the corresponding system can be implemented as a single device or apparatus (such as a workstation or server) or implemented in a single device or apparatus (such as a workstation or server). The device may be an embedded device. The device or apparatus may include one or more microprocessors that execute appropriate software. For example, the processor subsystem of the corresponding system may be embodied by a single central processing unit (CPU), or by a combination or system of such CPU and / or other types of processing units. The software may have been downloaded and / or stored in a corresponding memory, for example, a volatile memory such as RAM or a non-volatile memory such as Flash. Alternatively, the processor subsystem of the corresponding system may be implemented in a device or apparatus in the form of programmable logic (for example, as a field programmable gate array (FPGA). In general, each functional unit of the corresponding system can be implemented in the form of a circuit. The corresponding system can also be implemented in a distributed manner, for example involving different devices or apparatuses, such as distributed local servers or cloud-based servers.

[0051] Figure 2aA detailed but non-limiting embodiment of a cryptographic computing system 199 is shown. The system 199 is configured to perform cryptographic computing using homomorphic encryption (eg, fully homomorphic encryption).

[0052] The cryptographic computing system 199 may include a cryptographic system 112 configured to perform cryptographic computing using homomorphic encryption. The cryptographic computing system 199 may also include a compiler system 111 configured to determine a representation of a computer program optimized for an execution environment provided by homomorphic encryption. For example, the compiler system 111 may be based on Figure 1 The encryption computing system 199 may also include a data provider device 113. The corresponding systems 111, 112, 113 in this embodiment may be generally based on Figure 1 The hardware configuration of the device 100 of a may include, for example, Figure 1 Processor systems, data and / or communication interfaces in.

[0053] The system 112 may be configured to receive encrypted data items from the data provider 113. At least one or more data items may be received in encrypted form. One or more additional data items may be received in plain text format. The system 112 may be configured to receive key material (e.g., a bootstrap key and / or a key switching key) for performing cryptographic calculations, for example, from a data provider system or from a key generation system.

[0054] The system 112 can perform cryptographic computations on received data items, and possibly also on stored data items. Interestingly, computations can be performed by the device on encrypted data without decrypting the data, e.g., without converting the encrypted data items to data in a plain text format. The system 112 can perform cryptographic computations based on the transformed representation determined by the compiler system 111.

[0055] The compiler system 111 can be combined with the data provider system 113, in which case the data provider system 113 can, for example, provide a transformed representation of the computer program together with the data items used to execute the computer program. Alternatively, the compiler system 111 can be combined with the cryptographic system 112, or called by the cryptographic system to perform optimizations, in which case the data provider system 113 can, for example, provide a computational graph to which the optimizations are applied, together with the data items on which the computer program is to be executed. For example, the computational graph can be in a format that is independent of the definition of the execution of the cryptographic computations (e.g., the ONNX format), and may not even be built for use with homomorphic encryption. However, due to the optimization, the computer program can be efficiently executed using homomorphic encryption, in particular reducing the number of table lookup operations.

[0056] Although not shown in the figure, the cryptographic computing system 199 may include multiple cryptographic systems, such as two, three, or more than three. The cryptographic calculations may be distributed between multiple cryptographic systems. The cryptographic systems may exchange intermediate calculation results (usually encrypted) with each other. Each cryptographic device may be implemented as a cryptographic system 112 and may perform cryptographic operations as described herein.

[0057] In general, various types of homomorphic encryption that support encrypted table lookup operations can be used. In many cases, homomorphic encryption has integer and / or polynomial plaintexts, which are defined modulo some modulus (in the literature, the modulus integer is sometimes referred to by the correspondence is equivalently described as a torus element).

[0058] In addition to supporting encrypted table lookup operations, homomorphic encryption generally supports a set of additional operations that do not involve table lookups. Such operations can be called native operations of homomorphic encryption, and may include, for example, addition operations and / or encrypted scalar multiplications. Such operations are generally more computationally efficient than table lookups. Therefore, if possible, native implementations of these operations are generally preferred over table lookups. Interestingly, however, in some cases, these operations can still be fused into previous and / or subsequent table lookup operations as described herein, resulting in overall efficiency improvements.

[0059] Specifically, the computation may use homomorphic lattice encryption, such as LWE encryption and / or NTRU encryption. For such encryption, it is known per se how to implement various operations (including encryption table lookup operations). For example, homomorphic encryption may be "TFHE-class" homomorphic encryption, since the table lookup operation is implemented as a so-called programmable bootstrap operation. The operation may perform a table lookup on the input ciphertext based on a blind rotation. Blind rotation may decrypt the input ciphertext in a monomial exponent. By multiplying the encrypted monomial by a polynomial representing a lookup table, an encrypted polynomial with the output of the lookup table as fixed coefficients may be obtained. For example, the polynomial coefficients may be extracted, or the encrypted polynomial may be further processed. Such TFHE-like programmable bootstrapping is known per se for LWE encryption, e.g. from I. Chillotti et al., “Programmable bootstrapping enables efficient homomorphic inference of deep neural networks” and from L. Ducas et al., “FHEW: bootstrapping homomorphic encryption in less than asecond”, Proceedings EUROCRYPT 2015; and also for NTRU encryption, e.g. from C. Bonte et al., “FINAL: Faster FHE instantiated with NTRU and LWE”, https: / / eprint.iacr.org / 2022 / 074, and K Kluczniak, “NTRU-ν-um: Secure Fully Homomorphic Encryption from NTRUwith Small Modulus”, https: / / eprint.iacr.org / 2022 / 089.

[0060] A programmable bootstrap (PBS) may take as input a ciphertext and output a ciphertext of the same message, or a function of the message (in other words, a lookup table applied to the message), with noise that is independent of the input. A PBS may include blind rotations that evaluate homomorphic decryption of the input ciphertext in a polynomial exponent. Specifically, a programmable bootstrap may take as input a ciphertext of an encrypted message m, a bootstrap key, and an encryption of a lookup table L. The programmable bootstrap may output an encryption of the message L[m] with a fixed noise level. Blind rotations are also called accumulator updates.

[0061] For example, the cryptographic computing system 199 can be configured to train a machine learning model, such as an image classifier, such as a medical model, without requiring the cryptographic computing device to have access to the plaintext data items. A neural network can be trained. For example, back propagation can be performed on the input data. The resulting model parameters can be returned to the entity that has the decryption key. By sending the data to a cloud provider, this enables multiple providers of medical data to pool their data. The cloud provider then returns the model parameters without having access to the plaintext data. The encryption key can be equal to the decryption key.

[0062] After the model is trained, the cryptographic computing system 199 can be used to provide the model (e.g., for use with medical data). This can be done with plaintext model parameters or encrypted model parameters, in both cases using encrypted data (e.g., encrypted input data, intermediate data, and input data). Using plaintext model parameters is generally more efficient. In both cases, the effect of the system is to perform calculations (such as image classification, such as medical image classification) without the computer knowing the plaintext data items. For example, a mammogram can be evaluated for cancer without the image appearing in plaintext at the cryptographic system 112, and without any cryptographic system 112 or combination of such devices knowing what the result of the cancer assessment was. From a privacy perspective, it is acceptable to run a plaintext model on encrypted privacy-sensitive data, but it is not acceptable to run it on plaintext privacy-sensitive data.

[0063] Figure 2b A detailed but non-limiting embodiment of a device 200 supporting a lookup table is shown. In this case, the lookup table is implemented in hardware.

[0064] Specifically, the figure shows a processor subsystem 240 in which a hardware lookup table LUT 241 is implemented. The processor subsystem 240 may be configured as for Figure 1 The processor subsystem 240 is described as a processor subsystem 140 of FIG. Specifically, the processor subsystem 240 may be implemented in programmable logic (e.g., as a field programmable gate array (FPGA)). The processor subsystem 240 may also be implemented in whole or in part as a so-called application specific integrated circuit (ASIC), such as an integrated circuit (IC) customized for its specific purpose. For example, the circuit may be implemented in CMOS, such as using a hardware description language (such as Verilog, VHDL, etc.).

[0065] As shown in the figure, the lookup table LUT can access an input IN, 231, and determine an output OUT, 233 based on accessing data 232 storing the lookup table at a memory location corresponding to the input IN. The data 231 to 233 shown here are stored in a separate memory 240, for example, corresponding to Figure 1The memory 021 of a. For example, the lookup table can be implemented by a RAM (random access memory) or a ROM (read only memory). The data 231 to 233 can also be accessed by the lookup table LUT in different ways, such as as a register value or via a bus.

[0066] As illustrated in the figure, the output OUT may directly correspond to the data stored at the memory location indicated by the input IN. However, alternative implementations of a hardware lookup table are also possible. For example, instead of the output, the data 232 may represent the parameters of a relatively efficient computation function. For example, according to known techniques, the high bits of the input value IN may be used to look up two values, which may then be interpolated using the low bits of the input IN to obtain the output OUT.

[0067] As a specific embodiment, device 200 can be used to perform neural network reasoning or training.For example, processor subsystem 240 can be a neural network accelerator, such as a dedicated circuit optimized for the calculation of operations performed by a neural network.

[0068] Figure 3a to Figure 3d Detailed embodiments of computation graphs that can be optimized using the techniques described herein are shown. These embodiments illustrate the application of the present invention in a computer program representing the evaluation of a machine-learnable model, in this embodiment an artificial neural network.

[0069] Specifically, Figure 3a The computational graph of shows the evaluation of neurons of a neural network. As is well known, a neural network can be used as a statistical model that learns to detect or generate patterns using training data. Such a neural network can include a plurality of neurons, for example at least 1000 or at least 10,000 neurons. The neuron can perform a weighted combination 312 of a plurality of inputs 310 according to corresponding weights 311, and an activation function 313 can be applied to the result. For example, the input 310 of the neuron can be raw data or the activation of a previous neuron. In this embodiment, the neuron combines m-1 inputs with weights and with a bias b k combination. (The figure shows the corresponding bias w k0 =b k ; such constant-valued nodes / edges are generally not considered part of the computational graphs treated in this paper. )

[0070] The activation function 313 can be a univariate function, e.g., with a single variable input (but possibly using a constant). In practice, the activation function can determine whether a neuron is activated, e.g., has a high output value 314. Thus, it can effectively determine whether a neuron's input to the network is important during prediction, e.g., using simpler mathematical operations. For example, the activation function can be a Rectified Linear Unit, a Sigmoid, or an Exponential Linear Unit.

[0071] Mathematically speaking, the computation of a neuron in a neural network can be expressed as follows:

[0072] y=φ(∑ i w i x i + b) = φ(s)

[0073] The dot product 312 between the weight 311 and the input 310 may also be referred to as a neuron accumulator s.

[0074] In a neural network, neurons can be arranged in consecutive groups called layers. For example, a neural network can include at least 10 layers or at least 100 layers. For example, a neural network can include one or more convolutional layers and / or one or more dense layers (also called linear layers). Regardless of the type of layer, its computation may involve the computation of a neuron accumulator.

[0075] s=∑ i w i x i + b,

[0076] where w i , 311 is the learning weight, x i , 310 is the input, and b is the bias value. For example, in the case of a convolutional layer, i=(c,y,x) can be a tuple of iterators over the number of channels, height, and width of the input tensor 310.

[0077] In addition to weights 312 and activation functions 313, the values ​​of the neural network may also undergo various other operations, such as: normalization, particularly batch normalization; scaling; and / or non-linear operations, applied to the output 314, for example, before being used as input in subsequent neural network layers.

[0078] Figure 3aThe embodiment of illustrative examples shows a scalar computation graph in which edges 310, 314 may represent scalar values ​​and nodes 312, 313 may represent operations applied to the scalar values. However, in many other cases, the computation graph may be a tensor computation graph. In such cases, the nodes of the graph may represent operations applied to one or more input tensors and produce output tensors, corresponding to the incoming and outgoing edges of the nodes, respectively.

[0079] In general, a tensor X can represent a set of values ​​indexed by a set of m indices Where m is also referred to as the dimension of the tensor. A scalar value can be represented by a tensor of dimension m=0. A vector can be represented by a tensor of dimension m=1. A matrix can be represented by a tensor of dimension m=2. A tensor computation graph can be defined as having at least one edge representing a tensor of dimension m>0. For example, a tensor computation graph can include one or more edges represented as tensors of at least two dimensions, and / or one or more edges represented as tensors of at least three dimensions. For example, the inputs and outputs of a neural network layer can be represented as two-dimensional tensors (for one-dimensional data with multiple channels); three-dimensional tensors (for two-dimensional data with multiple channels); or four-dimensional tensors (for three-dimensional data with multiple channels).

[0080] A tensor computation graph can represent a series of operations performed on data stored in a tensor. A tensor is a generalization of a digital matrix to more than 2 dimensions. In various embodiments, a tensor computation graph can be viewed as directed and non-cyclic. Typically, a tensor computation graph can represent a set of ordered operations performed on a tensor. An operation can have inputs calculated by other operations, or can be input to an entire computer program. An operation can also involve constants. For the purposes of this specification, an input to an operation refers to a non-constant input.

[0081] For example, the tensor computation graph described in this article can represent the evaluation of a neural network. However, it is well known that tensor computation graphs can also be used for other types of computations, such as in stream processing systems or databases.

[0082] In general, computation graphs can represent computations at different levels of abstraction. For example, an activation function can be represented in a computation graph as a single node in the graph, which can then be implemented in code as a series of basic tensor operations; or a series of more basic tensor computation graph operations, such as 'add', 'subtract', etc. The techniques presented can in principle be applied to graphs at various levels of abstraction, such as Figure 4 and Figure 7 As discussed further below. As a specific example, the provided techniques can be applied to graphs in the ONNX (Open Neural Network Exchange) format. This is a format for tensor computation graphs using a given set of operators.

[0083] For example, Figure 3b to Figure 3d A tensor computation graph representing a computer program that can be used as part of a neural network evaluation is shown. The figures provide a visual representation of a tensor computation graph specified in the ONNX format. As illustrated in the figures, the nodes of the tensor computation graph can represent basic operations, such as arithmetic operations like addition (Add) and subtraction (Sub), division (Div), multiplication (Mul), convolution (Conv), or matrix multiplication (MatMul). In the computation graph, non-constant inputs to the operation are represented as incoming edges, while constant inputs are contained in the operation nodes themselves.

[0084] Specifically, Figure 3b and Figure 3c A computation graph representing a quantized computation is shown. Quantized computations can be used for neural network evaluation. Neural networks can typically represent values ​​such as inputs, activations, and / or weights as floating point values. However, in various cases, neural network operations can be applied to integer values. For example, this can be for computational performance reasons, or to evaluate neural networks under encryption.

[0085] Quantization calculations can convert floating point values ​​into integers. Floating point numbers can have fine resolution to represent real numbers, but have a limited number of bits. In addition, the resolution of a floating point number can be inversely proportional to the absolute value of the number. When a neural network is quantized to use integers to improve computational performance, the number of bits of these integers can be lower, such as up to 12 bits, up to 8 bits, or up to 6 bits. There are various quantization techniques that aim to perform quantization with minimal quantization error.

[0086] For example, a subgraph of a computation graph can represent uniform quantization. In uniform quantization, discrete values ​​can be assigned to equal-sized intervals on the real axis. This is represented by Figure 3b As illustrated, in this embodiment, uniform quantization is applied element-by-element to an input tensor 320 and produces an output tensor 321, which is input to a matrix multiplication operation 322 of, for example, a linear layer of a neural network. Specifically, the calculations performed in the illustrated calculations can be mathematically represented as follows:

[0087]

[0088] where q x is the scaling factor, zp x is zero point, and and are limiting values ​​that restrict the range of real numbers quantized to integer values. For example, the values ​​of these parameters can be obtained by calculating them from a representative set of data.

[0089] The uniform quantization shown can be undone by performing an inverse quantization calculation (not shown in the figure). Thus, a quantized floating point value close to the original original value can be retrieved. Inverse uniform quantization can be mathematically represented as:

[0090] x f′ =(x q -zp x )*s f .

[0091] Figure 3c shows a computational graph for non-uniform quantization. In non-uniform quantization, the corresponding intervals mapped to corresponding integers do not all have the same size. Mathematically speaking, non-uniform quantization can correspond to computing

[0092]

[0093] in

[0094]

[0095] and β i Here, the first interval is [-∞,β0], the second interval is [β0,β1], and the last interval is

[0096] In the illustrated embodiment, non-uniform quantization is applied to a tensor 330 of n=2-bit integers. In this embodiment, the quantized output 331 is input to a convolution (Conv) operation 332, such as a convolution (Conv) operation 332 corresponding to a convolution layer of a neural network. i The values ​​correspond to the values ​​in the child nodes in the graph.

[0097] Figure 3d The computational graph of the normalization computation is shown. This embodiment illustrates batch normalization in a two-dimensional case. The computation can be mathematically represented as follows:

[0098]

[0099] Where y is the output tensor 344, x is the input tensor 341, β and γ are learned parameters, and μ c and σ c are the mean and standard deviation of channel c (e.g., obtained by computing them on a set of representative input tensors).

[0100] In this embodiment, batch normalization is performed on the output 341 of the matrix multiplication (MatMul) layer. The division operation 343 in this embodiment represents the division of the center input x-μ with the standard deviation 342 (here expressed as the square root of the variance: σ=Var 0.5). Although shown in this embodiment as an edge representing a variable input to division 342, standard deviation 342 may more typically be a constant incorporated into division 342 and thus may not be represented as an edge in the computation graph.

[0101] Instead of or in addition to batch normalization, the computer programs described herein may also include tensor normalization (e.g., using a single μ and σ for each tensor), layer normalization, scaling to 0-1, etc.

[0102] Figure 4 A detailed but non-limiting embodiment of an optimization computer program is shown. In this embodiment, the optimization involves converting a high-level computational graph into an implementation operation (also called a substitution operation) of a low-level computational graph. In this embodiment, nodes of the high-level graph are determined to be implementable by a table lookup; the implementation operation is based on this determination; and nodes of the low-level graph are fused into a single table lookup operation.

[0103] Specifically, the figure shows a computational graph CG1,411 representing a computer program to be optimized, for example, as Figure 3a to Figure 3d As discussed. The corresponding nodes of the computation graph represent corresponding operations. Specifically, the computation graph GC1 can be a tensor computation graph in which the corresponding operations are applied to one or more input tensors and produce output tensors. For example, the computation graph CG1 can be represented in the ONNX format, for example as a directed acyclic graph of tensor operations from the ONNX reference (available at

[0104] (available from https: / / github.com / onnx / onnx / blob / 55d3b80f3ea1261f3f74f2e35844c204de85835a / docs / Operators.md and incorporated herein by reference).

[0105] Various operations can be supported by optimization. Specifically, the overall group operation can include at least an addition operation, a multiplication by a constant, and a multiplication between two non-constants. The overall group operation can also include at least one nonlinear element-wise operation, such as activation function application, numerical comparison, etc. In general, several groups of operations that may occur in the tensor computation graph CG1 can be distinguished. Nodes representing operations from the first group may never be implemented by table lookup operations; nodes representing operations from the second group may be implemented by table lookup operations depending on the computation graph at hand; and nodes representing operations from the third group may always be implemented by table lookup operations. Specifically, nodes can be identified as belonging to a certain type depending on the type of operation and the input of the operation. It is worth noting that a node representing a certain operation (such as addition) can have different types depending on the context in the graph (for example, depending on its input), as discussed in more detail below. Specifically, the following types can be defined.

[0106] The first type of operation is a mixed operation (MIX). A mixed operation can be defined as an operation that modifies the value of its input tensor but is not applied to its input element by element. Specifically, a mixed operation can optionally combine different values ​​of a single input tensor from the operation by involving an additional constant tensor. For example, the operation can be a neuron accumulator operation, such as a convolution or a matrix multiplication with a constant tensor (for example, for implementing a dense neural network layer). Alternatively, a mixed operation can combine values ​​from two different graph input tensors, or values ​​from tensors generated by different MIX operations. For example, the operation can be an addition or concatenation of two graph inputs, or an arithmetic or concatenation operation applied to the tensors of the outputs of two MIX operations. In various embodiments, it is determined that the node representing the mixed operation is not implemented by a table lookup operation (although a table lookup operation can be used to pre-process and post-process the input and output of the operation, such as by quantization). For example, a given group of operations (such as convolution or dot product) can be determined as a MIX type, regardless of the other nodes of the computational graph, and therefore can not be implemented by a table lookup.

[0107] The second type of operation is a univariate operation (UNI). A univariate function can be applied element-wise to its input tensor, returning an output tensor of the same size (but possibly with a different value type). For example, the operation can be an activation, normalization, quantization, or an arithmetic operation with a scalar constant. Note that univariate computations applied to a tensor are univariate with respect to the tensor cells (scalars). Specifically, different functions can be applied to different cells in a tensor. The determination of a node as a univariate operation can be implemented by a table lookup operation. Specifically, as illustrated below, a node can be determined to be of type UNI despite having multiple incoming edges in the computation graph.

[0108] Interestingly, there may be a set of operations for which nodes may be determined to be either MIX or UNI type operations, depending at least on their inputs. For example, the set of operations may include addition (Add) and / or multiplication (Mul). For example, when the input type of such an operation is UNI, then the operation itself may also be determined to be UNI, depending on the operation. For example, an Add operation may be UNI if one of the inputs is generated by a single variable operation and the other input is a constant, or if the inputs are generated by a UNI operation calculated on the same single input. For example, an Add operation may be MIX if the inputs are generated by UNI operations on different inputs. Thus, operations from this set may or may not be implemented by table lookups, depending on the other nodes in the graph. Specifically, it can be determined whether the operation is a MIX or UNI operation depending on the previous operation. For example, if the previous operation is a UNI operation (e.g., sigmoid(x)), then the addition x+1 can be determined to be a UNI operation, and if the previous operation is a MIX operation (e.g., conv(x,w)), then the addition x+1 can be determined to be a MIX operation. In some embodiments, for simplicity, constant addition is always determined to be a UNI operation.

[0109] For various univariate operations, this may be the case where no native implementation is available in the execution environment, and they are always determined to be of type UNI and thus implemented via a lookup table. For example, this may be the case for non-linear functions such as activation functions, sine functions, etc., depending on the execution environment.

[0110] The third type of operation is a reshape operation (SHAPE). Such an operation may reorganize the elements of one or more input tensors (e.g., by reordering values ​​or changing tensor metadata such as shape). The values ​​in the tensor may remain unmodified. Examples of such operations may include transpose, reshape, flatten, etc. Since such operations may not affect the values, they are also typically implemented without using lookup tables.

[0111] An identification operation Ident, 420 may be applied to the computation graph CG1 to determine, for a corresponding node of the computation graph CG1, whether the corresponding node can be implemented by a table lookup operation, in particular by determining whether the node is of type UNI as discussed above. This may result in an updated computation graph CG2, 412, in which the corresponding node is marked as being implementable by table lookup or not. Such marking is also referred to as an execution mode, or as a CanFuse marking. A node that can be implemented by a table lookup is said to have a TLU (table lookup) execution mode.

[0112] In general, an identifier Ident can process a tensor operation graph CG1 and determine the execution mode of its corresponding operation by: (a) iterating the operation graph CG1, which iterations can generally be performed in any order and no matter how many times are needed; (b) evaluating certain compatibility conditions for each corresponding operation to verify whether the TLU execution mode is applicable to the operation; and (c) assigning the execution mode to the corresponding operation of the graph. This is illustrated in the figure by several nodes of the identified result CG2 being diagonally striped to indicate that they have TLU execution mode.

[0113] Interestingly, for a node with multiple inputs in graph CG1, identification may involve that it can be implemented through a table lookup operation. That is, it can be determined that the multiple inputs have a common prototype from which the output of the node can be calculated as an element-by-element function. This makes it possible to fuse the node with the previous node (and possibly the subsequent node) into a single table lookup, resulting in overall more efficient execution in the execution environment.

[0114] As a detailed embodiment, an embodiment of an algorithm is now provided to determine which operations can be fused, that is, for the operation at the corresponding index i in the operation graph, the algorithm can set CanFuse[i] to the corresponding execution mode: either in TLU (e.g., UNI type operation) or as a basic operation (e.g., SHAPE or MIX type operation). As illustrated, for one or more nodes, it can be determined that they are of MIX or SHAPE type regardless of their position in the graph (e.g., convolution operation or flattening operation), in other words, they can have a pre-forced operation type. For one or more other nodes, the operation type or the CanFuse tag generated in this embodiment can be determined based on their position in the graph.

[0115]

[0116]

[0117] After the identification operation Ident, an implementation operation Impl, 430 can be applied to the computation graph CG2. If the node has been marked as being implementable by a table lookup operation, the implementation operation can replace the node of the computation graph CG2 by a first subgraph representing a first implementation; and in other cases by replacing the node of the computation graph CG2 by a second subgraph representing a second different implementation, thereby producing an updated computation graph CG3. In this sense, the high-level computation graph CG2 can become a low-level computation graph CG3.

[0118] Specifically, the high-level computation graph CG2 may include one or more operations that cannot be directly executed by the execution environment; for example, operations on floating-point values. The implementation Impl may convert the high-level computation graph into a low-level computation graph CG3, which may include only basic operations, for example, operations that can be directly executed by the execution environment; for example, such basic operations may be lookup table compatible operations (on integers and / or floating-point numbers) and primitive operations on integers (although such operations may still be implemented by multiple primitive operations of the execution environment: for example, matrix multiplication, convolution, etc. may be performed by performing multiple multiplications of integers).

[0119] Typically, the corresponding implementation available for an operation may be a corresponding subroutine, which is represented as a subgraph by which the node of the operation can be replaced. At least one of these subroutines can be used to build a TLU. For example, an implementation may represent a series of basic operations (addition, subtraction, etc.).

[0120] Specifically, the first implementation can be a floating-point subroutine. Such a floating-point subroutine can be represented by a single node with floating-point input and output tensors. As an algorithm, the first implementation can be represented as:

[0121]

[0122] The second implementation can represent a series of subroutines in floating point and / or integer. Specifically, the implementation can be configured to quantize the input to integers (if necessary); apply the integer operation corresponding to the node to obtain integer output; and dequantize the integer output (if necessary). For example, for the arithmetic operation ⊕, the implementation can correspond to the following algorithm:

[0123]

[0124] More generally, the operations of computation graph CG2 can be implemented in the transformed computation graph CG3 as follows:

[0125]

[0126] Specifically, the second implementation can perform integer operations corresponding to the operations to be implemented. For example, matrix multiplication can be performed on quantized values, as discussed, for example, in B. Jacob et al., "Quantization and training of neural networks for efficient integer-arithmetic-only inference", doi: 10.1109 / CVPR.2018.00286 (incorporated herein by reference).

[0127] Specifically, as shown in the figure, in the transformed computation graph CG3, the nodes identified as fusible can be replaced by corresponding floating-point operation nodes (exemplified as being the same as the original nodes in the computation graph CG2), and the nodes identified as non-fusible can be replaced by corresponding subgraphs used for their integer implementations (exemplified as resulting in horizontal striped nodes).

[0128] After implementing the operation Impl, a fusion determination operation DFuse, 450, can be applied, wherein for nodes that can be implemented by a table lookup operation, it is determined that one or more other nodes of the computation graph CG3 can be fused into the table lookup operation. This is illustrated in the figure as a computation graph CG4, 414, wherein it is determined that node 415 can be fused into a single table lookup operation. Therefore, the figure illustrates different identification steps Ident, in which nodes are identified as fusible or non-fusible, and fusion determination steps DFuse, in which such nodes are grouped into subsets of fused nodes, such as set 415.

[0129] Specifically, the fusion determination step DFuse can identify connected components of the computation graph CG3 that share a common property that can be implemented by a table lookup operation, for example, it can be described as applying a univariate function to the output of the previous node. Interestingly, at this stage, it may be sufficient to identify only connected components that satisfy this property; it may no longer be necessary to check whether they have a common prototype.

[0130] As illustrated in the figure, the fusion determination step DFuse may include nodes in the fusion step that do not correspond to the fusible operations of the computation graph CG2. Specifically, they may originate from the implementation of non-fusible nodes. For example, as illustrated in the figure, the quantization operations originating from the subsequent non-fusible nodes of the computation graph CG2 may be fused with the fusible operations of the computation graph CG2 into a single lookup table.

[0131] Interestingly, the fusion determination step DFuse can be performed in such a way that a table lookup operation is not applied to the output of another table lookup operation. In this regard, the number of table lookups performed can be minimized.

[0132] Once it has been determined that nodes can be fused, an output write operation Write, 460, can be performed, where a transformed representation of a computer program can be output, in which node 415 is fused into a single table lookup operation. The transformed representation can take various forms. It can be explicitly stored as a computation graph or stored as a set of instructions INSTR, 415, for implicitly defining the target execution environment of the computation graph. The set of instructions can be in a form of instructions that are portable between various machine architectures or in a form of code executable on a specific machine architecture. As is known per se, the write operation Write can include constructing a lookup table by applying a univariate function represented by the nodes to be fused to a set of possible (e.g., integer) inputs to obtain corresponding (e.g., integer) outputs. Such a write operation is known, for example, from Zama's Concrete-Numpy (available from https: / / github.com / zama-ai / concrete-numpy).

[0133] The computation graph explicitly or implicitly defined by the write operation Write can have a certain canonical form. The computation graph can be an integer computation graph and can utilize table lookup operations. The table lookup (TLU) operation is applied element-wise to a single input tensor and can be defined as

[0134]

[0135] where LUT is a vector of M values, LUT i is the i-th element of the vector and 0 < x < M. This operation is also referred to as "Gather". As is known per se, such a TLU operation can be constructed by a write operation Write from the following:

[0136] (a) A series of basic operations of computation graph CG4, the series of basic operations together implementing a univariate function, e.g., for f(x) = sin(x / 10), the series of operations is y = x / 10, followed by sin(y)

[0137] (b) An integer array is the representative value to be processed by the TLU, allowing the computation of the function of the integer value v i in the input set: LUT i = f(v i ),

[0138] This canonical form can be referred to as the Form TLU-Shape canonical form. This canonical form can refer to an integer (usually tensor) computation operation graph having the following properties:

[0139] 1. The operation graph only contains operations of types MIX, SHAPE, and TLU;

[0140] 2.TLU operations have integer inputs and outputs, for example,

[0141] 3. The output of a MIX or SHAPE type operation is processed by at most one TLU operation before becoming the input of a subsequent MIX or SHAPE type operation, or returned as the final computation output.

[0142] Specifically, the figure illustrates a situation where a write operation Write, 460 generates a set of instructions INSTR, 415 for a target execution environment based on a computation graph CG4. In the set of instructions INSTR, nodes 415 are fused into a single table lookup operation. In other words, a single table lookup operation is used to implement the set of nodes. The set of instructions can take a variety of forms, for example, it can be an assembly; a set of instructions executable by a homomorphic computing engine; a circuit layout of a hardware circuit; a programming language code that compiles or interprets a programming language, etc.

[0143] Optionally, the optimization may include a sorting operation Ord, 440. As illustrated in the figure, the sorting operation Ord may be applied to the low-level computation graph CG3 resulting from the implementation operation Impl. The operation may also be applied to the high-level computation graph CG2 before performing the implementation operation Impl (but typically after identifying which nodes Ident may be implemented by a table lookup operation); but applying the sorting operation to the low-level computation graph has the advantage of identifying more cases where reordering may be applied.

[0144] The sorting operation Ord can operate on a reshape (SHAPE) node. Such a node can represent an operation that reorganizes the elements of one or more input tensors without changing the values ​​of the elements of one or more input tensors, such as a flattened transposition. The sorting operation can reorder the computational graph (e.g., graph CG3) to fuse at least one operation before the reshape node with at least one operation after the reshape node. Interestingly, such a reorganization may not affect the results of the calculations represented by the computational graph CG3. However, the reorganization can have the effect that the previous node and the next node in the graph are directly connected to each other, and therefore the fusion determination step DFuse can determine that these nodes can be fused into a single table lookup operation (possibly with additional nodes).

[0145] Specifically, after the sorting operation Ord, the computation graph CG3 may have the property that no reshaping operation is applied to the output of the table lookup operation. This may be achieved by moving the reshaping operation after the table lookup operation. Specifically, a sequence of nodes in the graph comprising a blending operation followed by one or more table lookup sequences followed by a reshaping (optionally followed by a table lookup) may be replaced by a blending operation, a corresponding reshaping and a corresponding table lookup. The corresponding table lookups may then be fused into a single table lookup operation (possibly with an additional subsequent table lookup) by a fusion determination step DFuse.

[0146] Specifically, the sorting operation Ord can recognize the pattern MIX-(TLU-SHAPE)+{-TLU}, where + indicates at least 1 repetition and {x} indicates an optional operation. This operation can replace such patterns with new patterns of type SHAPE+-TLU+{-TLU}, allowing adjacent TLU operations to be fused through the DFuse operation.

[0147] Specifically, a computation graph defined explicitly or implicitly by a write operation Write can adopt a canonical form referred to in this paper as a TLU-specific canonical form. This canonical form of the computation graph can be an integer (usually tensor) computation graph because, in addition to the properties of the TLU shape form described elsewhere, it also has the following properties:

[0148] 1. There is no TLU operation between MIX and SHAPE operations, or between two SHAPE operations;

[0149] 2. TLU operations cannot perform identity mapping between input and output. An implementation of the sorting operation Ord is as follows:

[0150]

[0151]

[0152] Figure 5a to Figure 5c A detailed embodiment of a transformed computational graph is shown. The transformed computational graph shown may correspond to the output of the optimization as described herein.

[0153] These transformed computational graphs show the computational complexity of implementing quantization (in Figure 5a and Figure 5b in) and standardized (in Figure 5b Quantization and normalization, as well as the application of activation functions, are all examples of operations on a single activation (e.g., a single neuron output) in a neural network. Therefore, these functions are univariate with respect to the activation and can therefore be implemented using a lookup table. Figure 5a to Figure 5c, where these operations are implemented by a single lookup table. Specifically, the subgraph is in TLU-SHAPE canonical form, as also described with respect to Figure 4 As discussed in .

[0154] Specifically, Figure 5a shows a quantized transform representation, in this case uniform quantization. This output can be obtained by applying the techniques presented in this article to Figure 3b The figure shows Figure 3b The part of edge 321 calculated from edge 320 in the computation graph of Figure 5a is implemented as a single lookup table. It is worth noting that Figure 3b The graph of contains several nodes with multiple inputs, notably two "Where" nodes; however, since it can be determined that these nodes have a common prototype (notably the earlier "Add" or "Where" node), it is interesting to note that these nodes are represented by Figure 5a The lookup table 523 in is implemented.

[0155] Similarly, Figure 5b shows a transform representation of non-uniform quantization, which is an application of the proposed technique to Figure 3c In this embodiment, Figure 3c The portion of the computation graph of that computes edge 331 from edge 330 is implemented as a lookup table 533. Also in this case, various nodes that themselves have multiple inputs are included in the lookup table, which is determined to be possible because these nodes share a common prototype, i.e., they are all computed as a univariate function of the output 330 of the multiplication operation.

[0156] Figure 5c Finally, a transformed representation of a normalization operation (in this case a batch normalization operation) is shown, which can be obtained by applying the provided technique to Figure 3d (where, as also discussed with respect to that figure, edge 342 is incorporated into Div node 343 as a constant input). In this case, applied to the output 341 of the matrix multiplication and producing output edge 344 Figure 3d The batch normalization is implemented as a single lookup table 545.

[0157] Figure 6a , Figure 6bA detailed embodiment of reordering a computational graph is shown. This embodiment shows how to reorder a subgraph of a computational graph to facilitate the fusion of table lookup operations. Specifically, the reordering can be applied to a reshape node. Such a node can reorganize the elements of an input tensor without changing their values. If such a node (or a sequence of multiple such nodes) is preceded and followed by a table lookup operation, then by moving the reshape node, the table lookup nodes may be directly connected in the subgraph and can be merged into a single table lookup operation (e.g., in a subsequent operation).

[0158] Figure 6a , Figure 6b An input computation graph is shown, and an output computation graph to which "Merge" (eg, reordering and subsequent fusion) has been applied.

[0159] Specifically, Figure 6a A computational graph is shown, which represents the corresponding application of tensors to convolution CONV, 601; table lookup TLU, 602; reshape RESHAPE, 603; another table lookup TLU, 604; and matrix multiplication MATMUL, 605. For simplicity, any other operands of convolution 601 and matrix multiplication 605 are not shown in this embodiment. In this embodiment, the table lookups 602 and 604 are respectively before and after the reshape operation 603. The reordering may include moving the reshape operation 603 before the table lookups 602 and 604. Adjacent table lookups may then be fused into a single table lookup 609. The reshape operation may also be moved after the table lookups 602 and 604, and thus a computational graph is obtained in which a single table lookup 609 is located before the reshape 603. In any case, by iteratively moving the reshape before or after the table lookup, a sequence of table lookups to be fused may be obtained. Operations 601 and 605 that are neither element-wise applied to their inputs nor represent reshapes may not be affected by the transformation.

[0160] Similarly, Figure 6bA computation graph is shown in which a series of operations are applied to a tensor (which is a convolution CONV, 611); a first table lookup TLU, 612; a reshape RESHAPE, 613; a second table lookup TLU, 614; a reshape operation of the form of a transpose TRANSPOSE, 615; a third table lookup TLU, 616; and a matrix multiplication MATMUL, 617 (for simplicity, any other operands of operations 611, 617 are not shown). In this case, the described transformation can be applied twice: once, swapping the order of the table lookup 612 and the reshape 613; and once again, swapping the order of the transpose and the table lookups 612, 614 (which became adjacent by the previous swap). After these swaps, the table lookups 612, 614, 616 are adjacent and can therefore be fused into a single table lookup 619. As described above, the transpose operation 615 may also be swapped with the table lookup 616, and the reshape operation 613 may be swapped with the table loops 614, 616, resulting in a transformed graph in which the fused table lookup 619 precedes the reshape 613 and the transpose 615. The non-element-wise and / or non-reshape operations 611, 617 remain unchanged.

[0161] It is worth noting that in the two embodiment subgraphs produced by the proposed reordering, there is no reshape operation applied to the output of the table lookup operation. Otherwise, the reshape operation and the table lookup may be swapped. Specifically, the resulting subgraph is in TLU-Only canonical form.

[0162] Figure 7 A detailed but non-limiting embodiment of an optimization computer program is shown. Figure 4 The implementation operation of transforming the high-level computation graph into the low-level computation graph in this way; for example, such an operation may have been previously applied to the technology described for the graph. Identifying whether the corresponding node of Ident, 420 can be implemented by table lookup operation, and then determining whether the DFuse, 450 node can be fused, can be performed on the computation graph without such implementation operation in between.

[0163] Specifically, a computation graph CG1, 711 is shown in the figure. The computation graph can be a low-level computation graph, which means that the corresponding nodes can represent the corresponding operations that are directly converted to the implementation in the execution environment. Specifically, the computation graph CG1 may include low-level operations that implement quantization and / or conversion (cast) operations when necessary. For example, in the case where the execution environment supports integer operations and table lookup operations (such as using homomorphic encryption), the computation graph may include nodes representing integer operations supported by the execution environment (e.g., integer addition and subtraction, scalar multiplication, matrix multiplication by a constant matrix, convolution with a constant matrix, etc.) and nodes representing operations that can be implemented with table lookup operations (e.g., floating-point operations, such as activation functions, quantization or a portion thereof, etc.). The low-level computation graph CG1 may also be referred to as a basic operation graph, and may be an operation graph of Zama's Concrete-Numpy (available at https: / / github.com / zama-ai / concrete-numpy).

[0164] Specifically, as for Figure 4 As discussed, the computation graph CG1 can contain many different types of nodes. Specifically, as also discussed about Figure 4 As discussed, a computational graph may include nodes of a mix type (MIX); nodes of a univariate function type (UNI); nodes of a reshape type (SHAPE); and / or nodes of a table lookup operation type (TLU). Figure 4 As discussed, for a given set of operations, their type may depend on other nodes of the graph, and thus the operation may be implemented as a table lookup or not. This set of operations may include at least addition and multiplication. Other sets of operations may always or never be implemented as table lookups.

[0165] An identification operation Ident, 720 is shown, which can determine whether the corresponding node of the computation graph CG1 can be implemented through a table lookup operation, for example, whether the node is of a mixed type MIX. The identification operation Ident can be configured to determine that a node with multiple inputs can be implemented through a table lookup operation based on determining that the multiple inputs have a common prototype node, and in particular, to make the output of the node writeable as an element-by-element function of the output of the common prototype node. Specifically, the figure shows a computation graph CG2, which is obtained by annotating the diagonal striped nodes in the computation graph CG1 in the form of horizontal striped nodes as being implementable through a table lookup operation based on determining that the multiple inputs of the table lookup operation have a common prototype. It should be noted that whether this is the case for a node with multiple inputs depends on the structure of the graph. Therefore, the identification operation Ident can determine whether a node can be implemented as a lookup table not only based on the node itself but also based on other parts of the subgraph.

[0166] After the identification operation Ident, the fusion determination operation DFuse, 750 can be used to determine that one or more additional nodes of the computation graph can be fused into the table lookup operation for the nodes that have been determined to be implementable by the table lookup operation. This is illustrated in the figure by the computation graph CG3, in which the fusible subgraphs have been annotated. In this case, the fusible subgraph is a subgraph that starts with the outgoing edge of the common prototype node and ends at the node itself.

[0167] In this case, as illustrated in the figure, in the computation graph CG3, the identified subgraph can be replaced by a node representing a single table lookup operation. This can be a TLU type node (also as for Figure 4 After this, for example, the optimization may iteratively apply the identification Ident and the fusion determination DFuse until no more nodes can be fused.

[0168] After the nodes have been fused, you can perform the write operation Write, as for Figure 4 As discussed, the write operation Write may output a transformed representation of the computer program in which nodes are fused and implemented by a single table lookup, as known per se.

[0169] The figure also shows the sorting operation Ord, as also shown in Figure 4 As discussed, the sorting operation Ord can reorder the computation graph CG2 to merge at least one operation before the reshape node with at least one operation after the reshape node. This operation can be as for Figure 4 Implemented as discussed.

[0170] Specifically, the following algorithm provides a detailed embodiment of how to implement the identification operation Ident and the fusion determination operation DFuse. This embodiment uses a general pattern detection technique to find nodes that can be implemented as a table lookup operation and then fused. Specifically, in the described algorithm, UnwantedProperty can be configured to determine whether a given operation is a MIX or SHAPE operation. ConformsToFusingRules can be set to determine whether there is more than one TLU operation in the subgraph; whether the operation in the subgraph produces a tensor of a shape corresponding to the subgraph input tensor; and / or whether the operation in the subgraph uses constants with the same shape as the input tensor.

[0171] In the following algorithm, specifically, the identification operation Ident can be implemented by the functions FindClosestAncestorsWithoutUnwantedProperty and FindLowestFusingPointCandidate, and the fusion determination operation DFuse can be represented by the function CreateFusableSubgraphFromTo. Accordingly, if the operation in the operation graph belongs to the subgraph found by Algorithm FindSubgraphToFuse, it can be implemented in TLU mode, otherwise, it can be implemented in native mode.

[0172]

[0173]

[0174]

[0175]

[0176]

[0177]

[0178]

[0179]

[0180]

[0181] Figure 8 A block diagram of a computer-implemented method 800 for optimizing a computer program for an execution environment that supports table lookup operations is shown. The method 800 may correspond to Figure 1 However, this is not limiting, as method 800 may also be performed using another system, device, or apparatus.

[0182] The method 800 may include accessing (810) a computer program as a representation of a computation graph in an operation entitled "ACCESS COMPUTATION GRAPH". In the computation graph, a corresponding node may represent a corresponding operation. The method 800 may include, in an operation entitled "DETERMINE TLU IMPLEMENTATION", determining 820, for a corresponding node of the computation graph, whether the corresponding node can be implemented by a table lookup operation. The method 800 may include, in an operation entitled "DETERMINE FUSING", determining 830, for a node that can be implemented by a table lookup operation, that one or more additional nodes in the computation graph can be fused into the table lookup operation. The method 800 may include outputting 840 a transformed representation of the computer program in an operation entitled "OUTPUT". In the transformed representation, the node and the one or more additional nodes may be fused into a single table lookup operation.

[0183] Specifically, the method 800 may be a compiler method. Specifically, the method 800 may take source code in a compiled or interpreted programming language (e.g., Python, C++, etc.) as input, and compile the source code to determine a computation graph; and output a transformed representation in a representation executable by an execution environment, such as a homomorphic executable file for a FHE computing engine.

[0184] It should be understood that, generally speaking, Figure 8 The operations of method 800 may be performed in any suitable order (eg, serially, concurrently, or a combination thereof), but where applicable, are subject to a specific order being desired (eg, by input / output relationships).

[0185] The method can be implemented on a computer as a computer-implemented method, as dedicated hardware, or as a combination of both. Fig. 9 As illustrated in , instructions (e.g., executable code) for a computer may be stored on a computer readable medium 900, for example in the form of a series 910 of machine-readable physical marks and / or as a series of elements having different electrical (e.g., magnetic) or optical properties or values. The medium 900 may be transient or non-transitory. Examples of computer readable media include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Fig. 9 An optical disc 1100 is shown. Alternatively, the computer readable medium 900 may include data 910 representing a transformed representation of a computer program determined according to the methods described herein.

[0186] The following numbered clauses include contemplated and non-limiting embodiments:

[0187] 1. A computer-implemented method of optimizing a computer program for an execution environment supporting table lookup operations, wherein the method comprises:

[0188] - accessing the computer program as a representation of a computational graph, wherein respective nodes of the computational graph represent respective operations;

[0189] - for a corresponding node of the computational graph, determining whether the corresponding node can be implemented by a table lookup operation;

[0190] - for a node that can be implemented by a table lookup operation, determining one or more additional nodes in the computation graph that can be fused into the table lookup operation;

[0191] - outputting a transformed representation of said computer program, wherein said node and said one or more further nodes are fused into a single table lookup operation.

[0192] 2. A method according to clause 1, wherein the execution environment is a computation performed by homomorphic encryption supporting at least encrypted addition operations, encrypted scalar multiplication operations, and encrypted table lookup operations.

[0193] 3. The method of clause 1, wherein the table lookup operation is a hardware table lookup.

[0194] 4. A method according to any of the preceding clauses, wherein one or more operations from the first group of operations are never implemented through the table lookup operation; one or more operations from the second group of operations are implemented through the table lookup depending on the computation graph; and one or more operations from the third group of operations are always implemented through the table lookup operation.

[0195] 5. The method of clause 4, wherein the second group of types includes at least addition operations.

[0196] 6. A method according to clause 4 or 5, comprising replacing the node by a first subgraph representing a first implementation if it is determined that the node can be implemented by a table lookup operation; and replacing the node by a second subgraph representing a second implementation in other cases.

[0197] 7. The method of clause 6, wherein the second implementation is configured to quantize an input to an integer; apply an integer operation corresponding to the node to obtain an integer output; and dequantize the integer output.

[0198] 8. A method according to any preceding clause, comprising: based on determining that the multiple inputs have a common prototype, determining that a node having the multiple inputs can be implemented by a table lookup operation.

[0199] 9. A method according to any preceding clause, wherein an operation is applied to one or more input tensors and produces an output tensor, and wherein a table lookup operation is applied element-by-element to a single input tensor.

[0200] 10. The method according to clause 9 includes obtaining a reshaping node representing an operation of reorganizing elements of one or more input tensors without changing the values ​​of the elements of the one or more input tensors, and reordering the computational graph to fuse at least one operation before the reshaping node with at least one operation after the reshaping node.

[0201] 11. A method according to any preceding clause, wherein the transformed representation of the computer program does not include a table lookup operation applied to the output of a further table lookup operation.

[0202] 12. The method of clause 11, wherein the transformed representation of the computer program does not include a reshape operation applied to an output of a table lookup operation, wherein the reshape operation rearranges one or more inputs without changing their values.

[0203] 13. A method according to any preceding clause, wherein the computer program represents an evaluation of a machine learnable model, such as an artificial neural network, a generalized linear model, a decision tree or an ensemble model.

[0204] 14. A compiler system (100) for optimizing a computer program for an execution environment supporting table lookup operations, wherein the system comprises:

[0205] - a data interface (120) for accessing the computer program as a representation of a computational graph, wherein respective nodes of the computational graph represent respective operations;

[0206] - a processor subsystem (140), the processor subsystem being configured to:

[0207] - for a corresponding node of the computational graph, determining whether the corresponding node can be implemented by a table lookup operation;

[0208] - for a node that can be implemented by a table lookup operation, determining one or more additional nodes of the computation graph that can be fused into the table lookup operation;

[0209] - outputting a transformed representation of said computer program, wherein said node and said one or more further nodes are fused into a single table lookup operation.

[0210] 15. A transitory or non-transitory computer-readable medium (900) comprising data (910) representing:

[0211] - instructions which, when executed by a processor system, cause the processor system to perform a computer-implemented method according to any of clauses 1 to 13; and / or

[0212] - A transformed representation of a computer program determined according to the computer-implemented method of any of clauses 1 to 13.

[0213] Examples, embodiments or optional features, whether or not indicated as non-limiting, should not be construed as limiting the invention as claimed.

[0214] It should be noted that the embodiments mentioned above illustrate rather than limit the present invention, and those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference numerals placed in brackets should not be interpreted as limiting the claims. The use of the verb "comprise" and its variants does not exclude the existence of elements or stages other than those stated in the claims. The article "one" or "an" in front of an element does not exclude the existence of multiple such elements. When an expression such as "at least one" is placed before a series or a group of elements, it means that all elements or any subset thereof are selected from the series or group. For example, the expression "at least one of A, B and C" should be understood to include only A, only B, only C, both A and B, both A and C, both B and C, or all of A, B and C. The present invention can be implemented by means of hardware including several different elements, and by means of a suitably programmed computer. In a device claim that lists several devices, several of these devices can be implemented by the same hardware item. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A computer-implemented method (800) of optimizing a computer program for an execution environment supporting table lookup operations, wherein the method comprises: - accessing (810) the computer program as a representation of a computational graph, wherein respective nodes of the computational graph represent respective operations; - for a corresponding node of the computation graph, determining (820) whether the corresponding node can be implemented by a table lookup operation, including determining that a node having the plurality of inputs can be implemented by a table lookup operation based on determining that the plurality of inputs have a common prototype; - if it is determined that the node can be realized by a table lookup operation, replacing the node by a first subgraph representing a first realization; and in other cases replacing the node by a second subgraph representing a second realization; - for a node that can be implemented by a table lookup operation, determining (830) one or more other nodes in the computation graph that can be fused into the table lookup operation, and determining one or more additional nodes from a second subgraph of another node that can be fused into the table lookup operation; - outputting (840) a transformed representation of said computer program, wherein said node and said one or more further nodes are fused into a single table lookup operation.

2. The method (800) according to claim 1, wherein the execution environment is a computation performed by homomorphic encryption supporting at least encrypted addition operations, encrypted scalar multiplication operations, and encrypted table lookup operations.

3. The method (800) of claim 1, wherein the table lookup operation is a hardware lookup table.

4. A method (800) according to any preceding claim, wherein one or more operations from the first group of operations are never implemented through the table lookup operation; one or more operations from the second group of operations are implemented through the table lookup depending on the computation graph; and one or more operations from the third group of operations are always implemented through the table lookup operation.

5. The method (800) of claim 4, wherein the second group of types includes at least addition operations.

6. A method according to any preceding claim, comprising iterating the computation graph to establish for each operation whether a table lookup operation can be used for that operation.

7. The method (800) of any preceding claim, wherein the second implementation is configured to quantize an input to an integer; apply an integer operation corresponding to a node to obtain an integer output; and dequantize the integer output.

8. A method (800) according to any preceding claim, wherein an operation is applied to one or more input tensors and produces an output tensor, and wherein a table lookup operation is applied element-wise to a single input tensor.

9. A method (800) according to any preceding claim, comprising obtaining a reshape node representing an operation that reorganizes elements of one or more input tensors without changing the values ​​of the elements of the one or more input tensors, and reordering the computational graph to fuse at least one operation before the reshape node with at least one operation after the reshape node.

10. The method (800) of any preceding claim, wherein the transformed representation of the computer program does not include a table lookup operation applied to an output of a further table lookup operation.

11. The method (800) of any preceding claim, wherein the transformed representation of the computer program does not include a reshape operation applied to an output of a table lookup operation, wherein the reshape operation rearranges one or more inputs without changing their values.

12. The method (800) according to any preceding claim, wherein the computer program represents an evaluation of a machine learnable model, such as an artificial neural network, a generalized linear model, a decision tree or an ensemble model.

13. A compiler system (100) for optimizing a computer program for an execution environment supporting table lookup operations, wherein the system comprises: - a data interface (120) for accessing the computer program as a representation of a computational graph, wherein respective nodes of the computational graph represent respective operations; - a processor subsystem (140), the processor subsystem being configured to: - for a corresponding node of the computation graph, determining whether the corresponding node can be implemented by a table lookup operation, comprising: determining that a node having the multiple inputs can be implemented by a table lookup operation based on determining that the multiple inputs have a common prototype; - if it is determined that the node can be realized by a table lookup operation, replacing the node by a first subgraph representing a first realization; and in other cases replacing the node by a second subgraph representing a second realization; - for a node that can be implemented by a table lookup operation, determining that one or more additional nodes of the computation graph can be fused into the table lookup operation, and determining that one or more additional nodes of a second subgraph from another node can be fused into the table lookup operation; - outputting a transformed representation of said computer program, wherein said node and said one or more further nodes are fused into a single table lookup operation.

14. A transitory or non-transitory computer-readable medium (900) comprising data (910) representing: - instructions which, when executed by a processor system, cause the processor system to perform a computer-implemented method according to any one of claims 1 to 12; and / or - A transformed representation of a computer program determined by the computer-implemented method according to any one of claims 1 to 12.

Images