Method for optimizing computer programs for table lookup operations

Abstract

The invention relates to a computer-implemented method (800) of optimizing a computer program for an execution environment supporting table lookup operations. The computer program is represented as a computation graph, wherein respective nodes of the computation graph represent respective operations. For a respective node, it is determined whether the respective node can be implemented by a table lookup operation. For nodes that can be implemented by a table lookup operation, it is determined whether one or more further nodes of the computation graph can be fused into the table lookup operation. A transformed representation of the computer program is output, wherein the node and the one or more further nodes are fused into a single table lookup operation.

Description

Technical Field

[0001] This invention relates to a method and system for optimizing a computer program for an execution environment that supports table lookup operations. The invention also relates to a computer-readable medium. Background Technology

[0002] Homomorphic cryptography allows one party to execute a computer program (e.g., circuit evaluation) on encrypted data, which that party cannot decrypt. For example, input data and computation results can be received and returned in encrypted form. Intermediate data (e.g., the internal state of a computation) can also be encrypted. Even if the output of the computer program is returned in encrypted form, the output is expected to be the same or very close upon decryption, just as if the operation were performed on unencrypted data. Homomorphic encryption can be used to protect privacy in outsourced storage and computation. This allows data to be encrypted and outsourced to a cloud environment for processing and / or storage while remaining encrypted at all times. Specifically, there exist homomorphic cryptographic techniques that can, at least in principle, be used to compute any function of encrypted data. This technique is called "fully homomorphic encryption" (FHE).

[0003] Homomorphic encryption typically provides an execution environment that supports a relatively finite set of operations. More complex operations can then be implemented based on this set of supported operations. Supported operations typically include addition and subtraction of integer ciphertext; and multiplication of integer ciphertext by a known scalar. For example, homomorphic encryption can also support multiplication of encrypted integer ciphertext.

[0004] The execution environments provided by various homomorphic encryption techniques also support table lookup operations. Specifically, table lookup operations can be programmable bootstrapping (PBS) operations. Such operations can not only perform table lookup operations but also restore noise present in the ciphertext to a fixed level. Homomorphic encryption techniques that support PBS operations are also called TFHE-like homomorphic encryption schemes. This scheme is described in I. Chillotti et al., “Programmable bootstrapping enables efficient homomorphic inference of deep neural networks,” Cyber ​​Security Cryptography and Machine Learning (CSCML 2021), Lecture Notes in Computer Science, Vol. 12716, pp. 1-19, Springer, 2021 (included in this paper by reference). Generally, TFHE-like homomorphic encryption schemes can be based on lattice encryption, such as LWE or NTRU encryption.

[0005] While the computational efficiency of homomorphic encryption has improved significantly in recent years, it still incurs considerable overhead, especially when applied to relatively large amounts of data and / or relatively complex computations. In particular, the overhead of performing table lookups remains substantial compared to other operations such as homomorphic addition or scalar multiplication. Therefore, computer programs need to be optimized for efficient execution in homomorphic encryption execution environments that support table lookup operations.

[0006] Lookup table operations are also used in various other execution environments. For example, in hardware, lookup tables can be used to compute arbitrary functions. Such hardware lookup tables can be implemented, for example, using a single piece of RAM (Random Access Memory) or ROM (Read-Only Memory). Lookup tables can be used to accelerate computations involving particularly large amounts of data, such as in neural network accelerator circuits used to train or evaluate neural networks. For example, the activation function of a neural network can be implemented as a lookup table. Again, in this case, it is important to optimize the computer program to improve execution efficiency. Summary of the Invention

[0007] The aim is to optimize computer programs used in the execution environment to support table lookup operations, particularly by minimizing the number of table lookup operations required to execute the computer program in the execution environment.

[0008] According to a first aspect of the invention, a method and a corresponding system for optimizing a computer implementation of a computer program are provided, as defined in claims 1 and 14, respectively. According to another aspect of the invention, a computer-readable medium as defined in claim 15 is provided.

[0009] Various implementation schemes involve optimizing the computer program used to support the execution environment for table lookup operations. To this end, the computer program can be obtained as a representation of a computation graph. The corresponding nodes of the computation graph can represent the corresponding operations. In many cases, the computation graph is a tensor computation graph, where nodes represent operations applied to one or more input tensors and producing an output tensor, corresponding to the edges of the graph; however, it is also possible to represent computation graphs that only represent scalars.

[0010] This representation can be transformed in such a way that multiple nodes of the computation graph are implemented as a single table lookup operation (where, in the case of a tensor computation graph, a table lookup operation can represent the element-wise application of a table lookup). Jointly implementing multiple nodes as a single table lookup operation can be called "merging" the nodes (whether this merging is done explicitly, for example, by merging nodes of the computation graph, or by outputting a set of instructions for the execution environment based on determining that a node should be implemented as a single table lookup operation). Optimization can output a transformed representation of the computer program that has applied the merging.

[0011] While it is known to manually annotate (single-input, single-output) operations of a computer program as fusionable and then fuse these operations into a single table lookup operation—for example, fusion representing operations applied consecutively to corresponding single-variable functions of the same input—the inventors envision improving this fusion. For this purpose, a computation graph can be obtained. A computation graph can, in principle, represent any computation that can be performed by an execution environment. Given this graph, it can be automatically determined for its corresponding nodes whether a node can be implemented via a table lookup operation. This determination can be based not only on the node itself but also, at least, on the incoming edges of the nodes in the computation graph. Specifically, the determination can be based on whether the output of a node can be expressed as a single-variable (element-wise) function of the output of another node, for example, even if the node has multiple inputs.

[0012] For a node that has been determined to be operable through a table lookup operation, one or more other nodes in the computation graph can be incorporated into the table lookup operation; for example, nodes between other nodes and the node itself. In the transformed representation of the output, these other nodes can be incorporated with the original node.

[0013] Specifically, a mechanism can be provided that transforms a computation graph (e.g., the computation graph of a general neural network) into a form that incorporates univariate operations as much as possible, and uses lookup tables to compute the resulting operations. For example, univariate operations can include, but are not limited to, quantization, normalization, and activation functions.

[0014] Typically, by automating the determination of whether nodes can be implemented via a lookup operation, fusionable nodes that might not be found using manual annotations, such as based on the operation itself, can be identified. For example, a node representing an operation applied (elementally) to multiple (tensor) inputs can be implemented via a lookup operation, depending on whether the inputs themselves depend (elementally) on a common previous input. Therefore, whether a node can be implemented via a lookup can depend on the other nodes it connects to. Determining this manually is impractical, especially for large computations. Furthermore, while some heuristics are known to fuse nodes, these heuristics are inherently only effective in specific cases and may miss fusion opportunities.

[0015] Therefore, by automatically determining whether a node can be implemented via a table lookup operation, and then merging such a node with other nodes into a single table lookup operation, in particular by reducing the number of table lookups performed when executing a computer program, the efficiency of computer programs in a lookup-based execution environment can be improved (as illustrated elsewhere, this does not mean that all nodes determined to be implementable via table lookups are actually implemented in this way; for example, scalar multiplications that are not before or after other nodes that are implementable via table lookups can be implemented natively, for example, native operations using homomorphic encryption).

[0016] This is particularly beneficial in several setups. One such setup is where the table lookup operation is a hardware lookup table, such as in a hardware accelerator. For example, a hardware accelerator with low-bandwidth registers (e.g., low-bandwidth registers of up to 16 bits or even up to 8 bits). Here, using table lookups to compute single-variable operands can be faster than performing a series of arithmetic operations that compute the same result. Furthermore, the resulting transformation representation is faster than performing consecutive table lookups (once for each single-variable operation).

[0017] Furthermore, when applied to tensor computation graphs performing quantized neural network predictions, the provided technique avoids the need for repeated requantization of the input to each individual univariate operation if several such operations are fused. Therefore, this method eliminates the need for individual quantization of univariate operations, and improves the accuracy of the final result because quantization can lead to accuracy loss due to approximation.

[0018] The technology also allows for the design of simpler hardware accelerators, such as dedicated neural network hardware accelerators for quantizing neural networks, thereby avoiding the inclusion of more complex circuitry for normalization, quantization, and activation functions.

[0019] Another particularly advantageous setting of the provided technology is that of homomorphic encryption (e.g., fully homomorphic encryption (FHE)). In this setting, encrypted values ​​can be stored as ciphertext consisting of one or more numerical values, such as LWE or NTRU encryption. The size of such ciphertext and the computational efficiency of the cryptographic operations defined on it can depend on the bit width of the input to the table lookup operator, meaning that using a lower bit width (e.g., a maximum of 8 bits or a maximum of 12 bits) for computation is advantageous.

[0020] In this setup, by applying the provided techniques, a transformation representation of the computer program can be obtained, which is optimized for execution through homomorphic encryption as follows. First, the number of lookup table operations can be minimized. In fact, lookup table operations are computationally expensive, and redundant LUTs further increase this cost. Furthermore, homomorphic encryption typically operates on encrypted integer values ​​and therefore often uses quantization. By avoiding encoding the quantizer, activation, and standard layers separately into the TLU, correlated quantization errors caused by individual quantization errors in each LUT are avoided. Moreover, applying TLU operations generally introduces small errors or noise into the ciphertext, meaning the TLU result may deviate from 1 with a very small probability. Therefore, chaining multiple TLUs increases the probability of error. The provided techniques reduce the number of TLUs used, thereby also reducing the probability of error.

[0021] The corresponding nodes in the computation graph can represent the corresponding operations. Optimization can be performed in such a way that operations in the first set of operations are never implemented via table lookups. This is especially relevant when the computation graph is defined on tensors, for example, in cases where operations (e.g., convolution) are applied to one or more input tensors in a non-elemental manner. Such operations can be implemented based on the native operations of the execution environment. (In this case, even if the operation itself is not implemented via a table lookup from a functional perspective of the computation graph, it is still possible that preprocessing and / or postprocessing (e.g., (inverse)quantization) uses a table lookup. This is explained in more detail elsewhere in this specification.)

[0022] Nodes representing operations from the second group of operations can be implemented as lookup tables depending on the computation graph (e.g., depending on their incoming and / or outgoing edges). For example, a native implementation of the operation might be available in the execution environment for these nodes, but these nodes can still be merged into a table lookup with other nodes if this makes the overall computation more efficient. This is the case, for example, for addition or scalar multiplication. Nodes representing operations from the third group of operations can always be implemented by table lookup operations, for example, because no native implementation of the operation is available in the execution environment, such as an activation function.

[0023] Optionally, optimization may involve substitution operations, where if a node is determined to be implementable via a lookup operation, it is replaced by a first subgraph representing a first implementation; and otherwise, it is replaced by a second subgraph representing a second implementation. The substitution operation can be performed after determining which nodes are implementable via a lookup operation but before determining that such nodes can be merged with other nodes. Specifically, nodes to be merged may include one or more nodes from a second subgraph of another node, for example, a node corresponding to the inverse quantization operation of the previous node or the quantization node of the next node. Thus, optimization can be combined with transformations that convert high-level computational graphs into low-level computational graphs, as discussed in more detail elsewhere in this specification.

[0024] Optionally, the second implementation used for the substitution operation can be configured to quantize the input (e.g., floating-point input) to integers; apply integer operations corresponding to the node to obtain an integer output; and dequantize the integer output (e.g., dequantize to floating-point). For example, the optimizer's input graph can allow different types of input for a particular operation (e.g., supporting both integer and floating-point inputs), where the substitution operation can cause the corresponding operation to only allow the corresponding type of input (e.g., only integer addition is supported, and the substitution operation includes quantization if the preceding output is not an integer). Depending on the inputs and / or outputs supported by the node, the subgraph can include nodes corresponding to quantization and / or dequantization.

[0025] Optionally, for a node with multiple inputs (e.g., multiple tensor inputs, in the case that the computation graph is a tensor computation graph), the node can be determined to be implemented by a table lookup operation based on the determination that the multiple inputs share a common prototype in the graph. For example, the (element-wise) addition of two (tensor) inputs, which itself represents a (element-wise) function applied to a common predecessor edge, can itself be represented as an element-wise function applied to a common predecessor edge, and therefore can be determined to be implemented by a table lookup operation. Since this determination cannot be performed manually, especially for large computations, and particularly based on the structure of the computation graph, this is a particular embodiment in which it is beneficial to automatically determine whether a node can be implemented by a table lookup.

[0026] A computation graph may include one or more reshaping nodes. Such reshaping nodes can represent operations that rearrange the elements of one or more input tensors without changing the values ​​of the elements of one or more input tensors, such as transpose, reshaping, flattening, etc. In this case, the computation graph can be reordered to merge at least one operation before a reshaping node with at least one operation after a reshaping node. For example, swapping a reshaping node with its preceding or following node may result in two adjacent nodes representing element-wise operations, which can be implemented using lookup tables and then merged into a single table lookup. When the optimization involves substitution operations as described herein, reordering is preferably performed after the substitution operation, but in principle it can also be applied before the substitution operation.

[0027] Optionally, the transformation representation of the computer program does not include table lookup operations applied to the output of additional table lookup operations. Instead, such subsequent table lookups can be merged, resulting in a more efficient implementation of the computer program. Optionally, the transformation representation of the computer program does not include reshaping operations applied to the output of table lookup operations. By swapping the TLU and the reshaping whenever a reshaping is applied to the output of the TLU, it can be ensured that there are no TLUs that have reshaping only between them; in other words, such TLUs are grouped together in the computation graph. This allows TLUs to be merged, for example, by selecting them as connected components of the graph.

[0028] Alternatively, the computer program can represent the evaluation of machine-learnable models, such as artificial neural networks, generalized linear models, decision trees, or ensemble models. This type of computation can be well described by (tensor) computation graphs, can be relatively complex, and can be repeatedly applied to different data, making it important to optimize the computer program for their efficient evaluation.

[0029] Typically, the input computation graph can be defined in a format and / or using a set of supporting operations independent of a specific execution environment. For example, the input computation graph can be a high-level graph defined, for example, in the ONNX (Open Neural Network Exchange) format. Transformation representations can be low-level representations specific to the optimized execution environment; for example, it can be an executable format that can be executed by the execution environment, or it can be a computation graph using a restricted set of nodes and / or edges supported by the execution environment. For example, substitution operations can be used to implement high-level operations based on low-level operations. For example, optimizers can be used in cryptographic computation systems to allow efficient execution of machine learning models or other types of computations specified based on high-level representations using homomorphic encryption.

[0030] Those skilled in the art will understand that two or more of the above-described embodiments, implementations, and / or optional aspects of the present invention can be combined in any manner deemed useful.

[0031] Those skilled in the art can make modifications and variations to any system and / or any computer-readable medium based on the present description, the modifications and variations corresponding to the modifications and variations of the corresponding computer-implemented methods described. Attached Figure Description

[0032] These and other aspects of the invention will become apparent and elucidated with reference to the embodiments described by way of example in the following description and with reference to the accompanying drawings, wherein:

[0033] Figure 1 The compiler system is shown;

[0034] Figure 2a An embodiment of the encrypted computing system is shown;

[0035] Figure 2b An embodiment of a device that supports a hardware lookup table is shown;

[0036] Figures 3a to 3d A detailed embodiment of the computation graph is shown;

[0037] Figure 4 Examples of how to optimize a computer program are shown;

[0038] Figures 5a to 5c A detailed embodiment of the computational graph of the transformation is shown;

[0039] Figures 6a to 6b A detailed embodiment of the reordering computation graph is shown;

[0040] Figure 7 Examples of how to optimize a computer program are shown;

[0041] Figure 8 The method implemented by a computer is shown;

[0042] Figure 9 A computer-readable medium containing data is shown.

[0043] It should be noted that these diagrams are illustrative only and are not drawn to scale. Components in the diagrams that correspond to those already described may have the same reference numerals. Detailed Implementation

[0044] Figure 1 A compiler system 100 is shown. System 100 can be used to optimize the execution environment for supporting table lookup operations (e.g., for...). Figure 2a The homomorphic encryption computation illustrated herein or used for, for example Figure 2b A computer program (the hardware implementation illustrated in the example).

[0045] System 100 may include a data interface 120 for accessing data 040 that represents a computer program as a computation graph. Corresponding nodes in the computation graph may represent corresponding operations. For example, the input computation graph may include at least 50, at least 250, or at least 10,000 nodes.

[0046] For example, similarly Figure 1 As illustrated, data interface 120 may be comprised of data storage interface 120, which can access data 040 from data storage 021. For example, data storage interface 120 may be a memory interface or persistent storage interface, such as a hard disk or SSD interface, or a personal area network interface, local area network interface, or wide area network interface (such as Bluetooth, Zigbee, or Wi-Fi interfaces, or Ethernet or fiber optic interfaces). Data storage 021 may be internal data storage of system 100 (such as a hard disk drive or SSD) or external data storage (such as network-accessible data storage). In some embodiments, the corresponding data may be accessed from different data storage devices (e.g., via different subsystems of data storage interface 120). Each subsystem may be of the type described above for data storage interface 120.

[0047] System 100 may further include a processor subsystem 140, which may be configured to, during operation of system 100, determine whether a given node in a computation graph can be implemented via a table lookup operation. Processor subsystem 140 may also be configured to, for nodes that can be implemented via a table lookup operation, determine whether one or more other nodes in the computation graph can be incorporated into a table lookup operation. Processor subsystem 140 may also be configured to output a transformation representation of a computer program, wherein the nodes and one or more other nodes are incorporated into a single table lookup operation.

[0048] System 100 may also include an output interface for outputting a transformed representation of a computer program. In this embodiment, the output interface is constituted by a communication interface 180, through which the transformed representation 126 can be output to another system, such as an encryption device that performs encryption calculations based on the output representation. The communication interface 180 can communicate internally with the processor subsystem 140 via data communication 125. The communication interface 180 can be arranged for direct communication with other systems, for example, using a USB, IEEE 1394, or similar interface. As illustrated, the communication interface 180 can also communicate via a computer network 010 (e.g., a wireless personal area network, the Internet, an intranet, a LAN, a WLAN, etc.). For example, the communication interface 180 may include connectors, such as wireless connectors, Ethernet connectors, Wi-Fi, 4G or 4G antennas, ZigBee chips, etc., to be suitable for a computer network. The communication interface 180 may also be an internal communication interface, such as a bus, API, storage interface, etc.

[0049] However, other methods of implementing the output interface are also possible. For example, the output interface may be constituted by a data interface 120, which in these embodiments is an input / output ('IO') interface, through which the transformed representation determined can be stored in the data memory 021. In some embodiments, the output interface may be separate from the data storage interface 120, but it can typically be of the type described above for the data storage interface 120.

[0050] Generally speaking, each system described in this specification (including but not limited to) Figure 1 System 100 can be implemented as a single device or apparatus (such as a workstation or server) or implemented within a single device or apparatus (such as a workstation or server). The device may be an embedded device. The device or apparatus may include one or more microprocessors executing appropriate software. For example, the processor subsystem of the corresponding system may be embodied by a single central processing unit (CPU), or by a combination or system of such CPUs and / or other types of processing units. The software may have been downloaded and / or stored in corresponding memory, such as volatile memory like RAM or non-volatile memory like Flash. Alternatively, the processor subsystem of the corresponding system may be implemented in the device or apparatus in the form of programmable logic (e.g., as a field-programmable gate array (FPGA)). Generally, each functional unit of the corresponding system may be implemented in the form of a circuit. The corresponding system can also be implemented in a distributed manner, such as involving different devices or apparatuses, such as distributed local servers or cloud-based servers.

[0051] Figure 2aA detailed, but non-limiting, embodiment of a cryptographic computation system 199 is shown. System 199 is configured to perform cryptographic computations using homomorphic encryption (e.g., fully homomorphic encryption).

[0052] The cryptographic computing system 199 may include a cryptographic system 112 configured to perform cryptographic computations using homomorphic encryption. The cryptographic computing system 199 may also include a compiler system 111 configured to determine a representation of a computer program optimized for the execution environment provided by homomorphic encryption. For example, the compiler system 111 may be based on… Figure 1 System 100. Encrypted computing system 199 may further include data provider device 113. The corresponding systems 111, 112, and 113 in this embodiment can be generally based on... Figure 1 The hardware configuration of device 100, for example, may include, for example, Figure 1 The processor system, data and / or communication interface in the system.

[0053] System 112 can be configured to receive encrypted data items from data provider 113. At least one or more data items can be received in encrypted form. One or more additional data items can be received in plaintext format. System 112 can be configured to receive key material (e.g., bootstrap keys and / or key switching keys) for performing cryptographic calculations, for example, from a data provider system or a key generation system.

[0054] System 112 can perform cryptographic calculations on received data items, and may also perform cryptographic calculations on stored data items. Interestingly, the device can perform calculations on encrypted data without decrypting the data, for example, without converting the encrypted data items into plaintext format. System 112 can perform cryptographic calculations based on a transformation representation determined by compiler system 111.

[0055] The compiler system 111 can be combined with the data provider system 113, in which case the data provider system 113 can, for example, provide a transformed representation of the computer program along with data items for executing the computer program. Alternatively, the compiler system 111 can be combined with, or be invoked by, the cryptographic system 112 to perform optimizations, in which case the data provider system 113 can, for example, provide a computation graph with applied optimizations, along with data items on which the computer program is to be executed. For example, the computation graph can adopt a format independent of the cryptographic computation execution definition (e.g., ONNX format), and may not even be constructed for the use of homomorphic encryption. However, due to optimization, the computer program can be executed efficiently using homomorphic encryption, particularly reducing the number of table lookup operations.

[0056] Although not shown in the figure, the cryptographic computation system 199 may include multiple cryptographic systems, such as two, three, or more. Cryptographic computation can be distributed among the multiple cryptographic systems. The cryptographic systems can exchange intermediate computation results (typically encrypted). Each cryptographic device can be implemented as cryptographic system 112 and can perform the cryptographic operations described herein.

[0057] Generally, various types of homomorphic encryption that support table lookup operations can be used. In many cases, homomorphic encryption has integer and / or polynomial plaintext, which is defined as taking the modulus of a certain number (in the literature, the modulus integer is sometimes referred to by a correspondence). Equivalently described as a toroidal element.

[0058] In addition to supporting encrypted table lookup operations, homomorphic encryption typically supports a set of additional operations that do not involve table lookups. Such operations can be called native operations of homomorphic encryption and may include, for example, addition and / or encrypted scalar multiplication. These operations are generally computationally more efficient than table lookups. Therefore, native implementation of these operations is generally preferable to table lookups when possible. Interestingly, however, in some cases, these operations can still be incorporated into preceding and / or subsequent table lookup operations as described in this paper, resulting in overall efficiency improvements.

[0059] Specifically, the computation can use homomorphic lattice encryption, such as LWE encryption and / or NTRU encryption. For such encryption, how to implement various operations (including encrypted table lookup operations) is itself known. For example, homomorphic encryption could be a "TFHE-class" homomorphic encryption because the table lookup operation is implemented as a so-called programmable bootstrapping operation. This operation can perform a table lookup on the input ciphertext based on blind rotation. Blind rotation can decrypt the input ciphertext in the monomial exponent. By multiplying the encrypted monomial by a polynomial representing the lookup table, an encrypted polynomial with fixed coefficients as the output of the lookup table can be obtained. For example, the coefficients of this polynomial can be extracted, or the encrypted polynomial can be further processed. This type of TFHE programmable bootstrapping is known for its use in LWE encryption, as seen in papers such as "Programmable bootstrapping enables efficient homomorphic inference of deep neural networks" by I. Chillotti et al. and "FHEW: bootstrapping homomorphic encryption in less than a second" by L. Ducas et al., EUROCRYPT 2015; and also for NTRU encryption, as seen in papers such as "FINAL: Faster FHE instantiated with NTRU and LWE" by C. Bonte et al., https: / / eprint.iacr.org / 2022 / 074, and "NTRU-ν-um: Secure Fully Homomorphic Encryption from NTRU with Small Modulus" by K. Kluczniak, https: / / eprint.iacr.org / 2022 / 089.

[0060] A Programmable Bootstrap (PBS) can take ciphertext as input and output ciphertext of the same message, or a function of that message (in other words, a lookup table applied to the message) with noise independent of the input. PBS can include blind rotations that evaluate the homomorphic decryption of the input ciphertext in a polynomial exponent. Specifically, a programmable bootstrap can take the ciphertext of an encrypted message m, a bootstrap key, and the encryption of a lookup table L as input. The programmable bootstrap can output an encryption of message L[m] with a fixed noise level. Blind rotations are also known as accumulator updates.

[0061] For example, a cryptographic computing system 199 can be configured to train machine learning models, such as image classifiers or medical models, without requiring the cryptographic computing device to have access to plaintext data items. Neural networks can be trained. For example, backpropagation can be performed on the input data. The resulting model parameters can be returned to an entity possessing the decryption key. By sending data to a cloud provider, this allows multiple providers of medical data to aggregate their data. The cloud provider then returns the model parameters without requiring access to the plaintext data. The cryptographic key can be equal to the decryption key.

[0062] After the model is trained, the encrypted computing system 199 can be used to provide the model (e.g., for use with medical data). This can be done with plaintext model parameters or encrypted model parameters, in both cases using encrypted data (e.g., encrypted input data, intermediate data, and data from the input). Using plaintext model parameters is generally more efficient. In both cases, the system's effect is to perform computations (e.g., image classification, such as medical image classification) without the computer knowing the plaintext data items. For example, a mammogram can be used to assess cancer without the image appearing in plaintext at the cryptographic system 112, and without any combination of cryptographic system 112 or such devices knowing the outcome of the cancer assessment. From a privacy perspective, running a plaintext model on encrypted privacy-sensitive data is acceptable, while running it on plaintext privacy-sensitive data is unacceptable.

[0063] Figure 2b A detailed, but non-limiting, embodiment of a device 200 supporting a lookup table is shown. In this embodiment, the lookup table is implemented in hardware.

[0064] Specifically, the figure illustrates a processor subsystem 240 in which a hardware lookup table (LUT) 241 is implemented. The processor subsystem 240 can be configured as follows: Figure 1 The processor subsystem 240 is described in the description. Specifically, the processor subsystem 240 can be implemented as programmable logic (e.g., as a field-programmable gate array (FPGA)). The processor subsystem 240 can also be implemented wholly or partially as a so-called application-specific integrated circuit (ASIC), such as an integrated circuit (IC) customized for its specific purpose. For example, the circuit can be implemented in CMOS, for example using a hardware description language (such as Verilog, VHDL, etc.).

[0065] As shown in the figure, the lookup table (LUT) can access the input IN, 231, and determines the output OUT, 233 based on the data 232 stored in the memory location corresponding to the input IN. The data 231 to 233 shown here are stored in a separate memory location 240, for example, corresponding to... Figure 1The memory of a is 021. For example, the lookup table can be implemented using a single piece of RAM (Random Access Memory) or ROM (Read Only Memory). Data 231 to 233 can also be accessed by the lookup table LUT in different ways, such as as register values ​​or via a bus.

[0066] As illustrated in the figure, the output OUT can directly correspond to the data stored at the memory location indicated by the input IN. However, alternative implementations using hardware lookup tables are also possible. For example, instead of the output, the data 232 could represent the parameters of a function that is computed relatively efficiently. For instance, according to known techniques, the high-order bits of the input value IN can be used to look up two values, and then the low-order bits of the input IN can be used to interpolate the two values ​​to obtain the output OUT.

[0067] As a specific embodiment, device 200 can be used to perform neural network inference or training. For example, processor subsystem 240 can be a neural network accelerator, such as dedicated circuitry optimized for computations performed by a neural network.

[0068] Figures 3a to 3d Detailed embodiments of computational graphs that can be optimized using the techniques described herein are shown. These embodiments illustrate the application of the invention in computer programs representing the evaluation of machine-learnable models (artificial neural networks in this embodiment).

[0069] Specifically Figure 3a The computational diagram illustrates the evaluation of neurons in a neural network. Neural networks are well known to be used as statistical models that learn from training data to detect or generate patterns. Such neural networks can include multiple neurons, for example, at least 1000 or at least 10000 neurons. A neuron can perform a weighted combination 312 of multiple inputs 310 according to corresponding weights 311, and an activation function 313 can be applied to the result. For example, the inputs 310 of a neuron can be raw data or activations of previous neurons. In this embodiment, the neuron combines m-1 inputs with weights and a bias b. k Combination. (This figure shows the corresponding bias w) k0 =b k The constant input x0 = +1; nodes / edges with such constant values ​​are generally not considered part of the computational graph processed in this paper.

[0070] The activation function 313 can be a univariate function, for example, with a single variable input (but may be constant). In practice, the activation function can determine whether a neuron is activated, for example, whether it has a high output value 314. Therefore, it can effectively determine whether a neuron is important to the network's input during the prediction process, for example, using simpler mathematical operations. For example, the activation function can be a rectified linear unit, a sigmoid, or an exponential linear unit.

[0071] Mathematically speaking, the computation of a neuron in a neural network can be represented as follows:

[0072] y=φ(∑ i w i x i +b)=φ(s)

[0073] The dot product 312 between weight 311 and input 310 can also be called the neuron accumulator s.

[0074] In a neural network, neurons can be arranged into consecutive groups called layers. For example, a neural network may include at least 10 layers or at least 100 layers. For instance, a neural network may include one or more convolutional layers and / or one or more dense layers (also called linear layers). Regardless of the type of layer, its computation may involve the computation of neuron accumulators.

[0075] s=∑ i w i x i +b,

[0076] Where w i 311 is the learning weight, x i 310 is the input, and b is the bias value. For example, in the case of a convolutional layer, i = (c, y, x) can be a tuple of iterators on the number of channels, height, and width of the input tensor 310.

[0077] In addition to weights 312 and activation function 313, the values ​​of the neural network may undergo various other operations applied to the output 314 before being used as inputs in subsequent neural network layers, such as: normalization, especially batch normalization; scaling; and / or nonlinear operations.

[0078] Figure 3aThe embodiment illustrates a scalar computation graph, in which edges 310 and 314 can represent scalar values, and nodes 312 and 313 can represent operations applied to the scalar values. However, in many other cases, the computation graph can be a tensor computation graph. In this case, the nodes of the graph can represent operations applied to one or more input tensors and produce output tensors, corresponding to the incoming and outgoing edges of the nodes, respectively.

[0079] Generally, a tensor X can represent a set of values ​​indexed by a set of m indices. Here, m is also called the dimension of the tensor. Scalar values ​​can be represented by tensors with dimension m = 0. Vectors can be represented by tensors with dimension m = 1. Matrices can be represented by tensors with dimension m = 2. A tensor computation graph can be defined as having at least one edge representing a tensor with dimension m > 0. For example, a tensor computation graph can include one or more edges representing tensors with at least two dimensions, and / or one or more edges representing tensors with at least three dimensions. For example, the input and output of a neural network layer can be represented as a two-dimensional tensor (for one-dimensional data with multiple channels); a three-dimensional tensor (for two-dimensional data with multiple channels); or a four-dimensional tensor (for three-dimensional data with multiple channels).

[0080] Tensor computation graphs can represent a series of operations performed on data stored in tensors. Tensors are a generalization of numerical matrices to dimensions two or more. In various implementations, tensor computation graphs can be viewed as directed and acyclic. Typically, a tensor computation graph can represent an ordered set of operations performed on tensors. Operations can have inputs that are computed by other operations, or they can be the input of the entire computer program. Operations can also involve constants. For the purposes of this specification, the inputs to operations refer to non-constant inputs.

[0081] For example, the tensor computation graph described in this paper can represent the evaluation of neural networks. However, it is well known that tensor computation graphs can also be used for other types of computation, such as in stream processing systems or databases.

[0082] Generally, computation graphs can represent computations at different levels of abstraction. For example, an activation function can be represented as a single node in a computation graph, and then that single node can be implemented in code as a series of basic tensor operations; or a series of more basic tensor computation graph operations, such as 'addition', 'subtraction', etc. The provided techniques can, in principle, be applied to graphs at various levels of abstraction, such as those related to... Figure 4 and Figure 7 As discussed further, as a specific embodiment, the provided technique can be applied to graphs in the ONNX (Open Neural Network Exchange) format. This is a format for computing graphs using tensors with a given set of operators.

[0083] For example, Figures 3b to 3d Tensor computation graphs representing computer programs that can be used as part of neural network evaluation are shown. These graphs provide a visual representation of tensor computation graphs as specified in the ONNX format. As illustrated in these graphs, nodes in a tensor computation graph can represent basic operations such as arithmetic operations like addition and subtraction, division, multiplication, convolution, or matrix multiplication. In the computation graph, non-constant inputs to operations are represented as incoming edges, while constant inputs are contained within the operation nodes themselves.

[0084] Specifically Figure 3b and Figure 3c A computational graph representing quantization is shown. Quantization can be used for neural network evaluation. Neural networks can typically represent values ​​such as inputs, activations, and / or weights as floating-point values. However, in various cases, neural network operations can be applied to integer values. This could be for computational performance reasons or to evaluate neural networks under encryption.

[0085] Quantization converts floating-point values ​​to integers. Floating-point numbers can have fine resolution for representing real numbers, but their bit length is limited. Furthermore, the resolution of floating-point numbers can be inversely proportional to the absolute value of the number. When quantizing neural networks to use integers to improve computational performance, these integers may have a low bit length, such as a maximum of 12 bits, a maximum of 8 bits, or a maximum of 6 bits. Various quantization techniques exist designed to perform quantization with minimal quantization error.

[0086] For example, a subgraph of a computation graph can represent uniform quantization. In uniform quantization, discrete values ​​can be assigned to intervals of equal size on the real axis. This is achieved by... Figure 3b As illustrated, in this embodiment, uniform quantization is applied element-wise to the input tensor 320, producing an output tensor 321, which is then input into a matrix multiplication operation 322, such as that of a linear layer of a neural network. Specifically, the computation performed in the illustrated calculation can be mathematically represented as follows:

[0087]

[0088] Where q x It is the scaling factor, zp x It is midnight, and and These are the limits that restrict the range of real numbers quantized into integer values. For example, the values ​​of these parameters can be obtained by calculating them from a representative set of data.

[0089] The uniform quantization shown can be canceled by performing an inverse quantization calculation (not shown in the diagram). This allows retrieval of a quantized floating-point value that approximates the initial original value. Inverse uniform quantization can be mathematically represented as:

[0090] x f′ =(x q -zp x )*s f .

[0091] Figure 3c The computational graph of non-uniform quantization is shown. In non-uniform quantization, the intervals mapped to the corresponding integers do not all have the same size. Mathematically, non-uniform quantization can correspond to computation...

[0092]

[0093] in

[0094]

[0095] and β i These are the endpoints of each interval. Here, the first interval is [-∞, β0], the second interval is [β0, β1], and the last is...

[0096] In the illustrated embodiment, non-uniform quantization is applied to tensor 330 of an n = 2-bit integer. In this embodiment, the quantized output 331 is input to a convolution operation 332, such as the convolution operation 332 corresponding to a convolutional layer in a neural network. The β above... i The value corresponds to the value in the child node of the graph.

[0097] Figure 3d A computational graph of the normalization calculation is shown. This embodiment illustrates batch normalization in a two-dimensional case. The calculation can be mathematically represented as follows:

[0098]

[0099] Where y is the output tensor 344, x is the input tensor 341, β and γ are the learned parameters, and μ c and σ c It is (e.g., obtained by computing it on a set of representative input tensors) the mean and standard deviation of channel c.

[0100] In this embodiment, batch normalization is performed on the output 341 of the matrix multiplication (MatMul) layer. The division operation 343 in this embodiment represents the division between the central input x-μ and the standard deviation 342 (here expressed as the square root of the variance: σ = Var). 0.5The standard deviation 342 is a division between two variables. Although shown in this embodiment as an edge representing the variable input of division 342, it is more typically a constant incorporated into division 342 and therefore may not be represented as an edge in the computation graph.

[0101] In addition to batch normalization, the computer programs described herein may also include tensor normalization (e.g., using a single μ and σ for each tensor), layer normalization, scaling to 0-1, etc.

[0102] Figure 4 Detailed, but non-limiting, embodiments of optimizing a computer program are illustrated. In this embodiment, optimization involves implementation operations (also known as substitution operations) that transform a high-level computation graph into a low-level computation graph. In this embodiment, nodes of the high-level graph are determined as to whether they can be implemented via table lookups; the implementation operation is based on this determination; and nodes of the low-level graph are merged into a single table lookup operation.

[0103] Specifically, the figure shows a computational graph CG1,411 representing the computer program to be optimized, for example, as per [reference to...]. Figures 3a to 3d The discussion focuses on the corresponding nodes in the computation graph, which represent corresponding operations. Specifically, the computation graph GC1 can be a tensor computation graph where corresponding operations are applied to one or more input tensors and produce an output tensor. For example, the computation graph CG1 can be represented in ONNX format, such as a directed acyclic graph of tensor operations referenced from ONNX (which can be found in...).

[0104] (Originally obtained from https: / / github.com / onnx / onnx / blob / 55d3b80f3ea1261f3f74f2e35844c204de85835a / docs / Operators.md and incorporated into this article via citation).

[0105] Optimization can support a variety of operations. Specifically, global group operations can include at least addition, multiplication by a constant, and multiplication between two non-constant numbers. Global group operations can also include at least one non-linear element-wise operation, such as activation function application, numerical comparison, etc. Generally, several groups of operations that may occur in the tensor computation graph CG1 can be distinguished. Nodes representing operations from the first group may never be implemented via table lookup operations; nodes representing operations from the second group may be implemented via table lookup operations depending on the computation graph at hand; and nodes representing operations from the third group can always be implemented via table lookup operations. Specifically, nodes can be identified as belonging to a certain type based on the type of operation and the input of the operation. It is worth noting that nodes representing a certain operation (e.g., addition) may have different types depending on the context of the graph (e.g., depending on its input), as discussed in more detail below. Specifically, the following types can be defined.

[0106] The first type of operation is a Mixed Operation (MIX). A Mixed Operation can be defined as an operation that modifies the value of its input tensor but is not applied element-wise to its input. Specifically, a Mixed Operation can optionally combine different values ​​of a single input tensor from the operation by involving additional constant tensors. For example, the operation could be a neuron accumulator operation, such as convolution or matrix multiplication with constant tensors (e.g., for implementing dense neural network layers). Alternatively, a Mixed Operation can combine values ​​from two different graph input tensors, or values ​​from tensors produced by different MIX operations. For example, the operation could be addition or concatenation of two graph inputs, or arithmetic or concatenation operations applied to tensors that are the outputs of two MIX operations. In various implementations, determining the node representing the Mixed Operation is not achieved through table lookup operations (although table lookup operations can be used for preprocessing and postprocessing of the inputs and outputs of the operation, such as through quantization). For example, a given set of operations (e.g., convolution or dot product) can be determined as a MIX type regardless of the other nodes in the computation graph, and therefore can be achieved without table lookup.

[0107] The second type of operation is the Univariate Operation (UNI). A univariate function can be applied element-wise to its input tensor, returning an output tensor of the same size (but potentially with different value types). For example, the operation could be activation, normalization, quantization, or arithmetic operations with scalar constants. Note that a univariate computation applied to a tensor is univariate relative to the tensor cell (scalar). Specifically, different functions can be applied to different cells within a tensor. Nodes identified as univariate operations can be determined through table lookup operations. Specifically, as illustrated below, even with multiple incoming edges in the computation graph, a node can still be identified as a UNI type.

[0108] Interestingly, there can be a set of operations for which nodes can be identified as either MIX or UNI type operations, depending at least on their inputs. For example, this set of operations could include addition and / or multiplication. For instance, when the input type of such an operation is UNI, the operation itself can also be identified as UNI, depending on the operation. For example, an Add operation can be UNI if one input is produced by a single-variable operation and the other is a constant, or if the input is produced by a UNI operation computed on the same single input. For example, an Add operation can be MIX if the input is produced by UNI operations on different inputs. Therefore, operations from this set can be implemented via table lookup or not, depending on the other nodes in the graph. Specifically, whether an operation is a MIX or UNI operation can be determined based on the preceding operation. For example, if the preceding operation is a UNI operation (e.g., sigmoid(x)), then addition x+1 can be determined to be a UNI operation, and if the preceding operation is a MIX operation (e.g., conv(x,w)), then addition x+1 can be determined to be a MIX operation. In some implementations, for simplicity, constant addition is always determined to be a UNI operation.

[0109] For various univariate operations, this might be the case where no native implementation is available in the execution environment, and they are always determined to be of type UNI and therefore implemented via lookup tables. For example, depending on the execution environment, this might be the case for nonlinear functions (such as activation functions, sine functions, etc.).

[0110] The third type of operation is the SHAPE operation. Such operations can reorganize the elements of one or more input tensors (e.g., by reordering values ​​or changing tensor metadata such as shape). The values ​​in the tensor can remain unchanged. Examples of such operations can include transpose, reshape, flatten, etc. Since such operations may not affect the values, lookup tables are not typically used for implementation.

[0111] The identification operation Ident, 420 can be applied to computation graph CG1 to determine whether a given node can be implemented via a table lookup operation, specifically by determining whether the node is of type UNI as discussed above. This may result in an updated computation graph CG2, 412, where the given nodes are marked as either implementable via a table lookup or not. Such marking is also called an execution mode, or as a CanFuse marker. A node that can be implemented via a table lookup can be said to have a TLU (table lookup) execution mode.

[0112] Generally, the identifier `Ident` can process the tensor operation graph CG1 and determine the execution mode of its corresponding operations by: (a) iterating over the operation graph CG1, which can typically be performed in any order, regardless of the number of iterations required; (b) evaluating certain compatibility conditions for each corresponding operation to confirm whether the TLU execution mode is applicable to that operation; and (c) assigning the execution mode to the corresponding operation of the graph. This is illustrated in the graph by the diagonal stripes of several nodes in the identified result CG2 to indicate that they have TLU execution modes.

[0113] Interestingly, for a node in graph CG1 with multiple inputs, identification might involve it being implemented via a table lookup operation. That is, it can be determined that multiple inputs share a common prototype from which the node's output can be computed as an element-wise function. This makes it possible to merge the node with preceding nodes (and possibly subsequent nodes) into a single table lookup, resulting in more efficient execution overall in the execution environment.

[0114] As a detailed embodiment, an embodiment of an algorithm is now provided to determine which operations can be fused, i.e., for the operation at the corresponding index i in the computation graph, the algorithm can set CanFuse[i] to the corresponding execution mode: either in the TLU (e.g., an operation of type UNI) or as a basic operation (e.g., an operation of type SHAPE or MIX). As illustrated, for one or more nodes, they can be determined to be of type MIX or SHAPE regardless of their position in the graph (e.g., a convolution operation or a flattening operation); in other words, they can have a pre-forced operation type. For one or more other nodes, the operation type or the CanFuse marker generated in this embodiment can be determined based on their position in the graph.

[0115]

[0116]

[0117] After identifying the operation Ident, the implementation operation Impli, 430 can be applied to the computation graph CG2. If a node has been marked as an implementation that can be found in a table, the implementation operation can replace the node of the computation graph CG2 by replacing it with a first subgraph representing the first implementation; and in other cases, it can replace the node of the computation graph CG2 by replacing it with a second subgraph representing the second different implementation, thereby producing an updated computation graph CG3. In this sense, the high-level computation graph CG2 can become the low-level computation graph CG3.

[0118] Specifically, the high-level computation graph CG2 can include one or more operations that cannot be directly executed by the execution environment; for example, operations on floating-point values. The implementation Impl can convert the high-level computation graph into a low-level computation graph CG3, which can include only basic operations, such as operations that can be directly executed by the execution environment; for example, such basic operations can be lookup table compatible operations (for integers and / or floating-point numbers) and primitive operations on integers (although such operations can still be implemented by multiple primitive operations of the execution environment: for example, matrix multiplication, convolution, etc., can be performed by performing multiple multiplications of integers).

[0119] Typically, the corresponding implementation available for an operation can be a subroutine, which is represented as a subgraph through which nodes of the operation can be replaced. At least one of these subroutines can be used to construct a TLU. For example, the implementation can represent a series of basic operations (addition, subtraction, etc.).

[0120] Specifically, the first implementation could be a floating-point subroutine. Such a floating-point subroutine can be represented by a single node with floating-point input and output tensors. As an algorithm, the first implementation can be represented as:

[0121]

[0122] The second implementation can represent a series of subroutines in floating-point and / or integer formats. Specifically, this implementation can be configured to quantize the input to integers (if needed); apply integer operations corresponding to the nodes to obtain integer outputs; and dequantize the integer outputs (if needed). For example, for the arithmetic operation ⊕, this implementation could correspond to the following algorithm:

[0123]

[0124] More generally, the operations of computation graph CG2 can be implemented in the transformed computation graph CG3 as follows:

[0125]

[0126] Specifically, the second implementation can perform integer operations corresponding to the operations to be performed. For example, matrix multiplication can be performed on quantized values, as discussed, for example, in B. Jacob et al., “Quantization and training of neural networks for efficient integer-arithmetic-only inference”, doi:10.1109 / CVPR.2018.00286 (included in this paper by reference).

[0127] Specifically, as shown in the figure, in the transformed computation graph CG3, nodes marked as merging can be replaced by their corresponding floating-point nodes (e.g., the same as the original nodes in computation graph CG2), while nodes marked as non-merging can be replaced by their corresponding subgraphs that are used for their integer implementations (e.g., nodes that result in horizontal stripe-like nodes).

[0128] After implementing operation Impl, a fusion determination operation DFuse, 450 can be applied, whereby for nodes that can be implemented through a table lookup operation, one or more other nodes of computation graph CG3 are determined to be fused into that table lookup operation. This is illustrated in the figure as computation graph CG4, 414, where node 415 is determined to be fused into a single table lookup operation. Thus, the figure illustrates different identification steps Ident and fusion determination steps DFuse, in which nodes are identified as fused or non-fused, and in the fusion determination steps DFuse, such nodes are grouped into subsets of the nodes to be fused, such as set 415.

[0129] Specifically, the fusion determination step DFuse can identify connected components of the computation graph CG3 that share common properties that can be achieved through table lookup operations, such as applying a univariate function to the output of the previous node. Interestingly, at this stage, it may be sufficient to identify only the connected components that satisfy this property; it may no longer be necessary to check whether they share a common prototype.

[0130] As illustrated in the figure, the fusion determination step DFuse may include nodes in the fusion step that do not correspond to fusionable operations in computation graph CG2. Specifically, they may originate from the implementation of non-fusionable nodes. For example, as illustrated in the figure, quantization operations originating from subsequent non-fusionable nodes in computation graph CG2 can be fused with fusionable operations in computation graph CG2 into a single lookup table.

[0131] Interestingly, the fusion determination step DFuse can be performed in such a way that table lookup operations are not applied to the output of other table lookup operations. In this respect, the number of table lookups performed can be minimized.

[0132] Once it has been determined that nodes can be fused, the output write operation Write, 460, can be performed, where a transformed representation of a computer program can be output, in which node 415 is fused into a single table lookup operation. The transformed representation can take various forms. It can be explicitly stored as a computational graph or stored as a set of instructions INSTR, 415, for implicitly defining the target execution environment of the computational graph. The set of instructions can be in a form of instructions that are portable between various machine architectures or in a form of code executable on a specific machine architecture. As is known per se, the write operation Write can include constructing a lookup table by applying a univariate function represented by the nodes to be fused to a set of possible (e.g., integer) inputs to obtain corresponding (e.g., integer) outputs. Such a write operation is known, for example, from Zama's Concrete-Numpy (available from https: / / github.com / zama-ai / concrete-numpy).

[0133] The computational graph explicitly or implicitly defined by the write operation Write can have a certain canonical form. The computational graph can be an integer computational graph and can utilize table lookup operations. The table lookup (TLU) operation is applied element-wise to a single input tensor and can be defined as

[0134]

[0135] where LUT is a vector of M values, LUT i is the i-th element of the vector and 0 < x < M. This operation is also referred to as "Gather". As is known per se, such a TLU operation can be constructed by a write operation Write from the following:

[0136] (a) A series of elementary operations of computational graph CG4, the series of elementary operations together implementing a univariate function, e.g., for f(x) = sin(x / 10), the series of operations is y = x / 10, followed by sin(y)

[0137] (b) An integer array is the representative value to be processed by the TLU, allowing the calculation of the function of the integer value v i in the input set: LUT i = f(v i ), [[ID=2�]]

[0138] This canonical form can be referred to as the Form TLU-Shape canonical form. This canonical form can refer to an integer (usually tensor) computational operation graph having the following properties:

[0139] 1. The operation graph only contains operations of types MIX, SHAPE, and TLU;

[0140] 2. TLU operations have integer inputs and outputs, for example,

[0141] 3. The output of a MIX or SHAPE type operation is processed by at most one TLU operation before becoming the input of a subsequent MIX or SHAPE type operation, or is returned as the final computation output.

[0142] Specifically, this diagram illustrates the case where the Write,460 operation generates a set of instructions INSTR,415 for the target execution environment based on the computation graph CG4. Within this set of INSTR instructions, node 415 is integrated into a single table lookup operation; in other words, the set of nodes is implemented using a single table lookup operation. This set of instructions can take various forms; for example, it can be assembly language; a set of instructions executable by a homomorphic computation engine; a circuit layout of hardware circuits; or programming language code that is compiled or interpreted.

[0143] Optionally, optimization may include the sorting operation Ord, 440. As illustrated in the figure, the sorting operation Ord can be applied to the low-level computation graph CG3 obtained by the implementation operation Imp. This operation can also be applied to the high-level computation graph CG2 before performing the implementation operation Imp (but typically after identifying which nodes can be implemented through the table lookup operation); however, applying the sorting operation to the low-level computation graph has the advantage of identifying more cases where reordering can be applied.

[0144] The sorting operation `Ord` can be performed on reshaping nodes. Such nodes can represent operations that reorganize the elements of one or more input tensors without changing the values ​​of those elements, such as a flattened transpose. The sorting operation can reorder a computation graph (e.g., graph CG3) to merge at least one operation preceding a reshaping node with at least one operation following it. Interestingly, such a reorganization may not affect the result of the computation represented by computation graph CG3. However, the reorganization can have the effect of directly connecting preceding and following nodes in the graph, and therefore the fusion determination step `DFuse` can determine that these nodes can be merged into a single table lookup operation (which may have additional nodes).

[0145] Specifically, after the sorting operation Ord, the computation graph CG3 can have the property that no reshaping operation is applied to the output of the table lookup operation. This can be achieved by moving the reshaping operation after the table lookup operation. Specifically, the node sequence in the graph can be replaced by a blending operation, the corresponding reshaping, and the corresponding table lookup, the node sequence comprising a blending operation, followed by one or more table lookup sequences, followed by a reshaping (optionally followed by a table lookup). Then, through the fusion determination step DFuse, the corresponding table lookup can be merged into a single table lookup operation (possibly together with additional subsequent table lookups).

[0146] Specifically, the sorting operation Ord can recognize the pattern MIX-(TLU-SHAPE)+{-TLU}, where + indicates at least one repetition and {x} indicates an optional operation. This operation can be replaced with a new pattern of type SHAPE+-TLU+{-TLU}, thereby allowing adjacent TLU operations to be merged via the DFuse operation.

[0147] Specifically, the computation graph defined explicitly or implicitly by the Write operation can adopt a canonical pattern, referred to in this paper as the TLU-specific canonical form. This canonical form of the computation graph can be an integer (typically tensor) computation graph because, in addition to having the properties of the TLU shape form described elsewhere, it also possesses the following properties:

[0148] 1. There is no TLU operation between the MIX and SHAPE operations, or between two SHAPE operations;

[0149] 2. TLU operations cannot perform identity mappings between inputs and outputs. An example implementation of the sorting operation Ord is as follows:

[0150]

[0151]

[0152] Figures 5a to 5c A detailed embodiment of the transformed computation graph is shown. The transformed computation graph shown can correspond to the optimized output as described herein.

[0153] These transformed computational graphs illustrate the methods used to achieve quantization (in... Figure 5a and Figure 5b (in China) and standardization (in Figure 5b The lookup table (in the context of neural networks) is used. Quantization and normalization, as well as the application of activation functions, are all examples of operations performed on individual activations (e.g., the output of a single neuron) in neural networks. Therefore, these functions are univariate with respect to activations and can thus be implemented using lookup tables. Figures 5a to 5cA computational graph defined according to the techniques described herein is shown, where these operations are implemented using a single lookup table. Specifically, the subgraph adopts the TLU-SHAPE canonical form, as is the case with respect to... Figure 4 As discussed in the article.

[0154] Specifically Figure 5a The transformation representation of quantization is shown, in this case, uniform quantization. This output can be obtained by applying the techniques presented herein. Figure 3b The computational graph is used to generate it. This graph shows... Figure 3b In the computation graph, the part of edge 321 calculated from edge 320 is in Figure 5a This is implemented as a single lookup table. It is worth noting that... Figure 3b The graph contains several nodes with multiple inputs, specifically two "Where" nodes; however, it is interesting that these nodes are derived from a common prototype (especially the earlier "Add" or "Where" node), since it can be determined that these nodes share a common prototype. Figure 5a The lookup table 523 in the code is used for implementation.

[0155] Similarly, Figure 5b The transformation representation of non-uniform quantization is shown, which applies the provided technique to... Figure 3c The result of the computational graph. In this embodiment, Figure 3c The portion of the computation graph from edge 330 to edge 331 is implemented as lookup table 533. Similarly, in this case, various nodes with multiple inputs are included in the lookup table, which is determined to be possible because these nodes share a common prototype, i.e., they are all computed as a single-variable function of the output 330 of the multiplication operation.

[0156] Figure 5c Finally, a transform representation of the normalized operation (in this case, batch normalization) is shown, which can be achieved by applying the provided techniques. Figure 3d The computational graph is used to obtain (where, as discussed regarding the graph, edge 342 is incorporated into Div node 343 as a constant input). In this case, the output 341 of the matrix multiplication is applied and produces the output edge 344. Figure 3d Batch standardization is implemented as a single lookup table 545.

[0157] Figure 6a , Figure 6bA detailed embodiment of reordering a computation graph is shown. This embodiment illustrates how subgraphs of a computation graph can be reordered to facilitate the fusion of table lookup operations. Specifically, reordering can be applied to reshaping nodes. Such nodes can reorganize the elements of input tensors without changing their values. If such nodes (or sequences of such nodes) are preceded and followed by table lookup operations, then by moving reshaping nodes, the table lookup nodes may be directly connected in the subgraph and can be (e.g., merged into a single table lookup operation in subsequent operations).

[0158] Figure 6a , Figure 6b The input computation graph is shown, as well as the output computation graph where "Merge" (e.g., reordering and subsequent merging) has been applied.

[0159] Specifically, Figure 6a A computation graph is shown, representing the corresponding applications of the tensor of convolution CONV, 601; table lookup TLU, 602; reshaping RESHAPE, 603; another table lookup TLU, 604; and matrix multiplication MATMUL, 605. For simplicity, no other operands of convolution 601 and matrix multiplication 605 are shown in this embodiment. In this embodiment, table lookups 602 and 604 precede and follow the reshaping operation 603, respectively. Reordering may include moving the reshaping operation 603 before table lookups 602 and 604. Adjacent table lookups can then be merged into a single table lookup 609. Alternatively, the reshaping operation can be moved after table lookups 602 and 604, thus obtaining a computation graph where the single table lookup 609 precedes the reshaping operation 603. In any case, the sequence of table lookups to be merged can be obtained by iteratively moving the reshaping before or after the table lookups. The fact that the reshaping operations 601 and 605 are not applied element-wise to their inputs does not mean that they are unaffected by the transformation.

[0160] Similarly, Figure 6bA computational graph is shown in which a series of operations are applied to a tensor (which is convolution CONV, 611); a first table lookup TLU, 612; a reshaping operation RESHAPE, 613; a second table lookup TLU, 614; a reshaping operation of the form TRANSPOSE, 615; a third table lookup TLU, 616; and a matrix multiplication MATMUL, 617 (for simplicity, any other operands of operations 611 and 617 are not shown). In this case, two transformations as described can be applied: first, by swapping the order of table lookup 612 and reshaping 613; and second, by swapping the order of transpose and table lookups 612 and 614 (which become adjacent through the previous swap). After these swaps, table lookups 612, 614, and 616 are adjacent and can therefore be merged into a single table lookup 619. As described above, the transpose operation 615 can also be swapped with the table lookup 616, and the reshaping operation 613 can be swapped with the table loops 614 and 616 to obtain a transformed graph in which the table lookup 619 is merged before the reshaping 613 and transpose 615. The non-element-by-element and / or non-reshaping operations 611 and 617 remain unchanged.

[0161] It is worth noting that in the two subgraphs resulting from the proposed reordering, no reshaping operation was applied to the output of the table lookup operation. Otherwise, the reshaping operation and the table lookup might have been swapped. Specifically, the resulting subgraphs are in TLU-Only canonical form.

[0162] Figure 7 Detailed, but non-limiting, embodiments of optimizing computer programs are illustrated. The figures do not show examples such as... Figure 4 Such implementation operations transform a high-level computation graph into a low-level computation graph; for example, such operations may have previously been applied to techniques described for that graph. Identifying whether the corresponding node (Ident, 420) can be implemented through a table lookup operation, and subsequently determining whether the node (DFuse, 450) can be merged, can be performed on the computation graph without requiring such implementation operations in between.

[0163] Specifically, the figure illustrates computation graph CG1,711. This computation graph can be a low-level computation graph, meaning that corresponding nodes can represent corresponding operations that are directly translated into implementations in the execution environment. Specifically, computation graph CG1 can include low-level operations that implement quantization and / or cast operations when needed. For example, where the execution environment supports integer arithmetic and table lookup operations (such as using homomorphic encryption), the computation graph can include nodes representing integer operations supported by the execution environment (e.g., integer addition and subtraction, scalar multiplication, matrix multiplication by a constant matrix, convolution with a constant matrix, etc.) and nodes representing operations that can be implemented using table lookup operations (e.g., floating-point operations, such as activation functions, quantization, or parts thereof, etc.). The low-level computation graph CG1 can also be called the basic computation graph and can be the computation graph of Zama's Concrete-Numpy (available at https: / / github.com / zama-ai / concrete-numpy).

[0164] Specifically, as in response to Figure 4 The computational graph CG1 discussed can contain many different types of nodes. Specifically, as regarding... Figure 4 The computation graph discussed may include nodes of mixed type (MIX); nodes of univariate function type (UNI); nodes of reshaping type (SHAPE); and / or nodes of table lookup operation type (TLU). Also, as regarding... Figure 4 The discussion focuses on a set of operations whose types may depend on the other nodes in the graph, and therefore the operation may or may not be implemented as a table lookup. This set of operations may include at least addition and multiplication. Other sets of operations may always or never be implemented as table looksups.

[0165] The diagram illustrates the identification operation Ident, 720, which determines whether a given node in computation graph CG1 can be implemented via a table lookup operation, for example, whether the node is a mixed type (MIX). The identification operation Ident can be configured to determine whether a node with multiple inputs can be implemented via a table lookup operation based on determining that multiple inputs share a common prototype node, specifically making the node's output writable as an element-wise function of the output of the common prototype node. Specifically, the diagram shows computation graph CG2, which is obtained by annotating diagonal striped nodes in computation graph CG1 as implementable via a table lookup operation, based on determining that multiple inputs share a common prototype. It should be noted that whether this is the case for nodes with multiple inputs depends on the graph structure. Therefore, the identification operation Ident can determine whether a node can be implemented as a lookup table not only based on the node itself but also based on other parts of the subgraph.

[0166] Following the Ident operation, the fusion determination operation DFuse, 750 can be used to determine one or more additional nodes in the computation graph that can be fused into the table lookup operation for nodes that have been identified as implementable through the table lookup operation. This is illustrated in the computation graph CG3, where fusionable subgraphs are annotated. In this case, a fusionable subgraph is a subgraph that begins with an outgoing edge from a common prototype node and ends at the node itself.

[0167] In this scenario, as illustrated in the figure, the identified subgraph in computation graph CG3 can be replaced by a node representing a single table lookup operation. This could be a TLU type node (as well as for...). Figure 4 (As described). Following this, for example, optimization can iteratively apply the Identifier and fusion to determine the DFuse until no more nodes can be fused.

[0168] After nodes have been merged, a write operation can be performed, as can be done for nodes that have been merged. Figure 4 As discussed, the Write operation can output a transformed representation of a computer program in which node merging is achieved through a single table lookup, as is known per se.

[0169] The diagram also illustrates the sorting operation Ord, as is the case with the... Figure 4 The sorting operation Ord discussed here can reorder the computation graph CG2 to merge at least one operation before a node reshaping with at least one operation after a node reshaping. This operation can be performed as follows: Figure 4 As discussed, it will be achieved.

[0170] Specifically, the algorithm below provides a detailed implementation of how to perform the Ident operation and the DFuse operation. This implementation uses a general pattern detection technique to find nodes that can be implemented as table lookup operations and then fused. Specifically, in the described algorithm, UnwantedProperty can be configured to determine whether a given operation is a MIX or SHAPE operation. ConformsToFusingRules can be configured to determine whether there is more than one TLU operation in the subgraph; whether the operation in the subgraph produces a tensor with a shape corresponding to the subgraph input tensor; and / or whether the operation in the subgraph uses constants with the same shape as the input tensor.

[0171] In the algorithm below, specifically, the identification operation `Ident` can be implemented using the functions `FindClosestAncestorsWithoutUnwantedProperty` and `FindLowestFusingPointCandidate`, while the fusion determination operation `DFuse` can be represented using the function `CreateFusableSubgraphFromTo`. Accordingly, if the operation in the computation graph belongs to a subgraph found by `Algorithm FindSubgraphToFuse`, it can be implemented in TLU mode; otherwise, it can be implemented in native mode.

[0172]

[0173]

[0174]

[0175]

[0176]

[0177]

[0178]

[0179]

[0180]

[0181] Figure 8 A block diagram is shown of a computer-implemented method 800 for optimizing a computer program for an execution environment that supports table lookup operations. Method 800 may correspond to... Figure 1 The operation of system 100. However, this is not a limitation, because method 800 can also be performed using another system, apparatus or device.

[0182] Method 800 may include accessing (810) a computer program as a representation of a computation graph in an operation titled “ACCESS COMPUTATION GRAPH”. In the computation graph, corresponding nodes may represent corresponding operations. Method 800 may include, in an operation titled “DETERMINE TLU IMPLEMENTATION”, determining, 820, whether a corresponding node in the computation graph can be implemented via a table lookup operation for that node. Method 800 may include, in an operation titled “DETERMINE FUSING”, determining, 830, that one or more additional nodes in the computation graph can be incorporated into a table lookup operation for nodes that can be implemented via a table lookup operation. Method 800 may include outputting, 840, a transformed representation of the computer program in an operation titled “OUTPUT”. In this transformed representation, nodes and one or more additional nodes can be incorporated into a single table lookup operation.

[0183] Specifically, method 800 can be a compiler method. Specifically, method 800 can take source code in a compiled or interpreted programming language (e.g., Python, C++, etc.) as input, compile the source code to determine the computation graph, and output the transformation representation in a representation executable by an execution environment, such as a homomorphic executable for the FHE computation engine.

[0184] It should be understood that, generally speaking, Figure 8 The operations of Method 800 can be performed in any suitable order (e.g., sequentially, simultaneously, or a combination thereof), but where applicable, are subject to a specific order required (e.g., by input / output relationship).

[0185] The method can be implemented on a computer as a computer-implemented method, as dedicated hardware, or as a combination of both. Also, Figure 9 As illustrated, computer instructions (e.g., executable code) may be stored on a computer-readable medium 900, for example, in the form of a series of machine-readable physical marks 910 and / or as a series of elements having different electrical (e.g., magnetic) or optical properties or values. Medium 900 may be temporary or non-temporary. Embodiments of computer-readable media include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Figure 9 An optical disc 1100 is shown. Alternatively, a computer-readable medium 900 may include data 910 representing a transformed representation of a computer program determined according to the methods described herein.

[0186] The following numbered clauses include conjecture and non-limiting embodiments:

[0187] 1. A computer-implemented method for optimizing a computer program for an execution environment supporting table lookup operations, wherein the method comprises:

[0188] - Access the computer program as a representation of a computation graph, wherein the corresponding nodes of the computation graph represent corresponding operations;

[0189] - For the corresponding node in the computation graph, determine whether the corresponding node can be implemented through a table lookup operation;

[0190] - For nodes that can be implemented through table lookup operations, determine one or more other nodes in the computation graph that can be incorporated into the table lookup operation;

[0191] - Output the transformation representation of the computer program, wherein the nodes and the one or more additional nodes are merged into a single table lookup operation.

[0192] 2. The method according to Clause 1, wherein the execution environment is used for computation via homomorphic encryption that supports at least encrypted addition, encrypted scalar multiplication, and encrypted table lookup operations.

[0193] 3. The method according to Clause 1, wherein the table lookup operation is a hardware lookup table.

[0194] 4. The method according to any of the preceding clauses, wherein one or more operations from the first group of operations are never implemented through the table lookup operation; one or more operations from the second group of operations depend on the computation graph being implemented through the table lookup operation; and one or more operations from the third group of operations are always implemented through the table lookup operation.

[0195] 5. The method described in Clause 4, wherein the type of the second group includes at least addition.

[0196] 6. The method according to Clause 4 or 5, comprising replacing the node with a first subgraph representing a first implementation if it is determined that the node can be implemented by a table lookup operation; and replacing the node with a second subgraph representing a second implementation in other cases.

[0197] 7. The method according to Clause 6, wherein the second implementation is configured to quantize the input into an integer; apply integer operations corresponding to the node to obtain an integer output; and dequantize the integer output.

[0198] 8. The method according to any of the preceding clauses, comprising: determining, based on determining that a node having said multiple inputs has a common prototype, that a node can be implemented through a table lookup operation.

[0199] 9. The method according to any of the preceding clauses, wherein the operation is applied to one or more input tensors and produces an output tensor, and wherein the table lookup operation is applied element-wise to a single input tensor.

[0200] 10. The method according to Clause 9, comprising obtaining a reshaping node representing an operation that reorganizes the elements of one or more input tensors without changing the values ​​of the elements of one or more input tensors, and reordering the computation graph to merge at least one operation before the reshaping node with at least one operation after the reshaping node.

[0201] 11. The method according to any of the preceding clauses, wherein the transformation representation of the computer program does not include table lookup operations applied to the output of other table lookup operations.

[0202] 12. The method according to Clause 11, wherein the transformation representation of the computer program does not include a reshaping operation applied to the output of a table lookup operation, wherein the reshaping operation rearranges one or more inputs without changing their values.

[0203] 13. The method according to any of the preceding clauses, wherein the computer program represents an evaluation of a machine-learnable model, such as an artificial neural network, a generalized linear model, a decision tree, or an ensemble model.

[0204] 14. A compiler system (100) for optimizing a computer program for an execution environment supporting table lookup operations, wherein the system comprises:

[0205] - Data interface (120), the data interface being used to access the computer program as a representation of a computation graph, wherein the corresponding nodes of the computation graph represent corresponding operations;

[0206] - Processor subsystem (140), the processor subsystem being configured to:

[0207] - For the corresponding node in the computation graph, determine whether the corresponding node can be implemented through a table lookup operation;

[0208] - For nodes that can be implemented through table lookup operations, determine one or more other nodes of the computation graph that can be merged into the table lookup operation;

[0209] - Output the transformation representation of the computer program, wherein the nodes and the one or more additional nodes are merged into a single table lookup operation.

[0210] 15. A temporary or non-temporary computer-readable medium (900) comprising data (910) representing the following:

[0211] - Instructions, when executed by a processor system, cause the processor system to perform a computer-implemented method according to any one of clauses 1 to 13; and / or

[0212] - A transformed representation of a computer program determined by a method implemented by a computer according to any one of Clauses 1 to 13.

[0213] Examples, implementations, or optional features, whether or not indicated as non-limiting, should not be construed as limiting the claimed invention.

[0214] It should be noted that the embodiments mentioned above are illustrative and not limiting of the invention, and those skilled in the art will be able to devise many alternative embodiments without departing from the scope of the appended claims. Any reference numerals placed in parentheses in the claims should not be construed as limiting the claims. The use of the verb "comprising" and its variations does not exclude the presence of elements or stages other than those stated in the claims. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. When an expression such as "at least one" precedes a series or group of elements, it indicates the selection of all elements or any subset thereof from said series or group. For example, the expression "at least one of A, B, and C" should be understood to include only A, only B, only C, both A and B, both A and C, both B and C, or all of A, B, and C. The invention can be implemented by means of hardware comprising several different elements, and by means of a suitably programmed computer. In the device claims enumerating several means, several of these means may be implemented by the same item of hardware. The mere fact that certain measures are described in mutually different dependent claims does not imply that a combination of these measures cannot be used to exert an advantage.

Claims

1. A computer-implemented method (800) for optimizing a computer program for an execution environment supporting table lookup operations, wherein the method comprises: - Access (810) the computer program as a representation of a computation graph, wherein the corresponding nodes of the computation graph represent corresponding operations; - For the corresponding node of the computation graph, determine (820) whether the corresponding node can be implemented by table lookup operation, including determining that the node with the multiple inputs can be implemented by table lookup operation based on determining that multiple inputs have a common prototype node; If it is determined that a node can be implemented through a table lookup operation, then the node is replaced by a first subgraph of the computation graph representing the first subroutine; and in other cases, the node is replaced by a second subgraph of the computation graph representing the second subroutine. - For a node that has been determined to be achievable through a table lookup operation, determine (830) that one or more other nodes from the first subgraph of the computation graph can be merged into the table lookup operation, and determine that one or more additional nodes from the second subgraph of another node can be merged into the table lookup operation. - Output (840) the transformation representation of the computer program, wherein the node and the one or more additional nodes are merged into a single table lookup operation.

2. The method (800) according to claim 1, wherein the execution environment performs computations using homomorphic encryption that supports at least encrypted addition, encrypted scalar multiplication, and encrypted table lookup operations.

3. The method (800) according to claim 1, wherein the table lookup operation is a hardware lookup table.

4. The method (800) according to any of the preceding claims, wherein several groups of operations occurring in the computation graph are distinguished, wherein one or more operations from the first group of operations are never implemented through the table lookup operation; one or more operations from the second group of operations depend on the computation graph being implemented through the table lookup operation; and one or more operations from the third group of operations are always implemented through the table lookup operation.

5. The method (800) according to claim 4, wherein the type of the second group includes at least addition operations.

6. The method according to any one of claims 1 to 3, comprising iterating the computation graph to determine for each operation whether a table lookup operation can be used for the operation.

7. The method (800) according to any one of claims 1 to 3, wherein the second subroutine is configured to quantize the input into an integer; apply integer operations corresponding to the node to obtain an integer output; and dequantize the integer output.

8. The method (800) according to any one of claims 1 to 3, wherein the operation is applied to one or more input tensors to produce an output tensor, and wherein the table lookup operation is applied element-wise to a single input tensor.

9. The method (800) according to any one of claims 1 to 3, comprising obtaining a reshaping node representing an operation that reorganizes the elements of one or more input tensors without changing the values ​​of the elements of one or more input tensors, and reordering the computation graph to merge at least one operation before the reshaping node with at least one operation after the reshaping node.

10. The method (800) according to any one of claims 1 to 3, wherein the transformation representation of the computer program does not include table lookup operations applied to the output of additional table lookup operations.

11. The method (800) according to any one of claims 1 to 3, wherein the transformation representation of the computer program does not include a reshaping operation applied to the output of a table lookup operation, wherein the reshaping operation rearranges one or more inputs without changing their values.

12. The method (800) according to any one of claims 1 to 3, wherein the computer program represents the evaluation of a machine-learnable model.

13. The method (800) according to claim 12, wherein the machine-learnable model is an artificial neural network, a generalized linear model, a decision tree, or an ensemble model.

14. A compiler system (100) for optimizing a computer program for an execution environment supporting table lookup operations, wherein the system comprises: - Data interface (120), the data interface being used to access the computer program as a representation of a computation graph, wherein the corresponding nodes of the computation graph represent corresponding operations; - Processor subsystem (140), the processor subsystem being configured to: - For a corresponding node in the computation graph, determining whether the corresponding node can be implemented through a table lookup operation includes: determining that a node with multiple inputs can be implemented through a table lookup operation based on the determination that multiple inputs have a common prototype node; If it is determined that a node can be implemented through a table lookup operation, then the node is replaced by a first subgraph of the computation graph representing the first subroutine; and in other cases, the node is replaced by a second subgraph of the computation graph representing the second subroutine. - For a node that has been determined to be implementable by a table lookup operation, determine that one or more additional nodes from the first subgraph of the computation graph can be merged into the table lookup operation, and determine that one or more additional nodes from the second subgraph of another node can be merged into the table lookup operation. - Output the transformation representation of the computer program, wherein the nodes and the one or more additional nodes are merged into a single table lookup operation.

15. A temporary or non-temporary computer-readable medium (900) comprising data (910) representing the following: - Instructions, when executed by a processor system, cause the processor system to perform a computer-implemented method according to any one of claims 1 to 13; and / or - A transformed representation of a computer program determined by the computer-implemented method according to any one of claims 1 to 13.

Images