Computational network transformation for fully homomorphic evaluation

Abstract

Some embodiments relate to a computer-implemented method of converting a first computing network configured to compute an operation of a function to a second computing network configured to compute an FHE operation of the same function. i ) can be determined for matrix operations in the first computational network. The real-valued matrix can be converted by scaling the real-valued matrix with its corresponding expansion factor and rounding. The precision measure of the second computational network can be iteratively optimized.

Description

Technical Field

[0001] The subject matter of the present disclosure relates to a computer-implemented method for converting a first computing network into a second computing network, a computer-readable medium, and a system for converting a first computing network into a second computing network, wherein the first computing network is a first computing network arranged to calculate operations of a function, and the second computing network is a second computing network arranged to calculate fully homomorphic encryption operations of the same function. Background Art

[0002] Since Craig Gentry's breakthrough paper "Fully Homomorphic Encryption Using Ideal Lattices" (full version in Commun. ACM 53(3):97-105, 2010; incorporated herein by reference), there has been an effort to make fully homomorphic encryption (FHE) secure and efficient enough for real-world applications. FHE allows one to perform computations (e.g., evaluate circuits) on encrypted data without being able to decrypt it. For example, input data and computation results can be received and returned in encrypted form. Intermediate data (e.g., the internal state of a computation) can also be in encrypted form.

[0003] Even if the result of a computation is returned in encrypted form, when decrypted, the output is the same as if the operation was performed on the unencrypted data. Homomorphic encryption can be used for privacy-preserving outsourced storage and computation. This allows data to be encrypted and outsourced to a cloud environment for processing and / or storage, all of which is encrypted.

[0004] For example, homomorphic cryptography can be applied in areas such as healthcare, where privacy rules may make it difficult to share plain data, but computation on encrypted medical data may be permitted. For example, a medical model developed for, for example, classifying medical data can be configured to receive medical data in encrypted form from a third party, such as a hospital. The medical model can, for example, classify medical data as, for example, normal or abnormal, or as having a specific medical syndrome, disease, or other condition. Using homomorphic encryption, the medical model can be applied to medical data received in encrypted form. This means that the party evaluating the medical model does not have access to the plaintext medical data corresponding to the encrypted medical data. Users of the service can decrypt the results of the application of the medical model.

[0005] Cryptographic systems that support arbitrary computations on ciphertext are called fully homomorphic encryption (FHE) schemes. Such schemes enable the evaluation of a wide range of functions and can be run on encrypted inputs to produce an encryption of the result. For example, once a combination of AND gates and OR gates is available, so-called functional integrity is achieved because this allows one to implement any Boolean circuit. Such a circuit can be run by an untrusted party without exposing its inputs, internal state, and output. Fully homomorphic encryption can be hierarchical, in which case the number of certain operations cannot exceed a predetermined threshold. Hierarchical fully homomorphic schemes can be converted to non-hierarchical FHE schemes by performing bootstrapping operations. Bootstrapping operations increase the number of operations that can be performed on encrypted data items.

[0006] Fully homomorphic encryption schemes may support computations expressed in some particular form (e.g., as Boolean circuits or arithmetic circuits). For example, hierarchical fully homomorphic encryption schemes may support evaluation of circuits of arbitrary but bounded, predetermined depth. Non-hierarchical fully homomorphic encryption (FHE) allows evaluation of arbitrary circuits of unbounded depth. Hierarchical FHE schemes may be converted to non-hierarchical FHE schemes by periodically performing bootstrapping operations on the data. It is known how to compile computations (e.g., functions) in the form of circuits.

[0007] Since Gentry's paper, many FHE schemes have been developed, reducing computation time by orders of magnitude. Many FHE schemes are now known. An example of a newer FHE scheme is described in the paper "TFHE: Fast FullyHomomorphic Encryption over the Torus" by Ilaria Chillotti et al. (J. Cryptology 33(1):34-91, 2020) (incorporated herein by reference).

[0008] A preferred FHE scheme is described in the paper "Programmable Bootstrapping Enables Efficient Homomorphic Inference of Deep Neural Networks" by Ilaria Chillotti, Marc Joye, and Pascal Paillier (see Cryptology ePrint Archive: Report 2021 / 091, incorporated herein by reference). This paper operates on torus elements, for example, it is a torus-type FHE scheme (TFHE).

[0009] As efficient FHE schemes become increasingly available, there is growing interest in converting computations originally developed for conventional non-encrypted evaluation to computations using FHE. Such conversions have always been possible, especially with the development of so-called programmable bootstrapping (PBS), where the bootstrapped computations also perform predetermined functions that can be selected by the designer of the FHE computation. Nevertheless, it is desirable to run as many operations as possible in the original computation as can be performed directly on encrypted values ​​without the use of programmable bootstrapping.

[0010] Unfortunately, many FHE schemes (particularly TFHE schemes) impose restrictions on the operations that can be performed directly on encrypted values, e.g., as the computational cost of the encryption operation is comparable to the original non-encrypted computation. For example, addition of two encrypted values ​​is generally directly possible, and in particular, this is possible in torus-based schemes. Multiplication between a plain integer and an encrypted value is also generally possible; this is to be expected, since multiplication by an integer n can be viewed as repeated addition of n copies of the encrypted value. For example, in TFHE (where "T" stands for torus), addition and integer multiplication are inherently possible. The encryption scheme TFHE uses a mathematical structure called a torus. One embodiment of a torus is For example, real numbers are taken modulo 1 (other moduli are possible, but 1 is convenient). In practice, values ​​on a torus are often represented as a suitable finite ring, e.g. Where m is the modulus, i.e. the integer modulo a modulus. The integer modulus m is typically a power of 2, e.g. 2 q , where q can be, for example, 32 or 64, and q is typically less than or equal to 128. The encryption of the torus value can be implemented as in a specific scheme, for example, as in the paper “Programmable Bootstrapping Enables Efficient Homomorphic Inference of Deep Neural Networks” cited above.

[0011] However, multiplication of a plaintext non-integer value with an encrypted value is not always possible. In particular, in torus-based schemes, a known difficulty is that multiplication with a non-integer as an encrypted value is not restricted.

[0012] Unfortunately, in a large class of computations called neural networks, multiplications with non-integer numbers occur frequently, for example as part of multiple sums, matrix multiplications, convolutions, batch normalization, etc. The conversion of computational networks such as neural networks to FHE computational networks requires improvements. Summary of the invention

[0013] It is desirable to use FHE operations to run a computational network on encrypted data, such as a first computational network, which is not necessarily limited to a neural network based on conventional arithmetic operations. In order to improve the efficiency of such a converted computational network (e.g., a second computational network), it is desirable that the operation is run directly on the encrypted value without the need to use programmable bootstrapping, for example.

[0014] For example, a first computing network using conventional operations may include a plurality of real-valued matrix operations (y=Wx) and a plurality of additional operations, wherein the plurality of real-valued matrix operations (y=Wx) include real-valued matrix multiplications, and the plurality of additional operations are specifically operations that use the results of the matrix operations as inputs. The purpose may be to convert the matrix operations so as to avoid multiplications between real numbers and encrypted values, and instead use integers to multiply the encrypted values. Operations converted to using only multiplications of encrypted values ​​with plaintext integers and additions of encrypted values ​​with plaintext integers or plaintext real numbers may be implemented in the second computing network as native FHE operations, such as FHE operations without programmable bootstrapping. The real numbers in the matrix operations (e.g., multiplications or additions) in the first computing network may be floating point numbers or fixed point numbers.

[0015] Typically, the real numbers that define the operations can be unencrypted in both the first computing network and the second computing network. In a typical embodiment, most of the real numbers in the operations in the first computing network are not integers, such as at least 50%, 75% or substantially all of the real numbers that define the operations (e.g., matrix operations). As shown herein, for linear operations, this conversion is always possible. For other operations, it is always possible or desirable to avoid multiplication with real numbers. Other operations that may exist in the first computing network can then be converted using one or more programmable bootstraps.

[0016] One way to transform matrix operations to avoid multiplication between real numbers and encrypted values ​​is to multiply the matrix of the matrix multiplication by a real-valued constant called an expansion factor (typically denoted alpha) and then round the matrix. Directly rounding the value may significantly reduce precision. Therefore, the value produced by the matrix multiplication is also multiplied by the expansion factor; however, other operations may be adjusted for this. For example, the real value that needs to be added to the multiplication result may also be multiplied by the expansion factor: for example, if the operation is a matrix multiplication with a bias (y=W.x+B), then B is also multiplied by the expansion factor, so that the value obtained by the matrix multiplication with a bias is still the original value multiplied by the expansion factor. Other operations (e.g., additional operations) may be adjusted to expect inputs that have been multiplied by the expansion factor. For example, such a function may be adjusted to be a combination of the original function divided by the expansion factor (e.g., the function A(x) may be replaced by A(x / α)). Note that FHE schemes may not directly support division by the expansion factor, but this is not a problem if the additional operation is implemented as programmable bootstrapping.

[0017] In one embodiment, a measure of precision of the converted second computing network is determined. Conversion by an expansion factor may introduce imprecision because rounding operations do not produce exactly the same value even after multiplication by the expansion factor. For larger expansion factors, this imprecision is less. In other words, increasing the expansion factor improves precision, for example, making the precision of the second computing network closer to the precision of the first network. On the other hand, at a certain point, if the expansion factor is too large, precision begins to decline because the value tends to become larger, thereby causing other precision problems. In addition, not all operations have the same impact on the final result. Therefore, for some operations, a high expansion factor is more important than other operations. A good set of expansion factors can be easily found by iteratively optimizing a set of expansion factors to obtain acceptable precision (e.g., precision close enough to the precision of the first network).

[0018] Converting computational networks is not limited to neural networks, although neural networks are an important class of functions that can be converted well in this way. Neural networks are generally insensitive to small inaccuracies introduced by the computational process. Therefore, using one embodiment, neural networks tend to be converted into a particularly efficient second computational network.

[0019] Encrypted values ​​in FHE schemes are typically restricted to a fixed number of bits. In this article, the term real number or real is used to indicate a number that is not necessarily an integer. Although the word "real" is used, in FHE, such numbers are typically represented in a digital format containing multiple digits. Typically, fixed-point precision is used to represent real numbers, rather than floating-point numbers. For example, in TFHE, encrypted values ​​can be numbers restricted to an interval (typically, between 0 (inclusive) and 1 (exclusive)). For example, a real number can be a value between 0 and 1 with a fixed precision (e.g., 32 bits or 64 bits). Even if the value on the torus is within an interval (e.g., modulo 1), the value on the torus can also be represented by a value outside the interval, but it should be understood that if two values ​​x and y are equal modulo 1, they represent the same torus.

[0020] In an embodiment, real numbers are represented by finite digit real numbers, typically representing a rational approximation of real numbers. Floating point or fixed point representation is a typical embodiment of this real number representation. Real value matrix multiplication may include real value matrix multiplication represented by a real value matrix, and alternatively real value vector addition represented by a real value vector. In one embodiment, one or more or all elements of a real value matrix are represented by finite digit real numbers. In one embodiment, one or more or all elements of a real value vector are represented by finite digit real numbers. In one embodiment, all elements of a real value matrix and a real value vector are represented by finite digit real numbers; however, this is not required, and a mixed representation matrix and / or vector is possible. Floating point format is described in, for example, IEEE 754.

[0021] Once a set of expansion factors is chosen, one can convert real-valued matrix multiplications to integer-valued matrix multiplications, real-valued additions to real-valued additions, but where the real multipliers may be multiplied by the expansion factors. The same transformation is performed on convolutions. A real-valued convolution with a real-valued kernel consisting of real numbers is converted to an integer-valued convolution with an integer-valued kernel consisting of integers. A real-valued kernel can be converted by multiplying it by the expansion factor and rounding. The same transformation can also be performed on batch normalization. Batch normalization is typically an affine function, such as y=f*x+b, where f and b are real-valued constants. Real-valued batch normalization can be converted to another affine function, such as y=f'*x+b', where f' is an integer constant and b' is a real constant. Most other operations can be converted to one or more programmable bootstraps. Some operations can still be performed on encrypted values; for example, operations that only change the order of values, such as flattening, concatenation operations, or linear operations (e.g., addition or subtraction of two vectors, etc.).

[0022] The expansion factor itself is typically a real-valued number. The expansion factor can be represented in fixed precision notation, floating point, etc. A real number including an expansion factor can also be represented as a pair of numerator and denominator indicating a rational number approximating the real number.

[0023] The expansion factor is usually greater than 1, but this is not essential. Typically, each operation in the first computation network has an expansion factor, which simply means integer-valued multiplications and integer or real-valued additions between the predetermined value and the encrypted value.

[0024] The expansion factor can be found by symbolically assigning the expansion factor and propagating the symbolic input through the first computational network. In an advanced embodiment, a so-called joint operation is solved, which is a specific operation that combines values ​​from different sources, but does not use programmable bootstrapping, such as addition, subtraction, and concatenation operations. The joint operation can also be an integer weighted addition, such as an integer linear combination. For example, the joint operation receives two or more encrypted value vectors from two or more operations. If the encrypted values ​​are multiplied by the same expansion factor, this joint operation can be performed directly on the encrypted values. Therefore, the joint operation imposes a constraint on the expansion factor that we need to solve to obtain a second network that is homogeneous and gives the correct result.

[0025] Real-valued matrix operations combine multiplication and addition. The matrix operations can be considered to have a linear part and an additive part. For example, the matrix operation can be WX+B or Wx+b, where W, X, B are matrices and x, b are vectors. Many operations can be considered as such matrix operations, for example, real-valued matrix operations can include one or more of the following: convolution with a real-valued kernel, matrix multiplication with a real-valued weight matrix, batch normalization with a real-valued scalar. For example, convolution can be considered as repeated multiplication between a predetermined kernel weight vector and a convolution input value vector (the latter being encrypted in a second computing network). If convenient, data can be represented in a variety of types of tables, such as 2-dimensional or two-dimensional or higher-dimensional kernels. The input values ​​can also be organized in higher-dimensional tensors. However, this different data organization is included in matrix operations.

[0026] One particularly useful application of the transformation is to allow computations to be performed on private data, such as medical evaluation of images or other computations on medical data.

[0027] Medical data may include medical images. Medical images may include, for example, multi-dimensional image data, such as two-dimensional (2D), three-dimensional (3D), or four-dimensional (4D) images acquired by a variety of acquisition modes, such as, but not limited to, standard X-ray imaging, computed tomography (CT), magnetic resonance imaging (MRI), ultrasound (US), positron emission tomography (PET), single photon emission computed tomography (SPECT), and nuclear medicine (NM).

[0028] An embodiment of the method may be implemented on a computer as a computer-implemented method, or implemented in dedicated hardware, or in a combination of the two. Executable code for an embodiment of the method may be stored on a computer program product. Examples of computer program products include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Preferably, the computer program product includes non-transitory program code stored on a computer-readable medium for performing an embodiment of the method when the program product is executed on a computer.

[0029] In one embodiment, the computer program comprises computer program code, which is adapted to perform all or part of the steps of an embodiment of the method when the computer program is run on a computer.Preferably, the computer program is embodied on a computer readable medium.

[0030] Another aspect is an electronic system or device, such as a computer, configured for conversion according to one embodiment. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Further details, aspects and embodiments will be described by way of example only with reference to the accompanying drawings. The elements in the drawings are illustrated for simplicity and clarity and are not necessarily drawn to scale. In the drawings, elements corresponding to elements already described may have the same reference numerals. In the drawings,

[0032] Figure 1a schematically illustrates an example of an embodiment of a system for performing computations using fully homomorphic encryption (FHE),

[0033] Figure 1b schematically illustrates an example of an embodiment of a system for performing computations using FHE,

[0034] Figure 1c schematically illustrates an example of an embodiment of a system for performing computations using FHE,

[0035] Figure 1d schematically illustrates an example of an embodiment of a system for converting computations for FHE,

[0036] Figure 2 schematically shows an example of an embodiment of a conversion system,

[0037] Figure 3 schematically illustrates an example of an implementation of a first computing network,

[0038] Figure 4a schematically illustrates an example of an implementation of a first computing network,

[0039] Figure 4b schematically illustrates an example of an implementation of a first computing network,

[0040] Figure 5a schematically illustrates an example of an implementation of a first computing network,

[0041] Figure 5b schematically illustrates an example of an implementation of a first computing network,

[0042] Figure 6 An example of an embodiment of a conversion method is schematically shown,

[0043] Figure 7a schematically shows an embodiment of a computer-readable medium having a writable portion, the writable portion comprising a computer program according to an embodiment,

[0044] Figure 7b A representation of a processor system according to one embodiment is schematically shown.

[0045] Reference numerals list

[0046] The following list of symbols and abbreviations is provided to facilitate the explanation of the drawings and should not be construed as limiting the claims. Figures 1a to 5b as well as Figures 7a to 7b .

[0047] Floors 1-99

[0048] 110 FHE Computing System

[0049] 111-113 FHE Computing System

[0050] 114 Conversion system

[0051] 130, 134 Processor System

[0052] 140, 144 Memory

[0053] 150, 154 Communication interface

[0054] 160 Data Provider System

[0055] 200 Conversion system

[0056] 210 First Computing Network

[0057] 220 Second Computing Network

[0058] 211-214 Operations

[0059] 221-224 Operation

[0060] 230 expansion factor unit

[0061] 231 Optimization Unit

[0062] 240 conversion unit

[0063] 250 precision units

[0064] 251 Sample memory

[0065] 300 First Computing Network

[0066] 410 First Computing Network

[0067] 411 Input

[0068] 412-414 Operation

[0069] 415 Output

[0070] 420 First Computing Network

[0071] 421 Fork Operation

[0072] 422 Joint Operation

[0073] 510 First Computing Network

[0074] 511 Fork Operation

[0075] 512 Joint Operation

[0076] 520 First Computing Network

[0077] 525 Calibration Operation

[0078] 1000 Computer readable medium

[0079] 1010 Writable part

[0080] 1020 Computer Programs

[0081] 1110 Integrated Circuit

[0082] 1120 Processing Unit

[0083] 1122 Memory

[0084] 1124 ASIC

[0085] 1126 Communication Components

[0086] 1130 Interconnect

[0087] 1140 Processor System DETAILED DESCRIPTION

[0088] While the subject matter of the present disclosure may be embodied in many different forms, one or more specific embodiments are shown in the drawings and will be described in detail herein, it being understood that the disclosure should be considered as an exemplification of the principles of the subject matter of the present disclosure and is not intended to be limited to the specific embodiments shown and described.

[0089] In the following, for ease of understanding, various elements of the embodiments are described in operation. However, it is apparent that the corresponding elements are arranged to perform the functions described by them.

[0090] Furthermore, the subject matter of the disclosure is not limited to the embodiments but also comprises every other combination of the features described herein or recited in mutually different dependent claims.

[0091] Figure 1a An example of one embodiment of a FHE computing system 110 (e.g., a system that performs computations using fully homomorphic encryption (FHE)) is schematically illustrated. Figure 1a The system 110 can be used to perform computations on data even if the data is received in encrypted form, for example from a data provider.

[0092] The system 110 may include a processor system 130, a memory 140, and a communication interface 150. The memory 140 may include a local memory, such as a local hard drive or an electronic memory. The memory 140 may include a non-local memory, such as a cloud memory. In the latter case, the memory 140 may include a memory interface with the non-local memory. For example, the memory 140 may store encrypted data items, which are, for example, received from one or more data providers or generated as intermediate or final results (e.g., outputs) of a calculation. Typically, most or all of the data items (on which the calculations of the system 110 are performed) are encrypted with one (or more) keys unknown to the system 110, i.e., the system 110 may not be configured to obtain plaintext data items corresponding to the encrypted data items (e.g., such as stored in the memory 140). The decryption key in plain form is kept secret from the system 110, even if the encryption / decryption key is available in encrypted form. For example, the processor system may be configured to perform a sequence of FHE operations that may include such arithmetic operations as addition and multiplication on encrypted values, and may include arithmetic operations on encryption polynomials.

[0093] Performing computations using FHE operations can be accomplished by stringing a series of FHE operations one after another, with some FHE operations operating on input values ​​and other FHE operations operating on intermediate values. Typically, most or all of the operations are performed on encrypted values, making the FHE system unaware of the confidential values ​​it is processing.

[0094] One source of FHE computations is to start with a regular computation (e.g., a sequence of non-FHE operations) and convert the regular computation to an FHE computation. A particularly suitable source of computations that can be converted is a so-called computation network (also referred to as a computational network). In a computation network, nodes represent operations, and edges in the network indicate which operations take values ​​from other operations as input.

[0095] In one embodiment, a computation network may be represented as a directed acyclic graph, where nodes represent operations and directed edges indicate which operations take input from which previous operations. For example, a computation network may be represented in the form of a text file (e.g., as a sequence of operators). A computation network may also be represented as a connection table and information indicating the type of operator.

[0096] Input nodes in a computation network may represent external outputs, while output nodes may represent output values. This concept of a computation network may be extended while still being easily converted by implementation. For example, although in one implementation, the computation network does not include conditional jumps, conditional assignments may be allowed because conditional assignments may be expressed as FHE operations. More precisely, conditional assignments may be expressed as y=c? a:b (if c is true, then y=a is set, or if c is false, then y=b is set), and it is feasible to replace this operation in FHE by an equivalent operation y=c*(ab)+b, where c is restricted to be an encrypted bit 0 or 1. The computation network may also not include loops, for example, it may be acyclic, but this is not necessary, as loops with a fixed number of iterations may also be accommodated in FHE networks (e.g., by unrolling the loop). If there is an upper limit on the number of loops in the loop, even loops with a dynamically determined number of iterations may be converted; in the latter case, the dynamic loop may first be converted to a loop with a fixed number of iterations.

[0097] Figure 1dAn example of an embodiment of a system 114 for converting a computation for FHE is schematically shown. The system 114 may be configured to convert a first computation network, the first computation network being a first computation network arranged for computing an operation of a function, into a second computation network, the second computation network being a second computation network arranged for computing an FHE operation of the same function. Similar to the system 110, the system 114 includes a processor system 134, a memory 144, and a communication interface 154. In fact, the system 110 and the system 114 may be the same system, but this is not required. For example, the system 114 may receive a representation of a first computation network and convert it into a second computation network for FHE execution. For example, the system 114 may receive the first computation network from a third party (e.g., from a data provider system 160, or a similar system). Alternatively, the first computation network may originate from elsewhere, for example, a conventional compiler may receive a high-level description of the desired computation and generate the first computation network, for example, even multiple first computation networks each for a certain aspect of the computation, and perform the overall computation. Once converted, the second computation network may be transmitted to the system 110 for computation. If system 110 and system 114 are identical, the second computing network may be executed on the same system. For example, system 114 may be provided by the same party that provides computing system 110. Alternatively, a user of system 110 may install system 114 at his own location, such as by downloading appropriate software and converting his computing network using that software, which in turn may be uploaded to system 110 for execution.

[0098] System 110 and system 114 can communicate internally with each other via a computer network, and communicate with another system, external memory, input device, output device and / or one or more sensors. The computer network can be the Internet, an intranet, a LAN, a WLAN, etc. The computer network can be the Internet. The system includes a connection interface, which is arranged to communicate internally in the system or externally in the system as needed. For example, the connection interface may include a connector, such as a wired connector (e.g., an Ethernet connector, a fiber optic connector, etc.) or a wireless connector (e.g., an antenna (e.g., a Wi-Fi, 4G or 5G antenna)). Communication (e.g., internal communication) can use other communication protocols or media, such as an internal data bus.

[0099] In systems 110 and 114, communication interfaces 150 and 154 may be used to send and / or receive digital data. For example, system 110 may be configured to receive encrypted data items from an external computer (e.g., a data providing computer). For example, system 110 may be configured to transmit computational results to an external computer, typically in an encrypted format. For example, communication interface 150 may be used for internal communications in system 110, for example, for distributing computations across multiple computing entities (e.g., computing devices). For example, system 114 may be configured to receive a first computing network and / or transmit a converted second computing network.

[0100] The execution of the system 110 and / or 114 may be implemented in a processor system (e.g., one or more processor circuits (e.g., microprocessors)), embodiments of which are described herein. The system 110 and / or 114 may include multiple processors that may be distributed at different locations. For example, the system 110 and / or 114 may use cloud computing.

[0101] Some of the figures herein show functional units that may be functional units of a processor system. For example, the figures may be used as blueprints of possible functional organizations of a processor system. In most of the figures, the processor circuits are not shown separately from the units. For example, Figure 2 The functional units shown in (see below) may be implemented in whole or in part as computer instructions stored at a system (such as system 110 and / or 114) (e.g., stored in an electronic memory of the system) and executable by a microprocessor of the system. In a hybrid embodiment, the functional units are implemented partially in hardware (e.g., as a coprocessor, such as an arithmetic and / or cryptographic coprocessor) and partially in software stored and executed on system 110 and / or 114.

[0102] Figure 1b An example of one embodiment of a system for performing computations using FHE is schematically illustrated. Figure 1b In the context of the data provider system 160 Figure 1a The FHE system 110 is configured to perform computations using fully homomorphic encryption (FHE) cryptography.

[0103] For example, the system 110 can be configured to receive encrypted data items from the data provider 160. At least some of the data items can be received in encrypted form. Some of the data items can be received in plain text format. Computations can be performed on the received data items and can also be performed on the stored data items. Interestingly, computations can be performed on the encrypted data without decrypting the data, e.g., without converting the encrypted data items to data in plain text format.

[0104] The system 110 may be configured for a FHE scheme, e.g. for several FHE operations, e.g. sometimes referred to as gates. For example, the FHE system may be configured for so-called NAND-gates. For example, the FHE system may have addition operations and multiplication operations, e.g. in finite fields or finite rings, etc. The operations of the FHE scheme are in principle sufficient to perform a wide range of computations, except that the number of consecutive operations may be limited, e.g. if the FHE scheme is a hierarchical scheme without bootstrap operations or without employing bootstrap operations.

[0105] Typically, encrypted data in FHE schemes and implemented in FHE systems includes some measure of noise. For example, encryption of a data item may include mapping the data item to points in a key-dependent lattice, to which some noise is added.

[0106] When the data item is just encrypted, the noise is low (the encryption is recent). For example, the amount of noise is so low that if the data item is decrypted, the noise can be removed at some point in the decryption process (e.g., by rounding). On the other hand, the noise should be high enough to make attacks on the system difficult enough. For example, many FHE schemes can be attacked with linear algebra or other efficient algorithms (e.g., lattice-based algorithms) assuming that there is no noise. When the data item is encrypted, noise is added, and the noise is selected to make the attack difficult when the FHE operation can still be performed. Most FHE operations will increase the noise inherent in the encrypted FHE data item. When many such operations are performed, the noise will reach a level that makes correct decryption no longer possible. At this point, the scheme fails. Generally speaking, two methods have been developed in the art to deal with this phenomenon. The first is a hierarchical FHE method. A hierarchical FHE scheme can perform a finite number of operations one after another. The hierarchical FHE scheme is designed so that the worst-case noise at the end of such a number of operations is lower than the limit required for decryption. Another method is the so-called bootstrap operation. A bootstrap operation can reduce noise in an encrypted data item by returning a more recent encryption of the same data with less noise. Bootstrapping is significantly possible if the FHE scheme is sufficiently capable of homomorphically evaluating its decryption algorithm in the encryption domain (sometimes referred to as a bootstrappable FHE scheme). For example, the bootstrap operation receives auxiliary data related to the encryption key that allows noise to be removed without decrypting the encrypted data item. Typically, the auxiliary data is an encrypted version of the key used to decrypt the data item. It should be noted that in the case of symmetric key-based FHE, the decryption key may be the same as the encryption key, or in the case of, for example, an asymmetric key FHE scheme, the decryption key may be different from the encryption key. In addition to encryption keys and decryption keys, FHE schemes may use other keys, such as keys for homomorphic evaluation. The latter is typically a public key corresponding to a private key, which may be stored at the data provider.

[0107] Although the bootstrap operation performs decryption of the encrypted data item, counterintuitively, the decryption is performed in the encryption domain, so homomorphic decryption does not actually disclose any information about the plaintext data item. The bootstrap operation then performs noise removal (typically through homomorphic rounding). The result is an encrypted data item with a lower, fixed noise level. Typically, the noise present in the ciphertext produced by the bootstrap is caused by the bootstrap operation.

[0108] After the bootstrap operation, a new sequence of FHE operations can be performed until the noise level becomes so high that a new bootstrap operation is required.

[0109] Determining when bootstrapping is needed can be done by keeping track of how large the noise level will be, for example, assuming a worst-case scenario. Instead of a worst-case scenario, an average case scenario can be assumed, although this may increase the risk of computational results that cannot be deciphered. For example, an average-case analysis of noise propagation can be used to perform computations that are inherently robust to occasional errors, such as neural network evaluation.

[0110] Noise analysis (e.g., worst-case analysis or average-case analysis) can be performed in advance and can be performed dynamically. The increase in noise may depend on the specific calculation. For example, FHE scalar multiplication will increase the noise by an amount that depends on the multiplicand. One approach is to perform the bootstrap operation frequently, such as after each operation, or after every set number of operations (e.g., every two operations). Particularly useful are programmable bootstrap operations that, in addition to computing a function, can also reduce noise. For example, the function can be encoded as a lookup table, which is called a test vector for the programmable bootstrap.

[0111] The FHE scheme can be applied in many environments. For example, the FHE system 110 can be operated by a cloud provider. A cloud provider can provide computing and storage services to its customers. By adopting FHE encryption, data providers 160 (e.g., customers of the cloud provider) can send their data in encrypted form. The cloud provider can still perform the required computations, and / or the required storage, but will not be able to know the corresponding plaintext data, nor the results of the computations. For example, the data provider 160 can encrypt the data item using an encryption key of a type corresponding to the specific FHE system used. When the data provider 160 receives the computation results from the FHE system 110, the encrypted data item can be decrypted using the corresponding decryption key. The encryption key and the decryption key can be the same, which is typically the case.

[0112] For example, system 110 may execute a machine learning model, such as an image classifier, such as a medical model, without access to plaintext data input items. A user of system 110 may provide input values ​​in encrypted form and receive encrypted output values ​​from system 110. The machine learning model may be obtained from system 114. FHE system 110 may be used to provide a model (e.g., for medical data). This may typically be done using plaintext model parameters, but may also be done using encrypted data (e.g., encrypted input data, intermediate data, and output data). The effect of the system is to perform computations, such as image classification, such as medical image classification, without the computer being aware of plaintext data items. For example, a mammogram may be evaluated for cancer without the image being in plaintext at system 110, and without the system 110 knowing what the result of the cancer assessment was.

[0113] Other applications involve database services, such as looking up encrypted data in an encrypted database; for example, the computation may be a comparison between an input item and a database item. For example, multiple computations may be combined to produce a database index that matches an index. For example, the database may be a genomic database, and the input is a gene sequence. For example, system 110 may be used for protective control of equipment. For example, equipment (even large equipment such as a power plant) may send sensor values ​​to system 110 and receive encrypted control signals in return; the control signals are calculated from the sensor signals. An attacker of the system may be able to determine the content of data entering and leaving system 110, or even have access to intermediate data of system 110, but since the data is encrypted, this will not help him. Since the decryption key is unknown to system 100, even a complete cracking of system 110 will not disclose the data. Calculating the control signal may involve such mathematical operations as linear algebra, averages, matrix multiplication, polynomial evaluation, etc., all of which can be performed using FHE operations.

[0114] For example, a large number (e.g., a collection in a memory) of encrypted data items may be maintained in a FHE system; some of these may be received and some may be the results of FHE calculations, such as intermediate results. For example, a FHE system may be configured to apply a FHE operation to one, two or more encrypted data items in a memory, such as a collection of input values ​​and / or intermediate values ​​and / or output values. The result will be a new encrypted data item that can be stored in a memory. The encrypted data store may be stored in a memory of the FHE system. This may be a local memory or a distributed memory. In the latter case, it may occur that one or more encrypted data items are represented multiple times in the encrypted data store. If the value of the encrypted data item is needed elsewhere, the encrypted data item may be sent from one computing device to another computing device. The encrypted data store may be implemented in a variety of ways, such as as a register file, an array, a variety of data structures, etc.

[0115] For example, in a FHE scheme based on the learning-with-errors (LWE) problem, such as Torus-FHE (TFHE) mentioned in the background, the encryption key can be a string of n digits s i , the ciphertext can be a tuple (a1, ..., a n , b), where In the latter, + and · respectively represent the addition between torus elements and the product between an integer and a torus element, α i are n torus elements, μ is a plaintext data item that is a torus element, and e is noise (e.g., extracted from a probability distribution (e.g., Gaussian distribution)). Conventionally, the secret digits s1, ..., s n bits; but this is not required. j It can have more than two values, such as three values ​​(ternary digits), four values, or even more.

[0116] This approach naturally extends to other mathematical structures (including those based on polynomials). Thus, the number s i , a i , b, u, e may be taken from different mathematical structures. Not all encrypted data items need to be encrypted according to the same key, and in fact, re-encryption with a different key is a possible FHE operation. In order to switch from one key to another, a so-called key switching procedure may be used, which may use a so-called key switching key. The key switching key provides information about how an element encrypted according to a first key may be re-encrypted according to a second key without revealing either key.

[0117] Encrypted data items may represent all kinds of data. For example, an encrypted data item may represent a number that needs to be averaged or used for linear regression, etc. For example, an encrypted data item may represent an image. For example, each pixel of an image may correspond to one or more encrypted data items. For example, a grayscale pixel may be represented by a grayscale level, which in turn may be represented by a single encrypted data item. For example, 256 grayscale levels may be encoded as a single encrypted data item. For example, a color pixel may be represented by multiple color levels (e.g., RGB levels), which in turn may be represented by a tuple of encrypted data items. For example, three 256-level colors may be encoded by three encrypted data items. The number of encrypted data items used to represent a certain type of data depends on the capacity of the FHE scheme. For example, a more restrictive FHE scheme may only be able to encode one bit per encrypted data item. In this case, one color pixel may require 24 encrypted data items.

[0118] While it may not be possible to tell exactly how much noise there is without access to the decryption key, one can typically constrain the noise, e.g., because the initial noise level is known for a fresh encryption and the noise increase for various operations is known. The noise increase may depend on the type of operation (e.g., addition vs. multiplication) and other parameters (if any). For example, an FHE operation may be a multiplication by a known, e.g., plaintext value (e.g., multiplication by 2), a multiplication by a known polynomial, and so on. For example, multiplication by a larger value may add more noise than multiplication by a smaller value. Exactly how much noise increases, given the operations performed, can be calculated mathematically or estimated empirically. While some operations may add a lot of noise, other operations may not add noise, e.g., addition by a pure constant.

[0119] A set of FHE operations may be defined for a computation. For example, from a target FHE operation, a network or circuit of operations may be constructed that collectively implement the computation. For example, the operations may be Boolean operations. For example, the operations may all be NAND operations. The manner in which the FHE operations are combined (e.g., which operation is applied to which operand in the encrypted data store) determines the computation being performed. For example, a computation may be represented as a list of FHE operations to be performed and an indication of which FHE encrypted data item the list of FHE operations are to be performed on.

[0120] As operations are performed, the noise associated with newly calculated (non-recent) encrypted data items will grow. This is not a problem as long as the noise remains within the limits required for decryption. If more operations are to be performed, a bootstrap operation can be performed.

[0121] The size of encrypted data items in FHE schemes can be quite large. Furthermore, the more operations an FHE scheme performs without bootstrapping operations, the larger the size of the encrypted data items will typically be. Likewise, greater resistance to attack can be achieved with larger FHE parameters and, in turn, larger encrypted data items. For example, a data item can be mapped to an FHE system characterized by larger FHE parameters, so that more noise can be added while still being able to successfully perform operations. Increasing the range of possible values ​​for the key bits increases entropy, which helps improve the security of FHE.

[0122] Figure 1c An example of one embodiment of a system for performing computations using FHE is schematically illustrated. Figure 1c 1 is a data provider system 160, and three FHE devices: devices 111, 112, and 113. Each of devices 111, 112, and 113 is capable of performing FHE operations on encrypted data items. These three devices together constitute a FHE system. There may be two or more FHE devices cooperating to form a FHE system.

[0123] exist Figure 1c In the case of , the computation is distributed across multiple FHE devices (in the illustrated embodiment, three FHE devices). For example, one of the multiple FHE devices may transmit an encrypted data item to one or more of the other multiple FHE devices, such as a received encrypted data item or an intermediate encrypted data item, such as a partial computation result. Likewise, each of the multiple FHE devices may be configured to receive an encrypted data item from the other devices.

[0124] The primary embodiment of a computational network is a neural network, and it is desirable to convert the computational network from conventional computation to FHE computation. In one embodiment, a system for conversion takes as input a first neural network (an embodiment of a first computational network) and generates as output a second neural network (an embodiment of a second computational network). The second computational network may include representations of FHE operators, their parameters, etc. The second computational network computes the function represented by the first computational network, or typically an approximation of the function. In particular, for neural networks, computational approximations are acceptable.

[0125] Evaluating a neural network as a second computational network of FHE operations is useful to improve the privacy of a user. Other embodiments of the computational network include: (i) performing medical tests on a person's private information (e.g., DNA data) without revealing the private information or the test results, (ii) spam detection: a person can detect whether an email is spam without having a key to decrypt the email (e.g., without having to view the email).

[0126] An embodiment of the foregoing system may include a first device in which medical data is obtained, such as from a plurality of sensors. The sensors may include, for example, temperature sensors. The sensors may include image sensors. The data is encoded according to an input encoding and encrypted using a key. The encrypted data may then be sent to the medical device where a second computing network is applied to the encrypted data. The encrypted output may be a classification of the input, such as an image classification. The encrypted result may be sent back to the medical device for decryption and decoding.

[0127] An embodiment of the latter system may include a first mail device that receives an email, which may then be encrypted according to the FHE scheme, for example using a key known at the first mail device. The first mail device may first encode the email into a series of torus points, for example using an input encoding, and then encrypt the series into a series of encrypted values, for example using an LWE representation. The encrypted sequence is then sent to a spam detector device in which a second computing network is installed. The encrypted values ​​may also include other information, such as metadata, such as a starting address and / or a destination address. The second computing network is applied to the series of encrypted values ​​and produces an encrypted result, such as a binary spam or no spam result, or, for example, a real value representing the probability that the email is spam. The second computing network is obtained as a transformation of the first computing network, for example trained for spam detection.

[0128] Then, one (or more) encrypted output values ​​are sent from the spam detector device to the first mail device, where the output is decrypted, for example, using a key, and decoded, for example, using output decoding. Based on the result, the first mail device may or may not discard the mail. For example, the first mail device may decide whether to display the mail based on the decrypted output, or decide whether to forward the mail to a second mail device.

[0129] Interestingly, the use case is executing machine learning (ML) algorithms on untrusted platforms while maintaining user privacy: by using FHE in this case, one can keep the user's data or its results confidential.

[0130] Fully homomorphic encryption (FHE) is an encryption scheme in which one can perform computations on encrypted data using only public information. The FHE scheme does not require knowledge of, for example, the user's private key, and does not require decryption of the ciphertext. There are a variety of FHE schemes, and the TFHE (Torus-FHE) scheme is currently an efficient choice. The TFHE scheme supports efficient bootstrapping (BS), which allows noise reduction in the scheme, thereby supporting homomorphic evaluation of neural networks (NN) as deeply as possible. In addition, the bootstrap operation can be combined with any function that needs to be performed on the data, thereby combining function evaluation with noise reduction. This function is called programmable bootstrapping (PBS). For example, programmable bootstrapping can reduce noise and simultaneously execute, for example, activation functions.

[0131] A preferred class of FHE schemes is a torus-based scheme, referred to as a TFHE scheme. In a torus-based scheme, values ​​can be represented modulo 1. For example, the TFHE scheme can be based on the learning with errors (LWE) problem mentioned above. A preferred implementation of the TFHE scheme and programmable bootstrapping is provided in the paper "Programmable Bootstrapping Enables Efficient Homomorphic Inference of Deep Neural Networks", which is incorporated herein by reference.

[0132] TFHE supports some operations directly on encrypted data, without having to perform bootstrapping. These operations include modular addition (usually with 1 as the modulus), and multiplication between (plaintext) integers and encrypted values ​​(represented as torus elements). Encrypted values ​​in the torus scheme are mapped to real-valued numbers modulo 1, e.g., between 0 (inclusive) and 1 (exclusive). Although real numbers are mentioned, these values ​​are typically represented with fixed precision (e.g., 32 or 64 bit numbers).

[0133] One of the limitations of TFHE is that multiplication of two torus values ​​is unrestricted, since multiplication of real numbers by torus elements is unrestricted. Advantageously, multiplication by integers can be accomplished without programmable bootstrapping. In one embodiment, the second computation network is based on TFHE, and computations are performed on neural network encrypted data when executing the second computation network. Parameters in the second computation network are allowed to be plaintext values, but they operate on encrypted values. The encrypted values ​​are values ​​on the torus, e.g., real values ​​modulo 1. For example, a value on the torus (μ) can be encrypted as LWE ciphertext, e.g., by masking the value with a mask (b), the mask (b) being a series of key digits (s i ) and a series of mask values ​​(a i ) is obtained, where the noise (e) is: Conveniently, torus points are mapped to discrete rings.

[0134] The operations performed in the first computational network must be converted to the operations of the second computational network. Several objectives need to be considered. It is desirable to avoid programmable bootstrapping because they are much more expensive than operations that can be performed directly on encrypted data. Secondly, the resulting second computational network should be sufficiently accurate. For most practical applications, whether it is neural network evaluation or it calculates control signals from input sensor values, an exact copy is usually not required (although it is possible). Allowing some differences between the functions represented by the first and second computational networks allows for a much more efficient network, preferably, the accuracy should be kept within a certain limit. Whatever that limit is, it will depend on the application.

[0135] The difficulty in transforming a first computation network such as a neural network into a FHE computation network is that the first computation network may contain multiplications between encrypted values ​​and non-integer values. Such operations can be done, but only using programmable bootstrapping. For example, in a neural network, one may encounter an operation like this:

[0136] • Convolution, in which a small real-valued matrix called the kernel is multiplied by multiple selected portions of the input, which are shifted in the input.

[0137] Matrix multiplication, where two matrices are multiplied; in the NN example, one might contain ciphertext, and the other might be a constant matrix of real numbers. Matrix multiplication occurs, for example, in so-called fully connected layers or dense layers, sometimes called Gemm layers (for general matrix multiplication).

[0138] Matrix multiplication, such as matrix multiplication between two two-dimensional matrices or between two two-dimensional matrices and a vector, is an embodiment of matrix multiplication. Matrix operations can include matrix multiplication and matrix addition. Matrix multiplication can be more than two dimensions, and higher-dimensional matrices are usually called tensors.

[0139] Batch normalization, which can be viewed as a linear function f(x)=ax+b, where a and b are real numbers. Batch normalization can be viewed as a matrix operation, where the matrix is ​​al, where I is the identity matrix. Alternatively, batch normalization can be given an expansion factor, where batch normalization is replaced by f(x)=Round(αa)x+αb.

[0140] It is worth noting that an important class of neural networks (convolutional neural networks) use many convolutional as well as dense layers.

[0141] When transforming the first computational network, one strategy may be to replace nonlinear functions by corresponding programmable bootstrapping on encrypted values, and to replace linear operations by corresponding linear operations on encrypted values. Although a single bootstrapping operation is mentioned, the actual code may include multiple bootstrapping operations, or the same bootstrapping operation may be used multiple times.

[0142] Unfortunately, this approach cannot be used directly because linear operations typically include non-integer factors, which are not directly supported in FHE operations, for example, although they can be transformed using programmable bootstrapping. For example, for matrices, multiplications will typically involve many non-integer values. Therefore, it is desirable to transform the first computational network matrix operations with real-valued weights into TFHE matrix operations with integer-valued weights, thereby avoiding bootstrapping operations. This problem applies to matrix multiplication, convolution, matrix multiplication, batch normalization, and more generally, to any function whose output is a linear combination of some inputs.

[0143] Operations in the first computational network are generally not constrained to remain within a specific small interval, but the encrypted values ​​in the FHE operations are constrained to the torus and are thus modulo a modulus, typically 1. One solution to this is to scale all values ​​in the first computational network to values ​​that are small enough, between 0 and 1, and remain between 0 and 1. If the values ​​increase too much, they can occasionally be scaled to smaller values. Nonlinear operations can be adapted to the expected scaled values ​​rather than the original values. Interestingly, matrix multiplication is linear and applies to scaled values ​​as well as unscaled values. Matrix addition can be scaled proportionally to the scaling variable.

[0144] It is important to encode the values ​​as encrypted values ​​between 0 and 1. Although more optimal and suboptimal encodings can be designed, one can usually achieve a working encoding by direct scaling and shifting so that all values ​​lie in a small enough subinterval between 0 and 1. If the values ​​become too large, they may be scaled down occasionally. Note that scaling down a value can be done by programmable bootstrapping.

[0145] Figure 2 Schematically, an example of an implementation of a conversion system 200 is shown. The system 200 is configured to convert a first computational network, the first computational network being a first computational network arranged to compute an operation of a function, into a second computational network, the second computational network being a second computational network arranged to compute an FHE operation of the same function. An important class of networks are neural networks, but computational networks may also represent other functions. Figure 2 A first computing network 210 is schematically shown, showing operations 211, 212, and 214 of the first computing network 210. Typically, the first computing network 210 will include more than 3 operations, such as more than 10, more than 100, and so on.

[0146] The first computing network 210 includes a plurality of real-valued matrix operations, including real-valued matrix multiplication and at least a plurality of other operations. For example, the real-valued matrix multiplication may be a matrix multiplication, such as y=Wx or y=W.x+b, etc. In this embodiment, for vector x. Matrix multiplication may also be a multiplication with another matrix, such as y=WX+B. Matrix multiplication may include multiplication between two matrices. Matrix operations may also include matrix addition. In addition to matrix operations, the first computing network 210 also includes a plurality of other operations. Another operation receives the result of at least one of the plurality of real-valued matrix operations as input. There may be more operations, such as operations that receive input values ​​from other operations, operations that combine values ​​from different sources, and the like. One embodiment of other operations is an activation function, such as a sigmoid function, such as a function of the form z=Activation(y), where y may be the result of the above-mentioned matrix multiplication.

[0147] The first computational network (on which an embodiment may run) may be a very large and complex deep neural network. Figure 3 Schematically illustrated is an example of an implementation of a first computational network 300. The first computational network 300 is an example of a neural network that is successfully transformed into a second computational network by the implementation.

[0148] Figure 3 is a fragment of a network called GoogLeNet or Inception v1, which was the winner of ILSVRC 2014. Inception is a deep convolutional neural network architecture for object detection as well as image classification. More details of the network are available in the paper "Going deeper with convolutions" (incorporated herein by reference) and on the corresponding page of inception_v1 on the Open Neural Network Exchange (onnx) currently hosted on GitHub.

[0149] Figure 3 A plurality of operations and the layers between them are shown in FIG. The layers are represented by reference symbols between 1 and 99. For example, Figure 3 A concatenation operation is shown at the top of . A concatenation operation is indicated as 'Concat_xx', where xx indicates a specific concatenation operation. The output of a concatenation operation is the input of several other operations. The values ​​that are the output of one operation and the input of the next operation are collectively referred to as a layer. In this example, the concatenation operation Concat_52 provides a value for layer 53. A concatenation operation can be directly converted into an FHE operation because it only changes the order of the values, which can be done on plaintext values ​​as well as encrypted values. A concatenation operation is an embodiment of a so-called join operation because it combines values ​​from different operations. Join operations may require additional attention, as will be discussed below.

[0150] Figure 3 Also shown in FIG. 1 are convolution operations; indicated by 'Conv_xx'. Convolution is an embodiment of a matrix operation. Real-valued matrix operations include one or more convolutions with real-valued kernels. They can be replaced by matrix operations, operating on encrypted values; however, it must be noted that the real numbers in the convolution kernels must be replaced with integers.

[0151] Figure 3 A ReLU operation is also shown in . ReLU is an embodiment of a nonlinear operation, typically another operation, and ReLU can also take input from the concatenation operation, for example. In the conversion, ReLU as a nonlinear operation will be replaced by programmable bootstrapping.

[0152] Figure 3 Also shown in FIG. 1 is a pooling operation, in this example MaxPool. A pooling operation is a discretization process in which a set of input values ​​(e.g., an image, a hidden layer output matrix, etc.) is downsampled, thereby reducing its dimensionality. In the example of MaxPool, a subset of values ​​will be replaced by their maximum value. The MaxPool operation can be implemented by using a bootstrap operation, which takes the maximum of two values ​​and repeatedly applies it to obtain the maximum value of a subset.

[0153] Note that MaxPool can be implemented using ReLU, for example, MaxPool of two variables a and b can be implemented as a+ReLU(ba). MaxPool with more than two values ​​can be implemented as multiple MaxPools, each with more than two values.

[0154] There are other embodiments of pooling operations, in particular AveragePool, where a subset of values ​​is replaced by their average. AveragePool can be implemented as a summation, which can be performed directly on the encrypted values, and a division, which can be implemented as programmable bootstrapping.

[0155] Other operations that may occur in computational networks such as neural networks include flattening, transposing, and reshaping operations. These operations move values ​​around and can easily be adapted to work on encrypted values.

[0156] Dropout is an operation used during neural network training, but is typically replaced by the identity during inference.

[0157] The Add operation represents adding a layer; it is a linear operation, but is typically a union operation, so care needs to be taken as discussed in this article.

[0158] Figure 3 Not shown in , but also possible to use (among others) matrix multiplication of matrices with real-valued weights, and batch normalization with real-valued scalars are also possible operations. Matrix operations are indicated by Gemm_xx.

[0159] Other embodiments of the activation function include: sigmoid function, softmax function, softplus function, softsign function, tanh function, seLU function, eLU function, exponential function, LeakyReLU layer, PReLU layer, ThresholdedReLU layer.

[0160] Back to Figure 2 The first computing network (e.g. Figure 3) has been provided to system 200 and successfully converted to a second computational network, including removal of real-valued multiplications. For example, a file including the types of operations in the network, their parameters (e.g., weights), and connections can be provided in a digital format. For example, system 200 can receive the first computational format from an external source external to system 200. For example, system 200 can obtain the computational network as an output of a local compiler or an output of a neural network training unit, either of which can be installed on system 200.

[0161] To convert the first computational network, the matrix operations are converted into corresponding operations that can be performed on the TFHE encrypted value. The system 200 includes an expansion factor unit 230. The expansion factor unit 230 is configured to determine the expansion factor (α i ). Each matrix operation corresponds to an expansion factor. The idea is to convert a real-valued matrix (W) to, for example, a matrix multiplication with an integer-valued matrix by scaling the real-valued matrix with the expansion factor corresponding to the real-valued matrix and rounding it. For example, the matrix W can be replaced by the matrix W'=round(αW), where all entries are rounded individually. Rounding will typically be to the nearest integer, but alternatively, rounding down or rounding up is also possible.

[0162] This operation has several advantages. First, the multiplication with W' can be done directly on the encrypted value, because integer multiplication is supported, even if real-valued multiplication is not supported. Therefore, programmable bootstrapping is not required to support this operation, which is much more efficient. Secondly, since the expansion factor α can be arbitrarily selected, the loss of precision caused by replacing W with W' can be controlled. The higher the value of the expansion factor α, the more accurate the conversion. Typically, the expansion factor will be at least 1, but for matrices that happen to have large terms and relatively small effects on the overall function, values ​​less than 1 can also be used.

[0163] Once the matrices in a matrix operation are replaced by their scaled and rounded functions, the remaining operations can be adapted to the new matrices.

[0164] For example, the matrix operation Wx+B can be replaced by W′x+(αB). Note that there is no reason to round vector B because the addition of encrypted values ​​to plaintext real-valued numbers is supported. Other operations that receive inputs from matrix operations will now receive values ​​that are approximately α times larger. This can be taken into account by replacing function A(x) with function A(x / α). Since function A will be implemented by programmable bootstrap operations, adding additional division operations will not be a problem. In short, after adjusting the matrix operations so that the matrix only includes integers, additional operations (especially other operations) are adjusted to take into account the change in the size of the matrix. At this point, a second computing network can be created from the first computing network: all matrix operations are replaced by their integer matrix variants. Nonlinear operations can be replaced by one or more programmable bootstrap operations.

[0165] In other words, people can multiply the multiplication part of matrix operation with expansion factor (afterwards rounding), but this will cause the modification of the calculated value in the whole computing network part to be modified matrix operation.If matrix operation also has addition part, then the latter addition term can be multiplied by the same expansion factor, but rounding is not needed here.The output that matrix operation produces now increases to be similar to the multiple of expansion factor.Other operations adopting the input of modified matrix operation can be adjusted to receive the scaling result of scaling matrix operation as input.If the second matrix operation is after the modified first matrix operation, the second matrix operation can be adjusted by reducing the second matrix with the expansion factor of the first matrix operation and then the expansion factor of the second matrix operation itself.

[0166] The second computation network can now be constructed by implementing other operations (e.g., nonlinear operations as one or more programmable bootstraps). Matrix operations can be replicated in the second computation network because FHE supports real-valued addition as well as integer multiplication without programmable bootstrapping.

[0167] System 200 includes a conversion unit 240. For example, conversion unit 240 may receive the expansion factor from expansion factor unit 230. Conversion unit 240 may be configured to perform a conversion from a conventional operation in a first computing network to a FHE operation in a second computing network. Figure 2A second computing network 220 including FHE operations 221-224 is shown. In one embodiment, there is a correspondence between the first computing network and the second computing network, wherein matrix operations in the first computing network correspond to matrix operations on encrypted values ​​in the second computing network, and wherein other operations are mapped to one or more programmable bootstrap sets. In the conversion, most operators in the first computing network correspond to an operator in the second network. However, sometimes, it is better to introduce an additional operator. This is shown in the second network 220, which schematically has an additional operator 223. In practice, the first network 210 and the second network 220 typically have more operators, such as more than 10, more than 20, etc.

[0168] Note that in the above process, there are many choices for the expansion factor. A higher expansion factor ensures that the fidelity of the corresponding matrix operation in the second computing network is higher than that in the first computing network. However, a higher expansion factor also has disadvantages. The encrypted values ​​in the FHE scheme represent values ​​between 0 and 1, which means that the values ​​in the first computing network are represented by scaled values ​​in the second computing network, where the scaled values ​​fit between 0 and 1. In fact, most FHE operations have more stringent requirements, for example, requiring values ​​to fit between 0 and a certain upper limit, which can be 1 / 2 or even 1 / 4. Although this can be solved by sufficiently shrinking all values ​​so that each value fits in the required interval, this itself has the consequence of reducing the accuracy with which the second network can approximate the first computing network. Higher expansion factors make this problem worse by increasing the size of the values, thereby increasing the reduction ratio required to fit the values ​​in the specified interval. Therefore, the expansion factor both improves the accuracy of the second network, but also indirectly reduces the accuracy by increasing the requirements for encoding. To complicate this problem, the operations in the network have different effects on the final result. In other words, by applying larger expansion factors to some matrix operations and smaller expansion factors to other matrix operations, the accuracy of the second network is improved.

[0169] System 200 includes an accuracy unit 250 that can access sample storage 251 (e.g., is connected to sample storage 251, or includes sample storage 251). Accuracy unit 250 is configured to determine an accuracy measure of the second computing network to indicate the accuracy of the second computing network. There are many ways to perform the accuracy measure.

[0170] In a first approach, both the first and second computational networks are evaluated for the same plurality of sample values ​​(often referred to as a data set). The second computational network is then rated with respect to how close the output of the second computational network is to the output of the first computational network. For example, the accuracy measure may be mean squared difference, etc. - there are many accuracy measures that can be used. The input values ​​may be randomly selected, but more realistic results are achieved by testing the first and second computational networks with representative inputs. For example, in one embodiment, the sample memory 251 includes a plurality of input values. Preferably, the sample input values ​​represent the type of input that can be expected during computational use of the second computational network.

[0171] For example, if the first computational network is a neural network that expects images as input (e.g., medical images), the accuracy is preferably evaluated on similar medical images.

[0172] A second approach is to evaluate only the second network. In this case, a set of input values ​​and corresponding expected output values ​​may be provided to the system 200, for example, from the data provider 160. These outputs may be the same as the outputs that the first computational network would provide; however, this sample set may also be obtained independently of the first computational network. For example, the outputs may be reference values ​​or golden truth values. This approach has the advantage that the second network is evaluated with respect to its proximity to the function that the first network should evaluate, and is not penalized for failing to accurately reproduce the characteristics of the first computational network. For example, the second network may be evaluated on samples used to train or validate the first computational network.

[0173] Interestingly, in order to evaluate the second computational network, it is not necessary to actually perform operations on the encrypted values. Alternatively, the second computational network can be simulated by having the second computational network operate on plaintext values, simulating the same noise additions to the second network on plaintext values, etc. For example, matrix operations in the second computational network can be simulated by performing the same matrix operations on plaintext values ​​scaled in the same manner. For example, programmable bootstrap operations in the second computational network can be evaluated by using programmable bootstrap test vectors as a lookup table that maps plaintext values ​​to plaintext values. Obviously, this approach is not suitable for computational purposes, but it provides the same or similar precision values ​​at a significantly lower computational cost.

[0174] After obtaining the precision value, the second computing network can be improved by changing the expansion factor. For example, the system 200 may include an optimization unit 231, which is configured to iteratively optimize the expansion factor (α i) to increase the precision measure. The precision measure may take into account the precision loss due to the large expansion factor. Alternatively, the optimization may optimize the precision measure while also optimizing other objectives (such as the execution speed of the second computational network). A variety of optimization schemes may be used. For example, the optimization may use hill climbing, simulated annealing, genetic programming, etc. For example, for the expansion factor (α i ) optimization may include one or more of the following: exhaustive search, logarithmic search, probabilistic method, gradient search.

[0175] Various details, alternatives, and implementations are discussed in further detail below. Consider first a simple fragment of a neural network. Assume that in a first computational network, one has y=W.x+B, followed by z=Activation(y), where W is a constant real-valued matrix called a weight matrix, and B is a constant real-valued matrix called a bias matrix. In order to perform equivalent computations on encrypted data, the values ​​operated on the first computational network are scaled down to fit in the interval 0-1. A suitable amount of scaling can be obtained from the first computational network, for example by evaluating on multiple inputs. The real-valued matrix W is converted to an integer-valued matrix, such as a matrix of integers. Once this is done, matrix multiplication is defined in a matrix based on FHE (such as TFHE). Set an expansion factor α (the expansion factor α can be a real number) and replace the computation with y′=W′.x+B′ and z′=Activation′(y′), where:

[0176] W′ is an integer-valued constant matrix defined by W′=Round(α*W), where Round() is a rounding function to the nearest integer,

[0177] B′ is a real-valued constant matrix, defined by B′=α*B,

[0178] Activation′ is the function Activation′(x)=Activation(x / α),

[0179] x has been encoded in the torus, while y' and z' are encoded on the torus.

[0180] This is a valid modification because:

[0181] Now, the weight matrix W′ consists only of integers and is therefore compatible with TFHE operations,

[0182] The calculation is approximately correct because as long as α is large enough, and Note that y' is approximately α times larger than y. By scaling the initial input small enough, the intermediate values ​​will also become small. The activation function restores a value to its original range; if this range is too large, an additional scaling factor can be incorporated into the activation function, for example replacing the function A() with A() / d+s.

[0183] The values ​​in the modified segments are approximately in a fixed relationship to the values ​​in the original network. As noted above, the larger the expansion factor α, the more accurate the new calculation is relative to the original calculation. However, a large expansion value α means that the calculated values ​​are in a larger range, which may cause accuracy problems when performing FHE calculations on the lowest values ​​in this range. Therefore, the goal is to find the minimum α that makes the accuracy of the modified neural network close enough to the original accuracy. Various types of search algorithms can be used to find α.

[0184] This approach is particularly effective for neural networks because they inherently support noise. They can absorb a certain amount of modification and rounding in their layers without a large impact on the function. This reduces the impact of using rounding operations on the matrix. The activation function can be replaced by a PBS operator in the second computational network. The computational cost of supporting the original activation function or the adjusted activation function is the same. Finally, the modified neural network (i) is fully compatible with TFHE and (ii) has an accuracy close enough to that of the original NN.

[0185] In particular, real-valued weights are allowed in the first computational network. An alternative solution would be to use a different training algorithm that forces the entries of the weight matrix to be integers. Yet another option would be, given the first computational network, to train an intermediate computational network that uses only integer-valued weights, and then to convert the intermediate network. Both approaches are undesirable. Preferably, the developers of the first computational network are free to use the training scheme that best suits them, while retraining the network is computationally expensive. Another advantage of the conversion system is that it supports complex, state-of-the-art neural networks; for example, where multiple summations are not always followed by activations, but rather, for example, joint operations. Multiple summations refer to a general term for Gemm, matrix multiplication, Conv, or BatchNorm; that is, any operation y=Op(x) where for the elements y of y i , Op is limited to y i =∑ j c i，j x j , where c i，j is a constant multiplier. The constant multiplier may be a real value in the first computation network and replaced by an integer value multiplier in the second computation network.

[0186] Figure 4aSchematically illustrated is an example of one implementation of a first computing network 410. The first computing network 410 may also be considered a segment of a larger first computing network. Figure 4a 414, and output 415. At input 411 and output 415, input values ​​are received or generated, which will be encrypted values ​​in the transformed second computing network. In this embodiment, operation 412 is a flattening operation, operation 413 is a general matrix multiplication, and operation 414 is an activation function, which is a ReLU operation in this example. Figure 4a Also shown in FIG. 3 are three layers: layer 1, layer 2, and layer 3.

[0187] The relationship between the values ​​in the layers of the first computing network and the values ​​in the layers of the second computing network can be expressed as a layer factor. The layer factor is the multiplicative ratio between the values ​​in the layers of the first computing network and the corresponding values ​​in the corresponding layers of the second computing network.

[0188] Layers may be further specified as so-called tensors of neurons: a tensor is an array of any dimension, not necessarily limited to one or two dimensions, for example three or more dimensions. For example, one may have a layer of a tensor T of shape (2, 3, 5), which means that it corresponds to 30 = 2 × 3 × 5 neurons. For example, T[i, j, k], where i is in {0, 1}, j is in {0, 1, 2}, and k is in {0, 1, 2, 3, 4}. Using multidimensional representations is often helpful in better understanding the operation of neural network operations and / or their correspondence with real objects (e.g., images). However, tensor operations boil down to matrix operations and can be viewed as matrix operations. For example, the tensor defined in the first computational network may have real-valued elements, which may be converted to integer-valued elements according to one embodiment.

[0189] Matrix operations and expansion factor α i Associated, the expansion factor α i Typically a real-valued number. In matrix operations, the expansion factor multiplies the input layer factor by α i to get the output layer factor. Typically, the values ​​of neurons in a given layer have the same expansion factor and thus the same layer factor. Note that, strictly speaking, the input layer factor is not exactly multiplied by α due to rounding. i to get the output layer factor, as rounding will distort the relationship slightly. However, this distortion is usually small enough to be negligible.

[0190] In layer 1 as well as in layer 3, the layer factor is 1. This is the case for layer 1 because the flattening operation does not change the magnitude of the values. This is the case for factor 3 because the output of the activation function does not change, i.e. the output factor is 1. Because of the α0 expansion factor of the Gemm_1 layer, layer 2 is approximately multiplied by α0 in the modified network. The activation ReLU is replaced by Relu(x / α0) to accommodate the expansion factor of the layer. Note that Relu(x / α0) will be done by using programmable bootstrapping in the second computation network.

[0191] Fork and join layers

[0192] Computational networks can have complex topologies. To illustrate this, consider a neural network. A forked layer is a layer that serves as a source for at least two operations in the next layer, while a joined layer is a layer that uses values ​​produced by at least two previous layers, at least two previous operations, or more. Figure 4b An example of an implementation of a first computational network 420 with forked and joined layers is schematically shown. Here, layer 13 is a fork because it is used in layers 14 and 17, and layer 18 is a join because it uses layers 16 and 17 as sources. Forked and joined layers are important for deep networks such as Resnet, VGG, and MobileNet.

[0193] Typically, the joint operation combining two or more layers is addition, subtraction or concatenation etc. Addition, subtraction or concatenation can be performed directly on the encrypted value without the need for PBS.

[0194] When two or more layers are combined in a joint layer, their layer factors are preferably equal or substantially equal. If not, the result will not correspond to what is manageable for the next operation and the accuracy will drop significantly. All layers in the second computation network are preferably homogeneous, that is, each layer has a single expansion factor; note that the expansion factors of different layers can be different or equal. If all input layers of a joint operation have the same layer factor, the joint operation is said to be homogeneous. Therefore, when the first computation network includes bifurcations as well as joints, solving for the expansion factor is more complicated. In order to have equal layer factors in a joint operation, one can force the operations from which the layers originate to have equal expansion factors. One can also balance the layer factors by adding additional balancing operations in the second network.

[0195] Interestingly, one can propagate expansion factors in the first computation network even if it has a complex topology, and one can obtain the constraints they need to satisfy, in particular which expansion factors need to be equal. For example, the conversion system can be configured as:

[0196] Analysis: For all operations of a neural network / function that multiplies a real number, assign an expansion factor α i , and determine how the expansion factor propagates through the first computing network. When the expansion factors arrive at the same joint layer, constraints are introduced, for example, the corresponding expansion factors must be equal or made equal through additional equalization operations.

[0197] Solve: Solve the system of equations. For example, reduce the dimensionality of the problem by reducing the number of expansion factors, some of which become equal through the solution step. At the end of the solution step, the network is homogeneous.

[0198] · Search: Find the minimum variable {α i}, the minimum variable gives sufficiently good accuracy; when there are not too many α i This can be done by exhaustive search when , or by other gradient-based methods or fast-converging methods when there are a large number of parameters;

[0199] Replacement: Once α i It is known to replace the weights with their corresponding integer weights, modify the bias, and compensate for the computation in the next activation or function replaced by programmable bootstrapping.

[0200] To track the propagation of the expansion factor, one can calculate the layer factor. For example, this can be done using the following rule:

[0201] The factor of the starting layer or input layer is 1.

[0202] The factor after a real-valued operation is the factor before the operation multiplied by α i .

[0203] The factor after activation can be 1; in an advanced embodiment, the factor after activation itself can also have an expansion factor. In one optimization, the latter expansion factor can be directly set to the existing α in the solution part i One advantage is that it reduces the number of additional layers that need to be inserted for the transformation.

[0204] The input factors of the union layers are preferably the same; the outputs of these union layers are then the input factors of their sources.

[0205] For other operations, such as transpose, flatten, and reshape, the output factors are the input factors.

[0206] For pooling operations, the output factor is the input factor.

[0207] Back to Figure 4b , consider the forking layer 13 and the corresponding forking operation 421, and the joining layer 18 and the corresponding joining operation 422. Figure 4b In , expansion factors are introduced by operations Conv_13, Conv_15, and Conv_16, such as factors α0, α1, and α2, respectively. The Relu_14 operation is modified to accept inputs scaled by factor α0, but produce unmodified outputs. That is, after Relu_14, the expansion factor of the layer is 1.

[0208] One can see that layer 16 has a layer factor of factor α1, while layer 17 has a layer factor of α2. The join operation 422 (in this case an addition operation) requires equal layer factors, so the join operation 422 introduces the constraint that α1 = α2. As a result, one can no longer consider them independent, but equal.

[0209] This method is preferred because it is easy to implement. The disadvantage of solving the joint layer in this way is that it removes the expansion factor α to some extent. i This reduction in degrees of freedom can cause the optimization solution to find a larger value for the expansion factor or a lower-than-optimal accuracy.

[0210] Other options are possible, as will be further expanded below. For example, one can introduce an expansion factor at the output of Relu_14. In this case, one can let Relu_14'(x)=Relu-14(x)*(α2 / α1) to make the joint operation 422 homogeneous, where the expansion factor is α2. In this way, α1 is retained for optimization and can be selected independently of α2. Compared to the original ReLU in the first computational network, the ReLU function in the second computational network can also be made Relu_14′(x)=Relu_14(x / α0)*(α2 / α1) by adjusting the expansion factor. It is not necessary to introduce an expansion factor different from 1 at the output of another function (for example, a function implemented as a PBS), but this can further improve the second computational network.

[0211] Figure 5a An example of an implementation of a first computational network 510 is schematically shown. The first computational network 510 comprises a forking layer 3 with a forking operation 511 and a joining layer 7 with a joining operation 512. If one imposes a condition on the equality of the layer factors arriving at the operation 512, this may result in a condition where the layer factors of both layer 3 and layer 6 are 1. Layer 3 will have no expansion factor, or in other words, an expansion factor of 1, while layer 6 will have the expansion factor of Conv_5 as the layer factor. This in turn results in a non-free conversion of the conv_5 operation. This in turn may result in poor accuracy, especially when conv_5 happens to be an important operation. There are a number of ways to avoid this difficulty.

[0212] Figure 5bAn example of an implementation of a first computing network 520 which has been partially transformed by introducing a correction operation 525 is schematically shown. Figure 5b The other parts shown have Figure 5a To further transform network 520, matrix operations Conv_3 and Conv_5 can be replaced with corresponding matrix operations, but according to one embodiment, their matrices are converted to integers. Nonlinear operations Relu_2 and Relu_4 can be replaced with PBS types. Add_6 operation is also linear and can be replaced with corresponding addition (but on encrypted values). As in Figure 5a There is a condition that the input layer of operation Add_6 is homogeneous. Figure 5a and Figure 5b The difference between is the introduction of correction operation 525. Operation 525 is a correction operation inserted into the computation network to help make the second computation network homogeneous (joint operation Add_6 in this example) without having to lose too many degrees of freedom.

[0213] Correction operation 525 introduces an additional expansion factor of the value, such as a function α·x. Considering the constraints imposed by the join operation (here Add_6), operation 525 will be a function f(x)=α1*x, so that layer 3 and layer 6 both have an expansion factor of α1. Inserting a scaling operation between two consecutive operations introduces an additional factor that can be optimized. The inserted correction operation splits the layers, in this example, splitting layer 3 into 3 and 3'. For example, the system can be configured to introduce a correction operation such as operation 525 between the PBS and the join operation if the join operation forces the expansion factor in the matrix operation to be 1.

[0214] There are two options for operation 525. A first option may be that operation 525 may be an integer multiplication. This has the advantage of avoiding the costly PBS, but has the disadvantage that the introduced factors (e.g., α) are restricted to integers. This in turn typically means that the expansion factor α1 will be larger than it would otherwise need to be (to efficiently approximate the corresponding matrix operation), so that α1 is an integer, in this example the operation Conv_5. Larger expansion factors may themselves increase noise, for example by encoding values ​​into smaller intervals. Alternatively, larger parameters may be supported, but at an increased cost, for example by increasing the number of bits used to represent real numbers in the FHE scheme.

[0215] A second option may be that operation 525 may be implemented directly by a PBS (corresponding to f(x)=α*x). This has the advantage of not restricting the choice of α1 to be an integer, but at the expense of slower execution speed, since PBS is typically significantly slower than integer multiplication.

[0216] For example, in one embodiment, one or more correction operations are inserted into the second computational network to scale their inputs by correction factors. The correction operations may be programmable bootstrap and the correction factors may be real numbers, or the correction operations may be integer multiplications and the correction factors may be integers.

[0217] Note that, in general, people can insert to go from layer X with factor α0 to layer X' with factor α1. This is also useful, but it is best to avoid this solution if possible.

[0218] One way to obtain the expansion factors and optionally inserted correction operations is to symbolically associate the expansion factors to each matrix operation in the first computational network, then propagate the layer factors through the first computational network, and derive the conditions on the expansion factors from the operations in the first computational network applied to the outputs of the conversion operations, so that the network is homogeneous. A solution step may be involved here. For example, the conditions may be analyzed and the dimensionality of the expansion factors may be reduced, for example by imposing an equality condition between two or more expansion factors. For example, it may be verified whether the conditions are completely solvable, and if not, additional operations are inserted.

[0219] For example, the expansion factor (α) can be modified under the condition that the joint operation is homogeneous. i ) are iteratively optimized. The expansion factor can be optimized for precision. The optimization can collectively minimize the size of the expansion factor. For example, the expansion factor can be minimized under the condition of achieving a minimum level of precision. For example, one can optimize the weighted sum of the precision term and the expansion factor weight term.

[0220] For example, in one embodiment, a first optimization phase finds a set of optimized expansion factors without introducing correction operations. If the precision is not sufficient, correction operations can be introduced in a second optimization phase. The latter can be solved by allowing real values ​​for the expansion factors introduced by them. In the second phase, the expansion factors introduced by the correction operations can be replaced by integers, for example, by rounding the expansion factors of the correction operations, for example, rounding them up. If the replacement with integers does not reduce the precision too much, it can be kept as an integer multiplication, thus avoiding PBS. To perform integer multiplications, one can directly choose the expansion factors so that their ratio is an integer: in our embodiment, α0 and α1 can always be chosen so that is an integer to allow the correction operations to be integer multiplications. It can be decided on a case-by-case basis which correction operations are optimized to be implemented as integer multiplications and which are optimized to be implemented as PBSs, or it can be done by some algorithm (eg, a genetic algorithm).

[0221] In one embodiment, an additional factor is introduced that can be used for optimization without introducing a new correction operation. This can be done by adjusting the output of the other operation to modify the other operation. For example, in one embodiment, an expansion factor is applied to the output of the PBS; for example, for an expansion factor β, the function f(x) can be replaced by a function βf(x). If the input of the function is also adjusted to accommodate the expansion factor on the input layer, the function can be adjusted to βf(x / α) taking into account the expansion factor α. This approach is particularly useful for functions that are implemented as PBSs in the second computing network.

[0222] For example, for an activation function replaced by PBS, one can add an expansion factor to the output, e.g., for a certain β, replace Activation(x) with Activation′(x)=β*Activation(x). The value β can be obtained by optimization (e.g., by joint operation) under the constraints imposed by the network.

[0223] The expansion factor β can be any available α i The product of a single positive integer power or a single negative integer power. For example, one can have: β = α0, β = α0*α1, or β = α0 -1 *α1 -2 etc. to give some examples. Consider Figure 5a As an example, this segment can be optimized better without adding the correction operation 525 (eg, not performing PBS on the left branch).

[0224] For example, one can apply the factor This results in the layer factor of layer 5 being modified (compared to the original network) to After Conv_5, layer 6 now has a layer factor of 1, since Conv_5 itself had an expansion factor of α1. At the joint layer (e.g., layer 7), source 6 is now homogeneous with source 3, i.e., they have the same layer factor, so the joint operation calculation can be completed properly: Finally, layer 7 has a layer factor of 1.

[0225] Figure 5a Another option in would be to apply an expansion factor α1 to the output of ReLu_2, and consider the values ​​used as input in Relu_4 to have a layer factor of α0*α1. This would also make the input to Add_6 homogeneous, with a factor of α1. However, this approach may not be as good as just dealing with the expansion factor on the output of Relu_4, because this approach may cause some values, especially those of layer 4, to grow to a significant multiple (i.e., α0*α1), which may be bad for accuracy. Depending on the data set, this approach may also be better. This can be addressed in the optimization scheme.

[0226] In terms of performance, applying an expansion factor to the output of a PBS is better than adding a PBS or integer multiplication because it does not add any extra operations. In addition, this method maintains all expansion factors α i Independence makes it more likely that good solutions will be found.

[0227] In one embodiment, the optimization may follow the following stages. In the first stage, a solution is sought without introducing correction operations or new expansion factors, for example, in the output of the PBS. For example, for all operations involving real multiplications of the first computational network, an expansion factor α is assigned. i Conditions on the expansion factors are derived by determining how the expansion factors propagate. In particular, a join operation that requires homogeneous inputs (e.g., addition or subtraction or concatenation of join functions) implies conditions on the expansion factors of its inputs. The optimization makes the expansion factor {α i}, while giving sufficiently good accuracy. If the number of expansion factors is small, this can be done by exhaustive search, for example, optionally using logarithmic search. If the number of expansion factors is large, any existing optimization scheme can be used, for example, hill climbing, possibly combined with gradient-based methods, etc.

[0228] In the second stage, a correction operation is introduced and an improved solution is sought to allow the new expansion factor to be a real number. Finally, the correction operation is combined with the previous PBS if possible, or implemented as an insertion operation. In the latter case, the system attempts to round the new factor to an integer so that the correction operation can be implemented without a PBS. Instead of rounding, one can pick the expansion factors so that their ratio is an integer.

[0229] Finally, once the expansion factor is known, one can replace the matrices, for example replacing the weights with their corresponding integer weights, modifying the corresponding biases and compensating the computation in the next activation or function replaced by the PBS.

[0230] For example, once an acceptable set of expansion factors is found that makes the accuracy close enough to the accuracy of the original network, the matrix W can be replaced by Round(αW) by using the corresponding expansion factor α, and the addition part of the matrix operation, such as B, can be replaced by αB. Activation can be replaced by Activation′(x)=Activation(x / α in ) / α out , where α in is an appropriate factor of the input, α out is the appropriate factor of the output (if any).

[0231] The conversion systems 114 and 200 may include a communication interface. The communication interface may be selected from a variety of alternatives. For example, the interface may be a network interface leading to a local area network or a wide area network (e.g., the Internet), a memory interface leading to an internal data storage device or an external data storage device, a keyboard, an application program interface (API), etc. Specifically, the communication interface may be configured to be connected to a computer network. The communication interface may be used to receive a digital data set. The data set may include data items, such as parameters, values, etc. indicating a first computing network and on which the system will perform the conversion. The data may also include a sample set for evaluating an accuracy parameter. The data may also include a desired level of accuracy.

[0232] A conversion system such as system 114 and / or 200 may have a user interface, which may include well-known elements such as one or more buttons, a keyboard, a display, a touch screen, etc. The user interface may be arranged to accommodate user interaction to configure the system, apply the system to new data, etc. The memory may be implemented as electronic memory (such as flash memory), or magnetic memory (such as a hard disk or the like). The memory may include multiple discrete memories that together constitute the memory. The memory may include temporary memory, such as RAM. The memory may be cloud storage.

[0233] The system (e.g., systems 114 and 200) may be implemented in a single device. Typically, the system includes one or more microprocessors that execute appropriate software stored in the system; for example, the software may have been downloaded and / or stored in a corresponding memory, for example, a volatile memory such as RAM or a non-volatile memory such as Flash. Alternatively, the system may be implemented in whole or in part in programmable logic (e.g., as a field programmable gate array (FPGA)). The system may be implemented in whole or in part as a so-called application-specific integrated circuit (ASIC), for example, an integrated circuit (IC) customized for its specific purpose. For example, the circuit may be implemented in CMOS, for example, using a hardware description language such as Verilog, VHDL, etc. In particular, the system may include circuits for evaluating cryptographic primitives.

[0234] The processor circuit may be implemented in a distributed manner (e.g., as multiple sub-processor circuits). The memory may be distributed across multiple distributed sub-memories. Some or all of the memory may be electronic memory, magnetic memory, etc. For example, the memory may have a volatile portion and a non-volatile portion. A portion of the memory may be read-only.

[0235] Figure 6An example of an implementation of a conversion method 600 is schematically shown. The method 600 is preferably executed on a computer, which may be a single computer device or a system of multiple computer devices. The method 600 converts a first computing network into a second computing network, wherein the first computing network is a first computing network arranged to compute an operation of a function, and the second computing network is a second computing network arranged to compute an FHE operation of the same function. The method 600 includes:

[0236] - determining (610) a set of expansion factors, one expansion factor for each matrix operation,

[0237] - converting (620) a plurality of real-valued matrices (W) corresponding to a plurality of real-valued matrix operations into integer matrices by scaling the real-valued matrices with expansion factors corresponding to the plurality of real-valued matrices and performing rounding,

[0238] - adjusting (630) a plurality of further operations to receive as input a scaled result of the scaled matrix operation, the second computation network comprising the converted plurality of integer matrices and the adjusted plurality of further operations,

[0239] - determining (640) an accuracy measure of the second computing network indicative of the accuracy of the second computing network,

[0240] - Iteratively (650) optimize the set of expansion factors, thereby improving the accuracy measure. For example, optimization may use multiple parts of method 600, for example determining 610 may be done multiple times.

[0241] Many different ways of performing the method are possible, as will be apparent to one of ordinary skill in the art. For example, the order of the steps may be performed in the order shown, but the order of the steps may be changed, or some steps may be performed in parallel. In addition, other method steps may be inserted between the steps. The inserted steps may represent improvements to the method described herein, or may be unrelated to the method. For example, some steps may be performed at least partially in parallel. In addition, a given step may not be completely completed before the next step begins.

[0242] The embodiment of the present method can be performed using software, and the software includes instructions for causing the processor system to perform method 600. The software may only include those steps taken by the specific sub-entity of the system. The software can be stored in a suitable storage medium (such as, hard disk, floppy disk, memory, CD-ROM, etc.). The software can be sent as a signal by wired or wireless or using a data network (for example, the Internet). The software can be on a server for downloading and / or remote use. The embodiment of the present method can be performed using a bit stream, and the bit stream is arranged to be used to configure a programmable logic (for example, a field programmable gate array (FPGA)), so as to perform the present method.

[0243] It will be understood that the subject matter of the present disclosure is also extended to a computer program suitable for putting the subject matter of the present disclosure into practice, particularly a computer program on or in a carrier. The program can be in the form of source code, object code, code intermediate source, and object code such as partially compiled form, or any other form suitable for implementing the embodiment of the present method. The embodiment related to the computer program product includes computer executable instructions corresponding to each processing step of at least one method stated. These instructions can be subdivided into subroutines and / or stored in one or more files that can be statically or dynamically linked. Another embodiment related to the computer program product includes computer executable instructions corresponding to each device, unit and / or part of at least one system and / or product stated.

[0244] The following numbered clauses include contemplated and non-limiting embodiments:

[0245] 1. A computer-implemented method for converting a first computing network into a second computing network, wherein the first computing network is a first computing network arranged to compute an operation of a function, and the second computing network is a second computing network arranged to compute a FHE operation of the same function,

[0246] - the first computational network of operations comprises a plurality of real-valued matrix operations (y=Wx) and at least a plurality of further operations (z=Activation)(y)), the plurality of real-valued matrix operations (y=Wx) comprising real-valued matrix multiplications, the at least a plurality of further operations (z=Activation(y)) receiving as input a result of at least one of the plurality of real-valued matrix operations,

[0247] - Converting the first computing network comprises:

[0248] - Determine the expansion factor (α i ), each matrix operation corresponds to an expansion factor,

[0249] - converting a plurality of real-valued matrices corresponding to the plurality of real-valued matrix operations (W) into integer-valued matrices by scaling the real-valued matrices with corresponding expansion factors of the plurality of real-valued matrices (W) and rounding the real-valued matrices,

[0250] - adjusting the plurality of further operations to receive as input a scaled result of the scaled matrix operation, the second computation network comprising the converted plurality of integer-valued matrices and the adjusted plurality of further operations,

[0251] - determining an accuracy measure of the second computing network indicative of the accuracy of the second computing network,

[0252] - Iteratively optimize the expansion factor (α i ) to improve the accuracy measure.

[0253] 2. The conversion method as described in the preceding clause, wherein:

[0254] - the expansion factor is a real number, and / or

[0255] At least some of the further operations are implemented as programmable bootstrap in the second computing network.

[0256] 3. A system for converting a first computational network arranged to compute an operation of a function into a second computational network arranged to compute a FHE operation of the same function, the system comprising:

[0257] a communication interface for receiving a first computational network of operations, the first computational network of operations comprising a plurality of real-valued matrix operations (y=Wx) and at least a plurality of further operations (z=Activation(y)), the plurality of real-valued matrix operations (y=Wx) comprising real-valued matrix multiplications, the at least a plurality of further operations (z=Activation(y)) receiving as input a result of at least one of the plurality of real-valued matrix operations,

[0258] - a processor system configured to transform the first computing network, the transforming comprising:

[0259] - Determine the expansion factor (α i ), each matrix operation corresponds to an expansion factor,

[0260] - converting a plurality of real-valued matrices corresponding to a plurality of real-valued matrix operations into integer matrices by scaling the real-valued matrices with corresponding expansion factors of the plurality of real-valued matrices (W) and rounding the real-valued matrices,

[0261] - adjusting the plurality of further operations to receive as input a scaled result of the scaled matrix operation, the second computation network comprising the converted plurality of integer matrices and the adjusted plurality of further operations,

[0262] - determining an accuracy measure of the second computing network indicative of the accuracy of the second computing network,

[0263] - Iteratively optimize the expansion factor (α i ) to improve the accuracy measure.

[0264] Figure 7a A computer readable medium 1000 having a writable portion 1010 and a computer readable medium 1001 also having a writable portion are shown. The computer readable medium 1000 is shown in the form of an optically readable medium. The computer readable medium 1001 is shown in the form of an electronic memory, in this case a memory card. The computer readable media 1000 and 1001 can store data 1020, wherein the data can represent instructions that, when executed by a processor system, cause the processor system to perform an embodiment of a conversion method according to an embodiment. The computer program 1020 can be implemented on the computer readable medium 1000 as a physical mark or by magnetization of the computer readable medium 1000. However, any other suitable embodiments are also conceivable. In addition, it will be understood that although the computer readable medium 1000 is shown as an optical disk here, the computer readable medium 1000 can be any suitable computer readable medium, such as a hard disk, solid state memory, flash memory, etc., and can be non-recordable or recordable. The computer program 1020 includes instructions for causing the processor system to perform the conversion method. The computer-readable medium may also or instead include a result of performing a method according to an embodiment, such as a second computing network.

[0265] Figure 7b A schematic representation of a processor system 1140 according to one embodiment of a FHE device or system is shown. The processor system includes one or more integrated circuits 1110. The architecture of the one or more integrated circuits 1110 is schematically shown in FIG. Figure 7b. The circuit 1110 includes a processing unit 1120 (e.g., a CPU) for running a computer program component to perform a method according to an embodiment and / or implement its modules or units. The circuit 1110 includes a memory 1122 for storing programming code, data, etc. A portion of the memory 1122 may be read-only. The circuit 1110 may include a communication element 1126, such as an antenna, a connector, or both. The circuit 1110 may include an application specific integrated circuit 1124 for performing some or all of the processing defined in the method. The processor 1120, the memory 1122, the application specific integrated circuit 1124, and the communication element 1126 may be connected to each other via an interconnect 1130 (e.g., a bus). The processor system 1110 may be arranged for contact and / or contactless communication, using antennas and / or connectors, respectively.

[0266] For example, in one embodiment, the processor system 1140 (e.g., a FHE system or device) may include a processor circuit and a memory circuit, the processor being arranged to execute software stored in the memory circuit. For example, the processor circuit may be an Intel Core i7 processor, an ARM Cortex-R8, etc. In one embodiment, the processor circuit may be an ARM Cortex M0. The memory circuit may be a ROM circuit, or a non-volatile memory (e.g., flash memory). The memory circuit may be a volatile memory (e.g., SRAM memory). In the latter case, the device may include a non-volatile software interface (e.g., a hard disk, a network interface, etc.) arranged to provide the software.

[0267] Although device 1110 is shown as including one of each described component, multiple components may be repeated in multiple embodiments. For example, processor 1120 may include multiple microprocessors that are configured to independently perform the methods described herein, or are configured to perform the steps or subroutines of the methods described herein, so that multiple processors cooperate to implement the functions described herein. In addition, in the case where device 1100 is implemented in a cloud computing system, multiple hardware components may belong to separate physical systems. For example, processor 1120 may include a first processor in a first server and a second processor in a second server.

[0268] It should be noted that the above-mentioned embodiments illustrate rather than limit the presently disclosed subject matter, and that those skilled in the art will be able to design many alternative embodiments.

[0269] In the claims, any reference numerals placed in brackets shall not be interpreted as limiting the claims. The use of the verb "comprise" and its variants does not exclude the presence of elements or steps other than those stated in the claims. The article "a" or "an" in front of an element does not exclude the presence of multiple such elements. When an expression such as "at least one" is placed before a list of elements, it means selecting all elements or any subset thereof from the list. For example, the expression "at least one of A, B, and C" should be understood to include only A, only B, only C, both A and B, both A and C, both B and C, or all of A, B, and C. The subject matter of the present disclosure can be implemented by hardware including several different elements, and by a suitably programmed computer. In a device claim that lists several components, several of these components can be implemented by the same hardware item. The mere fact that certain measures are recorded in mutually different dependent claims does not indicate that the combination of these measures cannot be used to advantage.

[0270] In the claims, reference numerals in parentheses refer to reference symbols in the drawings of exemplary embodiments or formulas of embodiments, thereby improving the intelligibility of the claims. These reference numerals should not be construed as limiting the claims.

Claims

1. A computer-implemented method for converting a first computing network into a second computing network, wherein the first computing network is a first computing network arranged to compute an operation of a function, and the second computing network is a second computing network arranged to compute a fully homomorphically encrypted operation of the same function, - the first computational network of operations comprises a plurality of real-valued matrix operations and at least a plurality of further operations, the plurality of real-valued matrix operations comprising real-valued matrix multiplications, the at least a plurality of further operations receiving as input a result of at least one of the plurality of real-valued matrix operations, - Converting the first computing network comprises: - determining (610) a set of expansion factors, one expansion factor for each matrix operation, wherein the expansion factors are real numbers, - converting (620) a plurality of real-valued matrices corresponding to a plurality of real-valued matrix operations into integer-valued matrices by scaling the real-valued matrices by corresponding expansion factors of the plurality of real-valued matrices and rounding the scaled real-valued matrices, - adjusting (630) the plurality of further operations to receive as input a scaled result of the scaled matrix operation, the second computation network comprising the converted plurality of integer-valued matrices and the adjusted plurality of further operations, - determining (640) an accuracy measure of the second computing network indicative of the accuracy of the second computing network, - Iteratively (650) optimizing the set of expansion factors, thereby improving the accuracy measure.

2. The conversion method according to claim 1, wherein the first computing network and the second computing network are neural networks.

3. The conversion method according to claim 1, wherein the real-valued matrix operation further includes a real-valued matrix addition, and converting the real-valued matrix operation further includes scaling the matrix associated with the real-valued matrix addition with the same expansion factor.

4. The conversion method according to claim 1, wherein: At least some of the further operations are implemented as programmable bootstrap in the second computing network.

5. The conversion method according to claim 1, wherein determining a set of expansion factors comprises: - symbolically associating an expansion factor with a matrix operation in said first computation network, - said expansion factor is propagated through said first computation network and a condition on the expansion factor is derived from an operation applied in said first computation network to an output of a transformation operation. The conversion method according to claim 5 , wherein the set of expansion factors is iteratively optimized to satisfy the condition.

7. The conversion method according to claim 1 includes deriving a second computing network in the following manner: replacing a real-valued matrix with a corresponding scaled integer-valued matrix, replacing a real-valued matrix used for addition with a corresponding scaled matrix, and adjusting the operations in the additional operations.

8. The conversion method according to claim 5, wherein the first computing network and the second computing network include a plurality of layers between operations, each layer having an associated layer factor, the layer factor corresponding to a multiplication ratio between a value of a layer in the first computing network and a corresponding value of a corresponding layer in the second computing network, - said first computational network comprises a join operation receiving a first input and a second input from two different layers, said expansion factor being determined under the condition that two layer factors associated with said two different layers are equal, making said join operation homogeneous.

9. The conversion method according to claim 8, wherein the joint operation is one of addition, subtraction or concatenation, and wherein the joint operation is implemented in the second computing network as a corresponding addition, subtraction or concatenation on the encrypted value.

10. The conversion method according to claim 1, wherein the second calculation network comprises operations corresponding to the first calculation network and insertion operations, and correction operations are inserted into the second calculation network so that the expansion factor of the next operation is equal, - the correction operation is programmable bootstrapping and the correction factor is a real number, or - The correction operation is an integer-valued multiplication and the correction factor is an integer.

11. The conversion method of claim 1 , wherein adjusting the additional operation comprises: - remove the expansion factor from the input, and / or -Applies expansion factor to the output.

12. The conversion method according to claim 1, wherein: the accuracy measure is calculated by evaluating the first computational network and the second computational network on a set of sample inputs and comparing the evaluation results with a set of sample outputs to obtain a first accuracy of the first computational network and a second accuracy of the second computational network, the accuracy measure comparing the first accuracy with the second accuracy, and / or The accuracy measure is calculated by evaluating a second computational network on a set of sample inputs and comparing the evaluation result with a set of sample outputs to obtain a second accuracy of the second computational network, the accuracy measure comparing the second accuracy with a predetermined accuracy target.

13. A transitory or non-transitory computer-readable medium (1000) comprising data (1020), wherein the data represents any of the following: - instructions which, when executed by a processor system, cause the processor system to perform a method according to any one of claims 1 to 12; and - Data representing a second network obtained according to any one of claims 1 to 12.

14. A system (200) for converting a first computing network arranged to compute operations of a function into a second computing network arranged to compute fully homomorphically encrypted operations of the same function, the system comprising: a communication interface for receiving the first computational network of operations, the first computational network of operations comprising a plurality of real-valued matrix operations and at least a plurality of further operations, the plurality of real-valued matrix operations comprising real-valued matrix multiplications, the plurality of further operations receiving as input a result of at least one of the plurality of real-valued matrix operations, - a processor system configured to transform the first computing network, the transforming comprising: - determining (610) a set of expansion factors, one expansion factor for each matrix operation, wherein the expansion factors are real numbers, - converting (620) a plurality of real-valued matrices corresponding to a plurality of real-valued matrix operations into integer matrices by scaling the real-valued matrices by corresponding expansion factors of the plurality of real-valued matrices and rounding the scaled real-valued matrices, - adjusting (630) the plurality of further operations to receive as input a scaled result of the scaled matrix operation, the second computation network comprising the converted plurality of integer matrices and the adjusted plurality of further operations, - determining (640) an accuracy measure of the second computing network indicative of the accuracy of the second computing network, - Iteratively (650) optimizing the set of expansion factors, thereby improving the accuracy measure.

Images