Deep restricted knowledge distillation for inferring encrypted data

Abstract

The present disclosure provides a method, apparatus, and computer program product for creating a fully homomorphic encryption (FHE)-friendly machine learning model. The method herein utilizes a knowledge distillation framework, where the FHE-friendly (student) ML model closely mimics the predictions of a more complex (teacher) model, where the teacher model is a teacher model that is more complex relative to the student model and pre-trained on a large dataset. In the method herein, the distillation framework uses the more complex teacher model to facilitate training of the FHE-friendly model, but uses synthetically generated training data instead of the original dataset used to train the teacher.

Description

Technical Field

[0001] This disclosure generally relates to using fully homomorphic encryption operations to facilitate inference of encrypted data. Background Technology

[0002] In machine learning (ML), knowledge distillation refers to learning a student model that closely mimics the predictions of a teacher model. The teacher model is typically a complex ML model pre-trained on a large dataset. The student model is a target ML model with some desired characteristics, such as lower model complexity and better interpretability. Distillation is achieved by allowing the teacher to weakly supervise the student's learning process. In this context, it is not required that both the teacher and student be neural networks, and the original training data (used to learn the teacher) may not be usable for learning the student model.

[0003] Homomorphic encryption (HE) is a form of encryption that allows computation to be performed on ciphertext, producing an encrypted result that, when decrypted, matches the result of an operation performed on the plaintext. Homomorphic encryption schemes are cryptographic systems that allow computation to be performed on data without decryption.

[0004] Data privacy concerns often prevent cloud-based machine learning services from using sensitive personal data. While homomorphic encryption (HE) offers a potential solution by enabling computation on encrypted data, the challenge lies in obtaining an accurate machine learning model that works within the multiplicative depth constraints of hierarchical HE schemes. Existing methods for cryptographic inference either make temporary simplifications to pre-trained models at the expense of accuracy, or they use the original training set to directly train new, depth-constrained models. In machine learning models, these solutions do not work effectively when using encrypted data (e.g., for decision tree-based inference on encrypted data) because the required simplifications are more limited by the computational power of the hierarchical HE.

[0005] There is still a need to provide technology for building machine learning models that are easily evaluated in the FHE domain (hereafter sometimes referred to as FHE “friendly”), but without losing accuracy compared to arbitrary and more complex models that may otherwise be expected to be used to evaluate inputs of interest. Summary of the Invention

[0006] This disclosure provides a method, apparatus, and computer program product for creating Fully Homomorphic Encryption (FHE)-friendly machine learning models. The method in this paper utilizes a knowledge distillation framework, wherein the FHE-friendly (student) ML model closely mimics the predictions of a more complex (teacher) model, which is a more complex teacher model pre-trained on a large dataset relative to the student model. In the method of this paper, the distillation framework uses the more complex teacher model to facilitate the training of the FHE-friendly model, but uses synthetically generated training data instead of the original dataset used to train the teacher.

[0007] In one aspect, a method for creating a Fully Homomorphic Encryption (FHE)-compatible machine learning model begins by obtaining a first (teacher) machine learning model, which has been pre-trained using first training data. A set of input data distributions and FHE constraints is also received. FHE constraints typically include target multiplication circuit depth, security level, ciphertext size, and combinations thereof. Using these inputs, a second (student) machine learning model is then constructed. The second model has comparable accuracy to the first model, but unlike the first model, it also satisfies the set of FHE constraints. Specifically, preferably, the second ML model is constructed as follows: The input data distribution is used to generate a transport dataset. A synthetic training dataset is then generated by the first machine learning model making predictions on the transport dataset (instead of the original dataset used to train the first model). The synthetic training dataset is then used to train (i.e., learn parameters) the second model. The second model can be one of a set of candidate second machine learning models, each trained in this manner, wherein the best-fit candidate is then selected as the second model. The second machine learning model, constrained by the depth of the FHE constraints, comprises a shallow neural network with an FHE-compatible activation function.

[0008] The second machine learning model created in this way was then used for privacy-preserving homomorphic inference on encrypted data input.

[0009] The foregoing outlines some of the more relevant characteristics of this topic. These characteristics should be interpreted as illustrative only. Many other beneficial results can be obtained by applying the disclosed topics in different ways or by modifying the topics as described herein. Attached Figure Description

[0010] To gain a more complete understanding of the invention and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings.

[0011] In the attached diagram:

[0012] Figure 1 An exemplary block diagram of a distributed data processing environment that can implement exemplary aspects of the illustrative embodiments is shown;

[0013] Figure 2 This is an exemplary block diagram of a data processing system that can implement exemplary aspects of the illustrative embodiments;

[0014] Figure 3 An exemplary cloud computing architecture in which the disclosed topics can be implemented is shown;

[0015] Figure 4 A machine learning as a service (MLaaS) operating environment in which the techniques of this disclosure can be implemented is described;

[0016] Figure 5 This paper describes how to use the distillation framework disclosed herein to build an FHE-friendly student model;

[0017] Figure 6 The operation of the transport dataset generator disclosed herein is described; and

[0018] Figure 7 A representative neural network architecture from which a student model is constructed, based on a representative embodiment, is described. Detailed Implementation

[0019] Now refer to the attached diagram and see for details. Figure 1-2 Exemplary diagrams are provided illustrating a data processing environment that can implement the illustrative embodiments of this disclosure. It should be understood that... Figure 1-2 These are merely illustrative examples and are not intended to assert or imply any limitation on the environments in which the aspects or embodiments of the disclosed subject matter may be implemented. Many modifications may be made to the depicted environments without departing from the spirit and scope of the invention.

[0020] Client-server technology

[0021] Now refer to the attached diagram, Figure 1 A graphical representation of an exemplary distributed data processing system in which aspects of the illustrative embodiments can be implemented is depicted. The distributed data processing system 100 may include a computer network in which aspects of the illustrative embodiments can be implemented. The distributed data processing system 100 includes at least one network 102, which is a medium for providing communication links between different devices and computers connected together within the distributed data processing system 100. The network 102 may include connections such as wired, wireless communication links, or fiber optic cables.

[0022] In the depicted example, servers 104 and 106 are connected to network 102 along with storage unit 108. Clients 110, 112, and 114 are also connected to network 102. These clients 110, 112, and 114 can be, for example, personal computers, network computers, etc. In the depicted example, server 104 provides data such as boot files, operating system images, and applications to clients 110, 112, and 114. In the depicted example, clients 110, 112, and 114 are clients of server 104. The distributed data processing system 100 may include additional servers, clients, and other devices not shown.

[0023] In the depicted example, the distributed data processing system 100 is an internet with network 102, which represents a global collection of networks and gateways communicating with each other using the Transmission Control Protocol / Internet Protocol (TCP / IP) protocol suite. The core of the internet is the backbone of high-speed data communication lines between master nodes or host computers, composed of thousands of commercial, government, educational, and other computer systems routing data and messages. Of course, the distributed data processing system 100 can also be implemented as including multiple different types of networks, such as, for example, intranets, local area networks (LANs), wide area networks (WANs), etc. As described above, Figure 1 The intent is to be illustrative, not to be construed as an architectural limitation on different embodiments of the disclosed subject matter, and therefore, in Figure 1 The specific elements shown should not be considered as limitations on the environment in which the illustrative embodiments of the invention may be implemented.

[0024] Now for reference Figure 2 This diagram illustrates a block diagram of an exemplary data processing system that can implement various aspects of the illustrative embodiments. The data processing system 200 is a computer (such as...) Figure 1 Examples of client 110 in the present disclosure may be provided, in which computer-usable code or instructions for implementing the processing of illustrative embodiments of the present disclosure may be located.

[0025] Now for reference Figure 2 A block diagram of a data processing system that can implement illustrative embodiments is shown. The data processing system 200 is a computer (such as...) Figure 1 Examples of server 104 or client 110 in the illustrative embodiment may include computer-usable program code or instructions for implementing the process. In this illustrative example, the data processing system 200 includes a communication structure 202 that provides communication between processor unit 204, memory 206, persistent memory 208, communication unit 210, input / output (I / O) unit 212, and display 214.

[0026] Processor unit 204 is used to execute instructions for software that can be loaded into memory 206. Processor unit 204 may be a collection of one or more processors, or it may be a multiprocessor core, depending on the specific implementation. Furthermore, processor unit 204 may be implemented using one or more heterogeneous processor systems, in which the main processor and secondary processor reside on a single chip. As another illustrative example, processor unit 204 may be a symmetric multiprocessor (SMP) system containing multiple processors of the same type.

[0027] Memory 206 and persistent memory 208 are examples of storage devices. A storage device is any hardware capable of temporarily and / or permanently storing information. In these examples, memory 206 may be, for example, random access memory or any other suitable volatile or non-volatile storage device. Persistent memory 208 may take various forms depending on the specific implementation. For example, persistent memory 208 may include one or more components or devices. For example, persistent memory 208 may be a hard disk drive, flash memory, rewritable optical disk, rewritable magnetic tape, or a combination of the above. The medium used by persistent memory 208 may also be removable. For example, a removable hard disk drive may be used for persistent memory 208.

[0028] In these examples, communication unit 210 provides communication with other data processing systems or devices. In these examples, communication unit 210 is a network interface card. Communication unit 210 can provide communication using either or both physical and wireless communication links.

[0029] Input / output unit 212 allows data input and output to other devices that can be connected to data processing system 200. For example, input / output unit 212 can provide connectivity for user input via keyboard and mouse. Furthermore, input / output unit 212 can send output to a printer. Display 214 provides a mechanism for displaying information to the user.

[0030] Operating system and application or program instructions reside on permanent memory 208. These instructions may be loaded into memory 206 for execution by processor unit 204. Processor unit 204 may use computer-implemented instructions to perform processes of different embodiments, which may reside in memory (e.g., memory 206). These instructions are referred to as program code, computer-usable program code, or computer-readable program code that can be read and executed by a processor in processor unit 204. Program code in different embodiments may be implemented on different physical or tangible computer-readable media, such as memory 206 or permanent memory 208.

[0031] Program code 216 resides functionally on a selectively removable computer-readable medium 218 and can be loaded into or transferred to the data processing system 200 for execution by the processor unit 204. In these examples, program code 216 and computer-readable medium 218 form a computer program product 220. In one example, computer-readable medium 218 may be in a tangible form, such as an optical disc or disk inserted into or placed in a drive or other device that is part of persistent storage 208 for transfer to a storage device, such as a hard disk drive that is part of persistent storage 208. In a tangible form, computer-readable medium 218 may also take the form of persistent storage, such as a hard disk drive, thumb drive, or flash memory connected to the data processing system 200. The tangible form of computer-readable medium 218 is also referred to as a computer-recordable storage medium. In some instances, computer-recordable medium 218 may not be removable.

[0032] Alternatively, program code 216 can be transmitted from computer-readable medium 218 to data processing system 200 via a communication link to communication unit 210 and / or via a connection to input / output unit 212. In illustrative examples, the communication link and / or connection can be physical or wireless. The computer-readable medium can also take the form of intangible media, such as a communication link containing program code or wireless transmission. The different components shown for data processing system 200 do not imply an architectural limitation on the ways in which different embodiments can be implemented. Different illustrative embodiments can be implemented in data processing systems that include components other than or in lieu of those shown for data processing system 200. Figure 2 Other components shown may differ from the illustrative example shown. As an example, the storage device in data processing system 200 is any hardware device capable of storing data. Memory 206, persistent memory 208, and computer-readable medium 218 are examples of tangible storage devices.

[0033] In another example, a bus system can be used to implement communication structure 202 and may include one or more buses, such as a system bus or an input / output bus. Of course, any suitable type of architecture that provides data transfer between different components or devices attached to the bus system can be used to implement the bus system. Furthermore, the communication unit may include one or more devices for sending and receiving data, such as a modem or network adapter. Further, the memory may be, for example, memory 206 or a cache such as that found in the interface and memory controller hub that may be present in communication structure 202.

[0034] Computer program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java™, Smalltalk, C++, C#, Objective-C, etc., as well as conventional procedural programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0035] Those skilled in the art will understand that Figure 1-2 The hardware within can vary depending on the implementation. Besides or replacing... Figure 1-2 The hardware described herein can be replaced with other internal hardware or peripheral devices, such as flash memory, equivalent non-volatile memory, or optical disc drives. Furthermore, the processes of the illustrative embodiments can be applied to multiprocessor data processing systems, rather than the aforementioned SMP systems, without departing from the spirit and scope of the disclosed subject matter.

[0036] As will be seen, the techniques described in this article can be applied to, for example... Figure 1 The example illustrates collaborative operation within a standard client-server paradigm, where client machines communicate with an internet-accessible, web-based portal running on a set of one or more machines. End-users operate internet-connected devices (e.g., desktop computers, laptops, internet-enabled mobile devices, etc.) capable of accessing and interacting with the portal. Typically, each client or server machine is, for example... Figure 2 The diagram illustrates a data processing system comprising hardware and software, and these entities communicate with each other via networks such as the Internet, intranets, extranets, private networks, or any other communication medium or link. A data processing system typically includes one or more processors, an operating system, one or more applications, and one or more utilities. Applications on the data processing system provide native support for Web services, including but not limited to support for HTTP, SOAP, XML, WSDL, UDDI, and WSFL. Information on SOAP, WSDL, UDDI, and WSFL is available from the World Wide Web Consortium (W3C), which is responsible for developing and maintaining these standards; further information on HTTP and XML is available from the Internet Engineering Task Force (IETF). Familiarity with these standards is assumed.

[0037] Cloud computing model

[0038] An emerging information technology (IT) delivery model is cloud computing, through which shared resources, software, and information are provided on demand to computers and other devices via the Internet. Cloud computing can significantly reduce IT costs and complexity while improving workload optimization and service delivery. In this approach, application instances can be hosted and made available from Internet-based resources accessible via a regular web browser over HTTP. An example application could be one that provides a set of common messaging functionalities such as email, calendar, contact management, and instant messaging. Users then access the service directly via the Internet. Using this service, businesses place their email, calendar, and / or collaboration infrastructure in the cloud, and end users use appropriate clients to access their emails or perform calendar operations.

[0039] Cloud computing resources are typically housed in large server clusters running one or more web applications, often using a virtualization architecture where applications run on virtual servers, or so-called "virtual machines" (VMs), which are mapped to physical servers within a data center facility. VMs typically run on top of a hypervisor, which is the program that controls the allocation of physical resources to the VMs.

[0040] Cloud computing is a service delivery model designed to enable convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing power, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five features, at least three service models, and at least four deployment models, all of which are described and defined in more detail in the “NIST Working Definition of Cloud Computing” by Peter Mell and Tim Grance dated October 7, 2009.

[0041] Specifically, the following are typical characteristics:

[0042] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power, such as server time and network storage, as needed, without requiring human interaction with the service provider.

[0043] Extensive network access: Capabilities are available through networks and accessed via standard mechanisms that facilitate the use of heterogeneous thin client platforms or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0044] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically assigned and reassigned as needed. There is a sense of location independence because consumers typically do not have control or knowledge of the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0045] Rapid flexibility: The ability to provide capacity quickly and flexibly, automatically scaling down and up rapidly in some situations to scale up rapidly. For consumers, the available supply capacity often appears unlimited and can be purchased in any quantity at any time.

[0046] Metering services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0047] The service model is typically as follows:

[0048] Software as a Service (SaaS): This provides consumers with the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from different client devices via thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with possible exceptions such as limited user-specific application configuration settings.

[0049] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created or acquired by the consumer using programming languages ​​and tools supported by the provider onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of any application hosting environment.

[0050] Infrastructure as a Service (IaaS): The capabilities offered to consumers are processing, storage, networking, and other basic computing resources that enable consumers to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but rather have control over the operating system, storage, deployed applications, and potentially limited control over selected networking components (e.g., host firewalls).

[0051] The deployment model is typically as follows:

[0052] Private cloud: A cloud infrastructure that operates solely for an organization. It can be managed by the organization or a third party and can exist on-site or off-site.

[0053] Community cloud: A cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0054] Public cloud: Makes cloud infrastructure available to the public or large industry groups and is owned by an organization that sells cloud services.

[0055] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be portable (e.g., cloud bursting for load balancing between clouds).

[0056] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. The core of cloud computing is its infrastructure, which includes a network of interconnected nodes. Representative cloud computing nodes are shown above. Figure 2 As shown. Specifically, within a cloud computing node, there exists a computer system / server that can operate alongside many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with the computer system / server include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computers, and distributed cloud computing environments that include any of the above systems or devices. The computer system / server can be described in the general context of computer system executable instructions, such as program modules, executed by the computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. The computer system / server can be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can reside in local and remote computer system storage media, including memory storage devices.

[0057] See now Figure 3 With the added background, a set of functional abstraction layers provided by the cloud computing environment is illustrated. This should be understood beforehand. Figure 3 The components, layers, and functions shown are intended to be illustrative only, and embodiments of the invention are not limited thereto. As described, the following layers and corresponding functions are provided:

[0058] The hardware and software layer 300 includes hardware and software components. Examples of hardware components include mainframes, which in one example are... System; a server based on a RISC (Reduced Instruction Set Computer) architecture, in one example being an IBM... System; IBM System; IBM Systems; storage devices; networks and network components. Examples of software components include network application server software, one example being IBM. Application server software; and database software, in one example, IBM. Database software. (IBM, zSeries, pSeries, xSeries, BladeCenter, WebSphere, and DB2 are trademarks of International Business Machines Corporation registered in many jurisdictions worldwide.)

[0059] The virtualization layer 302 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers; virtual storage; virtual networks, including virtual private networks; virtual applications and operating systems;

[0060] And virtual clients.

[0061] In one example, management layer 304 can provide the following functionalities: Resource provisioning provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and pricing provides cost tracking as resources are utilized within the cloud computing environment and bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. The user portal provides consumers and system administrators with access to the cloud computing environment. Service level management provides the allocation and management of cloud computing resources to ensure that required service levels are met. Service level agreements (SLAs) plan and fulfill the pre-arrangement and procurement of cloud computing resources, anticipating future requirements for those resources according to the SLA.

[0062] Workload layer 306 provides examples of functionalities that can be utilized in a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation; software development and lifecycle management; virtual classroom education delivery; data analytics and processing; transaction processing; enterprise-specific functionalities in a private cloud; and, in accordance with this disclosure, techniques for privacy-preserving homomorphic inference of homomorphically encrypted data 308.

[0063] It should be understood in advance that while this disclosure includes a detailed description of cloud computing, the implementation of the teachings cited herein is not limited to cloud computing environments. Rather, embodiments of the disclosed techniques can be implemented in conjunction with any other type of computing environment now known or developed hereafter. These include stand-alone computing environments (e.g., on-site desktops), client-server based architectures, and so on.

[0064] Therefore, a representative cloud computing environment has a set of high-level functional components, including a front-end identity manager, a Business Support Service (BSS) functional component, an Operations Support Service (OSS) functional component, and a compute cloud component. The identity manager is responsible for interacting with requesting clients to provide identity management, and this component can be implemented using one or more known systems, such as the Tivoli Federation Identity Manager (TFIM) available from IBM in Armonk, New York. Where appropriate, TFIM can be used to provide Federation Single Sign-On (F-SSO) to other cloud components. The Business Support Service component provides certain management functions, such as billing support. The Operations Support Service component is used to provide provisioning and management of other cloud components, such as virtual machine (VM) instances. A cloud component represents a primary compute resource, typically multiple VM instances used to execute a target application accessible via the cloud. One or more databases are used to store directories, logs, and other working data. All these components (including the front-end identity manager) reside “within” the cloud, but this is not mandatory. In alternative embodiments, the identity manager can operate outside the cloud. Service providers can also operate outside the cloud.

[0065] Some clouds are based on non-traditional IP networks. Thus, for example, a cloud can be based on a two-tier CLOS-based network with a special single-tier IP routing that uses hashes of MAC addresses. The techniques described in this article can be used in such non-traditional clouds.

[0066] In general, cloud computing infrastructure provides a virtual machine hosting environment that includes hosts (e.g., servers or similar physical computing devices) connected via a network and one or more management servers. Typically, each physical server is adapted to dynamically provision one or more virtual machines using virtualization technologies such as VMware ESX / ESXi. Multiple VMs can be placed on a single host and share the host's CPU, memory, and other resources, thereby increasing the utilization of an organization's data center. Among other tasks, the management server monitors the infrastructure and automatically manipulates VM placement as needed, such as by moving VMs between hosts.

[0067] In non-restrictive implementations, representative platform technologies include, but are not limited to, IBM Systems with VMware vSphere 4.1 Update 1 and 5.0. server.

[0068] The aforementioned commercial implementation is not intended to be limiting, but is merely a representative example of a client application supported in a cloud computing environment that interacts with cognitive services.

[0069] Homomorphic encryption

[0070] Homomorphic encryption (HE) is a form of encryption that allows computation to be performed on ciphertext, thus generating an encrypted result that, when decrypted, matches the result of an operation performed on the plaintext. Homomorphic encryption schemes are cryptographic systems that allow computation on data without decryption. Homomorphic encryption is typically implemented by combining some homomorphic (SWHE) or hierarchical HE scheme with bootstrapping or re-encryption techniques. SWHE schemes can only support computations up to a preset complexity determined by their parameters. This is because the ciphertext is "noisy," and the noise increases with each HE computation; once the noise exceeds a parameter-related threshold, the ciphertext can no longer be decrypted. This problem can be addressed by bootstrapping, which refreshes the ciphertext and reduces its noise level, but at the cost of relying on cycle security. However, bootstrapping is a computationally intensive and time-consuming operation. Therefore, for practical feasibility, the number of bootstrapping operations should be kept to a minimum and avoided as much as possible.

[0071] Homomorphic encryption enables the construction of programs with any desired functionality, which can be run on encrypted input to produce an encrypted result. Because such a program never needs to decrypt its input, it can be run by an untrusted party without revealing its input and internal state. Toolkits for implementing homomorphic encryption are known. A well-known toolkit is HElib, an open-source project. The current version of HElib supports addition and multiplication of arbitrary numbers in binary representation, using bit-by-bit encryption.

[0072] As further background, several FHE schemes based on the difficulty of a computational problem known as Ring Learning with Error (RLWE) have been proposed. Prominent examples of these schemes include the BFV (Brakerski / Fan-Vercauteren), BGV (Brakerski-Gentry-Vaikuntanathan), and CKKS (Cheon-Kim-Kim-Song) schemes (named after their original proposers) which are homomorphic for addition and multiplication. While the BFV and BGV schemes are efficient for vector operations on integers, the CKKS scheme is better suited for “approximate” (finite-precision) floating-point operations. Furthermore, these schemes support Single Instruction Multiple Data (SIMD) operations, specifically by packing different plaintext values ​​into different slots within the ciphertext. While this ciphertext packing allows for parallelization of addition and multiplication operations, it is not possible to randomly access values ​​in the individual slots of the ciphertext after packing. Instead, only a finite number of operations (e.g., rotations within the slots of the ciphertext) are possible.

[0073] When using the CKKS scheme for HE operations (such as the one implemented in the HElib library), real-valued features can be directly input into the decision tree, i.e., there is no need to transform the input into integers.

[0074] Machine Learning as a Service with Homomorphic Encryption

[0075] Machine Learning as a Service (MLaaS) is now a popular paradigm where pre-trained models are hosted on the public cloud and inference is performed on a per-query basis. However, the use of MLaaS is limited in many application areas (e.g., financial services, healthcare) because the privacy (confidentiality) of the client data used to perform inference is of paramount concern. Fully homomorphic encryption (FHE) can address these privacy challenges by allowing certain types of computations to be performed on encrypted data without requiring decryption. As mentioned above, an FHE scheme can be defined as: Where ε and Let λ represent the encryption and decryption operations, respectively. λ is a security parameter, and EVAL is an evaluation function that takes an arbitrary function f and an encryption input ε(I) and returns the encryption result EVAL(f,ε(I)) = ε(f(I)). For ease of representation, it should be understood that both the ε operation and the EVAL operation require access to the public key and the private key, respectively.

[0076] Now for reference Figure 4 This describes a privacy-preserving inference scenario based on FHE, and its working principle is as follows: Client 400 encrypts its data x (using encryption function 401) and sends the encrypted data to a cloud-based machine learning service provider 402. A pre-trained model 404 is then provided. (in The service provider (where Θ represents the model architecture and parameters, respectively) performs inference computation in the cryptographic domain. The encrypted result is then returned to the client at 400 for decryption at 405. Since the service provider does not obtain any knowledge about the input (x) or the resulting output (y), complete client data privacy is guaranteed. Furthermore, the above scenario is described as non-interactive because it involves only a single round of communication between the client and the service provider.

[0077] Such as Figure 4 The fundamental challenges in cryptographic inference described in the text stem from severe limitations in the nature of feasible computation and the efficiency of computation in the cryptographic domain. Most FHE schemes are constructed based on hierarchical HE schemes, which only support computations up to a preset level of doubling depth determined by their parameters. This is because the “noise” in the ciphertext grows with each HE computation, and decryption fails once the noise exceeds a threshold. This depth constraint can be overcome using bootstrapping, which “refreshes” the ciphertext and reduces its noise level (at the cost of reliance on cycle security). However, bootstrapping is a very expensive and complex operation and must be avoided or used with caution for practically feasible computations. In the context of ML, the depth constraint problem is further exacerbated by the presence of nonlinear computations (e.g., comparing two numbers, activation functions), which require even greater depths because the underlying HE schemes typically only support addition and multiplication homomorphisms. For these reasons, it is rarely possible to use any pre-trained ML model for cryptographic inference. For example, while boosted decision trees remain the standard choice for classification and regression tasks on tabular datasets, they are difficult to apply directly to cryptographic data because they rely on comparison operations.

[0078] Recent work on cryptographic inference has attempted to address this problem in two ways. One approach is to train an unconstrained model and then simplify its architecture to reduce the multiplication depth without retraining the parameters. Another alternative is to define a depth-constrained model architecture and train it directly from scratch using available training data. While the former approach typically results in a significant loss of inference accuracy compared to the original model, the latter approach may be impractical because the training data used to train the original model may no longer be available. Ideally, and as... Figure 4 As described herein, the goal is to transform any given pre-trained model 404 into its depth-constrained version 406 without accessing the original training data, while minimizing any loss of accuracy.

[0079] With the above as background, the technology of the present invention will now be described.

[0080] Establish FHE-friendly machine learning models

[0081] Figure 5 The basic paradigm of this disclosure is depicted in the figure. As shown, the system (sometimes referred to herein as the knowledge distillation framework) receives a teacher model 500. The teacher model is unconstrained and has been trained using the original training set 502. The teacher model has high accuracy but is incompatible (friendly) with performing inference in the FHE domain. The goal is then to build an FHE-friendly student model 504 with comparable accuracy to the teacher model, i.e., it mimics the decision boundaries of the more complex teacher model while still allowing inference on encrypted data. In this way, the student model 504 acts as a proxy for the teacher model 500 with minimal accuracy loss.

[0082] According to this disclosure, student model 504 is constructed as follows. First, a transmission dataset 507 is generated by transmission set generator 506. The transmission dataset 507 is preferably randomly generated using a given input data distribution associated with the teacher model. It is not required that the given input data distribution have any associated labels. Furthermore, the input data distribution is also used to randomly generate a validation dataset (not shown), which, as described below, facilitates the selection of student models from a pool of student models compatible with a set of FHE constraints. These constraints can vary but typically include target multiplication circuit depth, security level, ciphertext size, etc. Multiplication circuit depth is, for example, the maximum number of consecutive multiplication operations required for computation. The security level reflects the difficulty of breaking the encryption, and this parameter can depend on the parameters of the underlying encryption scheme and the depth of computation at hand; the higher the security level, the higher the inference time, and wherein, for the same ciphertext parameters, a lower computational depth translates to higher security. The ciphertext size involves selecting hyperparameters of the ML model (e.g., the number of hidden nodes) to most efficiently utilize the ciphertext slots. Other possible FHE constraints (including combinations of those described above) may be considered.

[0083] Return to reference Figure 5In this method, preferably, a student model architecture is selected from an FHE-friendly hypothesis space (i.e., the set of all models satisfying the given FHE constraints). Then, and as depicted, the parameters of student model 504 are learned using a transfer dataset 507 and the predictions of a teacher model 500 on that transfer dataset as “synthetic” training data. Any supervised training algorithm (e.g., gradient descent) can be used for this purpose. One or more “candidate” student models can then be trained in this manner (i.e., using the teacher model’s predictions on the transfer dataset 507). Preferably, the fitness of each candidate student model is also computed using a validation dataset and the teacher model’s predictions on the validation dataset. Any standard fitness (goodness) measure (e.g., accuracy) can be used for this purpose. A loss estimator 508 evaluates the fitness computation and selects the student model with the best goodness measure as the FHE-friendly model. In this method, while the student model strives to minimize the loss between the teacher model and the student model’s predictions on the transfer dataset, preferably, the transfer set generator 506 is also configured to generate more challenging input samples that maximize this loss.

[0084] In short, and as Figure 5 As described, the method in this paper provides a technique for transferring knowledge embedded in a pre-trained first (teacher) model to a second, depth-constrained (student) model without accessing the original training data used to train the first model.

[0085] Now for reference Figure 7 In the above process, an FHE-friendly student architecture search can be used to select a student model. This assumes a mechanism for evaluating the FHE friendliness of the selected architecture. As shown in the figure, the student model 700 can be a multi-layer feedforward neural network. As depicted, the hyperparameter search space (from which the student model can be constructed) includes multiple layers, multiple hidden nodes in each layer, activation functions on each layer, etc. Then, depth-constrained knowledge distillation of HE inference for the decision tree set can be performed as follows: Given a multiplicative depth budget, a set of configurations of the neural network is computed along with a multinomial activation function that satisfies the depth budget. Then, as previously described, augmented data is generated from the given unlabeled data (e.g., a small portion of the original data), and labels are generated using this population. Then, the best possible neural network is searched for in terms of validation scores. This operation can be performed in an unencrypted space. Subsequently, HE inference is performed on the best network as a proxy for the original population.

[0086] The above method preferably minimizes the accuracy loss by searching for the optimal student model architecture that operates within a given depth constraint, and then preferably training the student model using only synthetic data sampled from the original training data distribution.

[0087] In FHE-based computation, several parameters define computational complexity. As mentioned above, these include the multiplication depth, the choice of security parameters, the choice of the number of ciphertexts, and others. Minimizing the use of bootstrapping techniques to reduce noise (since bootstrapping is very expensive), the technique presented in this paper constructs an ML inference model that is accurate and closer to the performance of the original complex model. To facilitate learning the student model, the framework assumes that the original training set used to train the complex model is unavailable. Instead, preferably, the method in this paper creates artificially trained feature data and then uses the original complex model to generate labels for the training features. Shallow networks are trained using FHW-perceptual computation to make simpler models easier to map to the FHE scheme.

[0088] Figure 6 A technique for generating transport datasets, implemented by a transport set generator, is described. In a first embodiment, as a static method, a small set of unlabeled data is input to a nonparametric density estimation function 600, although this is not mandatory. The resulting input data distribution is randomly sampled by a random sampling function 602 to generate the transport dataset. In an alternative embodiment, as a dynamic method, a small set of unlabeled data is input to the nonparametric density estimation function 600, and the resulting input data distribution is applied to an adaptive sampling function 604, which also receives the predicted difference between teacher and student models regarding previous samples as input. The adaptive sampling function 604 uses these inputs to generate the transport dataset.

[0089] Although not described, a validation dataset can also be generated using random or adaptive sampling, which is used to facilitate the selection of the most suitable candidate student model.

[0090] The following sections provide a more detailed description of the implementation of the above technologies.

[0091] Depth-constrained knowledge distillation for inference of encrypted data

[0092] As described above, for the purpose of obtaining a (multiplicative) depth-constrained student model, the technique applied in this paper employs a knowledge distillation framework that simulates the decision boundaries of complex teacher models while simultaneously enabling cryptographic inference. As will also be described, in one implementation, this framework provides an algorithm for searching for the optimal student model that can be accurately trained, even if the complete training set (used to train the original model) is unavailable.

[0093] Specifically, the present invention provides a method for constructing a model that is equivalent to (or closely mimics) a given pre-trained model. Deeply constrained machine learning models for decision-making behavior The technology. Original. and depth constraints Learners do not need to belong to the same family of ML models. Here, depth refers to the multiplication depth of the model, which is defined as follows:

[0094] definition (Multiplication depth of machine learning models)

[0095] Depend on The multiplication circuit depth of the machine learning model M is represented by the maximum number of multiplication operations (gates) required along any of its circuit paths. Let... For a pre-trained model, where and Let these represent the input and output spaces, respectively. Assume... It is used for learning models The training data for the parameter Θ, where It is the number of training samples, and p x Let represent the distribution of the training data. It is the distillation loss function, measured and Similar version The difference in predictions between them. The problem of learning deep constrained models can be formally stated as equation (1) (for ):

[0096]

[0097] in, This is the hypothesis space of the depth constraint, i.e., the set of all models whose multiplication depth is less than or equal to Ω. Since directly estimating the expected distillation loss in the above equation is usually impractical, it can be replaced with empirical risk. If the original training data... If available, the empirical risk minimization problem can be stated as equation (2) (for ):

[0098]

[0099] The method here assumes that, under normal circumstances, the original training data No longer available (or unavailable for training). In the context of cryptographic inference, this is often a valid hypothesis because the model... It is typically trained using sensitive personal information that cannot be retained indefinitely. Note that this assumption excludes "throwing" a given pre-trained model. Furthermore, it allows for the retraining of deeply constrained models from scratch. As will be described, the technique presented in this paper overcomes the unavailability of the original training data using a knowledge distillation framework. As mentioned above, knowledge distillation refers to the process of learning a student model that closely mimics the predictions of the teacher model, which is typically achieved by allowing the teacher to weakly supervise the student's learning process. Specifically, given a transition set (without true labels) of input data samples, the predictions made by the teacher model on that transition set are used as surrogate labels instead of the true labels to train the student model.

[0100] The following description is used to extract from Learning in China A representative algorithm. In a non-limiting embodiment, the input to this algorithm is a pre-trained model. Input data distribution p x Multiplication depth budget Ω, distillation loss function Transfer set size and the size of the validation set The output is a close approximation of Depth-constrained model and its verification score In step (1), a transfer set is generated. and verification set in In step (2), from the hypothesis space Select model architecture In step (3), Learning for In step (4), verify the score. Calculated as a verification set Candidate models The accuracy.

[0101] Transmission set generation

[0102] The first aspect of knowledge distillation is how to obtain the transfer set of unlabeled input samples required for distillation. In one known approach, the original training data (minus the output labels) is used as the transfer set. An alternative is to use synthetically generated pseudo-training samples that can be generated during distillation. Because the teacher model is trained only on the manifold px of the training data, its knowledge does not necessarily cover the entire input space. As a result, when random inputs not located on the manifold px are fed to the teacher model, the resulting predictions may be insufficient to reliably train the student model. Therefore, generating the synthetic transfer set may require some knowledge of the input data manifold. Recently, attempts have also been made to perform knowledge distillation without accessing any training data. However, these methods assume that the teacher model is a deep neural network with a final softmax layer capable of producing the label distribution as output. Here, it is assumed that the service provider retains minimal information about the training data distribution px. Furthermore, and in an example implementation in this paper (see example). Figure 6 p is obtained from a small subset (unlabeled) of the training data. x (x) Nonparametric density estimation algorithms (e.g., a modified version of the MUNGE algorithm) to generate large unlabeled sets that can be used for distillation.

[0103] In the method presented in this paper, only p is used. x Use knowledge to generate synthetic datasets Partly, this is because student models are typically constrained to the hypothesis space HΩ and synthetic datasets are limited by practical constraints. The size is finite. As a variant method, when the distillation loss is low, the transfer set generator can be used to generate more challenging input samples to distinguish the predictions of the teacher model and the student model.

[0104] Depth-constrained student architecture search

[0105] The second aspect of knowledge distillation is the appropriate selection of the student network architecture. Typically, the architecture of a student network is fixed based on the computational resources and memory requirements of the target environment in which the student is expected to be deployed. The method in this paper designs a student network with a finite multiplication depth. This depth constraint complicates the search for the optimal student model in the following way: Assume a multilayer perceptron with a sigmoid-shaped activation function is chosen as the student model. Given a finite multiplication depth (e.g., Ω), it is unclear whether the available depth should be used to more accurately approximate the sigmoid-shaped function (e.g., using a higher-order polynomial) or to add more hidden layers to the network. In the method presented in this paper, a shallow neural network is preferably used as the student model architecture.

[0106] This is driven by the concept that a fully connected sigmoid-shaped neural network with a single hidden layer can universally approximate any continuous univariate function. It has also been shown that this proof can be extended to any non-polynomial function as the activation function. Because the goal of this technique is to approximate the decision function learned by the teacher model, neural networks are a suitable choice for student models. That is, because other constraints or limitations may exist (e.g., using only non-linear activation functions, such as polynomial approximations of sigmoid and ReLU), it is preferable to add more hidden layers to the network to increase its modeling power. To determine the optimal student model architecture, preferably, for the following many choices... Repeat steps (2) to (4) of the above algorithm, and select the student model with the highest validation score.

[0107] Applications of decision tree set congruence inference

[0108] As an example application of the depth-constrained knowledge distillation framework presented in this paper, the following describes how to perform cryptographic inference against large clusters of decision tree classifiers. Unlike recent efforts to transfer knowledge from complex models (neural networks and augmentation trees) to a single decision tree to leverage the interpretability of decision trees, the technique presented in this paper distills knowledge from a collection of trees into a deep-constrained neural network (called DTNet) to facilitate inference on (homomorphic) cryptographic data. Specifically, preferably, the disclosed technique replaces the decision nodes with a comparator (in the original model) that has a simpler and shallower neural network (DTNet). In this replacement, preferably, a simpler (from the perspective of FHE) activation function is used instead of the standard activation function (in the original model).

[0109] The template structure of the DTNet neural network is now described. The input layer consists of d nodes for input data with d features. The hidden layer is a dense (fully connected) layer with a specified number of nodes (neurons) and a specified multinomial activation function. The output layer is a softmax layer with c nodes, where c is the number of classes in the classification task. When performing homomorphic inference, the final softmax computation can be delegated to the client.

[0110] DTNet has low computational depth, thus enabling efficient and accurate homomorphic inference on these models. In fact, DTNet's inference predictions in encrypted space are almost identical to those in unencrypted space. Furthermore, even with very limited training data, DTNet can very closely approximate inferences on assemblies of decision trees. Therefore, and in general, DTNet is used to efficiently approximate homomorphic inferences on assemblies of trees.

[0111] The technique of this invention offers significant advantages. It provides a systematic approach to knowledge distillation of deep constraints on multiplication, enabling efficient cryptographic inference. In one embodiment, the method enables efficient cryptographic inference to be performed on complex sets of decision trees. For example, the method in this paper provides an efficient way to perform cryptographic inference on complex sets of decision trees (e.g., 100 base estimators), achieving approximately three (3) orders of magnitude speed improvement over standard methods that directly replace hard comparisons at each decision node with soft comparisons. The system is highly scalable in multi-core cloud environments, and the method can be implemented using several real-world public datasets.

[0112] There are no limitations on potential use cases, as the technology can be used to support any cloud-based machine learning service that requires compliance with privacy rules or other security requirements. Many new industry applications can utilize this type of cloud service and thus provide enhanced privacy guarantees. Two examples are healthcare and finance, where compliance with customer privacy is paramount. As previously mentioned, the three (3) basic components in a machine learning-as-a-service scenario are the client, the server, and the communication infrastructure. Because the data and the results are encrypted, any attacks on the server or communication channel are effectively prevented.

[0113] As described, computing systems implementing this approach are typically implemented in software, for example, as a set of computer program instructions executed by one or more hardware processors. Specific tools or components within the system can include any number of programs, processes, execution threads, etc., along with appropriate interfaces and databases to support the data used or created by the tools or components. Tools or components can be configured or managed via command line or the like, along with a network-based front end. Tools or components can include one or more functionalities implemented programmatically or interoperable with other computing entities or software systems via application programming interfaces (APIs) or any convenient request-response protocol.

[0114] Any references to one or more commercial products or services in this document are exemplary and should not be construed as limiting the technology disclosed, which can be implemented on any system, device, appliance (or, more generally, machine) having the general features and operational functions described therein.

[0115] As mentioned above, a preferred implementation of this topic is as a service, but this is not a limitation. Inferences based on HE can be performed entirely on-premises or in a standalone operating environment. As previously noted, and without limitation, the topic can be implemented within or associated with a cloud deployment platform system or appliance, or using any other type of deployment system, product, appliance, program, or process. As already described, model building or inference system functionality can be provided as a standalone function, or it can leverage functionality from other products and services.

[0116] Representative cloud application platforms that can implement this technology include, but are not limited to, any cloud-supported application framework, product, or service.

[0117] In general, the technologies described herein can be implemented as management solutions, services, products, appliances, devices, processes, programs, execution threads, etc. Typically, these technologies are implemented in software as one or more computer programs executing in hardware processing elements, in conjunction with data stored in one or more data sources (such as a problem database). Some or all of the described processing steps can be automated and operate autonomously in conjunction with other systems. Automation can be complete or partial, and operation (complete or partial) can be synchronous or asynchronous, demand-based, or otherwise.

[0118] These components are typically implemented as software, that is, as a set of computer program instructions that execute in one or more hardware processors. Components are shown as distinct, but this is not necessary, as components can also be integrated with each other, either wholly or partially. One or more of the components may execute in a dedicated location or remotely from each other. One or more of the components may have sub-components that execute together to provide functionality. Since the functionality (or any aspect thereof) described here can be implemented elsewhere or in a system, it is not required that the specific functionality served by the generator be performed by the specific components described above.

[0119] Tools and response capabilities can interact or interoperate with security analytics systems or services.

[0120] As already described, the functionality described above can be implemented as a standalone method, such as one or more software-based functions executed by one or more hardware processors, or it can be available as a management service (including as a web service via a SOAP / XML interface). The specific hardware and software implementation details described herein are for illustrative purposes only and are not intended to limit the scope of the described subject matter.

[0121] More generally, computing devices in the context of the disclosed subject matter are data processing systems (such as hardware and software) Figure 2(As shown in the diagram), and these entities communicate with each other via networks such as the Internet, intranets, extranets, private networks, or any other communication medium or link. Applications on the data processing system provide native support for the Web and other known services and protocols, including but not limited to support for HTTP, FTP, SMTP, SOAP, XML, WSDL, UDDI, and WSFL. Information on SOAP, WSDL, UDDI, and WSFL is available from the World Wide Web Consortium (W3C), which is responsible for developing and maintaining these standards; further information on HTTP, FTP, SMTP, and XML is available from the Internet Engineering Task Force (IETF).

[0122] As noted, and in addition to cloud-based environments, the techniques described in this paper can be implemented in or in combination with different server-side architectures, including simple n-tier architectures, web portals, federated systems, and so on.

[0123] More generally, the subject matter described herein may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment that includes both hardware and software elements. In a preferred embodiment, the sensitive data detection service (or any component thereof) is implemented in software, including but not limited to firmware, resident software, microcode, etc. Furthermore, the download and deletion interface and functionality may take the form of a computer program product accessible from a computer-usable or computer-readable medium that provides program code for use by or in conjunction with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable medium may be any means that can contain or store programs for or in conjunction with an instruction execution system, apparatus, or device. This medium may be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device). Examples of computer-readable media include semiconductor or solid-state memory, magnetic tape, removable computer disks, random access memory (RAM), read-only memory (ROM), rigid disks, and optical discs. Current examples of optical discs include compact disc-read-only memory (CD-ROM), compact disc-read / write (CD-R / W), and DVDs. Computer-readable media are tangible, non-temporary items.

[0124] A computer program product may be a product having program instructions (or program code) for implementing one or more of the described functions. Those instructions or code may be stored in a computer-readable storage medium within a data processing system after being downloaded from a remote data processing system via a network. Alternatively, those instructions or code may be stored in a computer-readable storage medium within a server data processing system and are adapted to be downloaded via a network to a remote data processing system for use in a computer-readable storage medium within the remote system.

[0125] In a representative embodiment, the technology is implemented in a dedicated computing platform, preferably in software executed by one or more processors. The software is maintained in one or more data stores or memories associated with the one or more processors, and the software can be implemented as one or more computer programs. In general, the dedicated hardware and software include the functions described above.

[0126] While a specific order of operations performed by certain embodiments of the invention has been described above, it should be understood that such an order is exemplary, as alternative embodiments may perform operations in a different order, combine certain operations, overlap certain operations, etc. References to a given embodiment in the specification indicate that the described embodiment may include a particular feature, structure, or characteristic, but each embodiment may not necessarily include that particular feature, structure, or characteristic.

[0127] Finally, although the given components of the system have been described individually, those skilled in the art will understand that some functions can be combined or shared in a given instruction, program sequence, code section, etc.

[0128] Furthermore, FHE is merely a representative encryption protocol and is not intended to be restrictive.

[0129] Although the above method implements FHE-based inference in a client-server context, this is not a limitation. The above techniques can also be implemented within the client that accesses the tree.

[0130] Furthermore, while inference preferably occurs using the FHE protocol, the method presented herein can be utilized in conjunction with other multi-party secure computation techniques and privacy that are intended to be preserved, such as test points, models, or both.

[0131] The techniques described in this paper provide improvements to another technology or technology field (i.e., FHE-based inference tools and systems, and cloud-based systems that combine or disclose these techniques), as well as improvements to the computational efficiency of FHE systems and methods.

[0132] This disclosure is not limited to specific use cases or applications that use the framework.

[0133] Although the method described here envisions the server performing one or more of the operations described before receiving FHE-encrypted data points, this preprocessing is not necessarily required. An alternative approach is for the server to perform all of this processing on demand (i.e., dynamically) in response to a client query.

[0134] The subject matter has been described, and the claims are as follows.

Claims

1. A method for creating a fully homomorphic encryption (FHE) compliant machine learning model, comprising: A first machine learning model is obtained, which has been pre-trained using the first training data; Receive the input data distribution and the set of FHE constraints; and Construct a second machine learning model for the set that satisfies the FHE constraints. The second machine learning model is constructed as follows: Use the input data distribution to generate the transport dataset; and The parameters of the second machine learning model are learned using the transport dataset and the predictions of the first machine learning model on the transport dataset as synthetic training data.

2. The method according to claim 1, wherein, The parameters of the second machine learning model are learned without accessing the first training data.

3. The method according to claim 1, wherein, Constructing the second machine learning model further includes: Construct a second set of machine learning models, comprising a set of all models satisfying the FHE constraints, wherein each second machine learning model in the second set of machine learning models is constructed as follows: Use the input data distribution to generate the transport dataset; and The parameters of each second machine learning model are learned using the transmitted dataset and the predictions of the first machine learning model on the transmitted dataset as synthetic training data. Use the input data distribution to generate a validation dataset; For each candidate second machine learning model in the second machine learning model set, the fit of each candidate second machine learning model is determined using the validation dataset and the predictions of the first machine learning model on the validation dataset as synthetic validation data; and The candidate second machine learning model with the best fit is selected as the second machine learning model.

4. The method according to claim 1, wherein, The set of FHE constraints includes one of the following: target multiplication circuit depth, security level, ciphertext size, and combinations thereof.

5. The method of claim 1, further comprising: The second machine learning model is used to perform privacy-preserving homomorphic inference on encrypted data.

6. The method according to claim 5, wherein, The second machine learning model is used instead of the first machine learning model.

7. The method according to claim 1, wherein, The second machine learning model includes a shallow neural network with an FHE-compatible activation function.

8. An apparatus for creating fully homomorphic encryption (FHE) compliant machine learning models, comprising: processor; A computer memory containing computer program instructions executable by the processor, the computer program instructions being configured to create a fully homomorphic encryption (FHE) compatible machine learning model, the computer program instructions being configured to: A first machine learning model is obtained, which has been pre-trained using the first training data; Receive the input data distribution and the set of FHE constraints; and Construct a second machine learning model for the set that satisfies the FHE constraints. The second machine learning model is constructed as follows: Use the input data distribution to generate the transport dataset; and The parameters of the second machine learning model are learned using the transport dataset and the predictions of the first machine learning model on the transport dataset as synthetic training data.

9. The device according to claim 8, wherein, The parameters of the second machine learning model are learned without accessing the first training data.

10. The device according to claim 8, wherein, The computer program instructions for constructing the second machine learning model further include computer program instructions executed by the processor to perform the following operations: Construct a second set of machine learning models, comprising a set of all models satisfying the FHE constraints, wherein each second machine learning model in the second set of machine learning models is constructed as follows: Use the input data distribution to generate the transport dataset; and The parameters of each second machine learning model are learned using the transmitted dataset and the predictions of the first machine learning model on the transmitted dataset as synthetic training data. Use the input data distribution to generate a validation dataset; For each candidate second machine learning model in the second machine learning model set, the validation dataset and the predictions of the first machine learning model on the validation dataset are used as synthetic validation data to determine the fit of each candidate second machine learning model; as well as The candidate second machine learning model with the best fit is selected as the second machine learning model.

11. The device according to claim 8, wherein, The set of FHE constraints includes one of the following: target multiplication circuit depth, security level, ciphertext size, and combinations thereof.

12. The device according to claim 8, wherein, The computer program instructions are further executed by the processor to perform privacy-preserving homomorphic inference on the encrypted data using the second machine learning model.

13. The device according to claim 12, wherein, The second machine learning model was used to replace the first machine learning model.

14. The device according to claim 8, wherein, The second machine learning model comprises a shallow neural network with an FHE-compatible activation function.

15. A computer program product for a data processing system to create a fully homomorphic encryption (FHE) compatible machine learning model, the computer program product comprising computer program instructions that, when executed by the data processing system: A first machine learning model is obtained, which has been pre-trained using the first training data; Receive the input data distribution and the set of FHE constraints; and Construct a second machine learning model for the set that satisfies the FHE constraints. The second machine learning model is constructed as follows: Use the input data distribution to generate the transport dataset; and The parameters of the second machine learning model are learned using the transport dataset and the predictions of the first machine learning model on the transport dataset as synthetic training data.

16. The computer program product according to claim 15, wherein, The parameters of the second machine learning model are learned without accessing the first training data.

17. The computer program product according to claim 15, wherein, The computer program instructions for constructing the second machine learning model further include computer program instructions executed by the data processing system to perform the following operations: Construct a second set of machine learning models, comprising a set of all models satisfying the FHE constraints, wherein each second machine learning model in the second set of machine learning models is constructed as follows: Use the input data distribution to generate the transport dataset; and The parameters of each second machine learning model are learned using the transmitted dataset and the predictions of the first machine learning model on the transmitted dataset as synthetic training data. Use the input data distribution to generate a validation dataset; For each candidate second machine learning model in the second machine learning model set, the validation dataset and the predictions of the first machine learning model on the validation dataset are used as synthetic validation data to determine the fit of each candidate second machine learning model; as well as The candidate second machine learning model with the best fit is selected as the second machine learning model.

18. The computer program product according to claim 15, wherein, The set of FHE constraints includes one of the following: target multiplication circuit depth, security level, ciphertext size, and combinations thereof.

19. The computer program product according to claim 15, wherein, The computer program instructions are further executed by the data processing system to perform privacy-preserving homomorphic inference on the encrypted data using the second machine learning model.

20. The computer program product according to claim 19, wherein, The second machine learning model is used to replace the first machine learning model.

21. The computer program product according to claim 15, wherein, The second machine learning model includes a shallow neural network with an FHE-compatible activation function.

Images