Depth-Constrained Knowledge Distillation for Encrypted Data Inference

A knowledge distillation framework generates an FHE-friendly student model using synthetic data to maintain accuracy and comply with FHE constraints, addressing the challenges of performing machine learning on encrypted data with pre-trained models.

JP7578370B2Active Publication Date: 2024-11-06INTERNATIONAL BUSINESS MACHINE CORPORATION
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2022575921
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-06-22
Filing Date
2021-04-28
Publication Date
2024-11-06
Estimated Expiration
2041-04-28

AI Technical Summary

Technical Problem

Existing approaches for performing machine learning on encrypted data using fully homomorphic encryption (FHE) face challenges in maintaining accuracy due to depth constraints of leveled HE schemes, leading to either significant simplification of models or the inability to use pre-trained models without the original training data.

Method used

A knowledge distillation framework is used to generate a fully homomorphic encryption (FHE)-friendly student model that mimics the predictions of a more complex teacher model, utilizing synthetic training data and FHE constraints to build a depth-constrained student model without accessing the original training data.

Benefits of technology

The approach enables accurate privacy-preserving homomorphic estimation on encrypted data by creating a student model that maintains high accuracy while adhering to FHE constraints, overcoming the limitations of direct use of pre-trained models and simplification methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007578370000030
    Figure 0007578370000030
  • Figure 0007578370000031
    Figure 0007578370000031
  • Figure 0007578370000032
    Figure 0007578370000032
Patent Text Reader

Abstract

Depth-constrained knowledge distillation for,encrypted data inference. The present disclosure provides methods, apparatus, and computer program products for generating fully homomorphic encryption (FHE)-friendly machine learning models. The approach herein augments a knowledge distillation framework in which an FHE-friendly (student) ML model closely mimics the predictions of a more complex (teacher) model, where the teacher model has been pre-trained on a more complex and larger dataset than the student model. In the approach herein, the distillation framework facilitates training of a more complex teacher-model but FHE-friendly model using synthetically generated training data instead of the original dataset used to train the teacher.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates generally to the use of fully homomorphic encryption operations to facilitate inference on encrypted data. [Background technology]

[0002] In machine learning (ML), knowledge distillation refers to learning a student model that closely mimics the predictions of a teacher model. The teacher model is usually a complex ML model pre-trained on a large dataset. The student model is a target ML model that has some desired properties, such as less model complexity and better explanatory power. Distillation is achieved by allowing the teacher to weakly supervise the student's learning process. In this context, the teacher and student are not required to be neural networks, and the original training data (used for the teacher to learn) may not be available for the student model to learn from.

[0003] Homomorphic encryption (HE) allows computations performed on ciphertext to produce an encrypted result that, when decrypted, matches the result of an operation performed on the plaintext. Homomorphic encryption schemes are cryptosystems that allow computations to be performed without decrypting the data.

[0004] Data privacy considerations often hinder the use of cloud-based machine learning services for sensitive personal data. Homomorphic encryption (HE) offers a possible solution by allowing computations on encrypted data, but the challenge is to obtain accurate machine learning models that operate within the multiplicative depth constraints of leveled HE schemes. Existing approaches for encrypted estimation either perform ad-hoc simplifications on pre-trained models at the expense of accuracy, or they directly train new depth-constrained models using the original training set. In machine learning models, when encrypted data is used (e.g., determining tree-based inferences on encrypted data), these solutions cannot operate effectively because the required simplifications are more constraining to FHE (Fully Homomorphic Encryption) computational power.

[0005] There remains a need to provide techniques for building machine learning models that are easily evaluated within the FHE domain (hereinafter sometimes referred to as FHE "friendly") and without loss of accuracy in comparison to arbitrary, more complex models that are otherwise desired for use in evaluating inputs of interest. Summary of the Invention

[0006] The present disclosure provides methods, apparatus, and computer program products for generating fully homomorphic encryption (FHE)-friendly machine learning models. The approach herein augments a knowledge distillation framework such that an FHE-friendly (student) ML model closely mimics the predictions of a more complex (teacher) model, where the teacher model is more complex relative to the student model and has been pre-trained on a large dataset. In this approach herein, the distillation framework facilitates training an FHE-friendly model using a more complex teacher model, but using synthetically generated training data instead of the original dataset used to train the teacher.

[0007] In one aspect, the method generates a machine learning model compatible with fully homomorphic encryption (FHE) by obtaining a first (teacher) machine learning model pre-trained using first training data. A distribution of input data and a set of FHE constraints are also received. The FHE constraints typically include a target multiplier depth, a security level, a ciphertext size, and combinations thereof. Using these inputs, a second (student) machine learning model is then constructed. The second model has an accuracy compatible with the first model but also satisfies a set of FHE constraints but not like the first model. In particular, preferably the second ML model is constructed as follows: A transfer dataset is generated using the distribution of input data. A set of synthetic training data is then generated by the first machine learning model and predictions are made on the transfer dataset (instead of the original dataset used to train the first model). The set of synthetic training data is then used to train (i.e., learn parameters of) the second model. The second model can be one of a set of candidate second machine learning models, each trained in this manner, and the best fitting candidate can then be selected as the second model. The second machine learning model is depth-constrained with respect to the FHE constraints and includes a shallow neural network with an FHE-compatible activation function.

[0008] The second machine learning model generated in this manner is then used for privacy-preserving homomorphic inference on the encrypted input.

[0009] The foregoing has outlined some of the more pertinent features of the subject matter. These features should be construed as merely illustrative. Many other beneficial results can be achieved by applying the subject matter disclosed in different manners or by modifying the subject matter as described. For a more complete understanding of the present invention and its advantages, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which: [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an exemplary block diagram of a distributed data processing environment in which exemplary aspects of the exemplary embodiments may be implemented. [Diagram 2] FIG. 2 illustrates an example block diagram of a system in which example aspects of the example embodiments can be implemented. [Diagram 3] FIG. 3 illustrates an example block diagram of an example cloud computing architecture in which the disclosed subject matter can be implemented. [Figure 4] FIG. 4 illustrates a Machine Learning as-a-service (MLaaS) operating environment in which the techniques of this disclosure can be implemented. [Diagram 5] Figure 5 shows how the distillation framework of this disclosure is used to build an FHE-friendly student model. [Figure 6] Figure 6 illustrates the operation of the transfer data set generator of the present disclosure. [Figure 7] Figure 7 shows a representative neural network architecture from which the Student Model is constructed according to the respective embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Referring now to the drawings, and in particular to Figures 1-2, exemplary diagrams of data processing environments are provided in which exemplary embodiments of the present disclosure may be implemented. It should be appreciated that Figures 1-2 are exemplary only and are not intended to state or imply any limitations with respect to the environments in which aspects or embodiments of the subject matter of the present disclosure may be implemented. Many modifications to the illustrated environments may be made without departing from the spirit and scope of the present disclosure. Client-Server Technology

[0012] Referring now to the drawings, Figure 1 shows a diagrammatic representation of an exemplary distributed data processing system in which aspects of the exemplary embodiments may be implemented. The distributed data processing system 100 may include a network of computers in which aspects of the exemplary embodiments may be implemented. The distributed data processing system 100 includes at least one network 102, which is used to provide communication links between the various devices and computers connected to each other within the distributed data processing system 100. The network 102 may include connections such as wired, wireless communication links, or fiber optic cables.

[0013] In the illustrated embodiment, server 104 and server 106 are connected to network 102 along with storage unit 108. Additionally, clients 110, 112, and 114 are also connected to network 102. These clients 110, 112, and 114 may be, for example, personal computers, network computers, etc. In the illustrated embodiment, server 104 provides data such as boot files, operating system images, and applications to clients 110, 112, and 114. Clients 110, 112, and 114 are clients to server 104 in the illustrated embodiment. Distributed data processing system 100 may include additional servers, clients, and other devices not shown.

[0014] In the depicted embodiment, distributed data processing system 100 is the Internet with network 102 representing a worldwide collection of networks and gateways that use the Transmission Control Protocol / Internet Protocol (TCP / IP) suite of protocols to communicate with one another. At the heart of the Internet is a backbone of high speed data communication lines between major nodes or host computers, including thousands of commercial, governmental, and educational computers, and other computer systems that route data and messages. Of course, distributed data processing system 100 may also include numerous networks of different types, such as, for example, intranets, local area networks (LANs), wide area networks (WANs), and the like. As noted above, FIG. 1 is an example and is not intended to imply architectural limitations for different embodiments of the disclosed subject matter, and thus the specific elements illustrated in FIG. 1 should not be construed as limitations on the environments in which exemplary embodiments of the present invention may be implemented.

[0015] Referring now to Figure 2, there is shown a block diagram of an exemplary data processing system in which aspects of the exemplary embodiments may be implemented. Data processing system 200 is an example of a computer, such as client 110 in Figure 1, within which computer usable code or instructions implementing processes for the exemplary embodiments of the present disclosure may be located.

[0016] Referring now to Figure 2, there is shown a block diagram of a data processing system in which exemplary embodiments may be implemented. Data processing system 200 is an example of a computer, such as server 104 or client 110 of Figure 1, in which computer usable code or instructions implementing processes for exemplary embodiments may be located. In this exemplary embodiment, data processing system 200 includes a communication facility 202, which provides communication between a processor unit 204, a memory 206, a persistent storage 208, a communication unit 210, an input / output (I / O) unit 212, and a display 214.

[0017] Processor unit 204 provides execution of instructions for software that may be loaded into memory 206. Processor unit 204 may be a set of one or more processors or may be a multi-processor core, depending on the particular implementation. Additionally, processor unit 204 may be implemented using one or more heterogeneous processor systems in which a main processor resides on a single chip with secondary processors. As another illustrative example, processor unit 204 may be a symmetric multi-processor (SMP) system that includes multiple processors of the same type.

[0018] Memory 206 and persistent storage 208 are examples of storage devices. A storage device is any piece of hardware capable of storing information on either a temporary or permanent basis or both. Memory 206, in these embodiments, may be, for example, random access memory, or any other suitable volatile or non-volatile storage device. Persistent storage 208 may take a variety of forms, depending on the particular implementation. For example, persistent storage 208 may include one or more components or devices. For example, persistent storage 208 may be a hard drive, a flash memory, a rewritable optical disk, a rewritable magnetic tape, or some combination of the above. The media used by persistent storage 208 may also be removable. For example, a removable hard drive may be used for persistent storage 208.

[0019] Communications unit 210, in these examples, provides for communication with other data processing systems or devices. In these examples, communications unit 210 is a network interface card. Communications unit 210 can provide communications through either or both physical and wireless communications links.

[0020] The input / output unit 212 allows for the input and output of data to and from other devices that may be connected to the data processing system 200. For example, the input / output unit 212 may provide communication for user input through a keyboard and mouse. Additionally, the input / output unit 212 may send output to a printer. The display 214 provides a mechanism for displaying information to a user.

[0021] Instructions for the operating system and applications or programs are located on persistent storage 208. These instructions may be loaded into memory 206 for execution by processor unit 204. The processes of the different embodiments may be executed by processor unit 204 using computer-implemented instructions, which may be located in a memory, such as memory 206. These instructions are referred to as program code, computer usable program code, or computer usable program code, which may be read and executed by a processor in processor unit 204. The program code in the different embodiments may be embodied on different physical or tangible computer readable media, such as memory 206 or persistent storage 208.

[0022] The program code 216 is disposed in a functional form on a selectively removable computer readable medium 218 and can be loaded or transferred to the data processing system 200 for execution by the processor unit 204. The program code 216 and the computer readable medium 218 form a computer program product 220 in these examples. In one embodiment, the computer readable medium 218 can be in a tangible form, such as an optical or magnetic disk, that is inserted or placed into a drive or hard drive that is part of the persistent storage 208 for transfer to a storage device, such as a hard drive that is part of the persistent storage 208. In a tangible form, the computer readable medium 218 can take the form of a hard drive, thumb drive, or flash memory that is connected to the data processing system 200. The tangible form of the computer readable medium 218 is also referred to as a computer recordable storage medium. In some examples, the computer readable medium 218 may not be removable.

[0023] Alternatively, the program code 216 can be transferred to the data processing system 200 from the computer readable medium 218 through a communications link to the communications unit 210 or through insertion into the input / output unit 212, or both. The communications link, or connection, or both, can be physical or wireless in an exemplary embodiment. The computer readable medium can also take the form of a non-tangible medium, such as a communications link or wireless communication containing the program code. The different components illustrated for the data processing system 200 are not meant to provide architectural limitations to the manner in which different embodiments may be implemented. Different exemplary embodiments can be implemented in a data processing system that includes components in addition to or in place of those illustrated for the data processing system 200. Other components shown in FIG. 2 can vary from the illustrated exemplary embodiment. As one example, a storage device in the data processing system 200 is any hardware device capable of storing data. Memory 206, persistent storage 208, and computer readable media 218 are examples of storage devices in a tangible form.

[0024] In another embodiment, a bus system may be used to implement the communication function 202 and may be comprised of one or more buses, such as a system bus or an input / output bus. Of course, the bus system may be implemented using any suitable type of architecture that provides for a transfer of data between different components or devices attached to the bus system. Additionally, the communication unit may include one or more devices used for transmitting and receiving data, such as a modem or a network adapter. Furthermore, the memory may be, for example, a memory 206 or a cache such as found in an interface and memory controller that may be present in the communication function 202.

[0025] Computer program code for carrying out the operations of the present invention can be written in one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, C#, Objective-C, etc., and conventional procedural programming languages. The program code can run entirely on the user's computer as a stand-alone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (e.g., through the Internet using an Internet Service Provider).

[0026] Those skilled in the art will recognize that the hardware in Figures 1-2 may vary depending on the implementation. Other internal hardware or peripheral devices such as flash memory, equivalent non-volatile memory, or optical disk drives, etc. may be used in addition to or in place of the hardware depicted in Figures 1-2. Additionally, the processes of the exemplary embodiments may be applied to multi-processor data processing systems other than the previously described SMP systems without departing from the spirit and scope of the subject matter of the present disclosure.

[0027] As can be seen, the techniques disclosed herein can operate in combination with a standard client-server paradigm as illustrated in FIG. 1, where a client machine communicates with a web-based portal running on a set of one or more machines with Internet access. End users operate Internet-enabled devices (e.g., desktop computers, notebook computers, Internet-enabled mobile devices, etc.) that can access and interact with the portal. Typically, each client or server machine is a data processing system as illustrated in FIG. 2, including hardware and software, and these entities communicate with each other over a network, such as the Internet, an intranet, an extranet, a private network, or any other communication medium or link. A data processing system typically includes one or more processors, an operating system, one or more applications, and one or more utilities. Applications on the data processing system provide native support for Web services, including, but not limited to, support for HTTP, SOAP, XML, WSDL, UDDI, and WSFL, among others. Information on SOAP, WSDL, UDDI, and WSFL is available from the World Wide Web Consortium (W3C), which is responsible for developing and maintaining these standards; additionally, information on HTTP and XML is available from the Internet Engineering Task Force (IETF). Familiarity with these standards is assumed. Cloud Computing Model

[0028] An emerging information technology (IT) delivery model is cloud computing, whereby shared resources, software, and information are delivered on demand to computers and other devices over the Internet. Cloud computing can significantly reduce IT costs and complexity while improving workload optimization and service delivery. With this approach, application instances are hosted and made available from Internet-based resources accessible through a traditional Web browser over HTTP. An example application can provide a common set of messaging capabilities such as email, calendaring, contact management, and instant messaging. Users would then access the service directly over the Internet. Using this service, an enterprise deploys its email, calendar, or collaboration infrastructure in the cloud, and an end user would use an appropriate client to access his or her email or perform calendar operations.

[0029] Cloud computing resources are typically housed within large server farms that run one or more network applications using a virtualization architecture, where the applications run within virtual servers, or so-called "virtual machines" (VMs), that are mapped to physical servers in a data center facility. Virtual machines typically run on top of a hypervisor, which is a control program that allocates physical resources to the virtual machines.

[0030] Cloud Computing is a service delivery model for on-demand network access that provides convenient access to a shared pool of rapidly provisioned and open configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) with minimal administrative effort or interaction with a service provider. This cloud model includes at least five characteristics, at least three service models, and at least four deployment models, all of which are more particularly described and defined in “Draft NIST Working Definition of Cloud Computing” by Peter Mell and Tim Grance, dated October 7, 2009. In particular, the following are typical characteristics: On-Demand Self-Service: Cloud consumers are automatically provisioned with computing capacity - server time and network storage - on an as-needed basis without the need for human interaction with the service provider. Pervasive Network Access: Capabilities are available over the network and accessed through standard mechanisms facilitating use by different thin- and thick-client platforms (eg, mobile phones, laptops, and PDAs). Resource Sharing: Using a multi-tenant model, a provider's computing resources are shared to serve multiple consumers, with different physical and virtualized resources dynamically allocated and reallocated as needed. A sense of location independence exists, such that consumers generally have no control or knowledge of the exact location (e.g., country, state, or data center) of the resources provided, but can specify the location at a higher layer of abstraction. Rapid Elasticity: Capabilities can be provisioned quickly and elastically, in some cases automatically, to quickly scale out and quickly release to quickly scale in. To the consumer, the capabilities available for provisioning often appear unlimited and can be purchased in any quantity at any time. Metered Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at several levels of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled and reported to provide transparency to both providers and consumers of the services being used.

[0031] The service model is as follows: Software as a Service (SaaS): The functionality provided to the consumer is the use of the provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or individual application functionality, except for limited user-specific application configuration settings. Platform as a Service (PaaS): The capability offered to a consumer is to deploy applications that the consumer has created or acquired, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does control the deployed applications and, potentially, the configuration of the application hosting environment. Infrastructure as a Service (IaaS): The functionality provided to the consumer is the provision of processing, storage, network, and other basic computing resources onto which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, deployed applications, and possibly limited control over select networking components (e.g., host firewalls).

[0032] The deployment model is as follows: Private Cloud: The cloud infrastructure operates solely for one organization. It can be managed by that organization or a third party and can exist on-premises or off-premises. Community Cloud: The cloud infrastructure is shared by several organizations to support a specific community with common interests (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by those organizations or by a third party and can exist on-premises or off-premises. Public Cloud: Cloud infrastructure is made available to the public or large industry groups and is owned by organizations that sell cloud services. Hybrid Cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain unique entities but are bound to each other by standardized or proprietary technologies that allow portability of data and applications (e.g., cloud bursting for load balancing between clouds).

[0033] Cloud computing environments are service-oriented with a focus on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes multiple interconnected nodes. A representative cloud computing node is as shown in FIG. 2 above. In particular, there is a computer system / server that can operate with numerous other general-purpose or special-purpose computing system environments or configurations. Well-known computing systems, environments, or configurations, or combinations thereof, suitable for use with a computer system / server include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments, including any of the above systems or devices. A computer system / server may be described in the general context of computer system-executable instructions, such as program modules, executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system / server may be implemented in a distributed cloud computing environment where tasks are performed by remote processing system devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

[0034]

[0023] Referring now to Figure 3, for additional background, a set of functional abstraction layers provided by a cloud computing environment is shown. It should be understood that the components, layers, and functions shown in Figure 3 are intended to be illustrative only, and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided: The hardware and software layer 300 includes hardware and software components. Examples of hardware components can include a mainframe, in one embodiment, IBM® zSeries®, multiple servers based on RISC (reduced instruction set computer) architecture, in one embodiment, IBM pSeries® systems; IBM xSeries®; IBM BladeCenter® systems; storage devices; networks and networking components. Examples of software components include, in one embodiment, IBM WebSphere®, application server software; and database software, in one embodiment, IBM DB2® database software. (IBM zSeries, pSeries, xSeries, BladeCenter, WebShere, and DB2 are trademarks of International Business Machines Corporation, registered worldwide under various laws.)

[0035] The visualization layer 302 provides an abstraction layer from which are provided embodiments of virtual entities, as described below: virtual servers; virtual storage; virtual networks, including virtual private networks; virtual applications and operating systems; and virtual clients.

[0036] In one embodiment, the management layer 304 may provide the following functions: A resource provisioning unit provides dynamic acquisition of computing resources and other resources used to perform tasks within the cloud computing environment. A metering and pricing unit provides cost tracking as resources are used within the cloud computing environment and provides billing or charging for the consumption of these resources. In one embodiment, these resources may include application software licenses. A security unit provides identification and authentication of cloud consumers and tasks as well as protection of data and other resources. A user portal unit provides accessibility to the cloud computing environment and system administrators for consumers. A service level management unit provides allocation and management of cloud computing resources to meet required service levels. A service level agreement (SLA) planning and fulfillment unit provides pre-provisioning and acquisition of cloud computing resources as future demand requires according to SLAs.

[0037] The workload layer 306 provides examples of functionality for utilizing a cloud computing environment. Examples of workloads and functionality provided by this layer can include: mapping and navigation; software development and lifetime management; virtual classroom instructional delivery; data analytics processing; transaction processing; and, in accordance with the present disclosure, techniques for privacy-preserving homomorphic inference on homomorphically encrypted data 308.

[0038] Upfront, although this disclosure includes a detailed description of cloud computing, it is understood that implementation of the teachings recited herein is not limited to a cloud computing environment. Rather, embodiments of the invention can be implemented in combination with any other type of computing environment now known or later developed. These can include stand-alone computing environments (e.g., on-premises desktop machines), client-server based architectures, and the like.

[0039] Thus, a typical cloud computing environment has a set of high-level functional components including a front-end identity manager, a business support services (BSS) functional component, an operational support services (OSS) functional component, and a compute cloud component. The identity manager is responsible for interfacing with requesting clients to provide identity management, which may be implemented with one or more known systems, such as Tivoli Federated Identity Manager (TFIM) available from IBM Corporation, Armonk, New York. TFIM in an appropriate environment may be used to provide federated single sign-on (F-SSO) to other cloud components. The business support services component provides certain administrative functions, such as billing support. The operational support services component is used to provide provisioning and management of other cloud components, such as virtual machine (VM) instances. The cloud components represent the primary computing resources, which are typically multiple virtual machine instances used to run target applications that are made available for access via the cloud. One or more databases are used to store directories, logs, and other operational data. All of these components (including the front-end identity manager) are located "in" the cloud, although this is not required. In alternative embodiments, the identity manager could be run outside the cloud. The service provider could also be run outside the cloud.

[0040] Some clouds are based on non-traditional IP networks. Thus, for example, a cloud may be based on a two-tier CLOS-based network with a specialized single-layer IP that uses a hash of MAC addresses. The techniques described herein may be used with such non-traditional clouds.

[0041] Generally speaking, a cloud computing infrastructure provides a virtual machine hosting environment that includes host machines (e.g., servers or similar physical computing devices) connected via a network and one or more management servers. Typically, each physical server is adapted to dynamically host one or more virtual machines using virtualization technology such as VMware ESX / ESXi. Multiple VMs are deployed on a single host machine and share the host machine's CPU, memory, and other resources, thereby improving utilization of an organization's data center. Among other tasks, the management servers monitor the infrastructure and automatically manipulate VM deployment as needed, for example, by moving virtual machines between hosts.

[0042] In a non-limiting implementation, a representative platform technology is, but is not limited to, IBM System x servers with VMware vSphere 4.1, updates 1 and 5.0. The above-mentioned commercially available implementation is not intended to be limiting, but rather is merely one representative embodiment of a client application that may be supported in a cloud computing environment and interact with cognitive services. Homomorphic encryption

[0043] Homomorphic encryption (HE) allows computations to be performed on the ciphertext, thus producing a cryptographic result that, when decrypted, matches the result of the operation performed on the plaintext. Homomorphic encryption schemes are cryptosystems that allow computations to be performed on data without decrypting it. FHE is often achieved by utilizing Somewhat Homomorphic Encryption (SWHE), or leveled HE schemes, in combination with bootstrapping or re-encryption techniques. SWHE schemes can only support computations up to a pre-determined level of complexity determined by their parameters. This is because the ciphertext is “noisy” and this noise keeps growing with each HE computation, and once the noise grows beyond some parameter-dependent threshold, the ciphertext can no longer be decrypted. This problem can be addressed by bootstrapping, which refreshes the ciphertext and reduces the noise level, but at the cost of circular security. Bootstrapping, however, is a computationally expensive and time-consuming operation, and therefore, for all practical purposes, the number of bootstrapping operations should be kept to a minimum and avoided if possible.

[0044] Homomorphic encryption allows the construction of a program for any desired function that can be operated on encrypted inputs to produce encrypted results. Since such a program never needs to decrypt its inputs, it can be run by untrusted parties without revealing its inputs and internal state. Toolkits for implementing homomorphic encryption are known. A well-known toolkit is the open source project HElib. The current version of HElib supports arbitrary number addition and multiplication operations in binary representation using encryption of individual bits.

[0045] For further background, several FHE schemes have been proposed based on the difficulty of a computational problem known as Ring Learning with Errors (RLWE). Notable examples of such schemes include the Brakerski / Fan-Vercauteren (BFV), Brakerski-Gentry-Vaikuntanathan (BGV), and Cheon-Kim-Kim-Song (CKKS) schemes, which are both additively and multiplicatively homomorphic (named after their original proposers). The BFV and BGV schemes are effective for vector operations on integers, while the CKK scheme is more appropriate for “approximate” (precision-limited) floating-point operations. Furthermore, these schemes support Single Instruction Multiple Data (SIMD) operations, particularly by packing different plaintext values ​​into different slots in the ciphertext. This ciphertext packing allows parallelization of addition and multiplication operations, but it is not possible to randomly access values ​​in individual slots of the ciphertext after packing, instead only limited operations are possible, such as rotating slots within the ciphertext.

[0046] In the case of the CKKS scheme for HE operations (as implemented in the HElib library), real-valued features can be directly input to the decision tree, i.e., there is no need to convert the input to integers. Machine Learning as a Service using Homomorphic Encryption

[0047] Machine Learning as a Service (MLaaS) is now a popular paradigm, where pre-trained models are hosted in a public cloud and inference is performed on a pay-per-query basis. The use of MLaaS is, however, limited in many application domains (e.g. financial services, health care) because privacy of the client data on which the inference is performed is of paramount concern. Fully Homomorphic Encryption (FHE) can solve the above privacy challenge by allowing certain types of computations on encrypted data without the need for decryption. As mentioned above, the FHE scheme can be defined as follows:

[0048]

number

[0049] where ε and D denote the encryption and decryption operations, respectively, λ is a security parameter, and EVAL is the evaluation function that takes an arbitrary function f and an encryption input ε(I) and returns the encrypted result

[0050]

number

[0051] For ease of presentation, it will be understood that the ε and EVAL operations are required to have access to the public key, and D needs access to the private key.

[0052] Referring to FIG. 4, the FHE-based privacy-preserving estimation scenario is illustrated and works as follows: A client 400 encrypts its data x (using a cryptographic function 401) and sends the encrypted data ε(x) to a cloud-based machine learning service provider 402. The service provider receives a pre-trained model 404M. θ(where M and θ denote the model architecture and parameters, respectively), and in the encryption domain, the estimation calculation,

[0053]

number

[0054] and returns the encrypted result to the client 400 for decryption 405. Since the service provider does not obtain any knowledge about the input (x) or the resulting output (y), complete client data privacy is ensured. Furthermore, the scenario described above is referred to as non-interactive since it involves only a single round of communication between the client and the service provider.

[0055] The fundamental challenge in encrypted estimation as illustrated in Fig. 4 arises from significant limitations on the nature of suitable computations and the efficiency of computations in the encryption domain. Most FHE schemes are built on leveled HE schemes, which support computations only up to a pre-set level of multiplication depth determined by its parameters. This is because the “noise” in the ciphertext keeps growing with each HE computation, and once the noise grows beyond a threshold, decryption fails. This depth constraint can be overcome by using bootstrapping, which “refreshes” the ciphertext and reduces its noise level (at a cost that depends on the circular security). Bootstrapping, however, is a highly expensive and complex operation that must be avoided or barely used for practically suitable computations. In the ML context, the problem of limited depth is further amplified by the presence of non-linear computations (e.g., comparison of two numbers, activation functions), which require a larger depth since the underlying HE schemes often only support homomorphisms of addition and multiplication. For these reasons, it is rarely possible to directly use any pre-trained ML for encryption estimation. For example, boosted decision trees are still the standard choice for classification and regression tasks over tabular datasets, and applying them directly to encrypted data is difficult due to their reliability in comparison operations.

[0056] Recent work in cryptographic estimation has attempted to solve this problem in two ways. One approach is to train an unconstrained model and then simplify its architecture to reduce the multiplicative depth without retraining the parameters. Another alternative is to define a depth-constrained model and train it directly from scratch using available training data. The former approach typically results in a significant reduction in estimation accuracy compared to the original model, and the latter approach may not be feasible in practice since the training data used to train the original model is no longer available. Ideally, and as illustrated in Figure 4, we would like to reduce the multiplicative depth for any given pre-trained model 404M. θ and its depth-constrained version 406

[0057]

number

[0058] It would be preferable to convert it to and from the original training data, all without accessing the original training data, while minimizing any loss of accuracy.

[0059] With the above as background, the technology of the present disclosure will now be described. Building FHE-Friendly Machine Learning Models

[0060] The basic paradigm of the present disclosure is illustrated in FIG. 5. As illustrated, the system (sometimes referred to herein as the knowledge distillation framework) receives a teacher model 500. The teacher model is unconstrained and is trained using the original training set 502. The teacher model has high accuracy but is not compatible (friendly) for performing inference in the FHE domain. The goal is to then build an FHE-friendly student model 504 that has compatible accuracy to the teacher model, i.e., that mimics the decision boundaries of the more complex teacher model while still allowing inference on encrypted data. In this manner, the student model 504 serves as a proxy for the teacher model 500 with minimal loss in accuracy.

[0061] In accordance with this disclosure, a student model 504 is constructed as follows: First, a transfer data set 507 is generated by a transfer set generator 506. The transfer data set 507 is preferably generated randomly using a given input data distribution associated with the teacher model. It is not necessary that the given input data distribution have any associated labels. Additionally, the input data distribution is also used to randomly generate a validation data set (not shown), which is used to facilitate selecting a student model from a pool of student models compatible with a set of FHE constraints, as described below. These constraints may vary, but typically include a target multiplier circuit depth, security level, cipher size, etc. The multiplier circuit depth is, for example, the maximum number of consecutive multiplication operations required for a computation. The security level captures how hard it is to break, and this parameter may depend on the parameters of the underlying encryption scheme and the parameters of the computation depth close in time; the higher the security level, the longer the estimation time, and the lower the computation depth translates to higher security for the same ciphertext parameters. The size of the ciphertext involves choosing hyperparameters of the ML model (e.g., number of hidden nodes) to best utilize the ciphertext slots. Other possible FHE constraints (including combinations of the above) can be considered.

[0062] Referring back to FIG. 5, in this approach, preferably the architecture of the Student Model can be selected from an FHE-friendly premise space (i.e., the set of all models that satisfy a given FHE constraint). Thereafter, and as shown, the parameters of the Student Model 504 are learned using the Transfer Data Set 507 as “synthetic” training data and the predictions of the Teacher Model 500 on the Transfer Data Set. Any supervised training algorithm (e.g., steepest descent) can be used for this purpose. One or more “candidate” Student Models can then be trained in this manner, i.e., using the predictions of the Teacher Model on the Transfer Data Set 507. During this process, the fitness of each candidate Student Model is preferably also calculated using the Validation Data Set and the Teacher Model’s predictions on the Validation Data Set. Any standard fitness (goodness) criterion (e.g., accuracy) can be used for this purpose. The loss estimator 508 evaluates the fitness calculations and selects the best-performing Student model as the FHE-friendly model. In this approach, the Student model tries to minimize the loss between the teacher and Student model predictions on the transfer data set, but the Transfer Set Generator 506 is also configured to generate more challenging input samples that maximize this loss.

[0063] In summary, and as illustrated in FIG. 5, the approach herein provides techniques for transferring knowledge embedded in a pre-trained first (teacher) model to a second depth-constrained (student) model without access to the original training model used to train the first model.

[0064] Referring now to FIG. 7, a search for FHE-friendly Student architectures can be used in the above process for selecting a Student model. This assumes a mechanism for evaluating the FHE-friendliness of the selected architecture. As shown, the Student model 700 can be a multi-layer feed-forward neural network. As shown, the hyper-parameter search space (from which the Student model is constructed) includes a number of layers, a number of hidden nodes in each layer, activation functions at each layer, etc. Depth-constrained knowledge distillation for HE estimation of decision tree ensembles can then proceed as follows: For a given multiplicative depth budget, compute a set of neural network configurations with polynomial activation functions that satisfy the depth budget. Then, generate augmented data from the given unlabeled data (e.g., a small fraction of the original data) as previously described, and generate labels using the ensemble. Then, search for the most likely neural network in terms of validation score. This operation can be performed in the unencrypted space. Then, HE estimation is performed on the best network as a proxy for the original ensemble.

[0065] The above approach preferably minimizes accuracy loss by searching for the best student model that operates within a given depth constraint and then training this student model using only synthetic data preferably sampled from the original training data distribution.

[0066] In FHE-based computation, there are several parameters that govern the computational complexity. As mentioned above, these include multiplication depth, selection of security parameters, selection of number of ciphertexts, and others. To minimize the use of bootstrapping techniques to reduce noise (because bootstrapping is extremely expensive), the techniques herein build ML estimation models that are accurate and close to the performance of the original complex model. To facilitate learning of the student model, the framework assumes that the original training set used to train the complex model is not available. Instead, the approach herein preferably generates artificial training feature data and then generates labels for the features to be trained using the original complex model. A shallow network is trained with computations that understand the FHW, making it easier to map simpler models to the FHE scheme.

[0067] FIG. 6 illustrates a technique implemented by the transfer set generator to generate the transfer data set. In a first embodiment, this is a static approach, where a small set of unlabeled data is input to a non-parametric density prediction function 600, although this is not required. The resulting distribution of input data is randomly sampled by a random sampling function 602 to generate the transfer data set. In an alternative embodiment, this is a dynamic approach, where a small set of unlabeled data is input to the non-parametric density prediction function 600, along with the resulting distribution of input data applied to an adaptive sampling function 604, which also receives as input the predicted deviation between the teacher and student models in previous samples. The adaptive sampling function 604 uses these inputs to generate the transfer data set.

[0068] Although not shown, random or adaptive sampling is also used to generate a validation data set, which is used to facilitate selection of the best-fitting candidate Student model.

[0069] The following sections provide more detailed descriptions of implementations of the techniques described above. Depth-Constrained Knowledge Distillation for the Estimation of Cryptographic Data

[0070] As referenced above, the techniques herein apply a knowledge distillation framework with the goal of obtaining a (multiplicative) depth-constrained Student model that mimics the decision boundary of a complex Teacher model while allowing cryptographic estimation. Also, as described, in one implementation, the framework provides an algorithm to search for the best Student model that can be trained accurately even if the complete training set (used to train the original model) is not available.

[0071] In particular, the present disclosure relates to depth-constrained machine learning models.

[0072]

number

[0073] The present invention provides a technique for constructing a model that is equivalent to (or more closely mimics) the decision behavior of a given pre-trained model (M).

[0074]

number

[0075] The trained models do not need to belong to the same family of ML models. Here, depth refers to the multiplicative depth of the model, which is defined as: Definition:(Multiplicative depth of machine learning models)

[0076] The depth, denoted by φ(M), of a multiplication circuit of a machine learning model M is the maximum number of multiplication operations (gates) required along any circuit path of it, where M:x→y is a pre-trained model, where x and y represent the input and output spaces, respectively.

[0077]

number

[0078] Let be the training data used to learn the parameters θ of the model M, where

[0079]

number

[0080] is the number of training samples, and p x Let L denote the distribution of the training data, M and an approximate version of L,

[0081]

number

[0082] The problem of learning a depth-constrained model can be formulated in Equation (1),

[0083]

number

[0084]

number

[0085] Here, HΩ is the depth-constrained premise space, i.e., the set of all models whose multiplicative depth is less than or equal to Ω.

[0086] Since it is often not practical to directly predict the expected distillation loss in the above formula, it can be replaced by an empirical risk. If the original training data

[0087]

number

[0088] If is available, the empirical risk minimization problem can be written as equation (2).

[0089]

number

[0090]

number

[0091] The approach presented herein is based on the idea that, in the usual case, the original training data

[0092]

number

[0093] We assume that M is no longer available (or is not available for training). In the context of cryptographic estimation, this is typically a legitimate assumption since models are typically trained using sensitive personal information that cannot be retained indefinitely. This assumption is valid for a given pre-trained model M θNote that we ignore the possibility of “throwing away” the original training data and retraining a depth-constrained model from scratch. As described, the techniques herein overcome the unavailability of the original training data using a knowledge distillation framework. As mentioned above, knowledge distillation refers to the process of learning a student model that closely mimics the predictions of a teacher model, which is typically achieved by allowing the teacher to weakly supervise the learning of the student. In particular, for a given transfer set of input data samples (without true labels), the predictions made by the teacher on this transfer set are used as proxy labels to train the student model instead of the true labels.

[0094] The following is M θ from,

[0095]

number

[0096] In one non-limiting embodiment, the input to the algorithm is a pre-trained model M θ where the distribution of input data is px, the multiplicative depth budget is Ω, the distillation loss function is L, and the size of the transfer set is

[0097]

number

[0098] and the size of the validation set is v. The output is a depth-constrained model

[0099]

number

[0100] This means that M θ closely approximates, and its validation score is,ν. In step (1), the transfer set,

[0101]

number

[0102] and the validation set,

[0103]

number

[0104] where:

[0105]

number

[0106] In step (2), the model architecture

[0107]

number

[0108] In the premise space H Ω In step (3),

[0109]

number

[0110]

number

[0111] The students are trained as:

[0112] In step (4), the validation score ν is the accuracy of the candidate model,

[0113]

number

[0114] As a validation set,

[0115]

number

[0116] Generation of the transfer set

[0117] The first aspect of knowledge distillation is how to obtain the transfer set of unlabeled input samples required for distillation. In one known approach, the original training data (minus the output labels) is used as the transfer set. An alternative is to make use of synthetically generated pseudo training samples that can be generated at the time of distillation. Since the teacher model has been trained only on the px manifold of training data, its knowledge does not necessarily cover the input space completely. As a result, if random inputs that are not in the manifold px are fed to the teacher model, the resulting predictions cannot be trusted enough to train the student model. Therefore, some knowledge of the manifold of input data may be required for the generation of the synthetic transfer set. More recently, there have been attempts to perform knowledge distillation without access to any training data. These methods, however, assume that the teacher model is a deep learning neural network with a final softmax layer capable of generating a label distribution as output. Here, the service provider needs to know a minimum of information p about the distribution of the training data. x Assume that the following remains:

[0118] Additionally, and in one example implementation herein (see, e.g., Figure 6), an algorithm (e.g., a modified version of the MUNGE algorithm) that obtains a non-parametric density of px(x) from a small subset of the training data (unlabeled) is used to generate a large unlabeled set that can then be used for distillation.

[0119] In this approach, a synthetic dataset

[0120]

number

[0121] p x In part, this means that the Student model is typically generated using knowledge of the premise space H Ω and a synthetic dataset,

[0122]

number

[0123] As a variant approach, when the distillation loss is low, a transfer set generator can be used to generate more challenging input samples to differentiate between the predictions of the teacher and student models. Depth-constrained student architecture search

[0124] The second aspect in knowledge distillation is the appropriate choice of the architecture of the student network. Typically, the architecture of the student network is fixed based on the computational resources and memory requirements of the target environment in which the students are expected to be deployed. The approach herein designs a student network with a limited multiplicative depth. The depth constraint complicates the search for the optimal student model in the following ways: Suppose a multi-layer perceptron with a sigmoid activation function is selected as the student model. Given a limited multiplicative depth (called Ω), it is not clear whether the available depth should be earmarked to better approximate the sigmoid function (using a so-called higher order polynomial) or to add more hidden layers to the network. In the approach herein, a preferably shallow neural network is used as the architecture of the student model. This is motivated by the idea that a fully connected sigmoid neural network with a single hidden layer can universally approximate any continuous univariate function. It has also been shown that this robustness can be extended to any non-polynomial function as activation function. Since the goal of the present technique is to approximate the decision function learned by the teacher model, the neural network is an approximate choice for the Student model. That is, since there may be other constraints or limitations (e.g., using only polynomial approximations of nonlinear activation functions such as sigmoid and ReLU), preferably more hidden layers are added to the network to increase the modeling capability. To determine the best Student model architecture, steps (2) to (4) of the algorithm described above are preferably performed.

[0125]

number

[0126] The selection of α is repeated for many iterations and the selection of the Student model with the highest validation score is performed. Application of decision tree ensembles to homomorphic estimation

[0127] As one example application of the depth-constrained knowledge distillation framework herein, the following describes how to perform encryption inference on a large ensemble of decision tree classifiers. Rather than recent efforts to transfer knowledge from complex models (neural networks and boosted trees) to a single decision tree to enhance the interpretable nature of the decision tree, the technique herein distills knowledge from a tree ensemble to a depth-constrained neural network (referred to as a DTNet-like) to facilitate inference (homomorphically) of encrypted data. In particular, the disclosed technique preferably replaces decision trees with comparators (in the original model) with a simpler and shallower neural network (DTNet). In this replacement, preferably the standard activation function (in the original model) is replaced with a simpler activation function (from the FHE-transparent one).

[0128] The template structure of the DTNet neural network is described here: The input layer contains d nodes for input data with d features. The hidden layer is a dense (fully connected) layer with a specific number of nodes (neurons) and a specific polynomial activation function. The output layer is a softmax layer with c nodes, where c is the number of categories in the classification task. When performing homomorphic estimation, the final softmax calculation can be sent to the client.

[0129] DTNet has a low computational depth, which allows efficient and accurate homomorphic estimation to be performed on these models. Indeed, the inference predictions on DTNet in the encrypted space are virtually identical to those in the non-encrypted space. Moreover, even when the amount of available training data is very low, DTNet can very closely approximate the inference on an ensemble of decision trees. Thus, and generalizing, DTNet can be used to effectively approximate homomorphic estimation on an ensemble of trees.

[0130] The technique of the present disclosure provides significant advantages. It provides a systematic approach for performing multiplicative depth-constrained knowledge distillation that enables efficient cryptographic inference. In one embodiment, the approach enables performing highly efficient cryptographic inference on a complex ensemble of decision trees. For example, the approach herein provides a way to perform cryptographic inference on a complex ensemble of decision trees (e.g., hundreds of base estimates), achieving roughly three (3) orders of magnitude improvement in speed over standard approaches that directly replace hard comparisons in each decision tree with soft comparisons. The system is highly scalable in multi-core cloud environments, and the approach can be implemented on several real-world public databases.

[0131] There are no limitations on possible use cases, as the technology can be used to support any cloud-based machine learning service that does not need to comply with privacy regulations or other security requirements. Many new industrial applications can use this type of cloud service and thus provide improved privacy assurance. Two examples are healthcare and finance, where client privacy compliance is paramount. As mentioned, the three (3) basic components in a Machine Learning as a Service scenario are the client, the server, and the communication infrastructure. Since the data is encrypted and the results are returned encrypted, any attacks on the server or the communication channel are effectively thwarted.

[0132] As described, a computing system implementing the present approach is typically implemented in software, e.g., as a set of computer program instructions executed by one or more hardware processors. A particular tool or component within the system may include any number of programs, processes, threads of execution, etc., along with appropriate interfaces and databases supporting data used or generated by the tool or component. A tool or component may be configured or managed via a command line, a web-based front end, or the like. A tool or component may include one or more functions that are programmatically implemented or that interoperate with other computing entities or software systems via an application programming interface (API) or any convenient request-response protocol.

[0133] Any reference in this specification to one or more commercially available products is exemplary and should not be taken as a limitation of the disclosed technology, which can be implemented on any system, device, appliance (or more generally, machine) having the general characteristics and operational functions that have been described.

[0134] As mentioned, one preferred implementation of the subject matter is as-a-service, but this is not a limitation either. The HE-based estimation can be performed entirely on-premise or in a stand-alone operating environment. As previously mentioned, and without limitation, the subject matter can be implemented within or in connection with a cloud deployment platform system or appliance, or using any other type of deployment system, product, device, program, or process. As described, the model building or estimation system can be provided functionally as a stand-alone feature, or it can augment functionality from other products and services.

[0135] Exemplary cloud application platforms on which the technology may be implemented include, but are not limited to, any cloud-supported application framework, product, or service.

[0136] In generalization, the techniques herein can be implemented as management solutions, services, products, appliances, devices, processes, programs, execution threads, etc. Typically, the techniques are implemented in software as one or more computer programs executing on hardware processing elements in conjunction with data stored in one or more data sources such as databases of interest. Some or all of the processing steps described can be automated and operate autonomously in conjunction with other systems. The automation can be in whole or in part, and the operation (in whole or in part) can be synchronous or asynchronous, demand-based or otherwise.

[0137] Each of these above-mentioned components is typically implemented as software, i.e., as a set of computer program instructions executed in one or more hardware processors. Although the components are shown separately, this is not required as the components can also be integrated with each other in whole or in part. One or more components can be implemented in a dedicated location or remotely from each other. One or more components can have subcomponents that execute with each other to provide functionality. There is no requirement for the specific functionality of the generator service to be performed by a particular component as named above, as the functionality herein (or any aspect thereof) is implemented in another or in a system.

[0138] The tools and response functions may interact or interoperate with security analysis systems or services.

[0139] As described, the functionality described above may be implemented as a stand-alone approach, for example, as one or more software-based functions executed by one or more hardware processors, or it may be available as a managed service (including as a web service via a SOAP / XML interface). The specific hardware and software implementation details described herein are for illustrative purposes only and are not meant to limit the scope of the described subject matter.

[0140] More generally, within the context of the disclosed subject matter, a computing device is a data processing system (as shown in FIG. 2), each of which includes hardware and software, and these entities communicate with each other over a network, such as the Internet, an intranet, an extranet, a private network, or any other communication medium or link. Applications on the data processing system provide native support for the Web and other known services and protocols, including, but not limited to, HTTP, FTP, SMTP, SOAP, XML, WSDL, UDDI, and WSFL, among others. Information regarding SOAP, WSDL, UDDI, and WSFL is available from the World Wide Web Consortium (W3C), which is responsible for developing and maintaining these standards; further information regarding HTTP, FTP, SMTP, and XML is available from the Internet Engineering Task Force (IETF).

[0141] As mentioned, and in addition to cloud-based environments, the techniques described herein can be implemented within or in combination with a variety of server-side architectures, including simple n-tier architectures, web portals, federated systems, and the like.

[0142] Still further, and more generally, the subject matter described herein may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment including both hardware and software. In a preferred embodiment, the sensitive data detection service (or any component thereof) is implemented in software, which software includes, but is not limited to, firmware, persistent software, microcode, and the like. Furthermore, the download and delete interface and functions may take the form of a computer usable or computer readable medium that provides program code for use by or in connection with a computer or any instruction execution system. For purposes of this description, a computer usable or computer readable medium may be any apparatus that contains or can store a program for use by or in combination with an instruction execution system, apparatus, or device. The medium may be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device). Examples of computer readable media include semiconductor or solid state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), rigid magnetic disks, and optical disks. Current examples of optical disks include compact disk-read only memory (CD-ROM), compact disk-read / write (CD-R / W), and DVD. A computer-readable medium may be a tangible or intangible item.

[0143] A computer program product may be a product having program instructions (or program code) for implementing one or more of the described functions. These instructions or code may be stored in a computer readable recording medium at a data processing system after being downloaded over a network from a remote data processing system. Alternatively, these instructions or code may be stored in a computer readable recording medium in a server data processing system and adapted to be downloaded over a network to a remote data processing system for use in the computer readable recording medium in the remote system.

[0144] In an exemplary embodiment, the technology is implemented in a special purpose computing platform, preferably in software executed by one or more processors. The software may be implemented as one or more computer programs, with the software being held in one or more data stores or memories associated with the one or more processors. Collectively, this special purpose hardware and software includes the functionality described above.

[0145] Although the above describes a particular order of operations performed by certain embodiments of the present invention, it should be understood that such orders are exemplary, as alternative embodiments may perform operations in different orders, certain combinations of operations, superposition of certain operations, etc. Reference herein to a given embodiment indicates that the described embodiment may include a particular feature, structure, or characteristic, but all embodiments may not require the particular feature, structure, or characteristic.

[0146] Finally, although certain components of the system have been described separately, those skilled in the art will recognize that some functionality may be combined or shared within a given instruction, program sequence, code portion, etc.

[0147] Moreover, FHE is just an exemplary encryption protocol, which is not intended to be limiting.

[0148] The approach described above implements FHE-based estimation in a client-server context, but this is not a limitation. The techniques described above can also be implemented within a client that has access to the tree.

[0149] Furthermore, while the estimation preferably occurs using FHE protocols, the approach herein could be augmented with other multi-party secure computation techniques, and the privacy of the test points, or both, desired to be protected.

[0150] The technology herein provides improvements to another technology or technology area, namely, the computational efficiency of FHE systems and methods, as well as implementations of FHE-based estimation tools and systems, and cloud-based systems that incorporate or receive such technology.

[0151] The particular use case or application for which the framework should be used does not limit this disclosure.

[0152] Although the approach herein contemplates the server performing one or more of the described operations prior to receipt of the FHE-encrypted data points, this pre-processing is not required as a requirement. An alternative approach is for the server to perform all or some of this processing on-demand (i.e., dynamically) in response to receipt of a client query.

[0153] Having described the subject matter, what is claimed is:

Claims

1. 1. A computer-implemented method for generating a fully homomorphic encryption (FHE) compatible machine learning model, comprising: Obtaining a first machine learning model pre-trained using the first training data; receiving a distribution of input data and a set of FHE constraints; and and constructing a second machine learning model that satisfies the set of FHE constraints, the second machine learning model comprising: generating a transfer data set using the distribution of input data; and learning parameters of the second machine learning model using predictions of the first machine learning model on the transfer dataset as synthetic training data. A method constructed by

2. The method of claim 1 , wherein parameters of the second machine learning model are learned without access to the first training data.

3. Constructing the second machine learning model further comprises: generating a validation data set using the distribution of input data; For each set of candidate second machine learning models, determining a fitness of the second machine learning model using predictions of the first machine learning model on the validation data set as the combined validation data; and selecting the candidate second machine learning model having the best fit as the second machine learning model; The method of claim 1 , comprising:

4. 2. The method of claim 1, wherein the set of FHE constraints comprises one of a target multiplier depth, a security level, a ciphertext size, and combinations thereof.

5. 10. The method of claim 1, further comprising using the second machine learning model for privacy-preserving homomorphic inference on encrypted data.

6. The method of claim 5 , wherein the second machine learning model is used in place of the first machine learning model.

7. The method of claim 1 , wherein the second machine learning model comprises a shallow neural network with an FHE-compatible activation function.

8. An apparatus comprising: A processor; a computer memory holding a computer program for execution by the processor, the computer program instructions configured to generate a fully homomorphic encryption (FHE) compatible machine learning model, the processor comprising: Obtaining a first machine learning model pre-trained using the first training data; receiving a distribution of input data and a set of FHE constraints; and and constructing a second machine learning model that satisfies the set of FHE constraints, the second machine learning model comprising: generating a transfer data set using the distribution of input data; and learning parameters of the second machine learning model using predictions of the first machine learning model on the transfer dataset as synthetic training data. An apparatus for performing the above.

9. 10. The apparatus of claim 8, wherein parameters of the second machine learning model are learned without access to the first training data.

10. The processor that constructs the second machine learning model further comprises: generating a validation data set using the distribution of input data; For each set of candidate second machine learning models, determining a fitness of the second machine learning model using a prediction of the first machine learning model on the validation data set as the combined validation data; and The apparatus of claim 8 , further comprising: selecting a candidate second machine learning model having a best fit as the second machine learning model.

11. 9. The apparatus of claim 8, wherein the set of FHE constraints comprises one of a target multiplier circuit depth, a security level, a ciphertext size, and combinations thereof.

12. 10. The apparatus of claim 8, wherein the processor further comprises executing the second machine learning model for use in privacy-preserving homomorphic inference on encrypted data.

13. The apparatus of claim 12 , wherein the second machine learning model is used in place of the first machine learning model.

14. 10. The apparatus of claim 8, wherein the second machine learning model comprises a shallow neural network with an FHE-compatible activation function.

15. A computer program executed by a data processing system for generating a machine learning model compatible with Fully Homomorphic Encryption (FHE), the computer program comprising: Obtaining a first machine learning model pre-trained using the first training data; receiving a distribution of input data and a set of FHE constraints; and constructing a second machine learning model that satisfies the set of FHE constraints, the second machine learning model comprising: generating a transfer data set using the distribution of input data; and Training parameters of the second machine learning model using predictions of the first machine learning model on the transfer dataset as synthetic training data. Computer program.

16. 16. The computer program product of claim 15, wherein parameters of the second machine learning model are learned without access to the first training data.

17. The computer program for constructing the second machine learning model further comprises: generating a validation data set using the distribution of input data; For each set of candidate second machine learning models, determining a fitness of the second machine learning model using a prediction of the first machine learning model on the validation data set as the combined validation data; and 16. The computer program product of claim 15, further comprising: selecting as the second machine learning model a candidate second machine learning model having a best fit.

18. 16. The computer program product of claim 15, wherein the set of FHE constraints comprises one of a target multiplier circuit depth, a security level, a ciphertext size, and combinations thereof.

19. 16. The computer program product of claim 15, further comprising causing the data processing system to use the second machine learning model for privacy-preserving homomorphic inference on encrypted data.

20. 20. The computer program product of claim 19, wherein the second machine learning model is used in place of the first machine learning model.

21. 16. The computer program product of claim 15, wherein the second machine learning model comprises a shallow neural network with an FHE-compatible activation function.

Citation Information

Patent Citations

  • Data identification device, system, program, and method capable of access control of identification request source

    JP2020010217A

  • Support vector machine learning system and support vector machine learning method

    WO2015155896A1