Differential Private Split Vertical Learning
The system addresses computational and privacy issues in distributed learning by splitting models into worker modules with differential privacy, enabling secure and efficient collaborative training across data silos with customizable privacy settings.
Patent Information
- Application Number
- JP2024526513
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-03
- Filing Date
- 2022-11-02
- Publication Date
- 2024-11-19
AI Technical Summary
Existing distributed collaborative machine learning methods, such as federated learning, face challenges with computational power requirements at client nodes, especially for deep learning models, and privacy issues due to the transfer of entire models, which can be costly and insecure.
A system that splits a machine learning model into worker modules on local data silos and a server module, applying differential privacy through noise to gradients, allowing independent privacy settings based on epsilon values for each silo, ensuring data remains localized and secure.
Enables collaborative training of deep neural networks with privacy guarantees, reducing computational load and communication costs while maintaining data security and privacy across vertically partitioned data environments with varying privacy requirements.
Smart Images

Figure 2024543029000001_ABST
Abstract
Description
[Technical field]
[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 275,011, filed November 3, 2021. Such application is incorporated herein by reference in its entirety. [Background technology]
[0002] Distributed collaborative machine learning enables machine learning across a distributed data environment of client nodes without the need to transfer unprotected data from the client nodes to a central node or server. This feature increases privacy and security for the data being analyzed. Furthermore, the parties analyzing the results of data processing at the central node never have access to the raw data residing on the client nodes, instead only the smashed data (output of the last / cut layer of the local part of the model) during the training process is transferred to the central node where the local part of the trained model is passed for inference.
[0003] One approach to distributed collaborative machine learning is federated learning. Using federated learning, a central node transfers the entire machine learning model to each of the distributed client nodes, including its own local data, and then merges all the locally trained machine learning models from each client node to form a global model at the central node. This allows for parallel model training and increases the speed of operation of the system. However, a drawback of federated learning is that each client node must run the entire machine learning model. Client nodes in some real-world applications may not have enough computing power to process the entire machine learning model, which can be particularly challenging when the machine learning model is a deep learning model. Another drawback is that transferring the entire model can be expensive in terms of communication. There are also privacy concerns in giving each of the client nodes the entire machine learning model.
[0004] An alternative to federated learning is split learning. Split learning splits the entire machine learning model into multiple smaller parts and trains them separately. Allocating only a portion of the network for training at the client nodes reduces the processing load at each client node. The communication load is also improved since only smashed data is forwarded to the central node. This also improves privacy by preventing client nodes from having access to all machine learning models known to the central node or server.
[0005] Differential privacy is a method of protecting data privacy based on the principle that privacy is a property of a computation on a database or silo, as opposed to the syntactic quality of the database itself. Essentially, a computation is considered differentially private if it produces nearly identical results when applied to two databases that differ only by the presence or absence of a single record in the data. Differential privacy is powerful because of the mathematical, quantifiable guarantees it gives regarding the re-identifiability of the underlying data. Differential privacy differs from previous approaches because of its ability to quantify the mathematical risk of de-identification using an epsilon value that measures the privacy "cost" of a query. Differential privacy allows for recording the cumulative privacy risk for a data set across multiple analyses and queries.
[0006] A vertically partitioned decentralized data setting is one in which various databases or silos hold several different columns of data about the same individual or entity. Data silo owners may wish to collaborate and use the decentralized data to train machine learning models or deep neural networks to predict or classify some outcome under the constraint that the original data cannot be disclosed or exported from its original source. Furthermore, collaborating silos may have different risk tolerances to the privacy constraints of the contributing data silos. It would therefore be desirable to develop a system for applying machine learning models to a vertically partitioned decentralized data network to maintain privacy using differential privacy techniques while also allowing for different solutions afforded by machine learning processes.
[0007] Research papers on differential privacy and split learning include: Dwork, Cynthia, "Differential privacy: A survey of results", In International conference on theory and applications of models of computation, Springer, Berlin, Heidelberg, 2008, pp. 1-19 Abadi, Martin, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang, "Deep learning with differential privacy", In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, 2016, pp. 308-318 Abuadbba, S., Kim, K., Kim, M., Thapa, C., Camtepe, S. A., Gao, Y., Kim, H., and Nepal, S. (2020), "Can We Use Split Learning on 1D CNN Models for Privacy Preserving Training?", Proceedings of the 15th ACM Asia Conference on Computer and Communications Security, ASIA CCS 2020, pp. 305-318 https: / / doi.org / 10.1145 / 3320269.3384740 Thapa, C., Chamikara, M. A. P., and Camtepe, S. A., (2020) "Advancements of federated learning towards privacy preservation: from federated learning to split learning" Studies in Computational Intelligence, 965, pp. 79-109 https: / / arxiv.org / abs / 2011.14818v1 Thapa, C., Chamikara, MAP, Camtepe, S., and Sun, L. (2020) "SplitFed: When Federated Learning Meets Split Learning" https: / / arxiv.org / abs / 2004.12088v3 Li, O., Sun, J., Yang, X., Gao, W., Zhang, H., Xie, J., Smith, V., and Wang, C. (2021), "Label Leakage and Protection in Two-party Split Learning" http: / / arxiv.org / abs / 2102.08504 Vepakomma, Praneeth, Tristan Swedish, Ramesh Raskar, Otkrist Gupta, and Abhimanyu Dubey, “No peek: A survey of private distributed deep learning” arXiv preprint arXiv:1812.03288 (2018) The references cited in this background section are not admitted to be prior art to the present invention. [Prior art documents] [Non-patent literature]
[0008] [Non-Patent Document 1] Dwork, Cynthia, "Differential privacy: A survey of results," In International conference on theory and applications of models of computation, Springer, Berlin, Heidelberg, 2008, pp. 1-19. [Non-Patent Document 2] Abadi, Martin, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang, "Deep learning with differential privacy," In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, 2016, pp. 308-318. [Non-Patent Document 3] Abuadbba, S., Kim, K., Kim, M., Thapa, C., Camtepe, S.A., Gao, Y., Kim, H., and Nepal, S. (2020), "Can We Use Split Learning on 1D CNN Models for Privacy Preserving Training?", Proceedings of the 15th ACM Asia Conference on Computer and Communications Security, ASIA CCS 2020, pp. 305-318. https: / / doi.org / 10.1145 / 3320269.3384740 [Non-Patent Document 4] Thapa, C., Chamikara, MAP, and Camtepe, SA, (2020) "Advancements of federated learning towards privacy preservation: from federated learning to split learning" Studies in Computational Intelligence, 965, pp. 79-109 https: / / arxiv.org / abs / 2011.14818v1 [Non-Patent Document 5] Thapa, C., Chamikara, MAP, Camtepe, S., and Sun, L. (2020) "SplitFed: When Federated Learning Meets Split Learning" https: / / arxiv.org / abs / 2004.12088v3 [Non-Patent Document 6] Li, O., Sun, J., Yang, X., Gao, W., Zhang, H., Xie, J., Smith, V., and Wang, C. (2021), "Label Leakage and Protection in Two-party Split Learning" http: / / arxiv.org / abs / 2102.08504 [Non-Patent Document 7] Vepakomma, Praneeth, Tristan Swedish, Ramesh Raskar, Otkrist Gupta, and Abhimanyu Dubey, “No peek: A survey of private distributed deep learning” arXiv preprint arXiv:1812.03288 (2018) Summary of the Invention
[0009] In the machine according to the invention, the machine learning system or deep neural network is split into several "worker" modules and a single "server" module. The worker modules are independent neural networks initialized locally on each data silo. The server network receives the last layer output (herein called "smashed data") from each worker module during training, consolidates the results, and feeds them into its own local neural network. The server then calculates the error for the current prediction or classification task and commands the submodules to update their model parameters using the gradients to reduce the observed error. This process continues until the error is reduced to an acceptable level. A parameterized level of noise is applied to the worker gradients between each training iteration, resulting in a differentially private model. Each worker may parameterize the weighting of the amount of noise applied to its local neural network module according to its independent privacy requirements. Thus, the epsilon values (a measure of privacy loss for differential changes in the data) at each worker are independent. The present invention in some embodiments therefore represents the introduction of differential privacy in vertically partitioned data environments, where different silos with independent privacy requirements hold different sets of features / columns for the same data set.
[0010] One application of the present invention in its various embodiments is to enable collaborating parties to train a single deep neural network with privacy guarantees. Due to the modular nature of the neural network topology, the trained "worker" neural network module can be used as a privacy-preserving feature generator that can be used as input to other machine learning methods. The present invention thus enables cross-organizational and line-of-business collaborative machine learning in a coordinated, constrained data environment where each silo holds a different set of features.
[0011] These and other features, objects and advantages of the present invention will be better understood from a consideration of the following detailed description of the preferred embodiment and the appended claims, taken in conjunction with the drawings described below. [Brief description of the drawings]
[0012] [Figure 1] FIG. 2 is a swim lane diagram illustrating a process according to one embodiment of the present invention. [Diagram 2] FIG. 2 is a structural diagram illustrating a system for implementing the process of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] Before describing the present invention in greater detail, it is to be understood that the invention is not limited to the particular embodiments described, and that the terminology used in describing the particular embodiments is for the purpose of describing those particular embodiments only, and is not intended to be limiting, as the scope of the present invention will be limited only by the claims.
[0014] FIG. 1 shows a method according to an embodiment of the invention. The method is implemented using the structure of the architecture diagram, FIG. 2. The method can be used to train deep neural networks in a modular manner. In this network, each module lives in a data silo, i.e., the data is vertically partitioned or "cut" into modules. The raw data is never allowed to leave its own silo, and thus data privacy and security are protected, since data is never shared with a server or coordinator. The output of each module (at the silo level) feeds the input of a server node. The raw data is kept sufficiently "far away" from the module output layer, which encodes an intermediate representation / transformation of the data.
[0015] One component type is the worker nodes 10 that maintain independent local neural networks 18, as shown in FIG. 2. Although FIG. 2 shows three worker nodes 10, any number of two or more worker nodes 10 may be used in various implementations of the invention. A worker node 10 is installed on each data silo (or client) in the collaboration. A server node 12 maintains an independent fused neural network 20 and a set of labels used during the training process, as shown in FIG. 2. The server node 12 is responsible for integrating the output from each worker module 10 and coordinating the learning process between the local neural networks 18. A third component type is an optimization module 22 on each worker node 10 that applies noise during each parameter update iteration during training. This noise introduces a differential privacy element into the network. The fourth component type is an Application Programming Interface (API) 15, which allows the user to specify columns of distributed data that should be used during the training process and returns a unified / monolithic view of the trained network modules after training.
[0016] A problem in building models using neural networks is whether we can be sure that the model does not remember the underlying data, thereby compromising privacy. Known privacy attacks, such as membership interference attacks, are performed by making queries with specific inputs and observing the outputs, which may allow an attacker to identify privacy data even without direct access to the data silos. In the network of Figure 2, it can be seen that differential privacy (DP) is applied to each data silo or client node individually. Differential privacy is applied independently to each client node, which allows for independent selection of client epsilon values. Additionally, differential privacy is also applied on the first forward pass to the cut layer (herein referred to as the “smashed” epsilon values), as described below.
[0017] The process for applying machine learning using the system of FIG. 2 is shown in the swim lane diagram of FIG. 1. As mentioned above, this method allows for the splitting of epsilon (i.e., noise level) across each of the worker nodes. In this way, the level of privacy can be set explicitly for the data in each of these nodes. This allows for the handling of individual privacy requirements specific to each data silo. Data requiring more privacy can be set with a lower epsilon value, while other data can be set with a higher epsilon value, thus improving the results of the system since it is not necessary to use the minimum epsilon value across all of the data silos.
[0018] As shown in the swim lane diagram of FIG. 1, the process begins in step A where appropriate worker nodes 10 are set up in each data silo, servers are set up, and batch sizes for the neural networks are determined. Training in the training node 14 begins in step B using random batches of input data. The worker node 10 trains its vertical split of the input batch through one or more layers of the local neural network 18 in step D. This continues until a layer, referred to herein as the cut layer, is reached in step E. In step E, an optimization routine 22 adds a desired level of noise to the resulting smashed data. This is done only the first time a given batch is fed forward. The noise level is set independently in each of the worker nodes 10 so that the noise level can be configured for the desired level of privacy for this worker node's 10 particular vertical slice of the data set. The local process in the worker node then ends. The worker nodes 12 then send the smashed data with additive noise to the trainer 14 in step F. After the trainer 14 collects all of the noised smashed data from all of the worker nodes 10, it sends them to the trainer 14 in step G. The server 12 runs training (a forward pass in step G and a backpropagation in step H) on its own local neural network 20 using the labels for the input batch being processed. The server node 12 sends its backpropagation output (the smashed gradients) back to the trainer node 14 in step J. The trainer node 14 forwards the smashed gradients to all the worker nodes 10 in step K, and each worker node runs its own local backpropagation to get its local gradients.In step L, each of the worker nodes 10 applies differentially private noise to the local gradients and uses the resulting noisy smashed gradients to update its own local neural network weights. This process is repeated over batches of input data until network training is complete, based on which the error has reached an acceptable level.
[0019] A potential problem in this type of system is data leakage when applying noise only in the backpropagation stage, as shown in Figure 1. An attacker could attempt to infer client model parameters and recover the original input data by leaking to the server node 12 and back to the client. The solution shown in Figure 1 is to use an optimization module 22 to add an amount of noise to the cut layer (herein referred to as the "smashed" data) that is output during the first training epoch.
[0020] Traditional split neural networks are susceptible to model inversion attacks as described above. To reduce leakage, the distance correlation between the model inputs and the cut layer activations (i.e., raw and smashed data) can be minimized. This means that the raw and smashed data are maximally different. This technique has been shown to have negligible, if any, impact on model accuracy.
[0021] The systems and methods described herein may be implemented by any combination of hardware and software in various embodiments. For example, in one embodiment, the systems and methods may be implemented by a computer system or collection of computer systems, each of which includes one or more processors that execute program instructions stored on a computer-readable storage medium coupled to the processors. The program instructions may implement the functions described herein. The various systems and displays illustrated in the figures and described herein represent example implementations. The order of any method may be changed and various elements may be added, modified, or omitted.
[0022] The computing system or computing device described herein may implement the hardware portion of a cloud computing system or a non-cloud computing system as forming part of various implementations of the present invention. The computing system may be any of a variety of types of devices, including, but not limited to, a commodity server, a personal computer system, a desktop computer, a laptop or notebook computer, a mainframe computer system, a handheld computer, a workstation, a network computer, a consumer device, an application server, a storage device, a telephone, a mobile phone, or generally any type of computational node, compute node, compute device, and / or computing device. The computing system includes one or more processors (any of which may include multiple processing cores, which may be single-threaded or multi-threaded) coupled to a system memory via an input / output (I / O) interface. The computing system may further include a network interface coupled to the I / O interface.
[0023] In various embodiments, the computer system may be a single processor system including one processor, or a multiprocessor system including multiple processors. The processors may be any suitable processor capable of executing computational instructions. For example, in various embodiments, they may be general purpose processors or embedded processors implementing any of a variety of instruction set architectures. In a multiprocessor system, each of the processors may generally, but not necessarily, implement the same instruction set. The computer system also includes one or more network communication devices (e.g., network interfaces) for communicating with other systems and / or components over a communications network, such as a local area network, a wide area network, or the Internet. For example, a client application running on the computing device may use the network interface to communicate with a server application running on a single server or on a cluster of servers implementing one or more of the components of the system described herein in a cloud computing or non-cloud computing environment implemented in various subsystems. In another example, an instance of a server application running on a computer system may use a network interface to communicate with other instances of the application that may be implemented on other computer systems.
[0024] The computing device also includes one or more persistent storage devices and / or one or more I / O devices. In various embodiments, the persistent storage devices may correspond to disk drives, tape drives, solid state memory, other mass storage devices, or any other persistent storage devices. A computer system (or a distributed application or operating system running thereon) may store instructions and / or data in the persistent storage devices as desired and may retrieve the stored instructions and / or data as needed. For example, in some embodiments, a computer system may implement one or more nodes of a control plane or control system, and the persistent storage may include SSDs attached to the server nodes. Multiple computer systems may share the same persistent storage device or may share a pool of persistent storage devices, where the devices in the pool represent the same or different storage technologies.
[0025] A computer system includes one or more system memories that may store code / instructions and data accessible by a processor. The memory capabilities of a system may include multiple levels of memory and memory caches, for example in systems designed to swap information in memory based on access speed. Interleaving and swapping may be extended to persistent storage in virtual memory implementations. The technology used to implement memory may include, by way of example, static random access memory (RAM), dynamic RAM, read-only memory (ROM), non-volatile memory, or flash type memory. As with persistent storage, multiple computer systems may share the same system memory or may share a pool of system memory. The system memory or memories may include program instructions executable by the processor to implement the routines described herein. In various embodiments, the program instructions may be coded in binary, assembly language, any interpreted language such as Java, compiled languages such as C / C++, or any combination thereof, the specific languages listed herein being merely illustrative. In some embodiments, the program instructions may implement multiple separate client, server nodes, and / or other components.
[0026] In some implementations, the program instructions may include instructions executable to implement an operating system (not shown), which may be any of a variety of operating systems, such as UNIX, LINUX, Solaris, MacOS, or Microsoft Windows. Any or all of the program instructions may be provided as a computer program product or software, which may include a non-transitory computer-readable storage medium having stored thereon instructions that may be used to program a computer system (or other electronic device) to perform a process according to various implementations. A non-transitory computer-readable storage medium may include any mechanism for storing information in a form readable (e.g., software, processing application) by a machine (e.g., a computer). In general, a non-transitory computer-accessible medium may include a computer-readable storage medium or memory medium, such as a magnetic or optical medium, e.g., a disk or DVD / CD-ROM, coupled to a computer system via an I / O interface. A non-transitory computer-readable storage medium may also include any volatile or non-volatile medium, such as RAM or ROM, that may be included in some embodiments of a computer system as a system memory or another type of memory. In other implementations, the program instructions may be communicated using optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.) conveyed over a communications medium, such as a wired or wireless link, such as may be implemented over a network and / or a network interface. The network interface may be used to interface with other devices, which may include other computer systems or any type of external electronic device.Generally, system memory, persistent storage, and / or remote storage accessible on other devices over a network may store data blocks, replicas of data blocks, metadata related to the data blocks and / or their state, database configuration information, and / or other information usable in implementing the routines described herein.
[0027] In some implementations, the I / O interface may coordinate I / O traffic between the processor, system memory, and any peripheral devices in the system, including through a network interface or other peripheral interface. In some embodiments, the I / O interface may perform any necessary protocol, timing, or other data conversions to convert data signals from one component (e.g., system memory) into a format suitable for use by another component (e.g., processor). In some embodiments, the I / O interface may include support for devices attached through various types of peripheral buses, such as, for example, variations on the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard. Also, in some embodiments, some or all of the functionality of the I / O interface, such as the interface to the system memory, may be incorporated directly into the processor.
[0028] The network interface may enable data to be exchanged between the computer system and other devices attached to the network, such as, for example, other computer systems (which may implement one or more storage system server nodes, primary nodes, read-only nodes, and / or clients of the database system described herein). Additionally, the I / O interface may enable communication between the computer system and various I / O devices and / or remote storage. The I / O devices may, in some embodiments, include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other devices suitable for inputting or retrieving data by one or more computer systems. These may be directly connected to a particular computer system or may generally be connected to multiple computer systems in a cloud computing environment, grid computing environment, or other system involving multiple computer systems. Multiple I / O devices may be in communication with the computer system or may be distributed on various nodes of a distributed system that includes the computer system. The user interfaces described herein may be presented to a user using various types of display screens, which may include CRT displays, LCD displays, LED displays, and other display technologies. In some implementations, input may be received through a display using touch screen technology, and in other implementations, input may be received through a keyboard, mouse, touchpad, or other input technology, or any combination of these technologies.
[0029] In some embodiments, similar input / output devices may be separate from the computer system and may interact with one or more nodes of a distributed system that includes the computer system through a wired or wireless connection, such as via a network interface. The network interface may generally support one or more wireless networking protocols (e.g., Wi-Fi / IEEE 802.11, or another wireless network standard). The network interface may support communication over any suitable wired or wireless general data network, such as, for example, other types of Ethernet networks. Additionally, the network interface may support communication over a telecommunications / telephone network, such as an analog voice network or a digital fiber communications network, over a storage area network, such as a Fibre Channel SAN, or over any other suitable type of network and / or protocol.
[0030] Any of the distributed system embodiments described herein, or any of their components, may be implemented as one or more network-based services in a cloud computing environment. For example, read-write and / or read-only nodes in a database tier of a database system may present database services and / or other types of data storage services employing the distributed storage system described herein to clients as network-based services. In some embodiments, the network-based services may be implemented by software and / or hardware systems designed to support interoperable machine-to-machine interaction over a network. A web service may have an interface described in a machine-processable format, such as Web Services Description Language (WSDL). Other systems may interact with the network-based service in a manner defined by the description of the network-based service's interface. For example, the network-based service may define various operations that other systems may invoke and may define specific application programming interfaces (APIs) that other systems may be expected to follow when requesting various operations.
[0031] In various embodiments, a network-based service may be requested or invoked through the use of a message that includes parameters and / or data associated with the network-based service request. Such a message may be formatted according to a particular markup language, such as Extensible Markup Language (XML), and / or may be encapsulated using a protocol, such as Simple Object Access Protocol (SOAP). To perform a network-based service request, a network-based service client may assemble a message that includes the request and convey the message to an addressable endpoint (e.g., a Uniform Resource Locator (URL)) that corresponds to the web service using an Internet-based application layer transport protocol, such as Hypertext Transfer Protocol (HTTP). In some embodiments, a network-based service may be implemented using Representational State Transfer (REST) techniques rather than message-based techniques. For example, a network-based service implemented according to REST techniques may be invoked through parameters included within an HTTP method, such as PUT, GET, or DELETE.
[0032] Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs. Although any methods and materials similar or equivalent to those described herein can be used in the practice or testing of the present invention, a limited number of exemplary methods and materials are described herein. It will be apparent to those skilled in the art that many more modifications are possible without departing from the inventive concepts herein.
[0033] All terms used herein should be interpreted in the broadest possible manner consistent with the context. When groupings are used herein, all individual elements of the group and all possible combinations and subcombinations of the group are intended to be included individually. When a range is stated herein, the range is intended to include all subranges and individual points therein. All references cited herein are incorporated herein by reference to the extent that they are not inconsistent with the disclosure of this specification.
[0034] The present invention has been described in terms of certain preferred and alternative embodiments, which are illustrative only and not intended to limit the full scope of the invention.
Claims
1. 1. A system for differentially private split vertical learning, comprising: a server node comprising a server processor and a server neural network; a first data silo in communication with the server node over a network, the first data silo including a first subset of the data set; a second data silo in communication with the server over the network, the second data silo containing a second subset of the data set; a first worker node implemented in the first data silo, the first worker node including a first local neural network and a first optimization routine, the first worker node configured to generate a first cut layer data set from the first data subset using the first local neural network, and the first optimization routine configured to apply a first parameterized level of noise to the first cut level to generate a first smashed data set; a second worker node implemented in the second data silo, the second worker node including a second local neural network and a second optimization routine, the second worker node configured to generate a second cut layer from the second data subset using the second local neural network, the second optimization routine configured to apply a second parameterized level of noise to the second cut layer to generate a second smashed data set, the second parameterized level of noise being set independently from the first parameterized level of noise; Equipped with the server node is further configured to: combine the first and second smashed data sets; train on the first and second smashed data sets at least one additional layer in the server neural network to calculate an error on a prediction; and send a set of smashed gradients to the first and second worker nodes based on the error. A system for differentially private split vertical learning.
2. The system of claim 1 , wherein the first parameterized level of noise has a different value than the second parameterized level of noise.
3. 3. The system of claim 2, wherein the first subset of the data set comprises a first vertical split of the data set and the second subset of the data set comprises a second vertical split of the data set.
4. 4. The system of claim 3, wherein the first worker node prevents the server node from accessing the first subset of the data set, and further wherein the second worker node prevents the server node from accessing the second subset of the data set.
5. 5. The system of claim 4, further comprising an application programming interface (API) for communicating with the server node, the API configured to send a specified set of vertical columns to the server node and return a unified view of the server neural network from the server node after training.
6. the server node is further configured to backpropagate the error up to the cut layer in the server neural network, the first worker node is configured to further backpropagate past the cut layer in the first local neural network, and the second worker node is configured to further backpropagate past the cut layer in the second local neural network; The system of claim 5.
7. the server node is further configured to iteratively combine the first and second smashed data sets and calculate the error against the prediction until the error reaches an acceptable level. The system of claim 6.
8. 1. A method for differentially private split vertical learning, comprising: initializing a first worker node in a first data silo that includes a first data slice of a raw data set, the first worker node including a first local neural network that receives the first data slice as an input; and initializing a second worker node in a second data silo that includes a second data slice of the raw data set, the second worker node including a second local neural network that receives the second data slice as an input; training the first local neural network on the first data slice up to a cut layer to generate a first local neural network output, and training the second local neural network on the second data slice up to the cut layer to generate a second local neural network output; in a first optimization routine on the first worker node, applying a first noise level to the first local neural network outputs to generate a first smashed data set, and applying the first noise level to a first set of local gradients to generate a first set of weights; in a second optimization routine on the second worker node, applying a second noise level to the second local neural network outputs to generate a second smashed data set, and applying the second noise level to a second set of local gradients to generate a second set of weights, wherein the second optimization routine operates independently from the first optimization routine; merging the first smashed data set and the second smashed data set at a server node including a merged neural network; A method comprising:
9. The method of claim 8 , wherein the second noise level comprises a different value than the first noise level.
10. 10. The method of claim 9, further comprising training the combined neural network on the combined first smashed data set and the second smashed data set to calculate an error on a prediction.
11. 11. The method of claim 10, further comprising sending an updated set of parameters based on the error from the server node to the first worker node and the second worker node.
12. 12. The method of claim 11, wherein the first data slice is a first vertical slice of the data set and the second data slice is a second vertical slice of the data set that does not overlap with the first vertical slice of the data set.
13. 13. The method of claim 12, further comprising: blocking access to the first data slice by the server node at the first worker node; and blocking access to the second data slice by the server node at the second worker node.
14. 14. The method of claim 13, further comprising backpropagating the error in the unified neural network until it reaches the cut layer.
15. 15. The method of claim 14, further comprising backpropagating the error in the first local neural network starting from the cut layer, and backpropagating the error in the second local neural network starting from the cut layer.
16. 16. The method of claim 15, further comprising the steps of: backpropagating from the error in the first local neural network followed by feeding forward in the first local neural network to the cut layer; and backpropagating from the error in the second local neural network followed by feeding forward in the second local neural network to the cut layer.
17. 17. The method of claim 16, further comprising generating a subsequent first smashed data set and a subsequent second smashed data set.
18. 18. The method of claim 17, further comprising calculating a second error at the server node after receiving the subsequent first smashed data set and the subsequent second smashed data set.
19. 20. The method of claim 18, further comprising outputting the monolithic view of the unified neural network from an application programming interface (API).
20. 9. The method of claim 8, comprising: setting the first noise level based on a first privacy epsilon value for the first data slice; and setting the second noise level based on a second privacy epsilon value for the second data slice.
Citation Information
Patent Citations
Information processing system and operation method of the same
JP2020074211A
Method and apparatus for generating prediction information, electronic device, and computer-readable medium
WO2021203913A1
JP1145332026A
