Analytics platform for federated private data

Abstract

A data analytics platform provides secure access to federated data for advanced analytics and machine learning. Raw data is not exposed or moved outside of its original location, thereby providing data privacy. A coordinator in the provider cloud can communicate with runners in each client data silo. The runners ensure that raw private data is never exposed to the coordinator. A silo manager is implemented in the client data silo to remotely manage and maintain the client cloud components of the platform. In some embodiments, the platform can anonymize validated models for privacy and compliance purposes and allows users to export and deploy secure models outside of the original data location.

Description

Technical Field

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 080,333, filed on September 18, 2021. The above application is hereby incorporated by reference in its entirety.

Background Art

[0002] Information barriers limit the use of data analytics. These information barriers can take many forms. Data-privacy regulations, such as the General Data Protection Regulation (GDPR) in the European Union, restrict access to and movement of personal information. More broadly, by 2023, over 65% of the world's population will have their information handled under modern privacy regulations. Similarly, organizations may be subject to numerous data confidentiality contractual terms that restrict the use of data as a condition for gaining access to the data. The transfer of data between locally hosted environments and cloud environments also presents barriers. Various private agreements or best-practice restrictions can create barriers to the movement of data within an organization for confidentiality reasons. Also, "siloing" data between various departments within a company can limit the ability to analyze data across the organization.

[0003] A common reaction to information barriers is to centralize data. However, privacy regulations can make centralization infeasible or impossible. Even when centralization is possible, centralization can lead to an increased privacy risk by placing all data in a central repository, and an increased internal threat from analysts who are given access to all of this information. Centralization also increases the costs associated with the need for a centralized storage location and moving all of the data to a unified platform.

Summary of the Invention

[0004] The present invention relates to a federated data analytics platform for distributed and private data, which transforms, processes, and analyzes this private data in a decentralized dataset. In some embodiments, a multi-party analytics application programming interface (API) works to centralize sensitive and distributed data for fast access, agile analytics, and automated compliance. In some embodiments, the computing platform provides capabilities for securely accessing and using the data for querying, data transformation, and machine learning, while mitigating compliance risks. Computing platforms in some embodiments of this disclosure may use a combination of federated learning, federated query processing, differential privacy, and secure multi-party computation to process machine learning model training and analytical queries.

[0005] In various embodiments, raw data is neither exposed nor moved outside of its original location, thereby providing compliance with data privacy and localization laws and regulations. In some embodiments, the platform can anonymize validated models for privacy and compliance, and users can export and deploy secure models outside of the original data location.

[0006] In some embodiments of this disclosure, a secure and private gateway can be connected to a dataset, it can be exposed, and each action performed on the data can be permitted according to a custom access policy and recorded for use in audit trails and compliance reports. In some embodiments, a computing platform can differentially generate private, integrated data representing the underlying dataset. This allows data scientists and technicians to build data prep, data cleaning, and feature pipelines without ever referencing the raw data, thereby protecting privacy. In some embodiments, well-known libraries and frameworks can be used by data scientists to define machine learning models and queries. Users can engage with the platform in some embodiments by submitting simple commands using specific APIs. Requests can be broken down into an analytical graph that defines a set of federated and privacy-enhancing actions to ship to a local dataset.

[0007] These and other features, purposes, and advantages of the present invention will be better understood in consideration of the following detailed description of preferred embodiments and the appended claims, together with the drawings, as described below. [Brief explanation of the drawing]

[0008] [Figure 1] This is an architectural diagram showing an analysis platform according to one embodiment of the present invention. [Figure 2] This is an architectural diagram showing a coordinator and runner according to one embodiment of the present invention. [Figure 3] This is an architectural diagram showing a coordinator and runner in a siloed data environment according to one embodiment of the present invention. [Figure 4]This figure shows a multi-cloud computing environment according to one embodiment of the present invention. [Figure 5] This figure shows messaging and scheduling in a multi-cloud computing environment according to one embodiment of the present invention. [Modes for carrying out the invention]

[0009] Before the present invention is described in further detail, it should be understood that the scope of the present invention is limited only by the claims, and that the invention is not limited to the specific embodiments described, nor is it limited to the specific embodiments, and that the terms used in describing those specific embodiments are merely for the purpose of describing those specific embodiments and not to limit them.

[0010] Traditional machine learning models include tree-based models like random forests, neural networks, and relatively simple models like linear or logistic regression. In centralized machine learning, training data and computation data reside on the same machine. Distributed machine learning is built upon techniques developed for centralized machine learning, but its importance increases as model and data sizes grow to a point where it becomes impossible to fit the model and / or training data on a single machine. When training data is too large to fit in one place or to be efficiently read, analysts may use data-parallel distributed learning. In this case, data can be stored across many different machines, and learning occurs in a distributed manner.

[0011] Training data size is not the only issue with machine learning systems. Today, some machine learning models themselves are too large to fit on a single machine. When a neural network model is too large to fit on a single machine, one approach is to use parallel distributed model learning. In this approach, different layers of the network reside on different machines, and weights and gradients are passed between machines during the forward and backpropagation phases of model training. Ultimately, the underlying algorithms used here are the same as those used in centralized machine learning, but the distributed architecture allows for the use of massive models and datasets.

[0012] Decentralized machine learning differs from centralized and distributed machine learning in that it does not involve a centralized compute resource used to control model training. As an example, this can be understood in the context of learning across cell phones. In centralized and even distributed machine learning systems, if you want to build a machine learning model based on cell phone data, you send the data from the phone to an artificial intelligence (AI) architecture where the model is built. In decentralized learning, each phone is responsible for creating its own model based on its own data. The phone then sends that model to other phones. These phones look at the model sent to them and attempt to improve it by adding their own data to the original model. Each phone then sends this improved model to another phone. This process continues without a centralized service controlling model training.

[0013] The type of model training in decentralized machine learning is quite different from that used in decentralized learning, and therefore requires different techniques and algorithms. However, decentralized learning offers security benefits, namely that each endpoint (e.g., a phone) builds or improves the model using only its own data, so each data owner has control over how their data is used. That data never needs to be moved from its original location.

[0014] There are several challenges that arise when using decentralized machine learning if there is no single device to coordinate the process. For example, by default, devices with faster network connections will be able to send and receive models more quickly, and therefore those devices may have a greater impact on the final model state. However, the biggest problem with fully decentralized learning is scalability. A large portion of the protocol bandwidth is used to send and receive redundant messages. Nodes can become overwhelmed because there is no centralized rate limiting or job scheduling.

[0015] These limitations led to federative learning, where data is distributed across different devices or silos. Each device or silo uses its own data to build models, similar to the process in decentralized machine learning. However, in federative learning, these models are then sent to a coordinator, which aggregates the learning from each dataset to create a single universal model. This model is then sent back to the devices or silos. This model can be continuously improved by repeating this process. Federative learning offers the same security benefits as decentralized machine learning, namely, each data owner controls their own data, and that data is never moved. Federative learning also has the benefits of collaboration that decentralized machine learning has, and it is possible to avoid the scalability problems of decentralized machine learning.

[0016] There are three types of federative learning: lateral learning, vertical learning, and federative transfer learning. Lateral learning occurs when datasets share the same feature space or schema but differ in their samples. Lateral learning can be performed across devices or silos. One example of cross-device federative learning would be learning across cell phones, where each device holds data for a single user and there is no guarantee of persistent connectivity. Cross-silo learning refers to learning across data silos, where each silo holds data for many users and there is an expectation of consistent network connectivity. By analogy, lateral learning is like a database with the same columns but different rows. Since one device or silo operates only on the data that the device or silo has access to, and not on all the data in a central location, the main challenge of federative learning is the process of combining multiple partial models into a single complete model. In the case of lateral learning, federative averaging, federative SGD, and split learning are examples of effective optimization techniques.

[0017] Lateral learning can be applicable across many situations where sharing data is impossible or impractical. One example is similarly structured databases in different legal jurisdictions containing personal data. While the data may be legally required to remain where it is, in federated learning, these heterogeneous databases can be treated as a single, complete dataset. Another application is Internet of Things (IoT) devices. Federated learning enables learning across thousands or millions of devices and sensors to create a holistic picture of the ecosystem. Federated learning also enables industry-wide learning. The same concept applies to fraud detection. For example, federated learning allows a bank to learn from instances of fraud that have occurred at partner banks.

[0018] Vertical federation learning can be performed when different datasets share the same sample space but differ in their feature space. For example, this can be thought of as tables that have the same rows but different columns. Vertical federation learning can essentially be thought of as a join of two different database tables. The main difference is that in database join, the data is pulled together into a centralized location, whereas in vertical federation learning, the data never leaves their original locations.

[0019] In vertical federation learning, cryptographic keys are used to compare synthetic keys and generate a link table representing the synthesis across datasets. The synthesis logic is highly configurable in several implementations; that is, it can support exact matches, but also fuzzy matches (to capture misspelled names or other mistyped data), and multi-column synthesis. Just as with horizontal federation learning, algorithms for centralized machine learning cannot build models across this type of data without moving the data to a centralized location. Instead, vertical federation learning uses its own set of optimization algorithms, including, for example, Taylor gradient descent, SecureBoost, federated forests, and SplitNN. Using these algorithms, linear, tree, and neural network models can be trained across multiple related datasets as if they were located together in a single data warehouse.

[0020] One use case for vertical federated learning is line-of-business learning. Data sharing between lines of business (LoBs) is often restricted by internal policies or regulations. Vertical federated learning makes it possible to share insights across LoBs without ever sharing row-level data. Another use case is third-party data access. Consider a bank that wants to use third-party data to enhance its credit underwriting process, but bringing that data to the bank presents a risk and violates the principle of data minimization. Using vertical learning, the bank can gather insights from a combination of its data and third-party data without ever needing full access to the data. Vertical federated learning also allows a company to "try out" third-party data before deciding to purchase it. In federated learning, it is possible to determine the intersection of two datasets without exposing any of the underlying sensitive data.

[0021] The third type of associative learning is associative transfer learning. This type of associative learning focuses on using transfer learning techniques to leverage non-overlapping data in an associative setting. In vertical associative learning, for example, there may be many rows in each dataset that do not match when the datasets are synthesized, i.e., associative transfer learning uses those non-matching rows to improve the quality of the model.

[0022] These techniques for federated learning can be combined in various applications. For example, vertical and horizontal federated learning can be used to detect money laundering. Money laundering red flags can be classified using horizontal federated learning either across different jurisdictions of a single bank or across partner banks. The model is used to create suspicious activity reports (SARs) to help prevent fraud, but criminals understand that it is more difficult to stop them if they can disperse their activities across jurisdictions. However, due to the cross-silo synthesis at the core of vertical federated learning, the system can be assembled to eliminate cross-jurisdiction fraud.

[0023] Federated learning eliminates the need to move data between silos, but it provides no guarantees regarding who can build the models or the privacy of those models. In various embodiments of the present invention, data administrators can build on top of a default permission schema and define custom rules for determining who can create models for what data and how those models can be used.

[0024] Next, referring to FIG. 1, the main components of the platform according to an embodiment of the present invention can be described. In the client portal 10, data users can interact with the platform. The client portal 10 can be implemented using an application programming interface (API) or multiple individual APIs to enable data users to interact with the platform using different software tools. Examples of such tools in some embodiments include Spark and TensorFlow. Through the client portal 10, data users can use software tools to perform federated data analysis in a privacy-compliant manner.

[0025] Coordinator 12 functions to manage operations using data by means of the platform. Coordinator 12 ensures, for example, that data that cannot be deleted from its silo remains in that silo during all operations. Coordinator 12 also functions to ensure that the correct privacy approach is applied to data that exits its silo and has specific privacy approaches required for the data to be moved.

[0026] Coordinator 12 itself is divided into three basic operations, namely, logical planning, physical planning, and optimization. Logical planning is generally a determination of what needs to be done over different stages. Physical planning explains how Coordinator 12 divides that plan into increments that meet all applicable privacy requirements. The optimizer determines how all of these tasks should be performed most efficiently.

[0027] Coordinator 12 communicates with each of the silo managers 14. There is a silo manager 14 located in each data silo, and each data silo contains a dataset to be used by the platform in a particular analysis problem. Coordinator 12 communicates with each silo manager 14 and, in some cases, sends data back and forth between these two components until the desired task is completed. The silo manager 14 can spin up one or more runners 42 to complete the task, as will be described below.

[0028] Referring to FIG. 2, Coordinator 12 can be characterized essentially as a grouping of coordination components 16 that perform different functions in response to requests from the data user system 11 in the client portal 10. Each of the components of Coordinator 12 has its own coordinator backend 18. Similarly, each Silo Manager14 is, in essence, its own silo This is a grouping of request handler components 20, each having its own backend 22. Various coordinator backends 18, as well as corresponding silo The backend 22 can be used for different roles that the platform can perform. For example, these may include machine learning, identity resolution, data querying, and data transformation.

[0029] Figure 3 provides further details on the workings of the coordinator 12 and the silo manager 14. Data 24 is data in a specific silo where the silo manager 14 is installed. Data 24 may take the form of a data warehouse. The data user system 11 interacts with the coordinator 12 through the coordinator API 13 of the coordinator 12. The schema repository 26 stores metadata about the data 24 in the platform to ensure that the actions planned by the coordinator 12 are successfully executed by the silo backend 22. More specifically, the schema repository 26 is a collection of metadata about the data in the data silos, and therefore actions can be planned for this data. For example, if silo A has data including name, sex, and age, and silo B has name, sex, and weight, then the coordinator 12 has each of those data schemas in its schema repository 26. The metadata contains information about the data type, and therefore it knows that a "sum" operation can be performed on the "age" column in silo A and the "weight" column in silo B, but that it cannot perform the same mathematical function on the name and gender in each silo (because they are text fields). The coordinator authorizer 28 provides access control functions regarding the data user system 11's ability to make requests to the system, thereby providing authorization and privacy protection. The coordinator scheduler 30 performs scheduling functions for the various tasks requested by the coordinator 12. The artifact store 32 provides storage for calculated outputs that can be reused in later processing, thereby speeding up the re-execution of the same or similar tasks.

[0030] In the silo manager 14, there is a schema repository client, which sends metadata about the data 24 to the schema repository 26 of the coordinator 12. These interact through the runner API 15. The runner authorizer 36 ensures that tasks are effectively authorized, calling runners from the silo manager 14 for the coordinator 12 to execute. The service router 38 performs the function of routing command operations received from the scheduler 30 in the coordinator 12 to the appropriate backend 22. While the coordinator 12 backend 18 can communicate directly with the backend 22 corresponding to the correct backend for specific types of tasks and privacy needs during data processing, the initial execution of silo manager 14 tasks must be initiated through the service router 38.

[0031] Referring to Figure 4, the managed service system sits atop the platform shown in Figures 1-3. The managed service system facilitates the automated deployment of the platform shown in Figures 1-3 across heterogeneous computing infrastructures. More specifically, the purpose of the managed service system component is to enable customers using the system to perform minimal setup and maintenance to connect to it. This also improves system security, as fewer actors will have access to the secure system.

[0032] The control plane 40 is part of the overall system that resides within the provider's environment, which may be a cloud environment. The customer cloud 44 may be in a physically separate cloud environment and may use different cloud technologies or cloud providers. Within the customer cloud, a silo manager 14 may be created and given its own account within the cloud environment. A silo manager 42 is permitted to spin up (create) its own runner 14. To maximize privacy protection, a silo manager 42 may not have permission to access any of the data 24 in the silo associated with the silo manager 42, but the runner 14 is permitted to access the data 24. The silo managers 42 may have the ability to scale up and scale down the compute resources required for their operation, thereby conserving cloud compute resources. They may also perform upgrades and fault response operations on the system. The silo managers 42 may be controlled by a platform operator, thereby enabling the platform operator to manage this part of the customer's cloud on behalf of the customer. This significantly simplifies system deployment from a customer perspective. However, the separation between the silo manager 42 and the runner 14 means that the platform provider never has access to customer data 24, thereby ensuring privacy.

[0033] Figure 5 provides a more detailed diagram of the control plane 40 and customer cloud 44. In this case, there are multiple client portals 10 corresponding to multiple data users, each having access to the control plane 40 through the coordinator API 13. These communications are sent to the request scheduler service 46. In some embodiments, the request scheduler service 46 uses a compute graph for scheduling. A compute graph is a directed graph data structure where nodes correspond to operations or variables. The function of the request scheduler service 46 is to enable asynchronous communication with a separate customer cloud 44. The difficulty is that access and authorization are strictly controlled because these systems are in completely different hardware environments. Furthermore, the platform has no way of directly knowing the state of the customer cloud 44 at any given time; instead, it must send requests to determine whether the customer cloud 44 is, for example, working correctly and operational for any given task at that time, or whether it is available for use. The request scheduler service 46 controls these various operations through the coordinator 12 via the messaging service cluster 48. In the customer cloud 44, the silo manager 42 has its own messaging service to send asynchronous messages back and forth between the silo manager 42 and the messaging service cluster 48.

[0034] Other components shown in Figure 5 include cloud object storage 53 provided by the cloud vendor. The silo director 49 is responsible for deploying silo managers 42 within the customer cloud 44. The data collaboration service 50 manages the definition of policies applicable to data and how data is shared among clients. The claims check store 51 provides temporary storage for large objects for claims and returns tokens for later retrieval. The translation service 52 translates the Java Database Connectivity (jdbc) protocol into RESTful services. The cloud data warehouse 54 is where customer data is accessed.

[0035] The systems and methods described herein may be implemented in various embodiments by any combination of hardware and software. For example, in one embodiment, the systems and methods may be implemented by a computer system or a collection of computer systems, each including one or more processors that execute program instructions stored in a computer-readable storage medium coupled to the processors. The program instructions may implement the functions described herein. The various systems and representations described herein, as shown in the figures, represent exemplary implementations. The order of any method may be changed, and various elements may be added, modified, or omitted.

[0036] The computing systems or computing devices described herein may implement the hardware portion of a cloud computing system or a non-cloud computing system, forming part of various implementations of the present invention. A computer system may be any of various types of devices, including, but not limited to, commodity servers, personal computer systems, desktop computers, laptop or notebook computers, mainframe computer systems, handheld computers, workstations, network computers, consumer devices, application servers, storage devices, telephones, mobile phones, or generally any type of computing node, compute node, compute device, and / or computing device. A computing system may include one or more processors coupled to system memory via input / output (I / O) interfaces (which may include multiple processing cores, some of which may be single-threaded or multi-threaded). A computer system may further include network interfaces coupled to the I / O interfaces.

[0037] In various embodiments, a computer system may be a single-processor system containing one processor, or a multiprocessor system containing multiple processors. A processor may be any suitable processor capable of executing computing instructions. For example, in various embodiments, a processor may be a general-purpose processor or an embedded processor implementing one of several instruction set architectures. In a multiprocessor system, each processor may, though not necessarily, implement the same instruction set. A computer system may also include one or more network communication devices (e.g., network interfaces) for communicating with other systems and / or components across a communication network, such as a local area network, a wide area network, or the Internet. For example, a client application running on a computing device may use a network interface to communicate with a server application running on a single server or a cluster of servers implementing one or more components of the system described herein in a cloud computing or non-cloud computing environment, which are implemented in various subsystems. In another example, an instance of a server application running on a computer system may use a network interface to communicate with other instances of the application that may be implemented on other computer systems.

[0038] A computing device also includes one or more persistent storage devices and / or one or more I / O devices. In various embodiments, the persistent storage device may be a disk drive, tape drive, solid memory, other mass storage device, or any other persistent storage device. A computer system (or a distributed application or operating system running on it) may, as needed, store instructions and / or data in a persistent storage device and retrieve the stored instructions and / or data as needed. For example, in some embodiments, a computer system may implement one or more nodes of a control plane or control system, and the persistent storage may include SSDs mounted on its server nodes. Multiple computer systems may share the same persistent storage device or share a pool of persistent storage devices, where devices in the pool represent the same or different storage technologies.

[0039] A computer system includes one or more system memories capable of storing code / instructions and data accessible by one or more processors. The system's memory capabilities may include, for example, multiple levels of memory and memory caches in a system designed to swap information in memory based on access speed. Interleaving and swapping may extend to persistent storage in virtual memory implementations. Techniques used to implement memory may include, as an example, static random-access memory (RAM), dynamic RAM, read-only memory (ROM), non-volatile memory, or flash-type memory. As with persistent storage, multiple computer systems may share the same system memory or a pool of system memory. One or more system memories may contain program instructions executable by one or more processors to implement the routines described herein. In various embodiments, program instructions may be encoded in binary, assembly language, any interpreted language such as Java, a compiled language such as C / C++, or any combination thereof; the specific languages ​​given herein are merely examples. In some embodiments, program instructions may implement multiple separate clients, server nodes, and / or other components.

[0040] In some implementations, program instructions may include executable instructions for implementing an operating system (not shown), which may be one of various operating systems, such as UNIX®, LINUX, Solaris®, MacOS®, or Microsoft Windows®. Some or all of the program instructions may be provided as a computer program product or software, which may include a non-temporary computer-readable storage medium storing the instructions, and which may be used to program a computer system (or other electronic device) to perform a process according to various implementations. A non-temporary computer-readable storage medium may include any mechanism for storing information in a machine-readable form (e.g., software, processing applications). Generally, a non-temporary computer-accessible medium may include computer-readable storage media or memory media, such as magnetic or optical media, for example, a disk or DVD / CD-ROM coupled to a computer system via an I / O interface. Non-temporary computer-readable storage media may include any volatile or non-volatile media, such as RAM or ROM, which may be included as system memory or another type of memory in some embodiments of a computer system. In other implementations, program instructions may be communicated using optical, acoustic, or other forms (e.g., carrier waves, infrared signals, digital signals, etc.) of signals that are transmitted and propagated over a communication medium, such as a network and / or wired or wireless link, which may be implemented via a network interface. Network interfaces may be used to interface with other devices, which may include other computer systems or any type of external electronic device.In general, system memory, persistent storage, and / or remote storage accessible on other devices over a network may store data blocks, replicas of data blocks, metadata associated with data blocks and / or their states, database configuration information, and / or any other information that can be used when implementing the routines described herein.

[0041] In some implementations, the I / O interface can coordinate I / O traffic between the processor, system memory, and any peripheral devices in the system, including through a network interface or other peripheral interfaces. In some embodiments, the I / O interface can perform any necessary protocols, timing, or other data conversions to convert data signals from one component (e.g., system memory) into a format suitable for use by another component (e.g., the processor). In some embodiments, the I / O interface may include support for devices attached via various types of peripheral buses, such as variations of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard. Also, in some embodiments, some or all of the functionality of the I / O interface, such as an interface to system memory, may be directly integrated into one or more processors.

[0042] A network interface may enable the exchange of data between a computer system and other devices attached to the network, such as other computer systems (which may implement one or more storage system server nodes, primary nodes, read-only nodes, and / or database system clients as described herein). Furthermore, an I / O interface may enable communication between the computer system and various I / O devices and / or remote storage. In some embodiments, input / output devices may include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other devices suitable for inputting or retrieving data by one or more computer systems. These may connect directly to a specific computer system or, more generally, to multiple computer systems in a cloud computing environment, a grid computing environment, or other systems involving multiple computer systems. Multiple input / output devices may exist communicating with a computer system or be distributed across various nodes in a distributed system including a computer system. The user interfaces described herein may be visible to the user using various types of display screens, which may include CRT displays, LCD displays, LED displays, and other display technologies. In some implementations, input may be received through the display using touchscreen technology, and in other implementations, input may be received through a keyboard, mouse, touchpad, or other input technology, or any combination thereof.

[0043] In some embodiments, similar input / output devices may be separate from the computer system and may interact with one or more nodes of a distributed system, including the computer system, via wired or wireless connections, such as over a network interface. The network interface may typically support one or more wireless networking protocols (e.g., Wi-Fi / IEEE 802.11, or another wireless networking standard). The network interface may support communication over any suitable wired or wireless general data network, such as other types of Ethernet networks. Furthermore, the network interface may support communication over telecommunications / telephony networks, such as analog voice networks or digital fiber optic communication networks, over storage area networks, such as Fibre Channel SANs, or over any other suitable type of network and / or protocol.

[0044] Any of the distributed system embodiments described herein, or any of their components, may be implemented as one or more network-based services in a cloud computing environment. For example, read-write nodes and / or read-only nodes in the database tier of a database system may present database services and / or other types of data storage services employing the distributed storage systems described herein to clients as network-based services. In some embodiments, network-based services may be implemented by software and / or hardware systems designed to support interoperable machine-to-machine interaction over a network. Web services may have interfaces described in a machine-readable format, such as Web Services Description Language (WSDL). Other systems may interact with network-based services in a manner defined by the description of the network-based service's interface. For example, a network-based service may define various actions that other systems may invoke and define specific application programming interfaces (APIs) that other systems are expected to follow when requesting these actions.

[0045] In various embodiments, network-based services may be requested or invoked through the use of messages containing parameters and / or data associated with the network-based service request. Such messages may be formatted according to a specific markup language, such as Extensible Markup Language (XML), and / or encapsulated using a protocol, such as Simple Object Access Protocol (SOAP). To fulfill a network-based service request, a network-based service client may assemble a message containing the request and transmit that message to an addressable endpoint corresponding to the web service (e.g., a Uniform Resource Locator (URL)) using an Internet-based application layer transport protocol, such as Hypertext Transfer Protocol (HTTP). In some embodiments, network-based services may be implemented using Representational State Transfer (REST) ​​techniques rather than message-based techniques. For example, a network-based service implemented according to REST techniques may be invoked through parameters contained within HTTP methods, such as PUT, GET, or DELETE.

[0046] Unless otherwise specified, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art to the extent of this invention. Similar or equivalent methods and materials as those described herein may also be used in the implementation or testing of this invention, but only a limited number of exemplary methods and materials are described herein. It will be apparent to those skilled in the art that many more modifications are possible without departing from the inventive concept herein.

[0047] All terms used herein should be interpreted in the broadest possible form that is appropriate to the context. When grouping is used herein, all individual members of the group, as well as all possible combinations and subcombinations of the group, are included individually. When a scope is specified herein, the scope includes all subscopes and individual points within that scope. All references cited herein are incorporated herein by reference to the extent that they do not conflict with the present disclosure.

[0048] The present invention has been described in relation to several preferred and alternative embodiments, which are intended to be illustrative and not to be limited to the full scope of the invention as described in the appended claims.

Claims

1. An analytical platform for federated data, Multiple physically remote and separate silo managers, each configured to manage one of multiple data silos, Multiple runners, each configured to be located within one of the multiple data silos, perform analysis on a dataset within one of the multiple data silos, and send the results of the analysis to its corresponding silo manager, wherein each of the multiple data silos contains private data that cannot be shared with other data silos among the multiple data silos, and the results do not include raw private data from the data silos, and A coordinator configured to communicate with each of the aforementioned silo managers and to send commands to each of the aforementioned silo managers in order to perform data operations, A client portal that communicates with a coordinator, wherein the client portal includes at least one application programming interface (API), and Equipped with, An analytics platform in which each coordinator includes multiple coordinator backends, each silo manager includes multiple silo backends, and the coordinator is configured to select one of the multiple coordinator backends based on an action selected in the client portal API, wherein the action selected in the API is one or more of identity resolution, data querying, and data transformation.

2. The analytics platform according to claim 1, wherein each runner is configured to prevent the sharing of raw, private data from the dataset within the data silo with the coordinator.

3. The analytical platform according to claim 1, wherein the coordinator is further configured to implement a logical plan, and the coordinator determines which functions are performed at each stage of the analytical operation.

4. The analytical platform according to claim 3, wherein the coordinator is further configured to carry out a physical plan, and its operations are divided across each of the datasets in the data silo in order to carry out the logical plan.

5. The analysis platform according to claim 4, further comprising an optimizer configured to determine how the coordinator should efficiently execute the physical plan.

6. The analysis platform according to claim 1, wherein the operation to be selected in the API is identity resolution.

7. The analytical platform according to claim 1, wherein the operation to be selected in the API is a data query.

8. The analysis platform according to claim 1, wherein the operation to be selected in the API is data transformation.

9. The analysis platform according to claim 1, further comprising a coordinator schema repository comprising a plurality of coordinator schemas, each having metadata corresponding to data in the corresponding data silo.

10. The analysis platform according to claim 9, further comprising an artifact store configured to store calculated outputs for later reuse, wherein the coordinator.

11. The analysis platform according to claim 10, wherein at least one of the plurality of runners further comprises a runner schema repository having a runner schema for each coordinator schema.

12. A method for analyzing data using a platform, wherein the method is A receiving step of receiving a request for an action in a client portal via an Application Programming Interface (API), wherein the action includes one or more of identity resolution, data querying, and data transformation. A step of transferring the request from the client portal to the coordinator, wherein the coordinator is communicating with a number of physically remote and separate silo managers, The coordinator sends a command to each silo manager to perform the aforementioned operation, A step of creating at least one data runner in each silo manager, wherein each data runner is configured to perform the aforementioned operation on data contained in the data silo communicating with each silo manager. A step of sending the results of the operation from each data runner to its corresponding silo manager, and from each silo manager to the coordinator, wherein the results sent from each data runner to its corresponding silo manager do not include raw data from the data silo. Methods that include...

13. The method according to claim 12, wherein the coordinator is implemented in the provider cloud, and each silo manager is implemented in the client cloud along with its corresponding data silo.

Images