Reflective distributed data access

Abstract

The present disclosure relates to systems and methods for securely reflecting data from a local data storage. In one example embodiment, a user may connect a local data storage to an edge node, which may facilitate communication between a third party data management hub and the local data storage. A user operating a third party data management hub may request to read / write certain data at a local data store (owned by the user). Data from the local data storage may be generated in response to the request. Response data may be transmitted to the data management hub over the encrypted data session, where the response data is reflected to the user on the user interface. The response data is stored in the volatile memory store and is not removed from the persistent memory at the local data store. Upon termination of the encrypted data session, the volatile memory storage is erased to protect privacy / security of the underlying data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The entire disclosures of U.S. patent application serial number 16 / 776,293, entitled “SYSTEMS AND METHOD OF CONTEXTUAL DATA MASKING FOR PRIVATE AND SECURE DATA LINKAGE”; U.S. patent application serial number 17 / 103,751, entitled “SYSTEMS AND METHODS FOR UNIVERSAL REFERENCE SOURCE CREATION AND ACCURATE SECURE MATCHING”; U.S. patent application serial number 17 / 103,720, entitled “SYSTEMS AND METHODS FOR DATA ENRICHMENT”; and U.S. patent application serial number 17 / 219,340, entitled “SYSTEMS AND METHODS FOR AN ON-DEMAND, SECURE, AND PREDICTIVE VALUE-ADDED DATA MARKETPLACE” are hereby incorporated by reference. Technical Field

[0003] The present disclosure relates to distributed data and privacy protection designs. Background Art

[0004] Data profiling and data quality (DQ) are critical for entities with numerous large data sets and data sources. However, performing large-scale data profiling and ensuring data quality across such large amounts of data can be costly and take up scarce computing resources. Data profiling and quality analysis can also take a long period of time when performed by external or non-native applications. Data profiling and quality checks are typically serial and asynchronous with the process of data creation.

[0005] In one example, an entity that wishes to analyze or clean certain data may find that certain sensitive data storage is stored locally rather than on a server accessible via a network. In order to analyze and evaluate the quality of this local data, the entity will be required to set up an on-premise program to analyze the data (which takes a long time to set up and is expensive from a hardware and maintenance perspective) or to send sensitive local data upward to a server to be accessible on the network (which then increases the risk of privacy leakage because the data is no longer on a local and isolated computer). In some examples, certain sensitive data (such as personally identifiable information) may require storage only on a local storage device and not have a storage presence (i.e., a persistent presence at the disk level) on a third party / remote server. In order to circumvent these requirements, the client will need to create a bypass in the firewall, or rely on a cumbersome process / approval method such as establishing a virtual private connection (e.g., a virtual private link), which may trigger certain regulatory and compliance issues, thereby deepening the nature of sensitive data. Therefore, it is necessary to access local data (e.g., for analysis, quality assurance, analysis, etc.) in a fast, inexpensive and secure way without including the underlying privacy of sensitive data.

[0006] In another example, an entity that wishes to perform DQ analysis on certain data sets may find that the entity's data sources are scattered across many different data stores. In order to perform DQ analysis on each of these different data stores, the entity will need to perform independent analysis on each of those data stores because they are disconnected and isolated from the other data stores. Therefore, there is also a need to aggregate data from different data stores in a secure manner so that DQ analysis can be effectively performed across multiple different data stores.

[0007] It is with respect to these and other general considerations that the aspects disclosed herein have been made.In addition, although relatively specific problems may be discussed, it should be understood that the examples should not be limited to solving specific problems identified in the background or elsewhere in this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Non-limiting and non-exhaustive examples are described with reference to the following figures.

[0009] Figure 1 An example of a distributed system for securely reflecting data from a data store as described herein is shown.

[0010] Figure 2 An example input processor for implementing systems and methods for securely reflecting data from a data store is shown.

[0011] Figure 3 An example method for securely reflecting data from a data store as described herein is shown.

[0012] Figure 4 An example distributed environment for securely reflecting data from a data store is shown.

[0013] Figure 5 An example of a suitable operating environment is shown in which one or more of the present embodiments may be implemented. DETAILED DESCRIPTION

[0014] Various aspects of the present disclosure are described more fully below with reference to the accompanying drawings, which form a part of the present disclosure and illustrate specific exemplary aspects. However, different aspects of the present disclosure can be implemented in many different forms and should not be construed as being limited to the aspects set forth herein; on the contrary, these aspects are provided so that the present disclosure will be thorough and complete, and the scope of these aspects will be fully conveyed to those skilled in the art. These aspects can be practiced as methods, systems, or devices. Therefore, various aspects can take the form of hardware implementations, fully software implementations, or implementations combining software and hardware aspects. Therefore, the following detailed description should not be considered to be limiting.

[0015] Embodiments of the present application are intended to improve data analysis and data quality by realizing on-demand, real-time distributed data access, and protect the privacy and security of underlying data at the same time. Underlying data can also refer to source data, which can be defined as a data set, or a sample or subset thereof, submitted by a user to a third-party service for analysis, sampling, classification, cataloging or other forms of analysis. In the system and method described herein, the user controls which data is submitted to the system / can be accessed by the system. As used herein, the term metadata can also be referred to as platform data, which is defined as data or content or user's enterprise data structure (i.e., metadata) that classifies, organizes, defines or otherwise characterizes source data, which establishes a comprehensive data catalog, data management structure, business glossary, business process description, data steward roles and responsibilities, assets and domain lists, and similar data management concepts in a third-party system. Platform data also includes logs, insights, statistics or reports related to the performance, availability, use, integrity or security of third-party services generated by a third-party system. Platform data is not source data.

[0016] A metastore is an object store with database or file backing storage. A metastore holds metadata associated with the actual, underlying data stored in a datastore. With the proliferation of edge computing and edge / spoke sites as part of software-as-a-service offerings, some product components may no longer need to share the same network boundary for each client data source. In other words, a client that wishes to access each of its data sources can do so via a single unified user interface, regardless of whether the data source is stored on a remote server (i.e., cloud accessible) or a local database / storage.

[0017] In one example embodiment, a user (e.g., an entity) may have different data stores that contain sensitive data, such as PII (Personally Identifiable Information). The user may connect those data stores to an edge node, which is used to reflect requests from the user to be executed on the underlying data store. The edge node may also be used to reflect certain data from the data store back to the user, but without actually storing the data in a persistent storage device (e.g., a disk) external to the data store. In other words, the user may utilize the systems and methods described herein to access each of the user's data stores from a single unified user interface and perform analysis on the underlying data located in the user's data store without transmitting the underlying data in the data store to a separate storage device location external to the data store. Instead, the accessed data is only transmitted over the wire and temporarily stored in a volatile memory storage that is unique to the specific data session currently running. In other words, the underlying data in the data store is reflected to the user based on the user's instructions (which are transmitted to the edge node and then reflected to the underlying data store, whereby the data store then reflects the requested data back to the user through a secure channel established by the edge node).

[0018] In other embodiments, a cloud-based metastore supporting a cloud service that provides a single data view may include a data schema (e.g., platform data) that reflects the characteristics of the underlying data, such as data type, number of rows, number of columns, headers, etc. The client can access this metadata by viewing and requesting operations in real time in a single unified user interface. The cloud-based metastore can be communicatively coupled to the local data store (via the "data mirroring" mechanism described herein), so the requested data can be displayed on the user interface, but the underlying data displayed to the user is not yet stored in a persistent disk external to the data store (i.e., not transmitted and not stored in a third-party disk location). Instead, the underlying data is transmitted on the wire and temporarily stored in a volatile memory location that is unique to the data session that occurs in real time. Once the data session ends (or the user simply clicks away from the user interface), the data reflected from the underlying data store is removed from the volatile memory. In addition, the reflected data transmitted between the data store and the edge node is end-to-end encrypted (e.g., end-to-end encrypted using the TLS encryption protocol).

[0019] The reflective distributed data access architecture described herein has a number of technical benefits. The benefits include, but are not limited to: (1) enhanced security and privacy because data is not persistently stored outside of the data storage (i.e., ownership and storage device location of the underlying data is maintained only on a persistent disk within the client's data storage); (2) reduced latency because concurrent lookups via the distributed data access architecture are faster than independent sequential synchronous lookups of the actual underlying data at each unique data storage location; and (3) more efficient use of memory resources because the underlying data does not need to be copied and stored in a persistent storage device because the data accessed is reflected across the wire in volatile memory storage while the data session is uninterrupted.

[0020] Another technical benefit of the reflective distributed data access system described herein is an improved privacy protection design. Typical systems implement privacy features in a reactive or remedial manner; however, in the system and method described herein, privacy is embedded in the architecture itself. Specifically, the architecture described herein allows users to maintain ownership and control over their proprietary data in their own data storage. The underlying data in the data storage is not transmitted to a third-party persistent storage device to be accessed outside the user's data storage. Instead, the underlying data can be viewed and accessed via an end-to-end encrypted data session, in which the user is viewing the reflected data, which is temporarily stored in a volatile memory location reserved directly for the ongoing data session. In other words, accessing the reflected data will require decrypting the data transmission in real time while the data session is ongoing, or directly accessing a dedicated volatile memory location on the actual machine that the user is operating. Either case is highly unlikely.

[0021] In other example aspects, the meta-storage can be used to store data quality and data profiling results, as well as other data analysis operations. Consider a software as a service application that is intended to connect various data storages that hold source data at certain locations that are not directly accessible from the system that holds the source data. In order to enable access to the target secure data source, an edge node is set in a secure location where the target data is stored. The edge node can periodically perform data quality assessment activities on the underlying data at the data storage, and update the local meta-storage with the results of these activities. If an error / problem with the underlying data is detected, the problem can be reflected by the edge node (e.g., in the form of an error notification) from the location where the data is stored to a cloud-based user interface (and stored in the meta-storage). The user can then analyze the problem more deeply by requesting to see the source of the problem. A user request to view certain data (at the cloud instance of the application) is transmitted to the edge node, and then subsequently "reflected" to the data storage. The data storage receives the request, retrieves the underlying data, and returns the retrieved underlying data to the local edge node. The edge node securely transmits the data to the volatile memory location of the cloud instance of the application via an encrypted data session, thereby "reflecting" the underlying data to the user interface. After the user is done viewing the reflected data, the user can close the data session and erase the underlying data contained in the volatile memory. Note that under standard TCP protocol requirements, the underlying data presented to the unified user interface of the cloud-based data management application is presented in a unidirectional manner so that no data can be transmitted from the cloud-based data management application back to the underlying data storage. Note that reflecting data to the data storage and back to the volatile memory location of the cloud instance of the application does not require the use of a metastore.

[0022] In another example, data virtualization can be incorporated. The data storage device deployed at the edge can locally store data that is sensitive in nature from the data source. The underlying data of the data source can be shared to cloud-based applications on demand (via encrypted package delivery), and the unencrypted underlying data is never exposed to the persistent storage device in the cloud instance. In a data virtualization embodiment, the underlying data is copied to a volatile memory location isolated from the ongoing data session at the edge node. This allows the user to raise complex questions about the data set from the remote instance of the application, so that the edge node performs the required processing on the data in the volatile memory to answer the question, and then only "reflects" the limited data required by the response back to the user. The target data set is not permanently stored outside the underlying data storage device. In other words, the data management application that the user is operating can not access the actual underlying data (source data) in the data storage device due to the secure network perimeter (e.g., encrypted transmission of the underlying data to the edge node, and temporary storage in the volatile memory storage when the data session is active).

[0023] Figure 1 An example of a distributed system for securely reflecting data from a data store as described herein is shown. The presented example system 100 is a combination of interdependent components that interact to form an integrated whole for uniting and enriching data. The components of the system can be hardware components or software implemented on and / or executed by the hardware components of the system. For example, the system 100 includes client devices 102, 104, and 106, local databases 110, 112, and 114, one or more networks 108, and server devices 116, 118, and / or 120.

[0024] Client devices 102, 104, and 106 may be configured to receive and transmit data. For example, client devices 102, 104, and 106 may contain client-specific data. The client device may store certain client-sensitive data on databases 110, 112, and 114. The client device may also be communicatively coupled to a distributed meta-storage on an edge site via one or more networks 108, and one or more networks may be communicatively coupled to a central data management application or hub (e.g., a data control center). The client device may also be configured to receive data reflected from a data store based on certain requests received by the data store. In some examples, client-specific data stores may be required to be in local databases 110, 112, and 114. If the client-specific data contains PII, then the PII may be separated in local databases 110, 112, and 114. The client device may receive data reflected from one of the data stores 110, 112, and 114. In other examples, the client devices 102, 104, and 106 can be considered "edge nodes" that reflect data and commands back to the data stores 110, 112, and 114 and / or back to other devices connected to the one or more networks 108. The edge nodes and client devices can be connected to a central hub, to server(s) 116, 118, and / or 120 via the one or more networks 108 and / or satellites 122. The one or more servers 116, 118, and / or 120 can be third-party servers that can receive data reflected from the data stores 110, 112, and 114 and transmitted across the one or more networks 108 via the edge nodes. The edge nodes (e.g., client devices 102, 104, 106 or servers 116, 118, and / or 120) can receive commands from clients via a centralized data management application to read certain underlying data / data sets in the data stores to which the edge nodes are connected. The data may be displayed on a user interface at a central hub location (i.e., reflected or mirrored on the user interface), but the underlying data itself may be stored in local storage at the databases 110, 112, and 114. Specifically, certain requested underlying data stored in the data stores (e.g., 110, 112, and 114) may be encrypted and transmitted over one or more networks 108 to a volatile memory location tied to an active data session, in which the user may view the requested data as long as the data session is in operation. For example, a user may connect to one or more networks 108 via a device that is operating a central data management application. From the application, the user may request to receive status updates related to certain data quality tests run on data stores stored, for example, in the database 110.The edge node 102 connected to the data storage 110 can receive the command through one or more networks 108, and the edge node 102 can reflect the command to the data storage 110. The edge node 102 receives data from the data storage 110, which is then reflected back to the user device that is operating the central data management application. It is important to note that the actual underlying data stored in the data storage 110 is not transmitted to a persistent storage device outside the database 110. By using volatile memory, only the underlying data is temporarily reflected to the user device within the data management application. When the data session is closed, the data stored in the volatile memory location can be erased. Note that the requested underlying data encrypted and transmitted through one or more networks 108 is not stored in a persistent storage device (such as, databases 116, 118 and / or 120) at a third-party data location.

[0025] In some aspects, client devices (such as client devices 102, 104, and 106) can access one or more data sets or data sources and / or databases (e.g., source data) including client-specific data. In other aspects, client devices 102, 104, and 106 can be equipped to receive broadband and / or satellite signals from edge nodes that reflect client-specific read requests from central data management applications. Signals and information that client devices 102, 104, and 106 can receive can be transmitted from satellite 122. In addition to being able to communicate directly with client devices 102, 104, and 106, satellite 122 can also be configured to communicate with one or more networks 108. In some examples, client devices can be mobile phones, laptops, tablets, smart home devices, desktop phones, and wearable devices (e.g., smart watches), as well as other devices. In other example aspects, client devices 102, 104, and 106 can be considered to be edge nodes that are communicatively coupled to underlying data storage.

[0026] In some instances, a user may have write permissions for manipulating certain data stored locally at databases 110, 112, and / or 114. Write operations may be captured by edge sites (e.g., at client devices 102, 104, and 106) and then reflected to local data stores 110, 112, and / or 114 via one or more networks 108 and / or satellites 122. Such write operations may be serviced by local metastore instances to separate third-party data management applications from one or more underlying data stores. Read operations may also be serviced by local metastore instances, and access to sensitive data may be mirrored (or reflected) to a central data management application rather than being transmitted from its underlying location to a persistent storage location (e.g., a third-party disk storage device).

[0027] Figure 2 An example input processor for implementing systems and methods for securely reflecting data from a data storage device as described herein is shown. The input processor 200 can be embedded in a client device (e.g., client devices 102, 104 and / or 106), a remote web server device (e.g., devices 116, 118 and / or 120), and other devices capable of implementing systems and methods for securely reflecting data from a data storage device. The input processing system includes one or more data processors and is capable of executing algorithms, software routines, and / or instructions based on processed data provided by at least one client source and / or a third-party source. The input processing system can be a factory-assembled system or an add-on unit for a specific device. In addition, the input processing system can be a general-purpose computer or a special-purpose computer for a specific purpose. No limitation is imposed on the location of the input processing system relative to a client or remote web server device, etc. According to Figure 2 In the embodiment shown in , the disclosed system may include a memory 205, one or more processors 210, a communication module 215, a data mirror 220, and a read / write handler 225. The data mirror 220 can be configured to communicate with a local database (data storage) and generate metadata based on the underlying, locally stored data and user requests regarding (for example) DQ analysis, and reflect the data to a central data management application via an edge site. Other embodiments of the present technology may not include these modules and components or include some, all of these modules and components, as well as other modules, applications, data and / or components. However, some embodiments may merge two or more of these modules and components into a single module and / or associate part of the functionality of one or more of these modules with different modules.

[0028] The memory 205 may store instructions for running one or more applications or modules on one or more processors 210. For example, in one or more embodiments, the memory 205 may be used to accommodate all or some instructions required to perform the functions of the metastore generator 220, the read / write handler 225, and / or the data mirror 230. In general, the memory 205 may include any device, mechanism, or populated data structure for storing information. According to some embodiments of the present disclosure, the memory 205 may include, but is not limited to, any type of volatile memory, non-volatile memory, and dynamic memory. For example, the memory 205 may be a random access memory, a memory storage device, an optical memory device, a magnetic medium, a floppy disk, a tape, a hard disk drive, a SIMM, a SDRAM, a RDRAM, a DDR, a RAM, a SODIMM, an EPROM, an EEPROM, an optical disk, a DVD, and / or the like. According to some embodiments, the memory 505 may include one or more disk drives, a flash drive, one or more databases, one or more tables, one or more files, a local cache memory, a processor cache memory, a relational database, a flat database, and / or the like. Furthermore, those of ordinary skill in the art will appreciate that there are many additional devices and techniques for storing information that may be used as memory 205 .

[0029] The communication module 215 is associated with sending / receiving information (e.g., transmitting encrypted data from the data mirror 220, read / write commands from the read / write handler 225), commands received via a client device or server device, other client devices, remote web servers, etc. These communications can use any suitable type of technology, such as Bluetooth, WiFi, WiMax, cellular (e.g., 5G), single-hop communication, multi-hop communication, dedicated short-range communication (DSRC), or proprietary communication protocols. In some embodiments, the communication module 215 sends the information output by the data mirror 220 to a central data management application or hub. Specifically, the communication module 215 can facilitate "reflecting" data from the edge node to the central data management application. Metadata (e.g., data type, number of rows, number of columns, other modes, etc., also known as platform data) can represent the underlying actual data, but the underlying data (source data) is securely stored in a local database and is not transmitted across the network to a persistent storage device location located on a third-party server. In other examples, based on user requests reflected from edge sites to underlying data storage, certain data retrieved from the data storage (in response to user requests) may be transmitted to a volatile memory location via an encrypted data session to be reflected on a user interface managed by a central data management application or hub. When the data session is closed, the data stored in the volatile memory is erased. In other examples, the communication module 215 may transmit read / write commands from a central data management application, wherein a read command may be a command for displaying certain data sets, and wherein a write command may be a command for changing certain data or an underlying application program interface. Such communications may be transmitted to client devices 102, 104, and / or 106 and memory 205 to be stored for future use. In some examples, the communication module may be constructed using RESTful services based on the HTTP protocol, through one or more secure REST servers.

[0030] Data mirror 220 can be configured to facilitate the "mirroring" (or reflection) of data between devices and edge nodes. The data storage can be in the client's environment, between the edge computing site and the local computer cluster that holds local data. The edge site can include an edge proxy, an edge controller, a data access service, and a data quality agent. The edge representative can be defined as a component responsible for maintaining a secure channel for user instructions from a central data management application that affects the target location of the hosted target data source. The channel is logical, not permanent. The edge controller can be defined as a component that uses a secure channel provided by the edge representative to continuously poll (i.e., listen) data containing user instructions from the central data management application. Upon receiving user instructions targeting a specific edge site, the appropriate edge site receives those instructions and facilitates communication with a data access service (or data quality service) that ultimately processes the request locally at the target data storage. The data access service (or data quality service) can be defined as a service that processes user instructions. This is part of the data mirror 220. For example, instructions can be for data quality analysis, directory services, policy management, data virtualization, etc. Specifically, the data access service can deserialize the user instructions to execute them locally at the data storage level. The response transmitted by the data storage back to the data access service can be reflected back through the edge controller, the edge representative via the data mirror 220 module, and finally reflected to the central data management application, where the data can be displayed via an encrypted data session. The data quality agent can be used for an example use case of data quality analysis. The data quality agent is responsible for initiating data processing jobs and is controlled by the local DQ service on the client device. The DQ agent can execute DQ jobs consistent with the user instructions processed by the data access service. The data access service can be communicatively coupled to the data storage. The data quality service can also be communicatively coupled to the central data management application. The data mirror 220 can accurately transmit the user instructions received via the edge representative to the underlying data storage, and reflect the data received from the underlying data storage to the central data management application. For example, the edge site can be located in a virtual LAN, which can communicate with both the data quality module (e.g., the data quality meta storage) and the central data management application. The data mirror 220 may also be configured to receive information about the type of data to be reflected in the metadata from the communication module 215. For example, the data mirror 220 may be configured to not display certain sensitive information to the central data management application.

[0031] The read / write handler 225 may be configured to receive commands from a central data management application and replicate those commands locally at a local computer to read / write certain locally stored data. For example, a read command from a central data management application may include a request to access new data. The read / write handler 225 may include a data access service that receives user instructions from an edge representative and processes instructions to be executed at the local data storage level. The read / write handler 225 may first receive a user command via a data mirror 220 and pass the command to an edge site (via an edge representative and an edge controller). The edge site may then pass the command to a data storage communicatively coupled to a local computer (via a data access service) so that the read command is executed locally on the data stored on the local database. The read / write handler 225 may receive a response from a DQ service, wherein the response contains data of certain requests based on the user instructions. The data may then be encrypted and transferred to a volatile memory location, where the user may be able to view the reflected data at the central data management application.

[0032] Data mirror 220 is also configured to ensure that certain data from the local data store does not flow to a persistent storage location other than the authorized storage location. This is particularly important when a user accesses a data store via a reflective mirroring process (via data mirror 220), receives updates about the data quality of a certain data store, and / or issues commands that affect the underlying data in the data store. The underlying data is encrypted and not transmitted to a persistent storage device, but is temporarily reflected to the user via a volatile memory component.

[0033] Similarly, write commands from the read / write handler 225 can transmit write commands to the local database via the edge representative, edge controller, and data access service. From the user's perspective, the commands appear to occur synchronously, but due to edge sites and edge site deployment, the commands can occur asynchronously. This is to ensure that each command is accurately captured and transmitted while protecting the security and privacy of the underlying local data.

[0034] In some examples, read / write handler 225 receives commands from a central data management application via communications module 215 .

[0035] Figure 3 An example method 300 for securely reflecting data from a data store is shown. The method 300 begins at step 302 by receiving a request to access remote data. At step 302, a request to access a remote data store (such as from a remote data store) is received. Figure 1The request may be received from a central data management application that is communicatively coupled to an edge node that is communicatively coupled to a data store. In step 304, the request is analyzed to determine which data store the request is targeting, which is then used to determine the correct target edge node that is communicatively coupled to the relevant data store. Some preloaded requests may already exist so that when a request is received by the system, the request is automatically associated with a certain edge node without further analysis.

[0036] At step 306, the correct edge node receives the request. Between steps 304 and 306, there may be other sub-steps. For example, after identifying the correct edge node in step 304, the request can be forwarded to an edge gateway, where the request is marked for a target edge node. When the request is at the gateway, the channel from which the request comes remains open waiting for a response. In addition, the target edge node may include a controller that constantly polls the edge gateway (e.g., every 5 seconds or other duration). When the target edge node controller polls the gateway and sees a request for its edge node, the target edge node controller forwards the request to the edge node.

[0037] The reason why the target edge node polls the edge gateway is because the edge gateway cannot forward the request directly to the target edge node due to the lack of a network path between the target edge node and the edge gateway. The edge gateway cannot access secure, non-publicly addressable endpoints in the system. Instead, the user instruction is transmitted to a queue associated with the target edge node. The target edge node then polls the queue at the edge gateway until the user instruction is received at the queue. Once the user instruction is in the queue and the target edge node polls the queue, the user instruction is then transmitted to the target edge node.

[0038] At step 308, the edge node executes the request and retrieves the data from the data storage to which it is connected. In some instances, the data may be retrieved from the connected data storage, and in other instances, the data may be retrieved from the connected meta storage. The data is encrypted while being held at the edge node.

[0039] At step 310, the data acquired by the edge node is reflected back to the user device from which the central data management application is running. Before step 310 and after step 308, the edge node can query the edge controller, and the edge controller can open a secure data channel between the gateway and the edge node. The data reflected back to the user device from the edge node can be a status report on the data quality of the underlying data storage. If the status report indicates a problem, the user can then issue a subsequent request to obtain the original data at the data storage to see the problem. Data can be reflected from the edge node to the user device, but the data is only stored in a volatile memory location, not on a persistent storage device external to the data storage. Therefore, when the data channel is closed, the reflected data is immediately erased.

[0040] In some examples, from the client's perspective, the client may first receive a summary (e.g., a broadcast request) of each data storage device accessible to the client, regardless of whether the data storage device is located locally or externally. The client may then execute read and / or write commands via the user interface of the central data management application. Read commands to local data storage devices may pass through a distributed meta-storage device at the customer's edge site, which may then obtain the requested metadata and reflect it back to the user viewing the user interface of the central data management application. This process occurs without actually transferring any underlying local data to a persistent memory storage device location hosted by a third party. Instead, the data is transmitted via an end-to-end encrypted data session and stored in a volatile memory storage device location bound to the encrypted data session. At the end of the data session, the volatile memory storage device location is erased. In some examples, a data session may end when a user exits the central data management application, clicks away from the current screen on the user interface, and / or a certain duration of automatically ending the data session expires.

[0041] Figure 4An example distributed environment 400 for securely reflecting data from a data store is shown. The environment 400 shows a centralized data management application 402 (e.g., a data intelligence cloud, a data control center, etc.). Within the data management application, there is an edge management application 404, a data quality web application 406, and a data quality metastore 408. The DQ web application 406 is the main interaction point for the user. It cannot access the underlying data source. The DQ metastore 408 stores the data required for the operation of the DQ cloud application 402 (directories, users, management data, etc.), and the data required by the DQ cloud application to cache metadata from edge nodes / sites that is not considered sensitive data (aggregations, patterns, platform data, etc.). By storing such data in the DQ metastore library 408, read / write requests to the underlying local data store are reduced. Edge management 404 is the main gateway for users to interact with remote data sources. For example, when a user clicks on a data source or data from a user interface in the DQ web application 406, the DQ web application 406 sends a request to the edge management. Edge management queues the requests for pickup by target edge nodes / sites (eg, edge nodes marked as having / close to target data sources).

[0042] A wide area network (WAN) layer 410 allows communications to flow from edge sites 416 and 418 into the data management application 402. The customer network 412 includes at least one local computer cluster with a local data store and at least one edge site including a metastore. In this example environment 400, the customer network 412 includes a local computer cluster 414, which includes a local database. The network 412 also includes edge site A 416 and edge site B 418, both of which are located on a virtual local area network (LAN). Each edge site houses at least one edge controller, a data quality agent, a data quality service, and a metastore. The metastore is configured to communicate with the local data store and obtain metadata representing the underlying actual data stored in the local database. This metadata communication can then be transmitted back to the data management application 402 via the WAN layer 410. However, the underlying local data is maintained at the local computer cluster 414. The metastore is just a reflection of the underlying local data.

[0043] Edge nodes / sites 416 and 418 are where user instructions initiated in DQ cloud application 402 are actually executed. Figure 2 The components that run here are described in detail. The difference between edge nodes 416 and nodes 418 is what data the nodes are serving based on scoping requirements and / or scoping constraints set by the user. Depending on the number and location of each user's data stores, there may be one or more edge nodes.

[0044] Local computer clusters 414A and 414B represent computers that will perform the work of actually scanning the underlying data (source data) for quality. In some examples, as shown, there can be one local computer cluster for each edge node. In other examples, two different edge nodes can access the same local computer cluster, but have permissions to access only certain portions of the underlying data held in the local data storage (e.g., due to scoping constraints of each edge node).

[0045] The edge site metastore located in the edge node 416 / 418 can be configured to store data required to run the edge DQ! service. In addition, the metastore located in the edge node is where sensitive customer data extracted from one or more underlying data sources as a result of DQ checks and / or user instructions is stored. For example, a full-fidelity record example of a DQ check failure, particularly sensitive individual values ​​(such as account numbers, account balances, social security numbers, etc.) can be stored in the edge metastore.

[0046] The edge node / site metastore is isolated from the associated data store being served by the edge node, while the DQ metastore 408 serves all available data sources connected to the central data management application 402 .

[0047] The data quality stack represents both synchronous and asynchronous data flows with the metastore. The data flows can be distinguished by three different classes: user class, job class, and agent class. The user class can allow synchronous read / write operations performed by the user. An example of a synchronous read / write operation can be to extend the configuration via a user interface or an API (Application Programming Interface). The job class can allow asynchronous read / write operations. For example, a user who needs to view persistent data set profiles and statistics will use the job class read / write operations. The agent class can allow asynchronous read / write operations to manage job details (such as status updates).

[0048] In this example environment 400, jobs and proxy input and output operations may occur in the context of edge sites. For certain jobs, such as DQ check jobs, configuration from the metastore is required, such as connection details and model configuration. In other words, information about how the metastore is communicatively coupled to the local database and what type of metadata is being obtained may be required to establish certain classes of data flows.

[0049] In other examples, non-sensitive data may have a bidirectional flow between the data management application 402 and the customer's network 412 .

[0050] The metastore in the example environment 400 is used to serve the data management application 402 because it will contain metadata described in terms of the underlying local data. For non-sensitive data, read operations can be directly served from the local metastore hosted at the local computer cluster 414. For sensitive customer data, read operations can be delegated to the relevant edge sites depending on the scope characteristics of the data and the edge sites. Write operations can be essentially mirrored to the relevant edge sites. At the edge sites, write operations can be processed locally.

[0051] DQ metastore 408 mirrors the data management application 402 instance, except for certain data sets that do not reside at a given edge site. The local metastore on the local computer cluster 414 may contain sensitive customer data as a result of certain data processing and data profiling jobs. At the edge site, read operations can be served from the local metastore instance at the computer 414. Write operations can also occur locally. If the write operation targets non-sensitive data, the operation can then be mirrored to a persistent write-ahead log (WAL) backed by the local metastore instance. The published WAL can asynchronously publish entries to the data quality web application 406 running in the data management application 402 instance (i.e., the data management cloud or data intelligence cloud).

[0052] Figure 5 An example of a suitable operating environment in which one or more of the present embodiments may be implemented is shown. This is only an example of a suitable operating environment and is not intended to impose any limitation on the scope of use or functionality. Other well-known computing systems, environments, and / or configurations that may be suitable for use include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics (such as smart phones), network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0053] In its most basic configuration, operating environment 500 typically includes at least one processing unit 502 and memory 504. Depending on the exact configuration and type of computing device, memory 504 (which stores, among other things, information about detected devices, association information, personal gateway settings, and instructions for performing the methods disclosed herein) may be volatile (such as RAM), non-volatile (such as ROM, flash memory, etc.), or some combination of the two. Figure 5506. In addition, the environment 500 may also include storage devices (removable storage devices 508 and / or non-removable storage devices 510), which include but are not limited to magnetic or optical disks or tapes. Similarly, the environment 500 may also have one or more input devices 514 such as a keyboard, mouse, pen, voice input, etc. and / or one or more output devices 516 such as a display, speaker, printer, etc. The environment may also include one or more communication connections 512, such as LAN, WAN, point-to-point, etc.

[0054] Operating environment 500 typically includes at least some form of computer-readable media. Computer-readable media can be any available media that can be accessed by processing unit 502 or other devices including operating environment. As an example and not limitation, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other tangible medium that can be used to store the desired information. Computer storage media do not include communication media.

[0055] Communication media embody non-temporary computer-readable instructions, data structures, program modules or other data. Computer-readable instructions can be conveyed in a modulated data signal (such as a carrier wave or other transmission mechanism), and include any information delivery medium. The term "modulated data signal" means a signal, one or more of the characteristics of which are set or changed in a manner that encodes the information in the signal. As an example and not limitation, communication media include wired media (such as a wired network or a direct wired connection), and wireless media (such as acoustic, RF, infrared and other wireless media). Any combination of the above should also be included within the scope of computer-readable media.

[0056] Operating environment 500 can be a single computer operated in a networked environment using a logical connection to one or more remote computers. The remote computer can be a personal computer, a server, a router, a network PC, a peer device or other public network node, and generally includes some or all of the above elements and other elements not mentioned in this way. The logical connection can include any method supported by an available communication medium. Such a networked environment is common in offices, enterprise-wide computer networks, intranets and the Internet.

[0057] For example, various aspects of the present disclosure are described above with reference to the block diagrams and / or operational diagrams of the methods, systems, and computer program products according to various aspects of the present disclosure. The functions / actions marked in the boxes may not occur in the order shown in any flow chart. For example, depending on the functions / actions involved, two boxes shown in succession may actually be executed substantially simultaneously, or the boxes may sometimes be executed in reverse order.

[0058] The description and explanation of one or more aspects provided in this application are not intended to limit or constrain the scope of the disclosure claimed in any way. The aspects, examples and details provided in this application are considered to be sufficient to convey ownership and enable others to make and use the best form of the disclosure claimed. The disclosure claimed should not be interpreted as being limited to any aspect, example or detail provided in this application. Whether shown and described in combination or individually, it can be planned to selectively include or omit various features (structural and method features) in order to produce an embodiment with a specific feature set. Description and illustration of the application have been provided, and those skilled in the art can envision multiple variations, modifications and alternative aspects, which fall within the spirit of the broader aspects of the overall inventive concept embodied in this application, and do not depart from the broader scope of the disclosure claimed.

[0059] From the foregoing it will be appreciated that specific embodiments of the invention have been described herein for purposes of illustration, but that various modifications may be made without departing from the scope of the invention. Accordingly, the invention is not to be limited, except as by the appended claims.

Claims

1. A system for reflecting data from at least one data storage, comprising: a memory configured to store non-transitory computer-readable instructions; and a processor communicatively coupled to the memory, wherein the processor, when executing the non-transitory computer-readable instructions, is configured to: receiving at least one user instruction, wherein the at least one user instruction targets the at least one data store; transmitting the at least one user instruction to a queue associated with an edge node, wherein the edge node is associated with the at least one data store, and wherein the edge node polls the queue to receive the at least one user instruction; Initiating an encrypted data session with the edge node; receiving, via the encrypted data session, reflected data retrieved from the at least one data store, wherein the reflected data store is in at least one volatile memory location associated with the encrypted data session; and While the encrypted data session is ongoing, the reflected data is displayed.

2. The system of claim 1, the processor further configured to erase the reflected data from the at least one volatile memory location when the encrypted data session is disconnected. 3 . The system of claim 1 , wherein the processor is further configured to: receive at least one data quality result from a metastore. 4 . The system of claim 3 , wherein the processor is further configured to transmit a second user instruction to the edge node based on at least one data quality result from the metastore.

5. The system of claim 3, wherein the at least one data quality result is an error notification associated with the at least one data store.

6. The system according to claim 1, wherein: The at least one user instruction is deserialized at the edge node prior to execution at the at least one data store.

7. The system according to claim 1, wherein: The reflected data is encrypted prior to transmission over the encrypted data session.

8. The system according to claim 1, wherein: The at least one user instruction is at least one of: a read command and a write command.

9. The system according to claim 1, wherein the processor is further configured to: generate a meta-storage.

10. The system of claim 9, wherein the metastore includes at least one scoping requirement associated with underlying data stored in the at least one data store.

11. The system according to claim 9, wherein: The metastore is associated with a second data store.

12. The system of claim 1, wherein: The at least one user instruction is a view command for viewing information associated with a plurality of data stores in a unified user interface.

13. The system of claim 12, the processor further configured to: based on the view command, display platform data associated with the plurality of data storages in the unified user interface.

14. The system according to claim 9, wherein: The metastore includes at least one of: a pattern, a user instruction, a DQ analysis result, and platform data.

15. A method for reflecting data from a data storage, comprising: connecting at least one edge node to the data storage; transmitting a first user instruction to process data stored in the data storage for data quality; receiving at least one data quality result from the at least one edge node; transmitting a second user instruction to read a portion of the data stored in the data storage based on at least one data quality result received from the at least one edge node; Initiate an encrypted data session; receiving reflected data from the at least one edge node via the encrypted data session; and The reflected data is displayed on a user interface.

16. The method according to claim 15, wherein: While the reflected data is being displayed on the user interface, the reflected data is stored in volatile memory storage.

17. The method according to claim 16, further comprising: Upon disconnection of the encrypted data session, the reflected data is erased from the volatile memory storage.

18. The method according to claim 16, further comprising: Upon disconnecting the user interface, the reflected data is erased from the volatile memory storage.

19. A computer readable medium storing non-transitory computer executable instructions which, when executed, cause a computing system to perform steps for reflecting data from a data store, the steps comprising: connecting at least one edge node to a data storage; transmitting a first user instruction to process data stored in the data storage for data quality; receiving at least one data quality result from the at least one edge node; transmitting a second user instruction to read a portion of the data stored in the data storage based on at least one data quality result received from the at least one edge node; Initiate an encrypted data session; receiving reflected data from the at least one edge node via the encrypted data session; and The reflected data is displayed on a user interface.

20. The computer-readable medium of claim 19, wherein at least one data quality result from the at least one edge node is an error notification.

Images