Data integration for distributed and massively parallel processing environments

Abstract

Methods and systems for large scale data integration in distributed or massively parallel environments comprises a development phase wherein the results of a proposed jobflow can be viewed by the user during development, including the results of upstream units where the data sources and data targets can be any of a variety of different platforms, and further comprises the use of remote agents proximate to those data sources and data targets with direct communication between the associated agents under the direction of a topologically central controller to provide, among other things, improved security, reduced latency, reduced bandwidth requirements, and faster throughput.

Description

RELATED APPLICATION[0001]This application is a Continuation of U.S. patent application Ser. No. 16 / 611,199 filed on Nov. 5, 2019, which claims priority to and the benefit of International Patent Application No. PCT / US2018 / 031220 filed May 4, 2018, which claims priority to U.S. provisional patent application No. 62 / 502,594 filed 5 May 2017, each of which are incorporated herein by reference in their entirety.BACKGROUNDField of the Invention[0002]The present invention relates generally to data integration in either distributed or massively parallel processing environments, and, in one aspect, more particularly relates to interactive development of extract, load and transform data flows while, in another aspect, relates to the use of geographically dispersed agents to simplify extract, load and transform processes with enhanced security and improved data compression.Related Art[0003]More and more, data analysts require the use of data outside the control of their own organizations. Gre...

AI Technical Summary

Benefits of technology

The present invention provides a user interface and supporting data integration structure and process that allows users to easily build set-based transformation rules for integrating data from multiple sources. Users can understand the outcome of each transformation rule as it is applied. The invention also includes stateless agents that assist in extracting, loading, and transforming data in a highly distributed network of data systems, resulting in increased efficiency and speed without requiring that all data be kept in a single repository. The metadata rules can be maintained in any convenient location accessible by one or more controller components. Multiple agents can be implemented within a single server farm for load balancing and further efficiency. Overall, the invention provides an improved data integration solution.

Problems solved by technology

However, traditional database techniques have not generally focused on the challenges that result from trying to mine data from large repositories that are not organized for such searching, for linking what is found to other data, or for reusing and repurposing the data without massive effort.

An increasing number of challenges are not amenable to solution by these conventional techniques.

This conflict is magnified when the objective is to apply data analytics to unstructured or diversely structured datasets.

While sufficient data integration can overcome at least some of this conflict, conventional approaches to such data integration typically result in unworkable complexity and a lack of transparency that hinders or prevents successful debugging of transformation logic.

The result is that attempts at efficient integration of large datasets from diverse sources has been largely unsuccessful.

Further, data integration has typically involved moving large amounts of data across relatively long distances.

Given the confidential and proprietary nature of such data, these movements have historically run the risk of exposing confidential information to third parties.

While various encryption techniques have been used, the challenges of encrypting large data sets for transmission across long distances can be daunting.

Compression techniques have been used in the past, but again the challenges can become daunting because of the volume of compression typically needed and the security risks involved concerning both privacy and confidentiality.

Images