Behavior-consistent cluster-wide data wrangling based on locally processed sample data

Abstract

Exemplary embodiments include a system for behaviorally consistent data organization, a computer-readable storage medium storing at least one program, and a computer-implemented method. The local client device selects the original sample data set from the remote database. The local execution engine then applies one or more local data wrangling operations to the raw sample data. If the results of the local data wrangling operation are satisfactory, the local data wrangling operation may be transferred to the remote data wrangling cluster. A remote runtime engine run by a remote data wrangling cluster then applies the data wrangling operations to the larger raw data set from which the sample raw data was obtained. Because the remote execution engine and the local execution engine are of the same type, the data wrangling behavior exhibited by the local execution engine is reflected in the data wrangling behavior of the remote execution engine.

Description

technical field [0001] Exemplary embodiments of the present application relate generally to data processing, and more specifically, to large-scale data wrangling based on data wrangling performed on smaller subsets. Background technique [0002] Data wrangling is the process of transforming or mapping data from a "raw" form into a form that allows for more convenient use of the data. Such uses may include further wrangling, data visualization, data aggregation, and training statistical models, among many other possible uses. Data wrangling sometimes follows a collection of basic steps that begin with extracting data in raw form from a data source, "cleaning" the raw data using various hardware and / or software modules, parsing the data into predetermined data structures, and converting the resulting structured content to Stored in an accessible database for storage and future use. [0003] Data wrangling is typically performed on large data sets and can be performed by usin...

AI Technical Summary

Problems solved by technology

In other words, although one execution engine may be suitable for large data sets and a second execution engine may be suitable for small data sets, there is no guarantee that the behavior of the first and second execution engines will be equal

Therefore, even though the requested data wrangling operation is the same, the output of the first execution engine and the second execution engine may be different

Images