Flexible extract, transform, and load (ETL) processing
The flexible ETL process addresses inefficiencies in data integration by automating data mapping and lineage management within a data catalog, enhancing scalability and performance in big data environments.
Patent Information
- Application Number
- JP2024080610
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-19
- Filing Date
- 2024-05-17
- Publication Date
- 2024-11-29
AI Technical Summary
Existing ETL processes require extensive custom modeling and manual data lineage management, leading to inefficiencies and increased latency in data integration and analysis, especially in big data environments.
A flexible ETL process that configures data lineage within a data catalog to define data movement, automating the mapping of data from various sources to target schemas without custom coding, thereby reducing manual efforts and enhancing scalability and accuracy.
This approach reduces the need for custom coding, accelerates integration with new data sources, minimizes latency, and maintains data consistency, while improving performance and reducing resource requirements.
Smart Images

Figure 2024167074000001_ABST
Abstract
Description
[Technical field]
[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application is a non-provisional application of and claims the benefit of U.S. Provisional Patent Application No. 63 / 503,399, filed May 19, 2023, the entire contents of which are incorporated herein by reference.
[0002] FIELD OF THEINVENTION According to modern thinking in data analytics, the need to transform data from big data stores is greatly reduced by following the practice of extract, transform, and load (ETL). In general, ETL is a data integration process that combines data from multiple data sources into a single, consistent data store that is loaded into a data warehouse or other target system. The ETL process provides the basis for data analytics and machine learning workstreams. Through a set of logical rules, the ETL process cleanses and organizes data to address specific information needs as well as more advanced analytics that may improve back-end processes or end-user experience. Summary of the Invention
[0003] Disclosed herein are systems, as well as associated methods, computing devices, and computer-readable media. For example, in some embodiments, a method for ETL processing performed by a processing device includes receiving, from a data catalog, field mappings between application data and a target schema, receiving, from an ETL processing queue, a signal indicating that a record or data file including metadata and associated with the application data is ready for processing, determining source data by processing the metadata to identify a location of the record or data file and retrieving the record or data file from the identified location, and providing the source data to a big data table defined according to the target schema. [Brief description of the drawings]
[0004] BRIEF DESCRIPTION OF THE DRAWINGS The embodiments will be readily understood from the following detailed description taken in conjunction with the accompanying drawings, in which: For ease of description, like reference numerals denote like structural elements, and in which: FIG. [Figure 1] 1 illustrates an exemplary container diagram according to various embodiments. [Diagram 2] FIG. 2 is a block diagram of an example assistance module for flexible ETL processing according to various embodiments. [Diagram 3] FIG. 2 is a flow diagram of an example method for providing flexible ETL processing, according to various embodiments. [Figure 4] 1 is an example of a graphical user interface that may be used in performing some or all of the assistance methods disclosed herein, according to various embodiments. [Diagram 5] FIG. 1 is a block diagram of an example computing device that may perform some or all of the assistance methods disclosed herein, according to various embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0005] Data analytics typically requires extensive custom modeling to support specific use cases. Even in data warehousing, where online analytical processing (OLAP) structures are created to support a wide range of use cases, modeling these structures and populating the ETL pipeline requires specific knowledge of the data sources and custom modeling of the fact and dimension tables to provide that range of applicability. Furthermore, when platform teams are tasked with this custom modeling and specialized pipeline development, they risk getting caught in an endless cycle of custom development, complicating platform deployment and compromising platform adoption.
[0006] Moreover, moving data into big data stores often requires custom coding and scheduling of ETL jobs. Similarly, tracking data lineage in a data catalog requires linking the data catalog to data connectors and / or ETL tools to retrieve data mappings. Otherwise, manual curation of data lineage may be required. Typically, such techniques are employed in managing traditional enterprise big data stores. However, managing big data on a data platform by applying such enterprise data techniques is not particularly efficient or manageable. In some embodiments, the techniques described herein solve such problems by reversing the order of operations in traditional enterprise data techniques. For example, in some embodiments, data lineage is configured first in the data catalog, thereby defining the data movement that the ETL process implements. This allows for flexible definition of ETL jobs that can be broadly applicable and / or scalable across various data sources, data types, and / or target systems. Thus, the flexible ETL process described herein can reduce the amount of custom coding required. Furthermore, because data lineage is defined first in the data catalog, such curation of data relationships can remain accurate and up-to-date.
[0007] Thus, the described system and method provide a flexible ETL process that employs the data lineage capabilities of the data catalog. In general, the data lineage employed defines the data mapping required by the ETL process and tracks how data moves from data sources, such as files or operational databases, to, for example, big data tables. In some embodiments, the flexible ETL process uses metadata about data sources and targets from the data catalog to move input data from sources to targets without custom coding. Implementing a flexible ETL process reduces the amount of effort required to bring in new data sources for analysis and reporting purposes. Furthermore, implementing a flexible ETL process accelerates integration with new data sources, reduces the amount of code to maintain, and / or automatically aligns with new data catalogs. Furthermore, implementing a generic ETL reduces latency and memory requirements, and promotes optimization of operational databases.
[0008] In some embodiments, the described flexible ETL process is used to address the need to provide the ability to query data across many separate data files without having to read each file individually in a batch process, to efficiently combine data from various sources, and to provide the ability to visualize the data in a dashboard to address use cases that are not necessarily dependent on a particular application.
[0009] Implementing the flexible ETL process described herein improves many traditional information technology areas, such as ETL, data lakes, lakeshore marts, and business intelligence, all of which are typical components of enterprise computing. Deploying such technology can be difficult on platforms that utilize multiple deployment models and lack dedicated support staff to respond immediately to alerts, as may be common in enterprise environments. Furthermore, unlike enterprise deployments, where it may be relatively easy to add additional commodity hardware to a cluster as needed, scaling such deployments to improve performance can also be difficult. As a result, the flexible ETL process and associated architecture described herein maximizes performance using minimal resources, while significantly reducing and / or eliminating the need for periodic assistance to maintain data consistency and / or availability.
[0010] The need to track data movement from source files to queryable tables coincides with the need to describe data within the platform in the form of a data catalog. To address these and other technical issues, a flexible ETL process maintains a mapping between source data files and target tables, thereby enabling how data transfer to big data stores should be performed. In addition, the flexible ETL process provides provenance information about data movement within the platform. Furthermore, the representation of document types, associated schemas, and data fields in the data catalog are accompanied by semantic information that defines the meaning of the data, enabling interchange of data within and across platforms.
[0011] Specific Definitions In the following detailed description, reference is made to the accompanying drawings which form a part of this specification, in which like numerals designate like parts throughout and which show, by way of illustration, embodiments which may be implemented. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense.
[0012] Various operations may be described as multiple separate actions or operations, in the order that is most helpful in understanding the subject matter disclosed herein. However, the order of description should not be construed as implying that these operations are necessarily order dependent. In particular, these operations may not be performed in the order presented. The operations described may be performed in a different order than in the described embodiment. Various additional operations may be performed and / or described operations may be omitted in additional embodiments.
[0013] For purposes of this disclosure, the phrases "A and / or B" and "A or B" mean (A), (B), or (A and B). For purposes of this disclosure, the phrases "A, B, and / or C" and "A, B, or C" mean (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). Although some elements may be referred to in the singular (e.g., a "processing device"), any suitable element may be represented by multiple instances of that element, and vice versa. For example, a set of operations described as being performed by a processing device may be implemented with different ones of the operations performed by different processing devices.
[0014] This description uses the phrases "an embodiment," "various embodiments," and "several embodiments," each of which may refer to one or more of the same or different embodiments. Additionally, terms such as "comprising," "including," and "having" used with respect to embodiments of the present disclosure are synonymous. When used to describe a range of dimensions, the phrase "between X and Y" represents a range that includes X and Y. As used herein, an "apparatus" may refer to any individual device, a collection of devices, a portion of a device, or a collection of portions of devices. The drawings are not necessarily drawn to scale.
[0015] As used herein, the term "data catalog" includes, but is not limited to, a collection of metadata associated with file types of documents managed by document services and other services, including, for example, semantic definitions.
[0016] As used herein, the term "ontology" includes, but is not limited to, a set of entities, including classes of entities, and relationships between entities over well-defined properties.
[0017] As used herein, the term "knowledge graph" includes, but is not limited to, a collection of entity descriptions linked by meaningful relationships, typically stored in a graph database aligned to an ontology, for example.
[0018] As used herein, the term "triple store" or "semantic triple store" includes, but is not limited to, no structured query language (NoSQL) databases that store data primarily in the form of triples representing subject, predicate, and object, and provide query capabilities, typically using the SPARQL protocol, also known as "graph databases."
[0019] As used herein, the term "data catalog service" includes, but is not limited to, a service that manages metadata associated with a data set, such as a file stored in a document repository or a table in a database, including, for example, semantic definitions. A data catalog service may store metadata about how data should be moved, for example, to a big data store or a relational database server.
[0020] As used herein, the term "search index" includes, but is not limited to, a service that optimizes the storage of metadata for rapid identification of entities based on search criteria, such as full-text search.
[0021] As used herein, the term "ETL tool" includes, but is not limited to, software designed to extract, transform, and load data, for example.
[0022] As used herein, the term "ETL processing queue" includes, but is not limited to, a channel in a message broker, such as Kafka or RabbitMQ, that contains references to documents or datasets that need to be processed in an ETL pipeline.
[0023] As used herein, the terms "database server" or "relational database server" include, but are not limited to, systems that store and optimize tabular data for rapid querying, for example, using a defined query language.
[0024] As used herein, the term "big data store" includes, but is not limited to, a data repository for storing a large number of files for analysis, for example, typically in original data format, but possibly in optimized big data file formats such as ORC or Parquet. In various implementations, a "big data store" may also be referred to as a "data lake." In some embodiments, the large number of files is on the order of one million or more files.
[0025] Example Flexible ETL Process In an exemplary embodiment, the flexible ETL process can be outlined as follows: An application registers in the Data Catalog the data schema of its source data, the schema of the target big data table, and / or the field mapping between such source and target schemas. In general, the field mapping describes how persistent fields (e.g., from the sources) are mapped to the target schema. The ETL process reads these field mappings from the Data Catalog and creates the defined target big data tables (e.g., if they do not already exist) based on the field mappings. If a new source data file or record is available, the application registers it in the Data Catalog. The Data Catalog aligns the new data file or record with the appropriate schema (e.g., during configuration). If the new data file or record is associated with a schema mapped to a big data table, the Data Catalog inserts the record into the ETL processing queue. If the new data file or record is not associated with a schema mapped to a big data table, the Data Catalog registers the availability of the new data. The ETL tool reads the record from the ETL processing queue, which contains information about how the data should be mapped to the big data table. ETL tools process the input data into records and transfer the records to the appropriate big data tables.
[0026] Exemplary categories of source data for which a flexible ETL process should be employed include data in standard file formats such as CSV, JSON, and XML that can be easily parsed, data ingested from relational databases in the form of individual records, data in standard scientific file formats such as SPC or GAML for which data parsers are readily available, etc. In various implementations, the source data includes database files. For example, the source data may include relational database files such as MySQL® files, PostgreSQL files, Microsoft SQL Server files, Oracle database files, SQLite files, etc. In some examples, the database files are scientific instrument result files. For example, the result files may include Proteome Discoverer result files and / or Compound Discoverer™ result files.
[0027] Thus, the flexible ETL process improves computing technology by automating the mapping and / or import of dissimilar source data (e.g., source data having different file formats, different data structures, different schemas, etc.) to the same target schema (e.g., schema of a target database, target data table in a big data store) or to different target schemas (e.g., schema of different target databases, or different target data tables in the same big data store, etc.). By automating the mapping process between data fields of dissimilar source data and target schemas, the flexible ETL process reduces or eliminates errors that may occur due to laborious manual mapping processes, improving the overall reliability and speed of importing large and diverse source data into target data tables, databases, etc.
[0028] For example, the diverse source data may use different formats (e.g., CSV, JSON, XML, etc.) and / or have different structures (e.g., hierarchical, relational, or flat). The flexible ETL process systematically and automatically maps fields of the diverse data sources to the target schema, ensuring that different source data having different formats can be smoothly integrated into the target schema. Furthermore, as the diversity of source data increases, manually mapping various source data to the target schema may become unsustainable. Because the automatic mapping performed by the flexible ETL process can handle a diverse range of source data types, the flexible ETL process scales efficiently to increase the volume and type of source data being processed without a corresponding increase in error rates.
[0029] System Context In some embodiments, the flexible ETL process is provided via a platform that encompasses both the operational and analytical capabilities described herein. The operational capabilities include, for example, document services (see FIG. 1). In some embodiments, the analytical capabilities provided support multiple document repositories provided via a number of different platforms.
[0030] In some embodiments, applications store data in the platform via the document service. In some embodiments, the analytics function provides a default intelligence tool that pulls some of the data from the document service into a separate storage optimized for big data queries and allows the data to be presented on a dashboard for viewing by users of the application. In some embodiments, some applications may query the data within the analytics function optimized to meet a particular application use case. In some embodiments, there are several different paths through which data can be ingested or accessed for analytical purposes: (1) document data flow, (2) record data flow, and (3) virtual data flow. These are described separately in the following sections.
[0031] Document Data Flow FIG. 1 illustrates an exemplary container diagram 100 showing which elements of the analytics functionality 150 interact with the platform and its applications (not shown). In some embodiments, the document service 122 is responsible for registering documents and other datasets located within the platform with the data catalog service 104, including access URLs where the actual documents can be retrieved. In some embodiments, multiple data repositories within the platform or outside the platform register documents with the data catalog service 104. Other services may also register metadata with the data catalog service 104, for example, by linking the metadata to either a dataset or an entity related to the dataset, such as an injection or sequence. As illustrated, the primary data storage for the data catalog service 104 may be a semantic triple store 106. In various implementations, the semantic triple store 106 includes a web-based semantic triple store, such as an Apache Jena RDF triple store hosted by the Apache Jena Fuseki server. In some examples, the semantic triple store 106 includes a web-based triple store implemented using the Eclipse RDF4J framework. However, other database implementations may be employed depending on the details of the project.
[0032] In some embodiments, the search index 102 provides fast search capabilities across the indexed data sets in the data catalog (e.g., by keywords or search strings where the keys of the search index data are identifiers of entities in the semantic triple store 106). In some embodiments, the search index 102 is exposed to a platform, or an application 120 on that platform, to perform searches against metadata in the data catalog, and the results can be sent to a data catalog service 104 to pull associated metadata.
[0033] In some embodiments, the data catalog service 104 provides a signal to a downstream data pipeline that a new or updated document is ready to be processed. For example, the data catalog service 104 sends a message to the ETL processing queue 108, where the ETL tool 110 can pick up the message. In some embodiments, the ETL tool 110 (e.g., Apache NiFi) introduces the document to its pipeline by instantiating a listener, for example, via FTP, HTTP, or GRPC.
[0034] In some embodiments, the ETL tool 110 ensures that documents are queued for processing more reliably through an independent queue that decouples the data catalog service 104 from the ETL tool 110. In some embodiments, the ETL tool 110 processes source documents from one of the following: documents in a common file format such as CSV, JSON, or XML; documents in a standard analytical file format such as SPC, GAML, or Allotrope ADF; or documents or datasets in a proprietary format. In some embodiments, the first two categories are managed by a generic ETL pipeline, as long as the schema definitions for the source files and target tables are defined in the data catalog. In some embodiments, the third category is managed by a dedicated ETL pipeline for each proprietary format to be processed.
[0035] In some embodiments, the ETL tool 110 retrieves messages from the ETL processing queue 108 indicating that a new document needs to be processed. In some embodiments, the messages include information about the media type of the document, which defines what kind of document it is, and thus how the document is processed in the ETL pipeline. In some embodiments, for documents that can be processed through a generic ETL pipeline, the ETL tool 110 retrieves schema mapping information from the data catalog service 104 and directs the document to the appropriate path in the pipeline, or to a dedicated path in the pipeline if the document is in one of the proprietary formats. In some embodiments, the ETL tool 110 stores the data in a big data store 112.
[0036] Recording Data Flow The above description of the relationships between the containers in FIG. 1 focuses on ingestion of document-based data (e.g., documents and files). However, the same architecture can support data ingestion based on records, such as change data ingestion records from a relational database. In addition to metadata, other services that track data changes outside of files can also send records through the data catalog service 104, and in some embodiments, these records would not be processed into the semantic triple store 106, but would be passed to the ETL processing queue 108 as records to be processed by the ETL tool 110 and stored in the big data store 112. As a result, the example architecture illustrated in FIG. 1 can support new use cases, such as analytics or artificial intelligence (AI) / machine learning (ML) use cases, where operational data other than documents or files needs to be forwarded to the analytics function 150 to offload such query patterns from the source database, for example, to ensure that they do not interfere with the operational functions.
[0037] Virtual Data Flow Further, the architecture shown in FIG. 1 can be employed to support a virtual data flow model, which also employs an analytical engine 114 (e.g., Trino®) and a relational data store, such as an application database, to perform heterogeneous joins between the relational store and the data in the big data store 112. In some embodiments, the analytical engine 114 is designed to reduce or eliminate the need to move data from the relational store to perform certain analytical queries. In such embodiments, the configuration of the connection from the analytical engine 114 to the relational data store is performed at the time of deployment of the analytical function 150. In some embodiments, a service provided by the platform registers the definition of the relational data store with the data catalog service 104 to ensure that the data semantics are extended (by the analytical engine 114) into queryable elements. In some embodiments, an application 120 is employed that provides visualizations (e.g., charts) for the data in the big data store 112. However, in certain use cases, the application 120 can also be configured to directly query the analytical engine 114.
[0038] Many of the applications 120 described above correspond to software packages that are installed and configured (e.g., on the computing device 4000 of FIG. 5). For example, in some embodiments, the analytics engine 114 is configured to interface with the big data store 112 and the metastore service 118 (e.g., Hive®), while the intelligence tool 116 is configured to interface with the analytics engine 114 to provide chart and dashboard designs to the application user 101 (e.g., via the display device 4010 discussed herein with reference to FIG. 5). In some embodiments, the ETL jobs in the ETL tool 110 take the form of configurations in the intelligence tool 116.
[0039] FIG. 2 is a block diagram of an assistance module 1000 for performing an assistance operation, according to various embodiments. The assistance module 1000 may be implemented by a circuit (e.g., including electrical and / or optical components) such as a programmed computing device. The logic of the assistance module 1000 may be included in a single computing device or may be distributed across multiple computing devices in communication with each other as needed. Examples of computing devices that may implement the assistance module 1000, alone or in combination, are discussed herein with reference to the computing device 4000 of FIG. 5. In some embodiments, such computing devices are interconnected and the assistance module 1000 may be implemented across one or more of the computing devices.
[0040] The assistance module 1000 may include mapping logic 1002, receiving logic 1004, source data logic 1006, and / or persistence logic 1008. As used herein, the term "logic" may include an apparatus that performs a set of operations associated with the logic. For example, any of the logic elements included in the assistance module 1000 may be implemented by one or more computing devices programmed with instructions that cause one or more processing devices of the computing devices to perform the set of associated operations. In certain embodiments, the logic elements may include one or more non-transitory computer-readable media having instructions that, when executed by one or more processing devices of the one or more computing devices, cause the one or more computing devices to perform the set of associated operations. As used herein, the term "module" may refer to a collection of one or more logic elements that together perform one or more functions associated with the module. Different ones of the logic elements in a module may take the same form or different forms. For example, some logic in a module may be implemented by a programmed general-purpose processing device, while other logic in a module may be implemented by an application-specific integrated circuit (ASIC). In another example, different ones of the logic elements in a module may be associated with different sets of instructions executed by one or more processing devices. A module may not include all of the logic elements shown in an associated drawing, e.g., a module may include a subset of the logic elements shown in an associated drawing when that module performs a subset of the operations discussed herein with reference to that module.
[0041] The mapping logic 1002 may be configured to map fields between the application data and the target schema deployed in a data catalog or data catalog service (e.g., data catalog service 104 of FIG. 1). In some embodiments, the field mappings are stored in the data catalog by an application or server configured to use the application data to populate the data catalog with new records or new source data files, if available. In some embodiments, the records or data files are aligned with the target schema by the data catalog. In some embodiments, the mappings are defined according to the type of application data.
[0042] The receiving logic 1004 may be configured to receive a signal from an ETL processing queue (e.g., ETL processing queue 108) indicating that a record or data file that includes metadata and is associated with application data is ready for processing. In some embodiments, the signal is inserted into the ETL processing queue by the data catalog. In some embodiments, the signal is inserted by the data catalog based on a type of record or data file that matches a type of application data. In some embodiments, the signal includes a location of the record or data file.
[0043] The source data logic 1006 may be configured to determine the source data by processing records or data files. The persistence logic 1008 may be configured to provide the source data to big data tables defined according to a target schema. In some embodiments, the source data is provided to the big data tables using metadata provided by the data catalog.
[0044] 3 illustrates a flow diagram of a method 2000 for performing an assistive operation, according to various embodiments. The operations of the method 2000 may be illustrated with reference to certain embodiments disclosed herein (e.g., the analysis functionality 150 discussed herein with reference to FIG. 1, the GUI 3000 discussed herein with reference to FIG. 4, and / or the computing device 4000 discussed herein with reference to FIG. 5), but the method 2000 may be used in any suitable setting to perform any suitable assistive operation. Although the operations are illustrated in FIG. 2 once each and in a particular order, the operations may be appropriately reordered and / or repeated as desired (e.g., different operations performed may be performed in parallel, as appropriate).
[0045] In the method 2000, a first operation may be performed at 2002. For example, the mapping logic 1002 of the assistance module 1000 may perform the operation of 2002. The first operation may include receiving a field mapping between the application data and the target schema from a data catalog.
[0046] A second operation may be performed at 2004. For example, the receiving logic 1004 of the assistance module 1000 may perform the operation of 2004. The second operation may include receiving a signal from an ETL processing queue indicating that a record or data file, including metadata and associated with application data, is ready for processing.
[0047] A third operation may be performed at 2006. For example, the source data logic 1006 of the assistance module 1000 may perform the operation of 2006. The third operation may include determining the source data by processing the metadata to identify locations of records or data files and retrieving the records or data files from the identified locations.
[0048] A fourth operation may be performed at 2008. For example, the persistence logic 1008 of the assistance module 1000 may perform the operation of 2108. The fourth operation may include providing the source data to a big data table defined according to a target schema.
[0049] The assistance methods disclosed herein may include interactions with a human user. Such interactions may include providing information to the user or providing options for the user to enter commands (e.g., configurations for the generic ETL, or applications that provide records and data files to be processed by the generic ETL), queries (e.g., to a local or remote database), or other information. In some embodiments, these interactions may be implemented through a graphical user interface (GUI) that includes a visual display on a display device (e.g., display device 4010, discussed herein with reference to FIG. 5) that provides output to the user and / or prompts the user to provide input (e.g., via one or more input devices, such as a keyboard, mouse, trackpad, or touch screen included in other I / O devices 4012, discussed herein with reference to FIG. 5). The assistance systems disclosed herein may include any suitable GUI for interacting with a user.
[0050] 4 illustrates an exemplary GUI 3000 that may be used in implementing some or all of the assistance methods disclosed herein, according to various embodiments. As noted above, the GUI 3000 may be provided on a display device of a computing device (e.g., display device 4010 discussed herein with reference to FIG. 5), and a user may interact with the GUI 3000 using any suitable input device (e.g., any of the input devices included in other I / O devices 4012 discussed herein with reference to FIG. 5) and input technology (e.g., cursor movement, motion capture, facial recognition, gesture detection, voice recognition, button actuation, etc.).
[0051] 4 is exemplary only, and any number and arrangement of regions containing any desired characteristics may be included in the GUI 3000. The data display region 3002 is capable of displaying data.
[0052] The data analysis area 3004 may display results of the data analysis (e.g., results of analyzing the data shown in the data display area 3002 and / or other data). For example, the data analysis area 3004 may display source data from an application stored in a big data table. In some embodiments, the data display area 3002 and the data analysis area 3004 may be combined in the GUI 3000.
[0053] The control area 3006 may include options that allow a user to control the implementation of the flexible ETL. The settings area 3008 may include options that allow a user to control the features and functionality of the GUI 3000 (and / or other GUIs) and / or perform common computing operations related to the data display area 3002 and the data analysis area 3004 (e.g., saving the data on a storage device, such as the storage device 4004 discussed herein with reference to FIG. 5, sending the data to another user, labeling the data, etc.).
[0054] As described above, the assistance module 1000 may be implemented by one or more computing devices. Figure 5 is a block diagram of a computing device 4000 that may implement some or all of the assistance methods disclosed herein, according to various embodiments. In some embodiments, the assistance module 1000 may be implemented by a single computing device 4000 or multiple computing devices 4000.
[0055] 5 is illustrated as having several components, any one or more of which may be omitted or duplicated as appropriate for the application and setting. In some embodiments, some or all of the components included in computing device 4000 may be mounted on one or more motherboards and encapsulated in a housing (e.g., comprising plastic, metal, and / or other materials). In some embodiments, such components may be fabricated on a single system-on-a-chip (SoC) (e.g., an SoC may include one or more processing devices 4002 and one or more storage devices 4004). Additionally, in various embodiments, the computing device 4000 may not include one or more of the components illustrated in FIG. 5, but may include interface circuitry (not shown) for coupling to one or more components using any suitable interface (e.g., a Universal Serial Bus (USB) interface, a High-Definition Multimedia Interface (HDMI) interface, a Controller Area Network (CAN) interface, a Serial Peripheral Interface (SPI) interface, an Ethernet interface, a wireless interface, or any other suitable interface). For example, the computing device 4000 may not include a display device 4010, but may include display device interface circuitry (e.g., connectors and driver circuits) to allow the display device 4010 to be coupled.
[0056] The computing device 4000 may include a processing device 4002 (e.g., one or more processing devices). As used herein, the term "processing device" may refer to any device or portion of a device that processes electronic data from registers and / or memory and converts the electronic data into other electronic data that may be stored in registers and / or memory. The processing device 4002 may include one or more digital signal processors (DSPs), application specific integrated circuits (ASICs), central processing units (CPUs), graphics processing units (GSs), or other integrated circuits (ICs). The processing device may include a graphics processing unit (GPU), a cryptographic processor (a dedicated processor that executes cryptographic algorithms in hardware), a server processor, or any other suitable processing device.
[0057] The computing device 4000 may include a storage device 4004 (e.g., one or more storage devices). The storage device 4004 may include random access memory (RAM) (e.g., static RAM). The storage device 4004 may include one or more memory devices, such as a static RAM (SRAM) device, a magnetic RAM (MRAM) device, a dynamic RAM (DRAM) device, a resistive RAM (RRAM) device, or a conductive-bridging RAM (CBRAM) device, a hard drive-based memory device, a solid-state memory device, a network drive, a cloud drive, or any combination of memory devices. In some embodiments, the storage device 4004 may include memory that shares a die with the processing device 4002. In such embodiments, the memory may be used as a cache memory and may include, for example, an embedded dynamic random access memory (eDRAM), or a spin transfer torque magnetic random access memory (STT-MRAM). In some embodiments, the storage device 4004 may include a non-transitory computer-readable medium having instructions that, when executed by one or more processing devices (e.g., the processing device 4002), cause the computing device 4000 to perform any suitable method or portion of the methods disclosed herein.
[0058] The computing device 4000 may include an interface device 4006 (e.g., one or more interface devices 4006). The interface device 4006 may include one or more communication chips, connectors, and / or other hardware and software to manage communications between the computing device 4000 and other computing devices. For example, the interface device 4006 may include circuitry to manage wireless communications to transfer data to and from the computing device 4000. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communication channels, etc. that enable communication of data using modulated electromagnetic radiation through a non-solid medium. This term does not imply that the associated device does not include any wiring, although in some embodiments it may not. The circuitry included in the interface device 4006 for managing wireless communications may implement any of a number of wireless standards or protocols, including, but not limited to, Wi-Fi (IEEE 802.11 family), Institute for Electrical and Electronic Engineers (IEEE) standards including the IEEE 802.16 standard (e.g., the IEEE 802.16-2005 Amendment), the Long-Term Evolution (LTE) project with any amendments, updates, and / or revisions (e.g., the Advanced LTE project, the Ultra-Mobile Broadband (UMB) project (also referred to as “3GPP®2”), etc.).In some embodiments, the circuitry included in the interface device 4006 for managing wireless communications may operate in accordance with a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE network. In some embodiments, the circuitry included in the interface device 4006 for managing wireless communications may operate in accordance with an Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). In some embodiments, circuitry included in interface device 4006 for managing wireless communications may operate according to Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, as well as any other wireless protocols designated as 3G, 4G, 5G, and beyond. In some embodiments, interface device 4006 may include one or more antennas (e.g., one or more antenna arrays) for receiving and / or transmitting wireless communications.
[0059] In some embodiments, the interface device 4006 may include circuitry for managing wired communications, such as electrical, optical, or any other suitable communication protocol. For example, the interface device 4006 may include circuitry supporting communications according to Ethernet technology. In some embodiments, the interface device 4006 may support both wireless and wired communications, and / or support multiple wired and / or multiple wireless communications protocols. For example, a first set of circuits of the interface device 4006 may be dedicated to short-range wireless communications, such as Wi-Fi or Bluetooth, and a second set of circuits of the interface device 4006 may be dedicated to long-range wireless communications, such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, etc. In some embodiments, the first set of circuits of the interface device 4006 may be dedicated to wireless communications, and a second set of circuits of the interface device 4006 may be dedicated to wired communications.
[0060] The computing device 4000 may include a battery / power circuit 4008. The battery / power circuit 4008 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 4000 to an energy source separate from the computing device 4000 (e.g., AC line power).
[0061] The computing device 4000 may include a display device 4010 (e.g., multiple display devices). The display device 4010 may include any visual indicator, such as a heads-up display, a computer monitor, a projector, a touch screen display, a liquid crystal display (LCD), a light emitting diode display, or a flat panel display.
[0062] The computing device 4000 may include other input / output (I / O) devices 4012. The other I / O devices 4012 may include, for example, one or more audio output devices (e.g., a speaker, a headset, an earpiece, an alarm, etc.), one or more audio input devices (e.g., a microphone or microphone array), a location device (e.g., a GPS device that communicates with a satellite-based system to receive the location of the computing device 4000, as known in the art), an audio codec, a video codec, a printer, sensors (e.g., a thermocouple or other temperature sensor, a humidity sensor, a pressure sensor, a vibration sensor, an accelerometer, a gyroscope, etc.), an image capture device such as a camera, a cursor control device such as a keyboard, a mouse, a stylus, a trackball, or a touchpad, a barcode reader, a Quick Response (QR) code reader, or a radio frequency identification (RFID) reader.
[0063] The computing device 4000 may have any suitable form factor for the application and setting, such as a handheld or mobile computing device (e.g., a cell phone, a smartphone, a mobile Internet device, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a personal digital assistant (PDA), an ultra-mobile personal computer, etc.), a desktop or server computing device, or other networked computing component.
[0064] The following paragraphs provide various examples of the embodiments disclosed herein.
[0065] Example 1 is a method for ETL processing performed by a processing device, the method including: receiving, from a data catalog, field mappings between application data and a target schema, receiving, from an ETL processing queue, a signal indicating that a record or data file including metadata and associated with the application data is ready for processing, determining source data by processing the metadata to identify a location of the record or data file and retrieving the record or data file from the identified location, and providing the source data to a big data table defined according to the target schema.
[0066] Example 2 includes the subject matter of example 1 and further specifies that the field mappings are stored in a data catalog by an application or server.
[0067] Example 3 includes the subject matter of any of examples 1 and 2, further providing that the application is configured to use the application data.
[0068] Example 4 includes the subject matter of any of Examples 1-3, and further specifies that the application is configured to register new records or new source data files in the data catalog, if available.
[0069] Example 5 includes the subject matter of any of examples 1-4, further providing that the record or data file is registered in the data catalog as being ready for processing by the application.
[0070] Example 6 includes the subject matter of any of examples 1-5, further specifying that the records or data files are aligned with the target schema by a data catalog.
[0071] Example 7 includes the subject matter of any of examples 1-6, further specifying that the signal is inserted into the ETL processing queue by a data catalog.
[0072] Example 8 includes the subject matter of any of examples 1-7, further specifying that the mapping is defined according to a type of application data.
[0073] Example 9 includes the subject matter of any of examples 1-8, further specifying that the signal is inserted by the data catalog based on a record or data file type matching the application data type.
[0074] Example 10 includes the subject matter of any of examples 1-9, further specifying that the target schema is expanded into a target big data table in the data catalog.
[0075] Example 11 includes the subject matter of any of Examples 1-10, further providing that the record or data file is new or updated.
[0076] Example 12 includes the subject matter of any of examples 1-11, further providing that the signal includes a location of the record or data file.
[0077] Example 13 includes the subject matter of any of Examples 1-12, and further specifies that if the record or data file is not associated with a previously provided schema, the data catalog provides a mapping between the application data and the target schema.
[0078] Example 14 includes the subject matter of any of examples 1-13, further specifying that the source data is provided to the big data table using metadata provided by the data catalog.
[0079] Example 15 includes the subject matter of any of Examples 1-14, further specifying that the source data includes first source data and the target schema includes a first target schema, the method further including receiving from the data catalog a second field mapping between the second application data and the second target schema; receiving from the ETL processing queue a second signal including the second metadata and indicating that a second record or a second data file associated with the second application data is ready for processing; determining the second source data by processing the second metadata to identify a second location of the second record or the second data file from the identified second location, where the first source data and the second source data have different formats; and providing the second source data in a table defined according to the second target schema, where the first target schema and the second target schema are different schemas.
[0080] Example 16 includes the subject matter of any of Examples 1-15, further specifying that the source data includes first source data and the target schema includes a first target schema, the method further including receiving from the data catalog a second field mapping between the second application data and the second target schema; receiving from the ETL processing queue a second signal including the second metadata and indicating that a second record or a second data file associated with the second application data is ready for processing; determining the second source data by processing the second metadata to identify a second location of the second record or the second data file from the identified second location, where the first source data and the second source data have different formats; and providing the second source data in a second table defined according to the second target schema, where the first target schema and the second target schema are different schemas.
[0081] Example 17 is a system including a non-transitory computer-readable storage medium storing instructions and at least one electronic processor configured to execute the instructions to receive from a data catalog field mappings between application data and a target schema, receive from an ETL processing queue a signal indicating that a record or data file including metadata and associated with the application data is ready for processing, determine source data by processing the metadata to identify a location of the record or data file and retrieving the record or data file from the identified location, and provide the source data to a table defined according to the target schema.
[0082] Example 18 includes the subject matter of Example 17, and further specifies that the field mappings are stored in a data catalog by the application or server.
[0083] Example 19 includes the subject matter of any of examples 17 and 18, further providing that the application is configured to use the application data.
[0084] Example 20 includes the subject matter of any of Examples 17-19, and further specifies that the application is configured to register new records or new source data files in the data catalog, if available.
[0085] Example 21 includes the subject matter of any one of Examples 17-20, and further specifies that the record or data file is registered in the data catalog as being ready for processing by the application.
[0086] Example 22 includes the subject matter of any of Examples 17-21, further specifying that the records or data files are aligned with the target schema by a data catalog.
[0087] Example 23 includes the subject matter of any of examples 17-22, further specifying that the signal is inserted into the ETL processing queue by a data catalog.
[0088] Example 24 includes the subject matter of any of Examples 17-23, and further specifies that the mapping is defined according to a type of application data.
[0089] Example 25 includes the subject matter of any of Examples 17-24, further specifying that the signal is inserted by the data catalog based on a record or data file type matching the application data type.
[0090] Example 26 includes the subject matter of any of Examples 17-25, further specifying that the target schema is deployed to a target big data table in the data catalog.
[0091] Example 27 includes the subject matter of any of Examples 17-26, further providing that the record or data file is new or updated.
[0092] Example 28 includes the subject matter of any of examples 17-27, further providing that the signal includes a location of the record or data file.
[0093] Example 29 includes the subject matter of any of Examples 17-28, and further specifies that if the record or data file is not associated with a previously provided schema, the data catalog provides a mapping between the application data and the target schema.
[0094] Example 30 includes the subject matter of any of Examples 17-29, further specifying that the source data is provided to the big data table using metadata provided by the data catalog.
[0095] Example 31 includes the subject matter of any of Examples 1-14, and further specifies that the source data includes first source data and the target schema includes a first target schema, and the at least one electronic processor is configured to execute instructions to: receive from the data catalog a second field mapping between the second application data and the second target schema; receive from the ETL processing queue a second signal indicating a second record or a second data file associated with the second application data is ready for processing, the second signal including the second metadata, and the second record or the second data file is ready for processing; determine the second source data by processing the second metadata to identify a second location of the second record or the second data file from the identified second location, where the first source data and the second source data have different formats; and provide the second source data in a table defined according to the second target schema, where the first target schema and the second target schema are different schemas.
[0096] Example 32 includes the subject matter of any of Examples 1-14, and further specifies that the source data includes first source data and the target schema includes a first target schema, and the at least one electronic processor is configured to execute instructions to: receive from the data catalog a second field mapping between the second application data and the second target schema; receive from the ETL processing queue a second signal indicating a second record or a second data file associated with the second application data is ready for processing, the second signal including the second metadata, and the second record or the second data file is ready for processing; determine the second source data by processing the second metadata to identify a second location of the second record or the second data file from the identified second location, where the first source data and the second source data have different formats; and provide the second source data in a second table defined in accordance with the second target schema, where the first target schema and the second target schema are different schemas.
[0097] Example 33 is a non-transitory computer-readable storage medium including executable instructions that cause an electronic processor to receive from a data catalog field mappings between application data and a target schema, receive from an ETL processing queue a signal indicating that a record or data file including metadata and associated with the application data is ready for processing, determine source data by processing the metadata to identify a location of the record or data file and retrieving the record or data file from the identified location, and provide the source data in a table defined according to the target schema.
[0098] Example 34 includes the subject matter of example 33, and further specifies that the field mappings are stored in a data catalog by the application or server.
[0099] Example 35 includes the subject matter of any of Examples 33 and 34, further specifying that the application is configured to use the application data.
[0100] Example 36 includes the subject matter of any of Examples 33-35, and further specifies that the application is configured to register new records or new source data files in the data catalog, if available.
[0101] Example 37 includes the subject matter of any of Examples 33-36, and further specifies that the record or data file is registered in the data catalog as being ready for processing by the application.
[0102] Example 38 includes the subject matter of any of Examples 33-37, further specifying that the records or data files are aligned with the target schema by a data catalog.
[0103] Example 39 includes the subject matter of any of examples 33-38, further specifying that the signal is inserted into the ETL processing queue by the data catalog.
[0104] Example 40 includes the subject matter of any of Examples 33 to 39, and further specifies that the mapping is defined according to a type of application data.
[0105] Example 41 includes the subject matter of any of Examples 33-40, further specifying that the signal is inserted by the data catalog based on a record or data file type that matches the application data type.
[0106] Example 42 includes the subject matter of any of Examples 33-41, further specifying that the target schema is deployed to a target big data table in the data catalog.
[0107] Example 43 includes the subject matter of any of Examples 33-42, further specifying that the record or data file is new or updated.
[0108] Example 44 includes the subject matter of any of Examples 33-43, further providing that the signal includes a location of the record or data file.
[0109] Example 45 includes the subject matter of any of Examples 33-44, and further specifies that if the record or data file is not associated with a previously provided schema, the data catalog provides a mapping between the application data and the target schema.
[0110] Example 46 includes the subject matter of any of Examples 33-45, further specifying that the source data is provided to the big data table using metadata provided by the data catalog.
[0111] Example 47 includes the subject matter of any of Examples 1-14, and further specifies that the source data includes first source data and the target schema includes a first target schema, and the executable instructions cause the electronic processor to: receive from the data catalog a second field mapping between the second application data and the second target schema; receive from the ETL processing queue a second signal indicating a second record or a second data file associated with the second application data is ready for processing, the second signal including the second metadata and the second record or the second data file is ready for processing; determine the second source data by processing the second metadata to identify a second location of the second record or the second data file from the identified second location, where the first source data and the second source data have different formats; and provide the second source data in a table defined according to the second target schema, where the first target schema and the second target schema are different schemas.
[0112] Example 48 includes the subject matter of any of Examples 1-14, and further specifies that the source data includes first source data and the target schema includes a first target schema, and the executable instructions cause the electronic processor to: receive from the data catalog a second field mapping between the second application data and the second target schema; receive from the ETL processing queue a second signal indicating a second record or a second data file associated with the second application data is ready for processing, the second record or the second data file includes the second metadata, and determines the second source data by processing the second metadata to identify a second location of the second record or the second data file from the identified second location, where the first source data and the second source data have different formats; and provide the second source data in a second table defined in accordance with the second target schema, where the first target schema and the second target schema are different schemas.
Claims
1. A method for an extract-transform-load (ETL) process performed by a processing device, receiving a field mapping between the application data and a target schema from a data catalog; receiving a signal from an ETL processing queue indicating that a record or data file containing metadata and associated with the application data is ready for processing; determining source data by processing the metadata to identify locations of the records or data files and retrieving the records or data files from the identified locations; providing source data into a table defined according to the target schema.
2. The method of claim 1 , wherein the field mappings are stored in the data catalog by an application or server.
3. The method of claim 2 , wherein the application is configured to use the application data.
4. The method of claim 2 , wherein the application is configured to populate the data catalog with new records or new source data files, if available.
5. The method of claim 2 , wherein the record or data file is registered in the data catalog as being ready for processing by the application.
6. The method of claim 5 , wherein the records or data files are aligned with the target schema by the data catalog.
7. The method of claim 5 , wherein the signal is inserted into the ETL processing queue by the data catalog.
8. The method of claim 7 , wherein the field mappings are defined according to a type of the application data.
9. The method of claim 8 , wherein the signal is inserted by the data catalog based on a type of the record or data file that matches the type of the application data.
10. The method of claim 1 , wherein the target schema is deployed to a target table in the data catalog.
11. The method of claim 1 , wherein the record or the data file is new or has been updated.
12. The method of claim 1 , wherein the signal includes a location of the record or the data file.
13. The method of claim 1 , wherein if the record or data file is not associated with a previously provided schema, the data catalog provides the field mapping between the application data and the target schema.
14. The method of claim 1 , wherein the source data is provided to the tables using metadata provided by the data catalog.
15. The source data includes first source data, the target schema includes a first target schema, and the method includes: receiving a second field mapping between a second application data and a second target schema from the data catalog; receiving a second signal from the ETL processing queue indicating that a second record or a second data file including second metadata and associated with the second application data is ready for processing; determining second source data by processing the second metadata to identify a second location of the second record or the second data file from the identified second location, wherein the first source data and the second source data comprise different formats; 2. The method of claim 1, further comprising providing second source data to the table defined according to the second target schema, wherein the first target schema and the second target schema are the same schema.
16. The source data includes first source data, the target schema includes a first target schema, and the method includes: receiving a second field mapping between a second application data and a second target schema from the data catalog; receiving a second signal from the ETL processing queue indicating that a second record or a second data file including second metadata and associated with the second application data is ready for processing; determining second source data by processing the second metadata to identify a second location of the second record or the second data file from the identified second location, wherein the first source data and the second source data comprise different formats; 2. The method of claim 1, further comprising providing second source data to the second table defined according to the second target schema, wherein the first target schema and the second target schema are different schemas.
17. 1. A system comprising: A non-transitory computer-readable storage medium storing instructions; at least one electronic processor executing the instructions; receiving a field mapping between the application data and a target schema from a data catalog; receiving a signal from an ETL processing queue indicating that a record or data file containing metadata and associated with the application data is ready for processing; determining source data by processing the metadata to identify locations of the records or data files and retrieving the records or data files from the identified locations; and providing source data to tables defined in accordance with the target schema.
18. The system of claim 17 , wherein the field mappings are stored in the data catalog by an application or server.
19. A non-transitory computer-readable storage medium containing executable instructions, the executable instructions causing an electronic processor to: receiving a field mapping between the application data and a target schema from a data catalog; receiving a signal from an ETL processing queue indicating that a record or data file containing metadata and associated with the application data is ready for processing; determining source data by processing the metadata to identify locations of the records or data files and retrieving the records or data files from the identified locations; and providing source data to a table defined according to the target schema.
20. 20. The non-transitory computer-readable storage medium of claim 19, wherein the field mappings are stored in the data catalog by an application or server.
