Automated ETL workflow generation

Abstract

Exemplary embodiments disclose a system and method, computer program product, and computer system. Exemplary embodiments can include receiving a data analysis request; using a knowledge graph for determining a source data set based on the received data analysis request, wherein the knowledge graph represents an ontology based on extract, transform, and load (ETL), wherein the knowledge graph includes nodes representing entities and edges representing relationships between the entities, and wherein the entities are instances of concepts of the ontology based on ETL; constructing an ETL workflow for processing the source data set according to a data lineage associated with the source data set in the knowledge graph; and executing the ETL workflow.

Description

BACKGROUND

[0001] Exemplary embodiments relate to the field of digital computer systems, and more specifically, to a method for automated extract, transform, and load (ETL) workflow generation.

[0002] In computing, ETL can be the general process of copying data from one or more source systems into a destination system, which can represent the data differently or in a different context than the source. An ETL system can extract data from source systems, enforce data quality and consistency standards, make data consistent so that separate source systems can be used together, and finally deliver data in a presentation-ready format so that application developers can build applications and end users can make decisions. However, such ETL systems are only accessible to experts, which strongly limits their potential impact. SUMMARY

[0003] Exemplary embodiments disclose a system and method, computer program product, and computer system. Exemplary embodiments can include receiving a data analysis request; using a knowledge graph for determining a source dataset based on the received data analysis request, wherein the knowledge graph represents an ontology based on extract, transform, and load (ETL), wherein the knowledge graph includes nodes representing entities and edges representing relationships between entities, and wherein the entities are instances of concepts of the ontology based on ETL; constructing an ETL workflow for processing the source dataset according to data lineages associated with the source dataset in the knowledge graph; and executing the ETL workflow. BRIEF DESCRIPTION OF DRAWINGS

[0004] The following detailed description will best be understood in conjunction with the accompanying drawings, of which:

[0005] Figure 1A is a block diagram of a computer system according to exemplary embodiments.

[0006] Figure 1B depicts an ontology based on ETL according to exemplary embodiments.

[0007] Figure 2 is a flowchart of a method for automated workflow generation according to exemplary embodiments.

[0008] Figure 3 is a block diagram of a computer system according to exemplary embodiments.

[0009] Figure 4A is a flowchart of a method of performing data analysis according to exemplary embodiments.

[0010] Figure 4B is a block diagram illustrating a method for identifying source datasets according to example embodiments.

[0011] Figure 4C is a block diagram of a knowledge graph according to example embodiments.

[0012] Figure 4D depicts data lineage according to example embodiments.

[0013] Figure 4E is a block diagram illustrating a method for determining data transformations according to example embodiments.

[0014] Figure 4F is a block diagram illustrating dependency relationships between source datasets and target datasets according to example embodiments.

[0015] Figure 4G is a block diagram illustrating ETL execution phases according to example embodiments.

[0016] Figure 5 is a flow diagram of a method for implementing data access rules according to example embodiments.

[0017] Figure 6 is a flow diagram of a method for determining data transformations according to example embodiments.

[0018] Figure 7A is a flow diagram of a method for transforming transformation logic into an ETL workflow according to example embodiments.

[0019] Figure 7B is a block diagram illustrating a method for determining workflow execution phases according to example embodiments.

[0020] Figure 7C is a flow diagram of a method for workflow execution according to example embodiments.

[0021] Figure 7D is a flow diagram of a method for workflow phase execution according to example embodiments.

[0022] Figure 7E is a flow diagram of a method for processing datasets according to example embodiments.

[0023] Figure 8A is a flow diagram of a method for executing an ETL workflow according to example embodiments.

[0024] Figure 8B depicts an example of ETL workflow metadata according to example embodiments.

[0025] Figure 8Cis a block diagram illustrating a method for monitoring ETL workflow execution according to example embodiments.

[0026] Figure 9 is a block diagram of an execution environment according to example embodiments.

[0027] Figure 10 depicts an example block diagram of a computer system according to example embodiments.

[0028] Figure 11 depicts a cloud computing environment according to example embodiments.

[0029] Figure 12 depicts an abstraction model layer according to example embodiments.

[0030] The drawings are not necessarily to scale. The drawings are merely schematic representations, not intended to portray specific DETAILED DESCRIPTION

[0031] The description of various embodiments of the present invention will be presented for the purpose of illustration and description, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application, or improvement over technology found in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0032] An ETL-based ontology can contain representations, formal naming, and definitions of relationships between categories, attributes, and concepts. An ETL-based ontology can be provided by defining classes and class hierarchies of ETL-related tasks. A class can be defined using a general concept of an ETL-related task. The general concept can be, for example, a data set. Subclasses can be defined based on the class. For example, a data set class can have subclasses that specialize data sets. For example, a subclass can indicate a user role associated with a data set, a data lineage used to process the data set, and the like. Each class can be associated with attributes that describe the concept of the class. A knowledge graph can represent the concept terms and relationship terms of the ETL-based ontology.

[0033] A knowledge graph can be a graph. A graph can refer to a property graph, in which data values are stored as properties on nodes and edges. A property graph can be managed and processed by a graph database management system or other database system that provides a wrapper layer that converts the property graph to, for example, relational tables for storage, and converts the relational tables back to the property graph on read or query. The graph can be, for example, a directed graph. The graph can be a collection of nodes (also referred to as vertices) and edges. An edge of the graph connects any two nodes of the graph. An edge can be represented by an ordered pair of nodes (vi, v2) and can pass from node vi to node v2. A node of the graph can represent an entity. An entity can refer to a dataset, a user, and the like. An entity (and corresponding node) can have one or more entity attributes or properties, which can be assigned values. For example, entity attributes of a user can include a user ID, a location of the user, and the like. A property value representing a node is a value of an entity attribute of the entity represented by the node. An edge can be assigned one or more edge attribute values that indicate at least a relationship between the two nodes connected to the edge. A property value representing an edge is a value of an edge attribute. The relationship can include, for example, inheritance (e.g., parent and child) relationships and / or association relationships according to a particular hierarchy. For example, an inheritance relationship between nodes vi and v2 can be referred to as an "is-a" relationship between vi and v2, e.g., "v2 is-a parent of vi." An association relationship between nodes vi and v2 can be referred to as a "has-a" relationship between vi and v2, e.g., "v2 has-a relationship with vi" means that vi is a part of v2, vi is a component of v2, or vi is associated with v2.

[0034] The term "ETL workflow" can refer to a collection of operations that a system can perform for performing ETL tasks. The collection of operations can include sequential and / or parallel operations.

[0035] The term "dataset" as used herein refers to a collection of one or more data records. For example, a dataset can be provided in the form of a collection of related records contained in a file, e.g., a dataset can be a file containing records for all students in a class. A dataset can be, for example, one or more tables. The one or more tables can be tables of a database or files of a Hadoop file system, and the like. In another example, a dataset can include one or more documents such as HTML pages or other document types.

[0036] According to one embodiment, determining the source data set comprises performing natural language processing of the data analytics request and using the processed data analytics request for determining the source data set. The purpose of this embodiment can be to make ETL execution more user friendly without relying on specific expertise. This can increase availability of data as the number of users that can execute ETL tasks increases. This can be particularly advantageous for cloud environments where using the present subject matter can increase availability of data centers.

[0037] According to one embodiment, the method further comprises merging the constructed ETL workflow into the knowledge graph as a property of the source data set, as a property of the user that is the requester of the data analytics request, as an entity related to the user, or as an entity related to the source data set.

[0038] This embodiment can save processing resources that would otherwise be needed for constructing ETL workflows that are already stored in the knowledge graph.

[0039] According to one embodiment, the method further comprises repeating the method for further received data analytics requests. A set of ETL workflows of the knowledge graph that share one or more steps can be identified. The one or more steps can be executed and the set of ETL workflows can be adapted so that they use the results of the execution of the one or more steps without executing the one or more steps.

[0040] An advantage of this embodiment can be that it can identify identical or similar data constructed by multiple workflows and move it upstream so that the same data is not duplicated (storage optimization) and not constructed multiple times (CPU optimization). For example, if multiple users join the same table in their workflows, the system can pre-join them and then share across the workflows. Thus, in this example, the one or more steps involve joining of the same table.

[0041] This embodiment can enable identification of data processing patterns and provide recommendations to users for the data definition phase. For example, some data cleaning can be applied by users to certain columns in order to join with other tables. In the case of a new user that will export a corresponding table, the system can provide suggestions on how to modify the SQL statement to follow the common pattern, as well as the rationale of the suggested. In this example, the one or more steps involve application of cleaning, e.g., the cleaning can be applied once for the source data set that is commonly used by the set of ETL workflows.

[0042] Another example of sharing steps can be determined based on recommendations on how different data entities are commonly joined: for example, a hardware contract can use the contract delivery date and sales country columns to join with the company’s revenue data.

[0043] According to one embodiment, the method further comprises repeating the method for further received data analysis requests, wherein the building of the ETL workflow is performed using the ETL workflow of the knowledge graph. For example, the building of the ETL workflow comprises selecting one ETL workflow of the knowledge graph, wherein the built ETL workflow is the selected ETL workflow.

[0044] According to one embodiment, the knowledge graph further indicates data access rules associated with the user and the data set. The building of the ETL workflow comprises using the knowledge graph to identify data access rules that apply to the user as the requester of the data analysis request and / or that apply to the source data set, and including conditions in the ETL workflow for processing the source data set according to the identified data access rules.

[0045] This embodiment can embed governance rules and policies into the ETL workflow and can therefore guarantee their compliance. Thus, this embodiment can provide secure access to data.

[0046] According to one embodiment, the data access rules comprise governance policies and / or user access rights.

[0047] According to one embodiment, the identified data access rules indicate at least one of which part of the source data set can be accessed from the location of the user, which part of the source data set can be copied to the location of the target system, which target system is indicated in the data lineage, whether the user can write data to the target system, or whether the user can submit workloads.

[0048] According to one embodiment, the construction of an ETL workflow includes defining a target data set from the data lineage. The target data set has a direct dependency relationship with a source data set. The method further includes determining a refresh rate of the source data set, and based on the determined refresh rate, refreshing the source data set such that the refreshed source data set is processed according to the ETL workflow, wherein the refreshing of the source data set uses all data sets that depend on the source data set. The refreshing can or can not be included as part of the ETL workflow. For example, if the refresh rate is incompatible with a desired refresh rate, the refreshing of the source data set can be performed according to the desired refresh rate. In one example, if the determined refresh rate is less than a desired refresh rate, the refreshing of the source data set can be performed according to the desired refresh rate. In one example, the desired refresh rate can require the source data set to be refreshed every Tuesday of the week, while the determined refresh rate requires the source data set to be refreshed every Monday of the week; although both refresh rates have the same frequency, they are incompatible because the two refresh rates have different phases. The desired refresh rate can be, for example, higher than or equal to the refresh rate of the target data set, e.g., if the refresh rate of the target data set is once a month, the desired refresh rate can be once a week, once a day, etc. Determining the refresh rate of the source data set further includes prompting a user to provide an input indicating whether the source data set is to be refreshed based on the refresh rate, wherein the refreshing of the source data set is performed based on the input. In another example, determining the refresh rate of the source data set further includes comparing, by the computer system, the refresh rate to a predefined refresh rate threshold. The predefined refresh rate threshold can be, for example, a desired refresh rate defined by a user.

[0049] The refresh rate of a table defines how often the table is updated. The refresh rate can be part of the data set definition. For example, a data set can be automatically refreshed according to its associated refresh rate. A data set can be refreshed periodically (regular refresh) or manually on demand (one-time refresh). A user can define a desired refresh rate of a target data set that conflicts with the refresh rate of a direct dependency, e.g., in a data analysis request. In this case, the computer system can change the latter (if allowed by the owner of the direct dependency) or create an additional set of copies of the direct dependency with the desired refresh rate. When a user defines a derivation of a data set with a scheduled refresh, the computer system can recommend a refresh schedule for the target data set based on the schedule for the direct dependency. For example, if the refresh rate of a source data set is a week, which is longer than a desired refresh rate of, e.g., a day, the source data set can need to be refreshed according to the desired refresh rate before being used by an ETL workflow.

[0050] According to one embodiment, the method further includes creating ETL workflow metadata. The ETL workflow metadata includes execution metadata indicating a status of execution of the ETL workflow and dataset metadata indicating a status of execution of each subset of the source dataset. The ETL workflow metadata can be provided in association with the ETL workflow.

[0051] According to one embodiment, the method further includes detecting a failure of execution of the ETL workflow using the execution metadata, wherein one or more subsets of the source dataset are processed without failure prior to an occurrence of the detected failure of execution, and performing a partial re-execution of the ETL workflow for processing of unprocessed subsets of the source dataset. This embodiment can enable re-running of a failed ETL workflow from a failure point of the failed ETL workflow. This can save processing resources that would otherwise be required to re-run an entire failed ETL workflow.

[0052] According to one embodiment, a re-execution of the ETL workflow is performed if the source dataset is not flushed. This embodiment can prevent transformation of conflicting data, and thus can increase reliability of data produced according to the present subject matter.

[0053] According to one embodiment, the knowledge graph is generated using log data of ETL jobs of a plurality of ETL systems, wherein execution of the ETL workflow is performed on an ETL system that is part of the ETL system or is not part of the ETL system.

[0054] According to one embodiment, the method is performed automatically. For example, the ETL workflow can be automatically constructed and automatically executed upon receiving a data analysis request.

[0055] Figure 1A An example computer system 100 is depicted. The computer system 100 can be configured to perform, for example, master data management and / or data warehousing. The computer system 100 includes a data integration system 101 and one or more client systems 105 or data sources. The client systems 105 can include computer systems. The client systems 105 can communicate with the data integration system 101 via a network connection, including, for example, a wireless local area network (WLAN) connection, a WAN (wide area network) connection, a LAN (local area network) connection, the Internet, or a combination thereof. The data integration system 101 can control access (read and write access, etc.) to a central repository 103.

[0056] Data records stored in the central repository 103 can have values for a set of attributes 109A-P, such as a company name attribute. Although this example is described in terms of a small number of attributes, more or fewer attributes can be used.

[0057] Data records stored in the central repository 103 can be received from the client systems 105 and processed by the data integration system 101 prior to being stored in the central repository 103. The received records can or can not have the same set of attributes 109A-P. For example, data records received by the data integration system 101 from the client systems 105 can not have values for all of the attributes of the set of attributes 109A-P, e.g., the data records can have values for a subset of the attributes of the set of attributes 109A-P and can not have values for the remaining attributes. In other words, the records provided by the client systems 105 can have different completeness. Completeness is the ratio of the number of attributes of a data record that includes data values to the total number of attributes in the set of attributes 109A-P. Additionally, the records received from the client systems 105 can have a structure that is different from the structure of the stored records of the central repository 103. For example, the client systems 105 can be configured to provide records in an XML format, a JSON format, or other format that is capable of associating attributes and corresponding attribute values.

[0058] In another example, the data integration system 101 can import data records from the client systems 105 into the central repository 103 using one or more ETL batches or via hypertext transfer protocol (“HTTP”) communications or via other types of data exchange.

[0059] The data integration system 101 can be configured to receive requests from the users 110 to perform particular analyses. Upon receiving a request, the data integration system 101 can define and / or execute an ETL workflow corresponding to the request. The data integration system 101 can define the ETL workflow itself by constructing the ETL workflow according to the present subject matter or by requesting another system to construct the ETL workflow according to the present subject matter and receiving the constructed ETL workflow. For example, execution of the ETL workflow can cause the data integration system 101 to receive or retrieve records from one or more client systems 105, as described herein. The data integration system 101 can process the received records using one or more algorithms, such as the algorithms 120 that implement at least a portion of the present method. For example, the data integration system 101 can process data records received from the client systems 105 using the algorithms 120 according to the ETL workflow. The ETL workflow can be defined or constructed, for example, by the algorithms 120 using the knowledge graph 121. The knowledge graph 121 can represent, for example, an ETL-based ontology as described with reference to Figure 1B the ETL-based ontology.

[0060] Figure 1B An example of an ETL-based ontology 140 is shown. The ETL-based ontology 140 includes concepts and roles. Concepts and roles are examples of the terminology aspect of the ETL-based ontology 140. Concepts and roles can be expressed in various ways. Figure 1BConcepts and roles are shown in graphical form.

[0061] ETL-based ontology 140 can contain two kinds of terms: concept terms and relationship terms. Each node in ETL-based ontology 140 identifies a concept term. ETL-based ontology 140 can include, for example, concept terms 141, 142, 143, 144, 145, 146, 147, 148, and 149. Concept term 141 includes a dataset concept, while terms 142-149 include, respectively, a user, a governance rule, a permission, a workflow, a change history, a data lineage, a data entity, and a dataset storage concept. In this example, concept terms 142-147 can each be a subclass of concept term 141. Concept term 141 can be a superclass of concept terms 142-147.

[0062] Each of concept terms 141-149 can be associated with each other via one or more relationship terms. A relationship term can be a property that connects two concept terms. ETL-based ontology 140 can also include properties or attributes that describe concept terms and relationship terms. For example, a table attribute of a dataset concept can include a type of analysis previously performed on the dataset, a SQL statement indicating how the dataset was obtained from other datasets, a location of the dataset, and the like. A user property of a user concept can include, for example, a user ID of the user, a location of the user, and the like. A data lineage property of a data lineage concept can indicate data operations and target systems that store results of the data operations.

[0063] ETL-based ontology 140 can be used to create knowledge graph 121. To do so, data about ETL tasks can be collected. With this collected data and ETL-based ontology 140, specific instances of the terms of ETL-based ontology 140 can be created, and values of the properties of the terms can be determined, resulting in knowledge graph 121. Knowledge graph can thus represent ETL-based ontology 140. Knowledge graph includes nodes representing entities and edges representing relationships between entities. An entity is an instance of a concept of the ETL-based ontology.

[0064] Figure 2 is a flowchart of a method for performing ETL tasks according to an example of the present subject matter. For explanatory purposes, Figure 2 The method described in Figure 1A The system shown in Figure 2 The method of can be performed, for example, by data integration system 101.

[0065] A data analysis request can be received in step 201. The data analysis request is a request. The data analysis request can be, for example, a request to prepare data for subsequent analysis. In another example, the data analysis request can indicate a data analysis, where the data analysis can require preparation of source data prior to analysis. The data analysis request can be received from a user 110. The term "user" can refer to an entity, such as a person, a computer, or an application executing on a computer that issues the data analysis request. The data analysis request can be received via an ETL interface of the data integration system 101.

[0066] In one example, the data analysis request can be a SQL statement. The SQL statement can specify one or more tables. This can be particularly advantageous for expert users. In another example, the data analysis request can include natural language data. Accordingly, natural language processing of the data analysis request can be performed in step 201, resulting in a processed data analysis request. This can provide a user-friendly interface to ordinary users to perform ETL tasks without requiring specialized knowledge.

[0067] The received or processed data analysis request can be analyzed in order to determine or derive values for terms and / or properties of the ETL-based ontology 140. In particular, properties associated with data sets in the ETL-based ontology can be derived from the data analysis request. For example, based on the processed data analysis request, an analysis type and one or more data table names can be determined. The derived values for terms and / or properties can be advantageously used to perform ETL tasks.

[0068] The configuration of the ETL task can start from defining which tables will be used in the ETL task. To this end, the knowledge graph 121 can be used to determine or identify source data sets based on the data analysis request received in step 203. The properties derived from the data analysis request can be used to identify one or more nodes in the knowledge graph 121 representing the source data sets. For example, the analysis type or table names can be used to identify source data sets associated with the analysis type or table names in the knowledge graph.

[0069] An ETL workflow can be constructed or determined in step 205. The ETL workflow can be used to process the source dataset according to the data lineage associated with the source dataset in the knowledge graph. The ETL workflow can include steps of data operations that can need to be performed in order to complete the ETL task. For example, before the ETL task starts, tables can need to be created in the target relational database. These will be the destinations of the records produced by the ETL workflow. To this end, the data lineage associated with the source dataset in the knowledge graph can indicate the target relational database. Steps of creating and managing those target tables can be included in the ETL workflow. The data lineage can further be used to determine transformation operations, such as retrieving input data, reading input row by row, and mapping values (transformed or not) to output columns. During the transformation, data can be read from the source dataset, and data can be mapped to multiple output tables, depending on the transformation logic. The transformation operations can be included in the ETL workflow. Furthermore, data access to the source dataset and the target database can be constrained by one or more access rules. To this end, the knowledge graph can be used to identify access rules to the source dataset and / or the target database. These access rules can be implemented in the ETL workflow.

[0070] Thus, by simply sending an analytics request, a user can receive an ETL workflow that enables the user to prepare data for performing analytics. In one example, the ETL workflow can further include steps for performing analytics at the target database. This can enable a fully automated approach for performing analytics.

[0071] The ETL workflow can be executed in step 207. Furthermore, in step 207, the ETL workflow can be inserted in the knowledge graph for further use of the knowledge graph. The ETL workflow can be associated with the source dataset concept and / or the user concept in the knowledge graph. In one example, if a change occurs that can affect the source dataset determined in step 203, then step 207 can be repeated automatically. For example, if at least a portion (e.g., all) of the source dataset and the datasets it can depend on change, then step 207 can be repeated automatically.

[0072] In another example, steps 203-207 can be performed automatically upon receiving a data analytics request. Steps 203-207 can be repeated automatically upon receiving a further data analytics request. In each iteration, the step of constructing the ETL workflow can include determining whether an existing ETL workflow is associated with the source dataset or the user 110 in the knowledge graph, and if so, using that existing workflow as the constructed workflow, and otherwise constructing the ETL workflow as described with respect to step 205.

[0073] Figure 3is a block diagram of a computer system 300 according to an example of the present subject matter. The computer system 300 includes an automation engine 301 and an execution environment 305. The automation engine 301 includes an ETL interface 302 and a knowledge graph 303. The ETL interface 302 is configured to receive a data analysis request in the form of natural language or SQL statements. The automation engine 301 may, for example, be configured to use the knowledge graph 303 to perform the method steps 201 to 205 of Figure 2 . The resulting ETL workflow can be executed in the execution environment 305.

[0074] Figure 4A is a flowchart of a method for performing an ETL task according to an example of the present subject matter.

[0075] In step 401, a source dataset can be identified. The source dataset can be used to perform an analysis. For example, the source dataset can be identified in response to receiving a data analysis request. For example, the data analysis request can be received from a data scientist. The data analysis request can call for producing a recommendation, for example, to determine what customers can buy what products with an assessed probability and potential size of a transaction. As Figure 4B shown, the data analysis request 415 can include natural language input data, such as the keywords "transaction history" and "customer data". The data analysis request 415 can be parsed and processed in order to assess one or more properties of the ETL-based ontology (naming request properties). The request properties may, for example, include an indication of the user who submitted the data analysis request 415. The request properties can also include a role of the user. The request properties can also include a type of analysis requested, etc.

[0076] As Figure 4B shown, the data analysis request 415 can be used to identify candidate source datasets 417 using a knowledge graph 416. The knowledge graph 416 may, for example, represent an ETL-based ontology as described with reference to Figure 1B Each candidate source dataset 417 is associated with a data source in which it is located. For example, a portion of the assessed request properties, such as the type of analysis, can be used to identify candidate source datasets 417. By using further properties of the request properties, the search can be further refined. For example, based on the user role, some of the candidate source datasets 417 can be excluded. Since the user role relates to a data scientist, source datasets 421 and 422 can be selected among the candidate source datasets 417. In this example, each of the source datasets 421 and 422 includes two tables 421a-b and 422a-b, respectively. As with other tables of the knowledge graph 416, each of the identified source tables 421 and 422 can be associated with properties or metadata (naming table attributes). For the purpose of simplicity, Figure 4BOnly table properties 418a of source table 421a are shown. Table properties 418a can include, for example, a start date of the contract, an end date of the contract, a contract value, an ID of the item sold, an ID of the seller, a location of source table 421a, an owner of source table 421a, and a refresh rate of source table 421a. Each of the identified data tables 421a-b and 422a-b can be an entity of knowledge graph 416. Knowledge graph 416 can be constructed, for example, using ETL jobs performed by different users such as data scientists. That is, instances of terms of an ETL-based ontology can be defined using data indicative of ETL jobs of data scientists and other types of users.

[0077] Figure 4C A subgraph 420 of knowledge graph 416 is depicted. Subgraph 420 relates to users and data sets that match the "transaction history" keyword of data analysis request 415. Identified source tables 421a and 421b can be nodes of subgraph 420 of knowledge graph 416 as shown. Source tables 421a and 421b are linked to data sources 423 in which they reside and can retrieve their locations 426 from the data sources 423. Source tables 421a and 421b are linked to target tables 424a that have been produced from source tables 421a and 421b. Each of tables 421a, 421b, and 424a are also linked to other entities of subgraph 420 that further specify the table. For example, a user 425 that owns target table 424a and can read source tables 421a and 421b linked to the table. User 425 can also be linked to other entities that further specify characteristics of the user such as a location 426 of the user and a role 427 of the user. Figure 4C Figure 4C Additional example entities 423 to 436 that are directly or indirectly related to source tables 421a and 421b are shown. For example, a status of execution and update of the data sets can be provided by nodes 429, 432, and 433. Sample data entities 434 to 436 can be provided that indicate target data sets and source data sets, respectively.

[0078] Source tables 421a and 421b are those identified in step 401 in subgraph 420 because they are linked to user 425 that has a similar role to the user that sent the data analysis request and performed the analysis according to the requested data analysis. Similarly, source tables 422a and 422b can be identified in step 401 using knowledge graph 416.

[0079] ​After identifying the source datasets 421 and 422, the method proceeds to step 403, where data transformations can be determined. The data transformations can be determined such that they can be applied to the source tables 421a-b and 422a-b according to the data analysis request. The data transformations can be determined using the data lineage associated with the source tables 421a-b and 422a-b, e.g., 430. Figure 4D An example data lineage involving the target table 424a and the corresponding source tables 421a-b is shown. The data lineage indicates the relationship between the tables based on one of the table attributes of the tables, where the table attribute can be the value_usd attribute.

[0080] As Figure 4D shown, each of the source and target tables 421a-b and 424a is associated with a SQL statement. The source table 421a is associated with a SQL statement 441a that indicates that the value of the table attribute of the source table 421a is obtained from table 442a. The source table 421b is associated with a SQL statement 441b that indicates that the value of the table attribute of the source table 421b is obtained from table 442b. The target table 424a is associated with a SQL statement 440a that indicates the transformations applied to the source tables 421a-b in order to obtain the target table 424a. The data transformations can be determined based on the SQL statements associated with the tables 421a-b and 424a. In particular, as Figure 4E shown, the determined data transformations 440a and 440b can be used to obtain the target tables 424a and 424b, respectively, from the source tables 421a-b and 422a-b. For example, the data transformations 440a and 440b are applied to the source tables 421a-b and 422a-b in order to obtain the target tables 424a and 424b, respectively. Figure 5 and Figure 6 Implementation details of step 403 are provided.

[0081] After determining the data transformations, the method proceeds to step 405, where the transformation logic can be turned into an ETL workflow. The determined data transformations can be included in the ETL workflow. In addition, the dependencies of the target tables 424a and 424b can be determined. As Figure 4FAs shown, target table 424a has a direct dependency relationship with source tables 421a and 421b, where source tables 421a-b have dependency relationships with tables 442a-b, respectively. Similarly, target table 424b has a direct dependency relationship with source tables 422a and 422b, where source tables 422a-b can have no dependency relationships. Dependency relationships can be advantageously used to enhance the ETL workflow with additional data operations or steps that enable efficient processing of the energy data sets. For example, if the refresh rate of source tables 421a-b is less than (i.e., not as frequent as) a desired rate threshold, then a data operation of the ETL workflow can be defined such that the source tables 421a-b can be refreshed according to the desired rate before being transformed. The refresh of source tables 421a-b can use all of the dependency tables, i.e., 442a and 442b, respectively, of the source tables. Dependency relationships can also be used to determine the manner in which target tables 424a and 424b can be produced. This is illustrated, for example, in Figure 4G where stages of transformation are defined based on the dependency relationships of target tables 424a and 424b. The stages can be building blocks of the ETL workflow. Each of the stages can describe a data source, one or more data processing (or data transformation) steps, and / or a target system, and can define processing logic that moves data from an input link to an output link. The stages can include sequential and / or parallel stages. When two stages are sequential, the second stage can not begin executing until the first stage is completed. Stages can also be parallel. Following the present example, two parallel stages can be defined for execution, the first stage associated with target table 424a and the second stage associated with target table 424b. Figure 7A Implementation details of step 405 are provided.

[0082] After determining the ETL workflow, the method proceeds to step 407, where a workflow schedule is defined such that the ETL workflow can be executed according to the schedule in step 409. This can result in the production of target tables 424a and 424b. Analysis can then be performed on the produced data in step 411, and the results of the analysis can be populated into a result data set in step 413.

[0083] Figure 5 is a flowchart of a method 500 for implementing data access rules according to examples of the present subject matter. Method 500 can enable the production of an ETL workflow. The data access rules can be associated with source data sets 421 and 422 in knowledge graph 416, and can be considered when defining data transformations in step 403.

[0084] In step 501, a natural language query can be received from a user of a data scientist with a defined user ID (uid). As in step 401, in step 502, the natural language query can be parsed in order to determine source and target data sets and to produce a corresponding SQL statement that can be used to perform a data transformation from the source data set to the target data set. As indicated in block 503, step 502 can produce the user uid, source data sets 421 and 422, and target data sets 424a and 424b. In step 504, the location of the user and of the source and target data sets can be determined or retrieved. For example, it can be determined that the location of source data set 421 is data source 423 as shown in knowledge sub-graph 420. It can be determined (step 505) whether any portion of source data sets 421-422 can be accessed from the location of the user. In the case where a portion of source data sets 421-422 can not be accessible from the location of the user, an error can be generated in step 507, and the method can end in step 508. As shown in Figure 5 in this case, the ability to change governance policies is unlikely. In the case where any portion of source data sets 421-422 can be accessed from the location of the user, it can be determined (step 506) whether any portion of source data sets 421-422 can be copied to the location of target data sets 424a and 424b, respectively. In the case where a portion of source data sets 421-422 can not be copied to the location of target data sets 424a, the method proceeds to steps 507 and 508. As shown in Figure 5 in this case, the ability to change governance policies is unlikely.

[0085] In the case where any portion of source data sets 421-422 can be copied to the location of target data sets 424a and 424b, respectively, it can be determined (step 509) whether the user can write to a target database in which target data sets 424a and 424b can be located. In the case where the user can not write to the target database, a warning can be issued in step 510, and the method proceeds to step 511. In the case where the user can write to the target database, the method proceeds to step 511 in which it can be determined whether a name for target data set 424a is available. In the case where a target data set name is not available, a warning can be issued in step 512, and the method proceeds to step 513. In the case where a target data set name is available, the method proceeds to step 513 in which it can be determined whether the user can submit workloads. In the case where the user can not submit workloads, a warning can be issued in step 514, and the method proceeds to step 515. As shown in Figure 5As shown, reporting warnings allows the process to continue because the issues they relate to can be fixed before an actual commit. In cases where the user can commit a workload, this can indicate that the user can execute an ETL workflow or that the ETL workflow can be further defined, as indicated in step 515.

[0086] like Figure 5 The data access rules defined in [the documentation] can be used to define conditions that can be applied to data transformations in an ETL workflow (e.g., involving filtering and anonymization). For example, Figure 5 The steps may be included as such Figure 6 The steps of the ETL workflow are shown below. Step 601 provides the SQL statements that can be used to perform data transformation, as well as the user and the locations of the source and target datasets. In step 603, rows and columns of the source and / or target datasets that can be filtered can be detected. In step 605, rows and columns of the source and / or target datasets that can be anonymized can be detected. In step 607, based on, as... Figure 5 The data governance rules and strategies described herein apply filters to SQL statements. Figure 6 The steps of the method can be as follows: Figure 7A The method 609 described herein generates workflows. This allows SQL statements to be tailored to ensure compliance with governance rules and policies. Therefore, depending on the rules, the same query may return different results, depending on the target dataset, the location of the source dataset, and the user submitting the query. For example, European (EU) or US (US) data may be filtered according to the General Data Protection Regulation (GDPR) or federal laws.

[0087] Figure 7A This is a flowchart illustrating a method for transforming transformation logic into a workflow, based on examples from this topic. In step 701, dependencies can be pulled for source datasets 421-422. For example, step 701 could instruct source tables 421a-b (see...) Figure 4F The relationship is a direct dependency of target table 424a, and it depends on source tables 442a and 442b respectively. It can be determined (step 702) whether the dependency meets the refresh rate requirement. According to... Figure 4F For example, in step 702, it can be determined whether the refresh rate of each of the source tables 421a-b, 442a-b, and 422a-b meets the refresh rate requirement, for example, whether the refresh rate is greater than the expected rate threshold. If the dependencies meet the refresh rate requirement, in step 703 only direct dependencies can be selected, and the method proceeds to step 705. Figure 4FFor example, if the dependencies meet the refresh rate requirement, only the direct dependencies (of target tables 424a and 424b) including source tables 421a-b and 422a-b can be selected. If the dependencies do not meet the refresh rate requirement, all dependencies can be selected in step 704, and the method proceeds to step 705. Figure 4F For example, if the dependency only includes source tables 421a-b and does not meet the refresh rate requirement, source table 442a-b can be added to source table 421a-b to enable data transformation together. In one example, to meet the refresh rate requirement, the refresh rate of source datasets 421a-b and 442a-b can be changed for their locations if the owners of the source datasets 421a-b and 442a-b allow it. Otherwise, a replica dependency dataset of these source datasets can be created, and their refresh rates can be adjusted to meet the refresh rate requirement. Therefore, steps 704 or 703 can produce a dependency dataset that meets the refresh rate requirement and the corresponding SQL query, as shown in step 705. In step 706, it can be determined whether the data is sufficient for the final workflow generation. If the data is insufficient for the final workflow generation, in step 707, the user can be requested to provide the missing data, and the method proceeds to step 708. For example, if the content of the data at least indicates the source dataset, the target dataset, and the transformation operation from the source dataset to the target dataset, the data may be sufficient. If sufficient data is available for the final workflow generation, the method proceeds to step 708, where parallel and sequential execution phases can be defined. For example, it can be based on... Figures 4A-6 The transformations defined in the diagram are used to define the stages. After defining the parallel and sequential stages, the method proceeds to step 709, where connections are established between the stages. In step 710, the resulting ETL workflow can be stored, and the method can terminate in step 711. Figure 7B Figure E provides an example of the operations of the stored workflow.

[0088] Figure 7B An example of a batch defined for target dataset 424a is shown. Three batches can be defined based on dependencies. For example, batch 723 includes target dataset 424a. Direct dependencies of target dataset 424a as source tables 421a and 421b belong to another batch 722. Source tables 442a and 442b, which source tables 421a and 421b depend on, belong to another batch 721. Each of batches 721 to 723 can provide a file containing instructions for processing units on the tables within the batch. Tables within a batch can be built in parallel because they may not have dependencies on each other. Figure 7Cis a flowchart of a method 730 for executing an ETL workflow having stages according to examples of the present subject matter. A workflow stage 733 of the ETL workflow can be retrieved in step 731. For example, assuming the stages are sequential, it can be determined in step 735 whether there is an unexecuted stage of the workflow. In the absence of an unexecuted stage of the workflow, the method ends. In the presence of an unexecuted stage of the workflow, the method proceeds to step 737 in which the stage is executed. It can be determined in step 739 whether the execution of the stage was successful. In the event that the execution of the stage was successful, the stage can be marked as successful in step 741, and steps 735 through 743 can be repeated for the next stage. In the event that the execution of the stage was not successful, an error can be reported in step 743, and the method can end. Reference is made to Figure 7D The execution of the stage is further described. In step 751, the data sets 753 of the stage can be retrieved. For example, assuming the data sets can be processed sequentially, it can be determined in step 755 whether there is an unprocessed data set of the determined data sets. In the absence of an unprocessed data set of the determined data sets, the method ends. In the presence of an unprocessed data set of the determined data sets, the method proceeds to step 757 in which the unprocessed data set is processed. It can be determined in step 759 whether the processing of the data set was successful. In the event that the processing of the data set was successful, the data set can be marked as successfully processed in step 761, and steps 755 through 763 can be repeated for the next determined data set. In the event that the execution of the data set was not successful, an error can be reported in step 763, and the method can end. Reference is made to Figure 7EAn example data set processing method (step 757) is described. If one or more source data sets 771 that can need to be processed have not been retrieved, they can be retrieved in step 770. It can be determined (step 772) whether the retrieved source data sets are readable. In the case that the source data sets 771 are not readable, an error can be provided in step 773, and the method can end. In the case that the source data sets 771 are readable, an expected target schema 775 can be computed in step 774. It can be determined in step 776 whether a target data set exists. In the case that the target data set does not exist, it can be created in step 777, and the method proceeds to step 778. In the case that the target data set exists, the method proceeds to step 778. It can be determined in step 778 whether the target data set is compatible with the expected target schema. In the case that the target data set is not compatible with the expected target schema, an error can be provided in step 782, and the method can end. In the case that the target data set is compatible with the expected target schema, it can be determined in step 779 whether the target data set is writable. In the case that the target data set is not writable, an error can be provided in step 782, and the method can end. In the case that the target data set is writable, it can be truncated in step 780, and filled with new data resulting from the processing of the source data sets in step 781.

[0089] Figure 8A is a flowchart of a method for executing an ETL workflow according to examples of the present subject matter. For purposes of explanation, Figure 8A The method described in Figures 4A-7A may be implemented with the examples defined in

[0090] In step 801, ETL workflow metadata can be created. The ETL workflow metadata includes execution metadata indicating the execution status of the ETL workflow and data set metadata indicating the execution status of each subset of the source data sets. Figure 8B An example of ETL workflow metadata is shown. Execution metadata 820 can indicate the ETL workflow, and can include the name of the workflow, the owner of the workflow, the priority of the workflow, the status of the data sets involved in the workflow, and the status history of the workflow. Status history 823 indicates the time of status change and the status details. Each data set involved in the workflow 822 can be associated with corresponding data set metadata (data set status 821). The data set metadata may, for example, include table attributes such as the name of the data set, the schema of the data set, the key of the data set, the organization of the data set, the SQL statement to obtain the data set, the location of the data set, the owner of the data set, and the update time of the data set. The data set metadata can also include data set status 821. The data set status 821 can indicate the status history 823 of the data set. AsFigure 8B As shown, the execution metadata 820 and the dataset metadata can be linked according to a predefined structure that enables tracking of the progress state of the execution of the workflow.

[0091] In step 803, the ETL workflow can be executed, with the ETL workflow metadata used to monitor the execution. As Figure 8C As shown, the execution metadata 820 indicates that the execution failed due to a failed dataset state. The dataset metadata can be used to identify the failed dataset. As Figure 8C As shown, the dataset state associated with the target dataset 424a is successful. However, the dataset state associated with the target dataset 424b is a failed state. As Figure 8C As shown, one or more instances of the state history 823 associated with each dataset and ETL workflow indicate when the failure occurred, and which source tables were involved.

[0092] Using the ETL workflow metadata, the ETL workflow can be restarted in step 805, with only the failed stages re-run, and with the data in the successfully built datasets assumed to be sufficiently new for the requirements of the workflow.

[0093] Figure 9 is a block diagram of an execution environment 900 according to examples of the subject matter. Figure 9 A set of functional layers provided by the execution environment 900 is described. The set of functional layers includes a user layer 901, an ETL workflow generation layer 902, and an execution layer 903. The user layer 901 can include an ETL interface for receiving data analysis requests from users. The ETL workflow generation layer 902 includes services and corresponding data inputs or outputs. Each service can be configured to perform a respective step of an ETL workflow generation method. For example, a data discovery service can be configured to identify source datasets corresponding to a received data analysis request. A data lineage service can be configured to determine data transformations corresponding to the received data analysis request. A workflow compilation service can be configured to build an ETL workflow. A workflow execution service can be configured to execute the ETL workflow using the execution layer 903. Any of the services of the ETL workflow generation layer 902 can potentially use any of the listed metadata. For example, the data discovery service can use governance metadata or execution state to rank results.

[0094] Figure 10 A block diagram of devices within a system according to example embodiments is described. It should be appreciated that Figure 10 Only a few implementations are described and other implementations, modifications, and variations are possible.

[0095] The devices used herein can include one or more processors 02, one or more computer-readable RAM 04, one or more computer-readable ROM 06, one or more computer-readable storage media 08, device drivers 12, read / write drives or interfaces 14, network adapters or interfaces 16, all interconnected by a communication structure 18. The communication structure 18 can be implemented with any architecture designed for passing data and / or control information between processors (such as microprocessors, communication, and network processors, etc.), system memory, peripheral devices, and any other hardware components within a system.

[0096] One or more operating systems 10 and one or more application programs 11 are stored on one or more of the computer-readable storage media 08 for execution by one or more of the processors 02, via one or more of the respective RAMs 04, which typically include cache memory. In the illustrated embodiment, each computer- readable storage medium 08 can be a magnetic disk storage device of internal hard disk drive, a CD-ROM, a DVD, a memory stick, magnetic tape, a floppy disk, an optical disk, a semiconductor memory device such as RAM, ROM, EPROM, flash, or any other computer-readable tangible storage device that can store a computer program and digital information.

[0097] The devices used herein can also include read / write drives or interfaces 14 to read from and write to one or more portable computer-readable storage media 26, on which the application programs 11 on the devices can be stored, read via the respective read / write drives or interfaces 14, and loaded into the respective computer-readable storage media 08.

[0098] The devices used herein can also include network adapters or interfaces 16, such as a TCP / IP adapter cards or wireless communication adapters, such as a 4G wireless communication adapter using OFDMA technology. The application programs 11 on the computing devices can be downloaded to the computing devices from an external computer or external storage device via a network (e.g., the Internet, a local area network or other wide area network or wireless network) and the network adapters or interfaces 16. From the network adapters or interfaces 16, the programs can be loaded onto the computer-readable storage media 08. The network can comprise copper wires, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers.

[0099] The device as used herein can also include a display screen 20, a keyboard or keypad 22, and a computer mouse or touchpad 24. The device drivers 12 interface to the display screen 20 for imaging, to the keyboard or keypad 22, to the computer mouse or touchpad 24, and / or to the display screen 20 for pressure-sensitive alphanumeric character input and user selection. The device drivers 12, the R / W drive or interface 14, and the network adapter or interface 16 can include hardware and software (stored on the computer readable storage medium 08 and / or the ROM 06).

[0100] The programs described herein are identified based upon the application for which they are implemented in a particular example embodiment. It should be understood, however, that any particular program term(s) used herein is / are for convenience only, and thus the example embodiments are not limited to use solely in any specific application identified and / or implied by such term(s).

[0101] Based on the foregoing, a computer system, method, and computer program product are disclosed. Numerous modifications and alternatives, however, can be made without departing from the scope of the example embodiments. Accordingly, the example embodiments have been disclosed by way of example and not limitation.

[0102] It should be understood that while the present disclosure includes a detailed description on cloud computing, implementations of the teachings set forth herein are not limited to a cloud computing environment. Rather, example embodiments are capable of implementation in conjunction with any other type of computing environment now known or later developed.

[0103] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g. networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a provider of the service. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0104] The characteristics are as follows:

[0105] On-demand self-service: cloud consumers can unilaterally provision computing capabilities, such as server time and network storage, as needed automatically without requiring human interaction with the service's provider.

[0106] Broad network access: capabilities are available over a network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0107] Resource pooling: the provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to demand. There is a sense of location independence in that the consumer generally has no control or knowledge over the exact location of the provided resources but can be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter).

[0108] Rapid elasticity: in some cases, capabilities can be provisioned and released in a very short period of time (e.g., within minutes). Consumers can have a sense of location independence in that the physical location of the resource is generally transparent to them, but can be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter).

[0109] Measured service: cloud systems automatically control and optimize resource use by leveraging utilization of resources in an efficient manner. Consumers can acquire and rapidly provision what they need, in many cases only paying for drain they actually use.

[0110] Service models are as follows:

[0111] Software as a Service (SaaS): the capability provided to the consumer is to use the provider's applications running on a cloud infrastructure. The applications are accessible from various client devices through a thin client interface such as a web browser (e.g., web-based e-mail). The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0112] Platform as a Service (PaaS): the capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure including networks, servers, operating systems, or storage, but has control over the deployed applications and possibly application hosting environment configurations.

[0113] Infrastructure as a Service (IaaS): the capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include an operating system and applications. The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, deployed applications, and possibly limited control of select networking components (e.g., host firewalls).

[0114] Deployment models are as follows:

[0115] Private cloud: the cloud infrastructure is operated solely for an organization. It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0116] Community cloud: the cloud infrastructure is shared by several organizations and supports mission-critical enterprise resources. It can be managed by the organizations or a third party and can exist on-premises or off-premises.

[0117] Public cloud: the cloud infrastructure is made available to the general public or a large industry group and is owned by an organization selling cloud services.

[0118] Hybrid cloud: the cloud infrastructure is a composition of two or more clouds (private, community, or public) that remain unique entities but are bound together using standard or proprietary technologies that enable data and application portability.

[0119] A cloud computing environment is service-oriented, with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure comprising a network of interconnected nodes.

[0120] Referring now to the drawing Figure 11 , an illustrative cloud computing environment 50 is described. As shown, cloud computing environment 50 includes one or more cloud computing nodes 40 with which local computing devices used by cloud consumers, such as, for example, personal digital assistant (PDA) or cellular telephone 54A, desktop computer 54B, laptop computer 54C, and / or automobile computer system 54N can communicate. Nodes 40 can communicate with one another. They can be grouped (not shown) physically or virtually, in one or more networks, such as Private, Community, Public, or Hybrid clouds as described hereinabove, or a combination thereof. This allows cloud computing environment 50 to offer infrastructure, platforms and / or software as services with Figure 11 The types of computing devices 54A-N shown in

[0121] Referring now to the drawing Figure 12 , a set of functional abstraction layers provided by cloud computing environment 50 (best Figure 11 ) is shown. It should be understood that the components, layers, and functions shown in Figure 12 are intended to be illustrative only and that example embodiments are not limited thereby. As depicted, the following layers and corresponding functions are provided:

[0122] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include: mainframes 61; RISC (Reduced Instruction Set Computer) architecture based servers 62; servers 63; blade servers 64; storage devices 65; and networks and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0123] Virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 71; virtual storage 72; virtual networks 73, including virtual private networks; virtual applications and operating systems 74; and virtual clients 75.

[0124] In one example, management layer 80 can provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources that are utilized to perform tasks within the cloud computing environment. Metering and Pricing 82 provide cost tracking as resources are utilized within the cloud computing environment, and billing or invoicing for consumption of these resources. In one example, these resources can include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management such that required service levels are met. Service Level Agreement (SLA) planning and fulfillment 85 provide pre-arrangement for, and procurement of, cloud computing resources for which future requirements are anticipated in accordance with an SLA.

[0125] Workloads layer 90 provides examples of functionality for which the cloud computing environment can be utilized. Examples of workloads and functions which can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analysis processing 94; transaction processing 95; and automated extraction, transformation, and loading workflow generation 96.

[0126] The present application can be a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0127] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted via a wire cable.

[0128] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions into the computing / processing device for storage in a computer readable storage medium within the respective computing / processing device.

[0129] Computer readable program instructions for carrying out operations of the present application can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, microcode, firmware instructions, state-setting data, configuration data for an integrated circuit, or source code or object code written in any combination of one or more programming languages, including an object-oriented programming language such as Smalltalk, C++ or the like, and a procedural programming language such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0130] Aspects of the present application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0131] These computer readable program instructions can be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including

[0132] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0133] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

Claims

1. A computer-implemented method, the method comprising: receiving a data analysis request; using a knowledge graph for determining a source data set based on the received data analysis request, wherein the knowledge graph represents an extract, transform, and load (ETL) based ontology, wherein the knowledge graph comprises nodes representing entities and edges representing relationships between the entities, and wherein the entities are instances of concepts of the ETL based ontology; constructing an ETL workflow for processing the source data set according to a data lineage associated with the source data set in the knowledge graph; and executing the ETL workflow.

2. The method of claim 1, wherein determining the source data set based on the received data analysis request further comprises: performing natural language processing of the data analysis request; and determining the source data set based on the processed data analysis request.

3. The method of claim 1, further comprising: merging the constructed ETL workflow into the knowledge graph as a property of the source data set, as a property of a user submitting the data analysis request, as an entity related to a user, or as an entity related to the source data set.

4. The method of claim 3, further comprising: receiving a second data analysis request; using the knowledge graph for determining a second source data set based on the second data analysis request; constructing a second ETL workflow for processing the second source data set according to a data lineage associated with the second source data set in the knowledge graph; identifying that the second ETL workflow shares one or more steps with the executed ETL workflow; and adapting the second ETL workflow such that it uses results of the shared one or more steps within the executed ETL workflow.

5. The method of claim 1, wherein the knowledge graph further indicates data access rules associated with users and data sets, wherein the constructing of the ETL workflow comprises using the knowledge graph data to identify data access rules that apply to a user that is a requester of the data analysis request or to the source data set, and wherein conditions for processing the source data set according to the identified data access rules are included in the ETL workflow.

6. The method of claim 5, wherein the data access rules comprise governance policies and user access permissions.

7. The method of claim 5, wherein the identified data access rules indicate at least one of the items from a group comprising which portion of the source data set can be accessed from a location of the user, which portion of the source data set can be copied to a location of a target system, which target system is indicated in the data lineage, whether the user can write data to the target system, and whether the user can submit workloads.

8. The method of claim 1, wherein the constructing of the ETL workflow comprises: defining a target data set according to the data lineage, wherein the target data set has a direct dependency relationship with the source data set; and defining a target system according to the data lineage, wherein the target system has a direct dependency relationship with the source data set. ​ ​ determining a refresh rate for the source dataset, and refreshing the source dataset based on the determined refresh rate, wherein refreshing the source dataset includes processing the source dataset according to the ETL workflow, and wherein refreshing the source dataset uses all datasets that depend on the source dataset.

9. The method of claim 1, further comprising: creating ETL workflow metadata, wherein the ETL workflow metadata includes execution metadata indicating an execution status of the ETL workflow and dataset metadata indicating an execution status of each subset of the source dataset; and providing the ETL workflow metadata.

10. The method of claim 9, further comprising: detecting an execution failure of the ETL workflow using the execution metadata, wherein one or more subsets of the source dataset are processed without failure prior to the detected execution failure; and performing a partial re-execution of the ETL workflow for processing unprocessed subsets of the source dataset.

11. The method of claim 10, wherein the ETL workflow being executed is re-executed if the source dataset is not refreshed.

12. The method of claim 1, wherein the knowledge graph is generated using log data of ETL jobs of a plurality of ETL systems, and wherein the execution of the ETL workflow is performed on an ETL system that is part of the plurality of ETL systems.

13. The method of claim 1, wherein the knowledge graph is generated using log data of ETL jobs of a plurality of ETL systems, and wherein the execution of the ETL workflow is performed on an ETL system that is not part of the plurality of ETL systems.

14. A computer program product, the computer program product comprising: one or more non-transitory computer-readable storage media and program instructions stored on the one or more non-transitory computer-readable storage media capable of executing a method, the method comprising: receiving a data analysis request; using a knowledge graph for determining a source dataset based on the received data analysis request, wherein the knowledge graph represents an extract, transform, and load (ETL) based ontology, wherein the knowledge graph includes nodes representing entities and edges representing relationships between the entities, and wherein the entities are instances of concepts of the ETL based ontology; constructing an ETL workflow for processing the source dataset according to a data lineage associated with the source dataset in the knowledge graph; and executing the ETL workflow.

15. The computer program product of claim 14, wherein determining the source dataset based on the received data analysis request further comprises: performing natural language processing of the data analysis request; and determining the source dataset based on the processed data analysis request.

16. The computer program product of claim 14, further comprising: ​ incorporating the constructed ETL workflow into the knowledge graph as a property of the source dataset, as a property of a user that is a requester of the data analysis request, as an entity related to a user, or as an entity related to the source dataset.

17. A computer system comprising: one or more computer processors, one or more computer-readable storage media, and program instructions stored on one or more of the computer-readable storage media for execution by at least one of the one or more processors capable of performing a method comprising: receiving a data analysis request; using a knowledge graph for determining a source dataset based on the received data analysis request, wherein the knowledge graph represents an extract, transform, and load (ETL)-based ontology, wherein the knowledge graph comprises nodes representing entities and edges representing relationships between the entities, and wherein the entities are instances of concepts of the ETL-based ontology; constructing an ETL workflow for processing the source dataset according to data lineage associated with the source dataset in the knowledge graph; and executing the ETL workflow.

18. The computer system of claim 17, wherein determining the source dataset based on the received data analysis request further comprises: performing natural language processing of the data analysis request; and determining the source dataset based on the processed data analysis request.

19. The computer system of claim 17, further comprising: incorporating the constructed ETL workflow into the knowledge graph as a property of the source dataset, as a property of a user that is a requester of the data analysis request, as an entity related to a user, or as an entity related to the source dataset.

20. The computer system of claim 19, further comprising: receiving a second data analysis request; using the knowledge graph for determining a second source dataset based on the second data analysis request; constructing a second ETL workflow for processing the second source dataset according to data lineage associated with the second source dataset in the knowledge graph; identifying that the second ETL workflow shares one or more steps with the executed ETL workflow; and adapting the second ETL workflow such that it uses results of the shared one or more steps within the executed ETL workflow.

Images