Metadata-driven ingestion and data processing

Abstract

A method implemented by a data processing system to enable the system to pipeline or otherwise process data according to specified criteria by selecting data to be processed, determining metadata for the selected data, and providing a graphical user interface for automatically processing the selected data according to specified criteria based on the metadata.

Description

Technical Field

[0001] (Claim of Priority) This application claims priority to U.S. Patent Application No. 63 / 495,618, filed Apr. 12, 2023, and to U.S. Patent Application No. 18 / 496,543, filed Oct. 27, 2023, under 35 U.S.C. § 119(e), the entire contents of both of which are incorporated herein by reference.

[0002] (Field of the Invention) The present disclosure relates to techniques for enabling a data processing system to pipeline or otherwise process data according to specified criteria.

Background Art

[0003] Modern data processing systems manage vast amounts of data within an enterprise. For example, large enterprises may have millions of data sets. These data sets can support multiple aspects of an enterprise's operations. Complex data processing systems typically process data in multiple stages, with the results from one stage feeding into the next. The overall flow of information through such a system can be described in terms of a directed data flow graph, where nodes or vertices within the graph represent components (either data files or processes), and the links or “edges” within the graph indicate the flow of data between components. A system for performing such graph-based computations is described in U.S. Patent No. 5,966,072, titled “Executing Computations Expressed as Graphs,” which is incorporated herein by reference.

[0004] In many cases, corporate data is scattered across multiple different data sources, and companies need to consolidate this data to facilitate data storage and analysis. To do this, companies can use a data ingestion process in which data is moved from one or more data sources to a destination such as a data lake, data warehouse, or another data storage system. Once ingested, the data can be stored, analyzed, or used in other ways. [Overview of the Initiative]

[0005] Generally, in a first aspect, a method implemented by a data processing system for receiving instructions for a dataset to be processed and processing the dataset according to one or more operations specified in a specification includes: receiving a specification by the data processing system that specifies logical metadata and one or more operations to be performed on data associated with the logical metadata; providing a user interface by the data processing system for indicating one or more datasets to be retrieved and processed; receiving instructions for a given dataset from the user interface; and generating at least one or more instructions that are executable to process the given dataset according to the specification in response to the instructions, wherein generating the instructions recognizes the technical metadata about the given dataset. This includes separating data, accessing a metadata model from a data source that specifies relationships between logical metadata and technical metadata, traversing the metadata model to identify relationships between (i) logical metadata entries in the metadata model corresponding to the logical metadata of a specification and (ii) technical metadata entries in the metadata model corresponding to the technical metadata entries for a given dataset, and generating one or more instructions to perform at least one of one or more actions on the data of a given dataset according to the identified relationships between (i) logical metadata entries in the metadata model corresponding to the logical metadata of a specification and (ii) technical metadata entries in the metadata model corresponding to the technical metadata entries for a given dataset.

[0006] Generally, in the second aspect, a method implemented by a data processing system for processing data according to specified criteria, by selecting data to be processed, determining metadata for the selected data, and providing a graphical user interface for automatically processing the selected data according to specified criteria based on the metadata, includes: receiving a specification from the data processing system that specifies logical metadata and one or more actions to be performed on data associated with the logical metadata; providing a user interface for indicating one or more datasets to be retrieved and processed; receiving a user instruction for a given dataset from the user interface; and at least the user instruction The process includes: in response, generating one or more instructions that are executable to process a given dataset according to a specification; identifying technical metadata about a given dataset; accessing a metadata model that specifies relationships between logical metadata and technical metadata; traversing the metadata model to identify relationships between (i) logical metadata entries in the metadata model and (ii) technical metadata entries in the metadata model that correspond to technical metadata entries about a given dataset; and updating one or more instructions according to the identified relationships between (i) logical metadata entries in the metadata model and (ii) technical metadata entries in the metadata model that correspond to technical metadata entries about a given dataset.

[0007] In a third embodiment, which can be combined with the first or second embodiment, traversing includes traversing the metadata model to identify relationships between (i) logical metadata entries in the metadata model corresponding to logical metadata of a specification and (ii) technical metadata entries in the metadata model corresponding to technical metadata entries for a given dataset, and updating includes updating one or more instructions to specify that at least one of one or more operations is performed on data represented by technical metadata entries in the metadata model corresponding to technical metadata entries for a given dataset.

[0008] In a fourth embodiment, which can be combined with any of the first to third embodiments, the method includes, on the basis of traversing, identifying data quality controls to be applied to technical metadata items in a metadata model corresponding to technical metadata items for a given dataset, and updating includes updating one or more instructions with additional instructions for applying data quality controls to technical metadata items for a given dataset.

[0009] In a fifth embodiment, which can be combined with any of the first to fourth embodiments, the method includes updating a metadata model based on one or more instructions in an executable file; detecting that one or more instructions in an executable file cause an update to the metadata model; traversing the metadata model to identify one or more relationships between (i) data added to the metadata model based on the update and (ii) other data in the metadata model; and updating one or more instructions based on the identified relationships in accordance with one or more relationships between (i) data added to the metadata model based on the update and (ii) other data in the metadata model.

[0010] In a sixth embodiment, which can be combined with any of the first to fifth embodiments, the method includes outputting an executable file having updated instructions for execution, or storing the executable file for execution, based on the determination that there are no additional updates to the metadata model.

[0011] In a seventh embodiment, which can be combined with any of the first to sixth embodiments, the method includes receiving an identifier for data that is a candidate for processing from a metadata system, and causing a user interface to render a graphical visualization of the identifier.

[0012] In an eighth embodiment, which can be combined with any of the first to seventh embodiments, the method includes executing an updated instruction on a given dataset.

[0013] In a ninth aspect, which can be combined with any of the first to eighth aspects, the technical metadata includes metadata describing one or more physical attributes of the stored data, such as its technical name, structure, and / or storage location.

[0014] In a tenth embodiment, which can be combined with any of the first to ninth embodiments, the logical metadata includes metadata that provides meaning or context to the data, such as the meaning of the data or the business name and / or the relationship of the data with other data in the ontology.

[0015] In an eleventh embodiment, which can be combined with any of the first to tenth embodiments, the method includes executing updated instructions to process a given dataset in accordance with a specification.

[0016] In a twelfth embodiment, which can be combined with any of the first to eleventh embodiments, executing an updated instruction includes performing an operation on a given dataset.

[0017] In a thirteenth embodiment, which can be combined with any of the first to twelfth embodiments, the logical metadata is personally identifiable information or refers to personally identifiable information.

[0018] In a 14th embodiment, which can be combined with any of the 1st to 13th embodiments, the technical metadata identifies fields within a dataset, such as a given dataset.

[0019] In a 15th embodiment, which can be combined with any of the 1st to 14th embodiments, one or more operations include one or more data synthesis operations such as masking, hashing, reduce, generalize, and / or obfuscate.

[0020] In a 16th embodiment, which can be combined with any of the 1st to 15th embodiments, the logical metadata of a logical metadata item in the metadata model is specified by the specification, refers to personally identifiable information, and executing updated instructions includes performing one or more data synthesis operations specified by the specification on data specified by a technical metadata item for a given dataset corresponding to a technical metadata item in the metadata model related to a logical metadata item in the metadata model.

[0021] In a 17th embodiment, which can be combined with any of the 1st to 16th embodiments, generating one or more instructions to perform at least one of one or more operations on the data of a given dataset is done without additional user interaction.

[0022] Generally, in the 18th aspect, one or more machine-readable hardware storage devices for processing data according to specified criteria, by selecting data to be processed, determining metadata for the selected data, and providing a graphical user interface for automatically processing the selected data according to specified criteria based on the metadata, and storing instructions that can be executed by one or more processing devices to perform any of the operations in the first to 17th aspects.

[0023] In general, the 19th embodiment provides a system for processing data according to specified criteria, by selecting data to be processed, determining metadata for the selected data, and providing a graphical user interface for automatically processing the selected data according to specified criteria based on the metadata, comprising one or more processing devices and one or more machine-readable hardware storage devices that store instructions executable by one or more processing devices to perform any of the operations in the first to 17th embodiments.

[0024] One or more computer systems can be configured to perform a specific operation or action by installing software, firmware, hardware, or a combination thereof on the system that causes the system to perform an action while it is running. One or more computer programs can be configured to perform a specific operation or action by including instructions that cause the data processing device to perform an action when executed by the device.

[0025] One or more of the above embodiments may offer one or more of the following advantages:

[0026] The techniques described herein enable data processing in an efficient and reliable manner with lower latency, fewer errors, and higher accuracy compared to previously known methods. Through the use of blueprints that specify processing requirements (e.g., cleaning, fitting, and transformation requirements), the described system reliably and accurately applies those requirements to the data being retrieved and / or data stored in the system. The system achieves reduced latency because it can process this data in near real-time (with respect to when the request is sent), whereas otherwise, a lengthy code generation and debugging process would have to be waited for. Additionally, these techniques improve the accuracy of applying criteria (e.g., masking PII) because a metadata model is provided. Through the metadata model, the system contributes to assigning data to actions that should be applied to the data, and data quality rules, data types, and data controls can be set at the system-wide level or at the top level of the metadata model. Lower-level nodes automatically inherit these data quality rules, data types, and data controls.

[0027] The metadata model includes nodes that represent the type or name of data. In the metadata model, nodes are connected by edges that represent relationships between the nodes. In one example, a node within the metadata model may represent SSN data. This node is referred to as the SSN node. Next, the SSN node may be related to a node that specifies the name of a data field for storing the SSN. The names of these data fields may be hd73 and j343. The nodes representing these data fields are referred to as data field nodes. The SSN node is the parent node of the data field nodes. A parent node is a node at a level of the metadata model that is higher than the level of other nodes. Thus, the data field nodes inherit from the SSN node. In this example, the system creates a node within the metadata model, labels the created node as personally identifiable information (PII) (e.g., a PII node), and generates an edge between the SSN node and the PII node to specify that the SSN is PII. Here, the attribute of the SSN node is PII. The data field nodes inherit this attribute. Thus, each of the data fields is labeled here as PII, which keeps the data in a securely protected state. This contributes to the security of the data.

[0028] Inheritance refers to the association of the attributes of a parent node with the child nodes of that parent node. A child node is a node at a level within a metadata model that is lower than the level of another node. In this example, if a blueprint contains an instruction to "mask PII", the blueprint does not need to specify which fields within the imported dataset are PII. Rather, when the import process begins, the system described herein traverses the metadata model to identify nodes representing fields within the dataset and then traverses upward to inherit the attributes. If this dataset contains a field of hd73, this field inherits the attributes of the SSN node, and the field of hd73 is marked as PII (e.g., associated with an attribute having a PII value). Based on this traversal, the system updates the instruction to mask hd73 (based on the content of the blueprint). This inheritance process can set data types or attributes for the parent node, and when a dataset (or field) represents a child node related to the parent node, the fields (or the dataset itself) of the dataset automatically inherit the attributes, enhancing the accuracy of the import process.

[0029] Details of one or more embodiments of the present invention are described in the accompanying drawings and the following description. Other features, objects, and advantages of the present invention will become apparent from the specification, drawings, and claims.

Brief Description of the Drawings

[0030] [Figure 1] It is a block diagram of the system. [Figure 2A] It is a diagram of a system for self-service data pipelining. [Figure 2B] It is a diagram of a system for self-service data pipelining. [Figure 3A] It is a diagram of the system of Figure 2A at the stage of data pipelining. [Figure 3B] It is a diagram of the system of Figure 2A at the stage of data pipelining. [Figure 3C] This is a diagram of the system in Figure 2A during the data pipeline implementation phase. [Figure 3D] This is a diagram of the system in Figure 2A during the data pipeline implementation phase. [Figure 3E] This is a diagram of the system in Figure 2A during the data pipeline implementation phase. [Figure 3F] This is a diagram of the system in Figure 2A during the data pipeline implementation phase. [Figure 3G] This is a diagram of the system in Figure 2A during the data pipeline implementation phase. [Figure 3H] This is a diagram of the system in Figure 2A during the data pipeline implementation phase. [Figure 3I] This is a diagram of the system in Figure 2A during the data pipeline implementation phase. [Figure 3J] This is a diagram of the system in Figure 2A during the data pipeline implementation phase. [Figure 3K] This is a diagram of the system in Figure 2A during the data pipeline implementation phase. [Figure 3L] This is a diagram of the system in Figure 2A during the data pipeline implementation phase. [Figure 3M] This is a diagram of the system in Figure 2A during the data pipeline implementation phase. [Figure 3N] This is a diagram of the system in Figure 2A during the data pipeline implementation phase. [Figure 3O] This is a diagram of the system in Figure 2A during the data pipeline implementation phase. [Figure 3P] This is a diagram of the system in Figure 2A during the data pipeline implementation phase. [Figure 3Q] This is a diagram of the system in Figure 2A during the data pipeline implementation phase. [Figure 3R] This is a diagram of the system in Figure 2A during the data pipeline implementation phase. [Figure 3S] This is a diagram of the system in Figure 2A during the data pipeline implementation phase. [Figure 4A] This figure shows how to add transformation rules to the data pipeline. [Figure 4B] This figure shows how to add transformation rules to the data pipeline. [Figure 4C] This figure shows how to add transformation rules to the data pipeline. [Figure 4D] This figure shows how to add transformation rules to the data pipeline. [Figure 4E] This figure shows how to add transformation rules to the data pipeline. [Figure 4F] This figure shows how to add transformation rules to the data pipeline. [Figure 4G] This figure shows how to add transformation rules to the data pipeline. [Figure 5A] This figure shows the cleansing rules and a preview of their output. [Figure 5B] This figure shows the cleansing rules and a preview of their output. [Figure 5C] This figure shows the cleansing rules and a preview of their output. [Figure 5D] This figure shows the cleansing rules and a preview of their output. [Figure 6A] This figure shows data quality rules and their application. [Figure 6B] This figure shows data quality rules and their application. [Figure 6C] This figure shows data quality rules and their application. [Figure 7] This is an exemplary process flowchart for self-service data pipeline implementation. [Figure 8] This diagram shows the details of computer systems, such as data processing systems. [Modes for carrying out the invention]

[0031] Referring to Figure 1, an inefficient system for data ingestion is illustrated. In this example, a business user may request a new dataset, named "Active Customers." Additionally, a data quality and cleansing requirements setting system may specify data quality and cleansing requirements that must be met before data is ingested (e.g., for data security reasons). A programmer receives a request for a new dataset. Using technical metadata, the programmer attempts to identify all fields and datasets in "Active Customers" to which the data quality and cleansing requirements apply. After doing so, code is generated and sent to a Quality Assurance (QA) engineer. Due to the large amount of technical metadata, data quality rules, and cleansing rules that must be considered within the code, the QA engineer inevitably finds errors and notifies the programmer of these errors. The programmer then generates more code to address these errors, resulting in a large amount of code specifying the relationships between technical metadata and data quality, and the cleansing requirements, ultimately to ingest the requested dataset with cleansed conformances (e.g., to maintain data security). This cycle, where programmers fix errors and QA engineers find new ones, can last for months. In this example, after three months, the programmers finally produce code that the QA engineers determine to be error-free or nearly error-free. That code is then sent to an ingestion engine to run the code to ingest the dataset. The final ingested dataset has gaps in cleansing and data quality rules, and in addition, it suffers from high latency because it often takes weeks or even months from the time a request for ingesting a new dataset is made until it is actually ingested. Furthermore, this process is incredibly inefficient because there is no metadata inheritance or attributes as described herein. Overall, the ingestion system shown in Figure 1 is inefficient, inaccurate, and has extremely high latency.

[0032] Referring to Figure 2A, a system 10 for ingesting data in an efficient and reliable manner is shown. In this example, system 10 includes a pipeline executable generator 14 that, when executed, generates an executable file (e.g., code or other logic) that automatically ingests a dataset in a cleaned and adapted manner. The pipeline executable generator 14 includes a pipeline object generator 15. The pipeline object generator 15 generates a pipeline object that includes, for example, a data object or other data structure that specifies the actions to be performed when ingesting data. The pipeline executable generator 14 also includes a metadata inheritance engine 16, which retrieves data quality rules and controls associated with the ingested data from the metadata repository 24. The executable generator 17 retrieves a specified dataset and generates an executable file to perform actions on the dataset specified in the pipeline object. The metadata updater 18 updates the metadata repository 24 with information about the generated executable file and information specifying a new dataset or field that the executable generator 17 specifies should be generated. The metadata change analyzer 19 searches the metadata updater 18 for metadata updates, and if there are metadata updates, the metadata change analyzer 19 sends a request back to the metadata inheritance engine 16 to determine which data quality rules and / or controls are inherited by the new data specified or represented by the new metadata. The pipeline execution engine 21 executes the executable file generated by the executable file generator 17, for example, when there are no additional metadata updates. The optimizer 20 can optionally optimize the executable file before execution, as described in U.S. Patent Application No. 15 / 993,284, entitled "Systems and Methods for Dataflow Graph Optimization," the full details of which are incorporated herein by reference.

[0033] System 10 also includes a developer device 11 for generating blueprints. Generally, a blueprint (sometimes referred to as a specification) contains logic that specifies how data will be processed (e.g., cleaned and adapted) before it is stored (e.g., ingested). Because blueprints are specified before the time of ingestion or processing, datasets to be processed can be easily requested and processed automatically in real time according to the blueprint. As described herein, blueprints define logic with respect to logical metadata rather than technical metadata, and as a result, the logic can be described system-wide and independently of any particular dataset. Generally, technical metadata includes metadata that describes the physical attributes of stored data, such as its technical name (e.g., dataset name, field name, etc.), structure (e.g., record format), and storage location. Logical metadata includes metadata that gives meaning or context to the data, such as the meaning or business name of the data and the relationship of the data with other data in the ontology. System 10 also includes a blueprint engine 12 for sending blueprints or parts of blueprints to various other devices. System 10 includes a client device 13 for specifying one or more datasets to be ingested. System 10 also includes a metadata manager 23 and a metadata repository 24 which may contain a data catalog. System 10 also includes storage systems 22a to 22n.

[0034] Referring to Figure 2B, environment 25 demonstrates the automatic and efficient ingestion of datasets in response to simple user requests 27, such as those submitted by the user via client device 13. These datasets are represented as ingested datasets 29. The metadata manager 23 includes a metadata model 28. The metadata model 28 specifies relationships between different types of data, including datasets, data elements, business data elements, data applications, controls, and PII data. Generally, controls include logic and / or instructions that specify one or more rules and one or more actions to be taken. The metadata model 28 enables metadata inheritance, such that specific data represented in the metadata model is linked to or associated with controls or PII, for example. In this example, a blueprint developer uses developer device 11 to generate a blueprint 26. The blueprint 26 includes generation rules and templates. Generation rules specify the rules that apply to the data being ingested. Templates specify which parts of the generation rules are exposed to the user and which users can view and / or modify. The pipeline executable file generator 14 may read and process the metadata model 28 and / or receive information specifying attributes and / or rules associated with the dataset being processed. This is because generation rules can be defined with respect to logical metadata (represented by nodes) within the metadata model 28. For example, a generation rule may specify that PII should be masked. In this example, the generation rule does not actually specify which fields of the dataset should be masked. Therefore, when applying a generation rule, the metadata inheritance engine determines which data elements are associated with nodes representing PII in the metadata model 28.

[0035] Referring to Figure 3A, the graphical user interface 31 is displayed on the developer device. The graphical user interface 31 displays a rendering of the Blueprint Editor for Blueprint developers to specify generation rules and templates. Generally, the Blueprint Editor includes i) logic for specifying and storing generation rules, ii) logic for specifying and storing templates, and iii) logic for specifying that for each of the one or more generation rules created or stored, a portion of the template corresponds to that generation rule (for example, logic for exposing the generation rule (or a portion thereof) through the template, for example, for editing and / or viewing).

[0036] The graphical user interface 31 includes a section 32 for specifying and viewing generation rules. Generally, generation rules include rules that specify one or more actions to be performed on a dataset that is being ingested or processed. Rules are defined with respect to logical metadata (e.g., data that provides semantic meaning to technical metadata). For example, PII is logical metadata. A rule may be defined as "mask PII". This rule is referred to as a PII rule. The system described herein determines which fields of a dataset the PII rule should be applied to by traversing the metadata model and identifying the technical metadata associated with the PII logical metadata (identifying the fields of the dataset). In this way, generation rules provide abstractions that can be automatically applied to new datasets ingested into the system (when the metadata model is updated according to those new datasets).

[0037] Section 32 includes control 32a, which, depending on the selection, allows the user to add a new generation rule. The graphical user interface 31 includes section 33 representing a template. As previously mentioned, the template specifies which parts of the generation rule are exposed to the user. Additionally, the Blueprint Editor includes logic for generating one or more cells shown in section 33. For example, if the user creates a raw generation rule, the Blueprint Editor includes logic for generating a raw zone with a raw dataset. In this example, the contents of the graphical user interface 31 are a visual rendering of the logic in the Blueprint Editor.

[0038] Referring to Figure 3B, view 34 shows a blueprint 26 sent from the developer device 11 to the blueprint engine 12. The contents of the blueprint 26 are shown in visualization 35, which includes a section 35a representing the generation rules and a section 35b representing the template. Section 35a displays visual representations 32b to 32h, each representing the generation rule associated with it.

[0039] Referring to Figure 3C, the graphical user interface 36 shows an alternative form of blueprint. In this example, the blueprint includes generation rules 36a and templates 36b. The generation rules are shown graphically, illustrating the logic of each generation rule. Template 36b describes the functionality to be performed with respect to each or part of the generation rules.

[0040] In some cases, a single Blueprint may have multiple different modes, each used to generate a different executable file, thereby eliminating the need to create new Blueprints that perform variations of several operations. For example, a "Data Lake Inbound" Blueprint may support the following three modes (however, it may support different modes and / or different numbers of modes without departing from the scope of this disclosure): 1. Insertion (for example, inserting a new dataset into a data lake) 2. Time-varying (for example, comparing today's data with data from previous periods, performing change data capture (CDC), and outputting the time-varying dataset to a data lake) 3. Archive reload (for example, taking a user-specified data range and reloading the dataset from the archived data using a more efficient single-pass algorithm).

[0041] In this way, each mode generates a completely different executable file (e.g., a dataflow graph topology), while the logic controlling the generation of the executable files resides within a single blueprint. In addition, the selection between different modes can be controlled by metadata. For example, the user can interact with a dropdown list in the "Pipeline Graph" column (see column 40g in Figure 3E) to select which generation mode to choose and, therefore, which executable file to generate.

[0042] Referring to Figure 3D, view 37 shows the communication between the metadata manager 23, the blueprint engine 12, the client device 13, the pipeline object generator 15, and the executable file generator 17. The metadata manager 23 sends data 38a to the blueprint engine 12 specifying the names of available datasets, for example, those that are available for ingestion. In this example, the metadata manager 23 first determines the names or identifiers of these available datasets based on technical metadata. In this example, the metadata manager 23 receives technical metadata from storage systems 22a...22n (as shown in Figure 2A) that specify or otherwise identify the datasets of fields within those storage systems. Using this received technical metadata, the metadata manager 23 generates the names of the available datasets. The blueprint engine 12 uses the blueprint 26, which has data 38a specifying the names of the available datasets, to generate an instruction 38. In particular, instruction 38 is an instruction to render a visualization of the template using the available datasets. When generating instruction 38, the Blueprint Engine 12 utilizes the template of Blueprint 26 (as visually depicted in part 35b (Figure 3B)). The Blueprint Engine 12 also sends Blueprint 26 to the Pipeline Object Generator 15 and the Executable File Generator 17. In this example, the Blueprint Engine 12 is configured to use the template specified in Blueprint 26 and update it with renderings of available datasets that can be selected for ingestion, as will be explained later with reference to Figure 3F.

[0043] Referring to Figure 3E, the graphical user interface 40 shows the rendering of instruction 38. In particular, the graphical user interface 40 includes columns 40b, 40c, 40d, 40e, 40f, and 40g. In this example, column 40b specifies the name of the pipeline. Column 40c specifies the name of the source dataset being ingested into the system. Column 40d specifies that each source dataset must be copied before any additional functionality is performed. Column 40e specifies that the dataset to be ingested is cleansed. Column 40f specifies that the dataset to be ingested is fitted. Column 40g contains controls, the selection of which enables the generation of the underlying logic for actually performing the ingestion. In this example, each of these columns is specified in the blueprint. That is, in this example, the blueprint specifies the columns, and the user can specify the values ​​of those columns or view the attributes of those columns via the user interface 40. The graphical user interface 40 also includes a search box 40a from which the user can search for a specific dataset to be ingested.

[0044] Referring to Figure 3F, view 41 shows the sending of a request 42 about the pipeline from the client device 13 to the pipeline object generator 15. The client device 13 renders graphical user interface 40', which is a version of graphical user interface 40 in which the active customer dataset has been selected for ingestion. In this example, graphical user interface 40' includes a section 40'' that displays the datasets that are candidates for ingestion. The blueprint engine 12 generates data about section 40'' based on data 38a (Figure 3D) that specifies the names of the datasets that are candidates for ingestion.

[0045] Referring to Figure 3G, view 43 shows the generation of the pipeline object 44 and its transmission from the pipeline object generator 15 to the metadata inheritance engine 16. In response to request 42 (Figure 3F), the pipeline object generator 15 uses the blueprint 26 to generate the pipeline object 44, which contains a data structure that specifies i) the functions to be performed by the pipeline execution engine 21 when processing the data, and ii) the attributes, characteristics, or data values ​​associated with those functions. The pipeline object generator 15 generates the pipeline object 44 as follows, for example: The pipeline object generator 15 reads each generation rule (as shown in parts 32b-32h of Figure 3B), and for each generation rule, the pipeline object generator 15 assigns a portion of the pipeline object to that rule. In this example, based on the generation rules (shown in parts 32b-32h of Figure 3B), the pipeline object generator 15 generates portions 44a-44g. Each portion specifies a function and data related to that function. For example, part 44a specifies the functionality of the source dataset to be read (shown as Source Dataset in Figure 3F). The data associated with that functionality is "Active_cust.dat" included in request 42 (Figure 3F). Therefore, part 44a specifies that Active_cust.dat should be read. Each functionality in parts 44a to 44g corresponds to the functionality of the generation rule to which that part is assigned. For each part, its attributes are determined, among other things, from the input data, the generation rule itself, metadata attributes and / or inheritance. That is, the blueprint 26 defines the functionality specified in the pipeline object 44. Then, how that functionality is applied to a particular dataset to be ingested or processed is determined based on user input and metadata attributes and / or inheritance. In determining how the functionality is applied, the pipeline executable generator 14 populates each of parts 44a to 44g based on user input or metadata attributes and / or inheritance.For example, part 44a is populated in "active_cust.dat" based on user input in part 40'' (Figure 3F).

[0046] In this example, pipeline object 44 specifies the source dataset functions to be performed on its source dataset and the resulting dataset. For example, in part 44a, pipeline object 44 specifies that in this example, the source dataset is active_cust.dat. Part 44b specifies that a raw dataset is generated based on the raw generation rule specified by part 32c (Figure 3B). This raw dataset is also called Active_cust.dat. Part 44b includes the word "generated" to specify that this raw (or copied dataset) dataset is created, since the generation rule specifies that this raw (or copied dataset) dataset must be created. In this way, this raw dataset is generated. Generally, a raw dataset is a copy of the source dataset.

[0047] Part 44c specifies the cleansing rules to be applied to the raw dataset. Part 44d specifies the data quality rules to be applied to the raw dataset. Part 44e specifies the resulting dataset, which is the result of applying the cleansing and data quality rules, and is also named Active_cust.dat. Part 44f specifies the transformation rules to be applied to the cleansed dataset. Part 44g specifies the adapted dataset, which is the result of applying the transformed rules to the cleansed dataset, and is also named Active_cust.dat.

[0048] As mentioned above, it is the Blueprint itself that specifies that for a given source dataset, a raw dataset must first be generated, and then cleansing and data quality rules can be applied to it. It is the Blueprint itself that specifies that a new cleaned dataset is generated and stored after the cleansing and data quality rules have been applied. Additionally, it is the Blueprint itself that specifies that transformed rules are applied when specified. It is the Blueprint that a new adapted dataset is generated based on the application of transformed rules to the cleaned dataset. As illustrated in the preceding and subsequent diagrams, these parts of the pipeline object 44 are popularized as part of the data ingestion process. While this example describes pipeline ingestion of data, it will be understood by those skilled in the art that the pipeline object 44 may be equally applicable to any system or functionality for modifying data, applying cleansing rules to it, and adapting it, even to data already in the system.

[0049] Referring to Figure 3H, view 45 shows the update of pipeline object 44 based on metadata inheritance. In this example, the metadata inheritance engine 16 receives pipeline object 44 and detects that parts 44b, 44e, and 44g specify that a new dataset will be created. Therefore, the metadata inheritance engine 16 requests the record format of the active_cust.dat dataset from the metadata manager 23. Based on this request, the metadata manager 23 traverses the metadata model 47 and identifies node 47a, which represents the source dataset. Based on this, the metadata manager 23 begins traversing the metadata model 47, moving up the layers, and identifies nodes 47b, 47c, and 47d, which together represent the record format of active_cust.dat (e.g., field names and order). The field names are cem, pc05, and bdate14, each of which is an item of technical metadata. Business data elements (BDEs), such as names, represent metadata that provides semantic meaning to logical metadata, e.g., technical metadata. When generating metadata models, semantic discovery may be applied to already stored or ingested fields, as described in U.S. Patent Application No. 16 / 794,361, entitled "Discovering a Semantic Meaning of Data Fields from Profile Data of the Data Fields," the full details of which are incorporated herein by reference.

[0050] In this example, the metadata model 47 includes several layers that represent relationships between nodes and layers, or between nodes and layers, or between nodes and edges.

[0051] The metadata manager 23 sends the inherited metadata 46 to the metadata inheritance engine 16. Using the inherited metadata 46, the metadata inheritance engine 16 updates parts 44b, 44e, and 44g. These updated parts 44b, 44e, and 44g specify the format of these new datasets, and the raw dataset, cleansed dataset, and adapted dataset are generated when the data is ingested into the system, respectively.

[0052] Additionally, in parts 44b, 44e, and 44g, the generated words indicate that these datasets were generated based on the Blueprint's generation rules. That is, the Blueprint itself specifies that for each source dataset, that source dataset is copied, cleansed, and each resulting dataset is landed and fitted into the raw zone (raw dataset), cleansed zone (cleansed dataset), and fitted zone (fitted dataset).

[0053] Referring to Figure 3I, view 50 shows the generation of executable logic from pipeline object 44. In this example, the metadata inheritance engine 16 sends pipeline object 44 to executable file generator 17. Based on pipeline object 44, executable file generator 17 generates executable file 52. Executable file generator 17 may do this using the technique described in U.S. Patent Application No. 15 / 795,917, entitled "Transforming a Specification into a Persistent Computer Program," the full details of which are incorporated herein by reference. In one example, executable file generator 17 stores a template having components for reading a dataset and components for writing a dataset. Executable file generator 17 also includes the logic necessary to add additional components to the template, for example, based on the contents of pipeline object 44. These additional components include an "Apply" component. Executable file generator 17 is configured to add appropriate parameter values ​​to each of the components (when generating the graph) based on values ​​in the pipeline object. For example, for a read component in a template, the executable file generator 17 updates the read component with the value in active_cust.dat based on the value in part 44a (Figure 3G) of the pipeline object 44 (Figure 3G). Additionally, when parts 44c, 44d, and 44f are populated, the executable file generator 17 is configured to add components to the graph to perform the functions specified in those parts and to update the added components with the values ​​specified in those parts.

[0054] The executable file generator 17 sends the executable file 52 to the metadata updater 18. As described herein, the metadata updater 18 checks whether any additional updates need to be made to the executable file, based on the fact that the executable file itself generates a new dataset, and the new dataset may need to inherit attributes or rules based on the metadata model.

[0055] Referring to Figure 3J, view 53 shows the traversal of the metadata model for metadata and attribute inheritance. In this example, the metadata updater 18 sends metadata 54 for the pipelined dataset to the metadata manager 23. The metadata updater 18 identifies that the metadata manager 23 needs to be updated with the new metadata by identifying in the executable file 52 (referenced in Figure 3I) that the read data has been written to three new datasets: confirmed active_cust.dat, copy active_cust.dat, and cleansed active_cust.dat. In response to receiving the metadata 54, the metadata manager 23 updates the metadata model 47 with nodes representing the copy, cleansed, and adapted datasets, resulting in an updated metadata model 47'. This updated metadata model 47' includes nodes 47a' representing the source, copy, cleansed, and adapted datasets. Additionally, the metadata model 47 is updated with fields for each of the new datasets. Next, each of these new fields is linked to or otherwise associated with the business data element to which the corresponding field in the source dataset is associated. In particular, the metadata manager 23 generates new nodes 47b', 47c', and 47d', each of which corresponds to nodes 47b, 47c, and 47d for the source dataset. These new nodes 47b', 47c', and 47d' are associated with nodes 47e, 47f, and 47g in the metadata model 47'. Thus, the new datasets (e.g., copy, cleanse, and adapt datasets) are reflected here in the metadata model 47', and each of the fields in these new datasets is also reflected in the metadata model 47', and the fields in these new datasets are also correctly linked to the appropriate business data elements. For example, here, the cem field in the cleansed dataset is correctly linked to the email address business data element.

[0056] Referring to Figure 3K, view 55 shows the detection of new metadata, which triggers an iterative process in metadata inheritance. In this example, the metadata updater 18 also sends metadata 54 to the metadata change analyzer 19. When the metadata change analyzer 19 receives the metadata (such as metadata 54), it identifies that there has been a change in the metadata within the metadata model 47. Once the metadata change analyzer 19 identifies that there has been a change in the metadata within the metadata model 47, it initiates an iterative process to determine whether the new metadata in the metadata model needs to inherit any attributes, rules, or controls from the parent node. In this example, the metadata change analyzer 19 sends a notification 56 of the changed metadata to the metadata inheritance engine 16. This notification 56 specifies the new data represented in the metadata model 47'. In this example, the new data is the copied, cleansed, and adapted dataset 47a' and nodes 47b', 47c, and 47d' represented in the metadata model 47'.

[0057] Referring to Figure 3L, view 57 shows that the metadata inheritance engine 16 begins the process of identifying whether nodes in the metadata model 47' representing fields of the copied, cleansed, and adapted dataset need to inherit any data quality controls or any PII levels. In this example, the metadata manager 23 traverses the metadata model 47' and identifies that the cem field (in each of the copied, cleansed, and adapted datasets) is associated with an email address node, and the email address node is associated with a PII level node. Based on this identification of inherited metadata, the metadata manager 23 sends inherited metadata 58 to the metadata inheritance engine 16. Inherited metadata 58 specifies cem=PII. The metadata inheritance engine 16 then updates the pipeline object 44 by specifying “mask cem” in part 44c. In this example, the blueprint itself is specified to mask the PII. In this case, the metadata inheritance engine 16 has identified that the field labeled “cem” is PII. Therefore, the metadata inheritance engine 16 specifies that the cleansing rule is “mask cem”. Additionally, part 44c specifies that this cleansing rule is generated to indicate that it is based on the application of a generation rule, which is based on traversing a metadata model 47' to identify which fields are PII.

[0058] Referring to Figure 3M, view 59 shows an additional traverse of the metadata model 47'. In addition to inheriting metadata 58, the metadata inheritance engine 16 also inherits metadata 60 based on a further traverse of the metadata model 47' by the metadata manager 23. In this example, the metadata manager 23 identifies that a node representing the business name of an email address is associated with a control specifying that the email address is a required field, and this control is associated with a data application node, and this data application node is associated with two data quality controls specifying that fields with required=yes must be populated and that the email address must be valid. Thus, the metadata manager 23 identifies that each of the cem fields in the copy, cleanse, and adapt datasets must be populated and must have a valid email address. This information is specified in the inherited metadata 60. The metadata inheritance engine 16 updates part 44d of the pipeline object 44 with the information specified in the inherited metadata 60.

[0059] Referring to Figure 3N, view 61 shows an additional traverse of the metadata model 47'. In this example, the metadata model specifies that each bdate field in the copied, cleansed, and adapted datasets is associated with a business data element of date of birth and with a control specifying that the date of birth must be before or equal to January 1, 1900. Based on this traversal and the identification of the data quality control for the date field, the metadata manager 23 generates inheritance metadata 62 and sends it to the metadata inheritance engine 16. In response to receiving the inheritance metadata 62, the metadata inheritance engine 16 further updates portion 44d.

[0060] Referring to Figure 3O, view 63 shows the update of the executable file 52 based on the updated pipeline object 44, which is then updated by identifying the linkage in the metadata model 47' between the newly added data in the metadata model 47' and the associated nodes in the metadata model 47'. That is, for example, new data may be added to the metadata model 47 by adding a new node and linking that node to a node in a layer higher than the layer of the newly added node. However, that linkage is not reflected in the executable file 52 until the pipeline object 44 is updated according to that linkage, thereby updating the executable file 52. In this example, the executable file 52 is updated with components (as specified in part 44d of Figure 3M) that specify that the cleansing rule applies to "mask cem" and that the data quality rule applies, resulting in the executable file 52', which is a version of the executable file 52. The executable file generator 17 sends the executable file 52' to the metadata updater 18.

[0061] Referring to Figure 3P, view 64 shows the receipt of the executable file 52' by the pipeline execution engine 21. In this example, the metadata updater 18 detects in the executable file 52' that there are no changes to the metadata in the metadata model 47'. Therefore, the metadata updater 18 sends the executable file 52' to the metadata change analyzer 19. The metadata change analyzer 19 sends the executable file 52' to the pipeline execution engine 21. The metadata change analyzer 19 does not receive any additional metadata about the pipelined dataset from the metadata updater 18. Therefore, the metadata change analyzer 19 determines that there are no changes to the metadata model.

[0062] Referring to Figure 3Q, optimizer 20 generates executable file 52'', which is a version of executable file 52'. In executable file 52'', the component for applying data quality rules is moved to come before the component for applying cleansing rules. Optimizer 20 performs this move to improve processing efficiency, because masking an email address is inefficient, even if the email address is invalid.

[0063] Referring to Figure 3R, view 65 shows real-time data acquisition using the executable file 52''. In this example, the storage system 22a sends the source dataset 66 to the pipeline execution engine 21. The pipeline execution engine 21 executes the executable file 52''. View 68 shows the resulting data from the execution of each component in the executable file 52''. View 68 shows that data 69a results from the execution of component 68a. Data 69b results from the execution of component 68b. Data 69c results from the execution of component 68c. Data 69d results from the execution of component 68d. Data 69e results from the execution of component 68e. Data 69f results from the execution of component 68f. Following the execution of the executable file 52'', the pipeline execution engine 21 outputs the dataset 67 to the storage system 22n.

[0064] Referring to Figure 3S, view 70 shows the storage of metadata 70a resulting from the execution of the executable file 52'' by the pipeline execution engine 21. In this example, the metadata 70a specifies, for example, which datasets are being generated, which components are being executed successfully, and, for example, where the resulting actual fitted and / or cleansed data is stored. In some examples, the metadata 70a specifies the data quality results from the execution of the executable file 52'' by the pipeline execution engine 21.

[0065] Referring to Figure 4A, the graphical user interface 71 includes cells 71a to 71e, which display icons for specifying the pipeline name, source dataset name, raw dataset name, cleansed dataset name, and transformation rules, respectively. Cell 71d also specifies the name of the adapted dataset.

[0066] Referring to Figure 4B, the overlay 72 is displayed alongside or on top of the graphical user interface 71, for example, when a plus icon is selected in a portion or cell 71e (referenced in Figure 4A). The overlay 72 allows the user to input transformation rules to be applied during the import process and to apply those rules. In general, import refers to retrieving data from one or more data sources and performing one or more actions on that data before either outputting the data or storing the data.

[0067] Referring to Figure 4C, view 73 shows that pipeline objects are updated based on transformation rules entered into overlay 72 or otherwise described. In this example, client device 13 sends update 74 to object generator 15. Update 74 specifies the transformation rules entered into overlay 72 or otherwise described. In response to receiving update 74, pipeline object generator 15 updates part 44f to indicate that there is a transformation rule that reformats the name to format: last name, first name. In this example, part 44f includes the phrase “stated” to reflect the fact that this transformation rule is described in the system or otherwise entered, rather than being based on a generation rule that is part of the blueprint.

[0068] Referring to Figure 4D, view 75 shows the traverse of the metadata model 47' in the further generation of transformation rules. In this example, the metadata inheritance engine 16 sends a request to the metadata manager 23 that one or more fields are associated with the business data element "Name". In response, the metadata manager 23 traverses the metadata model 47' to identify that field pc05 is associated with the business data element "Name". Thus, the metadata manager 23 sends the inherited metadata 76 to the metadata inheritance engine 16. The metadata inheritance engine 16 updates part 44f to specify the field name that is actually being reformatted.

[0069] Referring to Figure 4E, view 77 shows the update to the executable file 52'' in part 44f of pipeline object 44 according to the transformation rules specified. The executable file generator 17 generates the executable file 52'''' and sends the executable file 52'''' to the metadata updater 18.

[0070] Referring to Figure 4F, view 78 shows the sending of the executable file 52''' to the pipeline execution engine 21. In this example, the metadata updater 18 sends the executable file 52''' to the metadata change analyzer 19. The metadata change analyzer 19 detects that there are no changes to the metadata model, since at least the metadata change analyzer 19 has not received instructions for those changes from the metadata updater 18. Therefore, the metadata change analyzer 19 sends the executable file 52''' to the pipeline execution engine 21. In this example, the metadata change analyzer 19 determines that there are no further updates to the metadata model, and therefore there is no additional reason to determine whether attributes and / or data quality rules need to be inherited by those new metadata items.

[0071] Referring to Figure 4G, view 79 shows the execution of executable file 52''. In this example, executable file 52'' contains updates that apply transformation rules. View 80 shows the data resulting from the execution of each component within executable file 52''.

[0072] Referring to Figure 5A, the graphical user interface 81 allows the user to see the cleansing rules applied to a particular dataset for selecting an icon in cell 81a. In this example, the name of the pipeline is US_customers_active. Generally, a pipeline refers to a final dataset that is cleansed, fitted, and / or transformed and stored in a specified storage system. In another example, a pipeline generally refers to the name given or assigned to a set of actions or tasks for retrieving data and applying specified cleansing, fitting, and / or transformation rules to that data. That is, the name of the pipeline refers to the name of the process for creating the data that is ultimately cleansed and fitted.

[0073] Referring to Figure 5B, an overlay 82 is shown on the graphical user interface 81. The overlay 82 displays the cleansing rules applied to the source dataset of active_cust.dat.

[0074] Referring to Figure 5C, selecting icon 81b provides a preview of the data being ingested into the system according to the cleansing, conforming, and transforming of data quality rules.

[0075] Referring to Figure 5D, overlay 83 shows a preview of the data records and source dataset that have been cleansed and adapted according to the blueprint and defined rules.

[0076] Referring to Figure 6A, the graphical user interface 84 provides a view of the underlying source dataset. In this example, the graphical user interface 84 includes a column 84a representing the source dataset in the raw zone. In this example, row 84b represents a specific pipeline. Cell 84c within row 84b contains a selectable portion, and its selection triggers the rendering of an overlay 85. The overlay 85 contains details of the source dataset being ingested by the pipeline, as specified by row 84b. In this example, the overlay 85 includes a selectable portion 85a, and its selection allows the user to see the controls applied to the dataset.

[0077] Referring to Figure 6B, the graphical user interface 86 is rendered following the selection of the selectable portion 85a. In this example, cell 87 displays a description of one of the controls applied to this particular dataset. Selecting cell 87 displays the graphical user interface shown in Figure 6C.

[0078] Referring to Figure 6C, the graphical user interface 88 is displayed. The graphical user interface 88 includes a section 88c that displays the business name of the field to which the control described in cell 87 applies. In this example, section 88c also indicates whether a particular field failed the control. In this example, the field represented by the business name shown in cell 88b has failed this control, as shown in cell 88a. By providing this real-time review of the data and indicating which controls succeeded or failed for which fields, the systems described herein provide a very granular level of results review.

[0079] Referring to Figure 7, a process 100 for generating an on-demand integration profile is shown. During operation, the data processing system receives a specification that specifies logical metadata such as PII and one or more data synthesis operations to be performed on the data associated with the logical metadata (e.g., masking, hashing, reduction, generalization, and / or obfuscation) (102). The data processing system provides a user interface for indicating one or more datasets to be retrieved and processed (103). The data processing system receives user instructions for a given dataset from the user interface (104). In response to at least the user instructions, the data processing system generates one or more instructions that can be executed to process the given dataset according to the specification, such as to maintain data security by performing data synthesis operations on the data including PII (105). The data processing system identifies technical metadata such as fields for a given dataset (106). The data processing system accesses a metadata model that specifies the relationships between logical metadata and technical metadata (107). The data processing system traverses the metadata model to identify relationships between (i) logical metadata items in the metadata model (e.g., PII) and (ii) technical metadata items in the metadata model (e.g., fields) that correspond to technical metadata items for a given dataset (108). The data processing system updates one or more instructions according to the identified relationships between (i) logical metadata items in the metadata model and (ii) technical metadata items in the metadata model that correspond to technical metadata items for a given dataset (109), so that, for example, the execution of an instruction to process a given dataset according to a specification maintains data security by performing a data synthesis operation on data including PII.

[0080] In some examples, traversing the metadata model involves identifying relationships between (i) logical metadata entries in the metadata model corresponding to logical metadata in a specification, and (ii) technical metadata entries in the metadata model corresponding to technical metadata entries for a given dataset. Based on the identified relationships, one or more instructions executable for processing a given dataset can be modified or otherwise updated to specify that at least one of the actions associated with the logical metadata in the specification will be performed on the data of the given dataset represented by the technical metadata entries in the metadata model corresponding to the technical metadata entries.

[0081] In some examples, traversing a metadata model involves identifying data quality controls or other controls that should be applied to technical metadata items in the metadata model that correspond to technical metadata items for a given dataset, and updating one or more instructions that can be executed to process a given dataset involves modifying or otherwise updating one or more instructions with additional instructions to apply data quality controls to the data corresponding to technical metadata items for a given dataset.

[0082] As described herein, a dataflow graph component includes data processing components and / or datasets. A dataflow graph can be represented by a directed graph containing nodes or vertices, where the nodes or vertices represent dataflow graph components, connected by directed links or dataflow connections, and represent the flow of work elements (i.e., data) between dataflow graph components. A data processing component includes code for processing data from at least one data input (e.g., a data source) and providing the data to at least one data output of the system (e.g., a data sink). Thus, a dataflow graph can perform graph-based computations on data flowing from one or more input datasets through graph components to one or more output datasets.

[0083] The system also includes a data processing system for executing one or more computer programs (such as a data flow graph) generated by transforming a specification into a computer program using the transformation generators and techniques described herein. The transformation generator transforms the specification into a computer program. In this example, selections made by the user through the user interface described herein form a specification that specifies which data sources are to be taken in. Based on the specification, the transformations described herein are generated.

[0084] A data processing system may be hosted on one or more general-purpose computers under the control of a suitable operating system, such as a UNIX operating system. For example, a data processing system may include a multi-node parallel computing environment that includes a computer system configuration using multiple central processing units (CPUs) that are local (e.g., microprocessor systems such as SMP computers), locally distributed (e.g., multiple processors linked as a cluster or MPP), remotely distributed (e.g., multiple processors linked via a LAN or WAN network), or any combination thereof.

[0085] The graph construction approach described above can be implemented using software to run on a computer. For example, the software may form a procedure in one or more computer programs running on one or more systems, e.g., computer-programmed systems or computer-programmable systems (which may be of various architectures such as distributed, client / server, or grid), each system including at least one processor, at least one data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device or port, and at least one output device or port. The software may form one or more modules of a larger program that provides other services related to the design and construction of data flow graphs, for example. The nodes and elements of the graph can be implemented as data structures stored in computer-readable media, or as other organized data conforming to data models stored in a data repository.

[0086] The software may be provided on a hardware storage device readable by a general-purpose or dedicated programmable computer, such as a non-temporary storage medium like a CD-ROM, or it may be delivered (encoded into a propagating signal) to a computer on which the software is executed via a network communication medium. All functions may be performed on a dedicated computer or using dedicated hardware such as a coprocessor. The software may be implemented in a distributed manner, with different parts of the data flow specified by the software being performed by different computers. Each such computer program is preferably stored or downloaded onto a non-temporary storage medium or hardware storage device readable by a general-purpose or dedicated programmable computer (e.g., solid-state memory or media, or magnetic or optical media), and the computer is configured to operate to perform the procedures described herein when the non-temporary storage medium or device is read by the system. The system may also be considered implemented as a computer-readable storage medium configured with a computer program, and such a storage medium operates the system in a specific and predefined manner to perform the functions described herein.

[0087] Exemplary computing environment Referring to Figure 8, an exemplary operating environment for carrying out embodiments of the present invention is shown, and is explicitly referred to as a computing device 120. Essential elements of the computing device 120, or a computer, data processing system, client, or server, are one or more programmable processors 122 for performing actions according to instructions, and one or more memory devices 124 for storing instructions and data. Generally, the computer also includes I / O components 126, such as a display device, a network / communication subsystem (not shown), and one or more mass storage devices 128 for storing data and instructions, etc., and a network / communication subsystem 130 powered by a power supply (not shown), or operably connected to them (via a bus 121, fabric, network, etc.). The memory devices 124 contain an operating system 124a and an application 124b for application programming.

[0088] Suitable devices for storing computer program instructions and data include, for example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory can be complemented by or incorporated into dedicated logic circuits.

[0089] To provide user interaction, embodiments of the subject matter described herein are implemented on a computer, having a display device (monitor) for displaying information to the user, a keyboard on which the user can provide input to the computer, and a pointing device (e.g., a mouse or trackball). In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user (for example, by sending a web page to a web browser on the user's user device in response to a request received from a web browser).

[0090] Embodiments of the subject matter described herein can be implemented in a computing system that includes a backend component (e.g., as a data server), a middleware component (e.g., an application server), a frontend component (e.g., a user computer having a graphical user interface or a web browser that allows a user to interact with the implementation of the subject matter described herein), or any combination of one or more such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad-hoc peer-to-peer networks).

[0091] A computing system can include users and servers. Clients and servers are generally geographically separated and typically interact through a communication network. The relationship between a client and a server is established by computer programs running on each computer that have a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to client devices (for example, to display data to a user interacting with a user device and to receive user input from the user interacting with the user device). Data generated on the client device (e.g., the results of user interaction) can be received by the server from the client device.

[0092] This specification includes many details of specific implementations, but these should not be construed as limitations on the scope of any invention or claim, but rather as descriptions of features specific to a particular embodiment of a particular invention.

[0093] Similarly, while the operations are depicted in a specific order in the drawings, it should not be understood that such operations must be performed in that specific or sequential order, or that all illustrated operations must be performed, in order to achieve the desired result. Furthermore, the separation of various system components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0094] Several embodiments have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of the techniques described herein. For example, some of the above steps may be sequence-independent and can therefore be performed in an order different from that described. In addition, any of the aforementioned techniques described in relation to data flow graphs can also be implemented and performed in relation to programs. Therefore, other embodiments are also within the scope of the following claims.

Claims

1. A method implemented by a data processing system for receiving instructions for a dataset to be processed and processing that dataset according to one or more operations specified in the specification, The data processing system receives a specification that specifies logical metadata and one or more actions to be performed on the data associated with the logical metadata. To provide a user interface for indicating one or more datasets that are retrieved and processed by the aforementioned data processing system, The user interface receives instructions for a given dataset, This includes, at least in response to the instruction, generating one or more instructions that are executable for processing the given dataset in accordance with the specifications, and the generation is Identifying technical metadata for the given dataset, Accessing a metadata model from a data source that specifies the relationship between logical metadata and technical metadata, (i) Traversing the metadata model to identify the relationship between the logical metadata items in the metadata model corresponding to the logical metadata of the specification, and (ii) the technical metadata items in the metadata model corresponding to the technical metadata items for the given dataset, A method comprising: (i) generating one or more instructions to perform at least one of the one or more operations on the data of a given dataset, in accordance with the identified relationship between the item of logical metadata in the metadata model corresponding to the logical metadata of the specification, and (ii) the item of technical metadata in the metadata model corresponding to the item of technical metadata for a given dataset.

2. The method according to claim 1, wherein generating one or more instructions to perform at least one of the one or more operations on the data of the given dataset is done without additional user interaction.

3. Based on the traversal, the further includes identifying data quality controls to be applied to the technical metadata items in the metadata model corresponding to the technical metadata items for the given dataset, Updating The method according to claim 1, comprising updating one or more of the instructions with additional instructions for applying the data quality controls to the items of technical metadata for the given dataset.

4. Updating the metadata model based on one or more instructions in the executable file, To detect that one or more instructions in the executable file cause the metadata model to be updated, (i) traversing the metadata model to identify one or more relationships between the data added to the metadata model as a result of the update, and (ii) other data in the metadata model. The method according to claim 1, further comprising: (i) updating the data added to the metadata model based on the update; and (ii) updating the one or more instructions in accordance with the one or more relationships between the other data in the metadata model, based on the one or more identified relationships.

5. Based on the decision that there will be no further updates to the metadata model, Based on the updated instructions, generate an executable file, The method according to claim 4, further comprising storing the executable file for execution.

6. The metadata system receives identifiers for data that are candidates for processing, The method according to claim 1, further comprising rendering a graphical visualization of the identifier on the user interface.

7. The method according to claim 1, further comprising executing one or more instructions on the given dataset.

8. One or more machine-readable hardware storage devices for receiving instructions for a dataset to be processed and processing the dataset according to one or more operations specified in the specifications, wherein the one or more machine-readable hardware storage devices store instructions that can be executed by one or more processing devices to perform the operations, and the operations are The data processing system receives a specification that specifies logical metadata and one or more actions to be performed on the data associated with the logical metadata. To provide a user interface for indicating one or more datasets that are retrieved and processed by the aforementioned data processing system, The user interface receives instructions for a given dataset, This includes, at least in response to the instruction, generating one or more instructions that are executable for processing the given dataset in accordance with the specifications, and the generation is Identifying technical metadata for the given dataset, Accessing a metadata model from a data source that specifies the relationship between logical metadata and technical metadata, (i) Traversing the metadata model to identify the relationship between the logical metadata items in the metadata model corresponding to the logical metadata of the specification, and (ii) the technical metadata items in the metadata model corresponding to the technical metadata items for the given dataset, One or more machine-readable hardware storage devices, comprising: (i) generating one or more instructions to perform at least one of the one or more operations on the data of a given dataset, in accordance with the identified relationship between the items of logical metadata in the metadata model corresponding to the logical metadata of the specification, and (ii) generating one or more instructions to perform at least one of the one or more operations on the data of a given dataset.

9. One or more machine-readable hardware storage devices according to claim 8, wherein generating one or more instructions to perform at least one of the one or more operations on the data of the given dataset is done without additional user interaction.

10. The aforementioned operation, Based on the traversal, the further includes identifying data quality controls to be applied to the technical metadata items in the metadata model corresponding to the technical metadata items for the given dataset, Updating One or more machine-readable hardware storage devices according to claim 8, comprising updating one or more of the instructions with additional instructions for applying the data quality control to the items of technical metadata for the given dataset.

11. The aforementioned operation, Updating the metadata model based on one or more instructions in the executable file, To detect that one or more instructions in the executable file cause the metadata model to be updated, (i) traversing the metadata model to identify one or more relationships between the data added to the metadata model as a result of the update, and (ii) other data in the metadata model. One or more machine-readable hardware storage devices according to claim 8, further comprising: (i) updating the data added to the metadata model based on the update; and (ii) updating the one or more instructions in accordance with the one or more relationships between the other data in the metadata model, based on the one or more identified relationships.

12. The aforementioned operation, Based on the decision that there will be no further updates to the metadata model, Based on the updated instructions, generate an executable file, One or more machine-readable hardware storage devices according to claim 11, further comprising storing the aforementioned executable file for execution.

13. The aforementioned operation, The metadata system receives identifiers for data that are candidates for processing, The machine-readable hardware storage device according to claim 8, further comprising rendering a graphical visualization of the identifier on the user interface.

14. The aforementioned operation, The machine-readable hardware storage device according to claim 8, further comprising executing one or more instructions on the given dataset.

15. A system for receiving instructions for a dataset to be processed and for processing that dataset according to one or more operations specified in the specifications, One or more processing devices, Includes one or more machine-readable hardware storage devices that store instructions that can be executed by one or more processing devices in order to perform an operation, wherein the operation is The data processing system receives a specification that specifies logical metadata and one or more actions to be performed on the data associated with the logical metadata. To provide a user interface for indicating one or more datasets that are retrieved and processed by the aforementioned data processing system, The user interface receives instructions for a given dataset, This includes, at least in response to the instruction, generating one or more instructions that are executable for processing the given dataset in accordance with the specifications, and the generation is Identifying technical metadata for the given dataset, Accessing a metadata model from a data source that specifies the relationship between logical metadata and technical metadata, (i) Traversing the metadata model to identify the relationship between the logical metadata items in the metadata model corresponding to the logical metadata of the specification, and (ii) the technical metadata items in the metadata model corresponding to the technical metadata items for the given dataset, A system comprising: (i) generating one or more instructions to perform at least one of the one or more operations on the data of a given dataset, in accordance with the identified relationship between the items of logical metadata in the metadata model corresponding to the logical metadata of the specification, and (ii) generating one or more instructions to perform at least one of the one or more operations on the data of a given dataset.

16. The system according to claim 15, wherein generating one or more instructions to perform at least one of the one or more operations on the data of the given dataset is done without additional user interaction.

17. The aforementioned operation, Based on the traversal, the further includes identifying data quality controls to be applied to the technical metadata items in the metadata model corresponding to the technical metadata items for the given dataset, Updating The system according to claim 15, comprising updating one or more of the instructions with additional instructions for applying the data quality controls to the items of technical metadata for the given dataset.

18. The aforementioned operation, Updating the metadata model based on one or more instructions in the executable file, To detect that one or more instructions in the executable file cause the metadata model to be updated, (i) traversing the metadata model to identify one or more relationships between the data added to the metadata model as a result of the update, and (ii) other data in the metadata model. The system according to claim 15, further comprising: (i) updating the data added to the metadata model based on the update; and (ii) updating the one or more instructions in accordance with the one or more relationships between the other data in the metadata model, based on the identified one or more relationships.

19. The aforementioned operation, Based on the decision that there will be no further updates to the metadata model, Based on the updated instructions, generate an executable file, The system according to claim 18, further comprising storing the executable file for execution.

20. The aforementioned operation, The metadata system receives identifiers for data that are candidates for processing, The system according to claim 15, further comprising rendering a graphical visualization of the identifier on the user interface.

21. The aforementioned operation, The system according to claim 15, further comprising executing one or more instructions on the given dataset.

Images