METADATA-CONTROLLED ACCOUNTING AND DATA PROCESSING

Abstract

A method implemented by a data processing system to enable a system to pipeline or otherwise process data in accordance with specified criteria by providing a graphical user interface for selecting data to be processed, determining metadata of selected data, and, based on the metadata, automatically processing the selected data in accordance with the specified criteria.

Description

priority claim

[0001] This application claims priority from U.S. patent application 18 / 496,543, filed on October 27, 2023, which claims priority from U.S. patent application 63 / 495,618, filed on April 12, 2023, under 35 USC §119(e), the entire contents of which are hereby incorporated by reference. TECHNICAL AREA

[0002] This disclosure concerns techniques to enable a data processing system to pipeline or otherwise process data in accordance with specified criteria. BACKGROUND

[0003] Modern data processing systems manage large volumes of data within an organization. A large organization, for example, might have millions of data records. These records can support multiple aspects of the organization's operations. Complex data processing systems typically process data in multiple stages, with the results produced by one stage being fed into the next. The overall flow of information through such systems can be described as a directed data flow graph, where nodes or vertices in the graph represent components (either data files or processes), and the connections or "edges" in the graph indicate flows of data between the components. A system for performing such graph-based computations is described in U.S. Patent No. 5,966,072, entitled "Executing Computations Expressed as Graphs," incorporated herein by reference.

[0004] In many cases, a company's data is distributed across multiple disparate data sources, and the company needs to consolidate this data to facilitate data storage and analysis. To achieve this, the company can employ a data ingestion process, which moves data from one or more data sources to a destination, such as a data lake, data repository, or other data storage system. Once ingested, the data can be stored, analyzed, or otherwise used. SUMMARY

[0005] In general, in a first aspect, a procedure implemented by a data processing system for receiving a specification of a data record to be processed and for processing that data record according to one or more operations specified in a specification comprises the following: Receiving, by the data processing system, a specification that specifies logical metadata and one or more operations to be performed on data associated with the logical metadata; Providing, by the data processing system, a user interface for specifying one or more data records to be retrieved and processed; Receiving, from the user interface, a specification of a given data record;and in response to at least the specification, generating one or more instructions executable to process the given data set according to the specification, comprising: identifying technical metadata for the given data set; accessing, from a data source, a metadata model that specifies relationships between logical metadata and technical metadata; traversing the metadata model to identify a relationship between (i) an element of logical metadata in the metadata model that corresponds to the logical metadata of the specification, and (ii) an element of technical metadata in the metadata model that corresponds to an element of the technical metadata for the given data set;and generating the one or more instructions to perform at least one of the one or more operations on data of the given dataset according to the identified relationship between (i) the element of logical metadata in the metadata model that corresponds to the logical metadata of the specification, and (ii) the element of technical metadata in the metadata model that corresponds to the element of technical metadata for the given dataset.

[0006] In general, in a second aspect, a procedure implemented by a data processing system for processing data in accordance with specified criteria comprises: providing a graphical user interface for selecting data to be processed; determining metadata of the selected data; and, based on the metadata, automatically processing the selected data in accordance with the specified criteria: receiving, by the data processing system, a specification that defines logical metadata and one or more operations to be performed on data associated with the logical metadata; providing, by the data processing system, a user interface for specifying one or more data records to be retrieved and processed; and receiving, from the user interface, a user specification of a given data record.and in response to at least the user's input, generate one or more executable instructions to process the given data set according to the specification; identify technical metadata for the given data set; access a metadata model that specifies relationships between logical metadata and technical metadata; traverse the metadata model to identify a relationship between (i) an element of logical metadata in the metadata model and (ii) an element of technical metadata in the metadata model that corresponds to an element of technical metadata for the given data set; and update the one or more instructions according to the identified relationship between (i) the element of logical metadata in the metadata model and (ii) the element of technical metadata in the metadata model that corresponds to the element of technical metadata for the given data set.

[0007] In a third aspect, which can be combined with the first or second aspect, traversing includes: traversing the metadata model to identify a relationship between (i) an element of logical metadata in the metadata model that corresponds to logical metadata of the specification, and (ii) an element of technical metadata in the metadata model that corresponds to an element of technical metadata for the given dataset; and updating includes: updating the one or more instructions to specify that at least one of the one or more operations is performed on data represented by the element of technical metadata in the metadata model that corresponds to the element of technical metadata for the given dataset.

[0008] In a fourth aspect, which can be combined with any of the first to third aspects, the procedure, based on traversing, includes identifying a data quality control to be applied to the element of technical metadata in the metadata model that corresponds to the element of technical metadata for the given dataset; and where updating includes updating the one or more instructions with additional instructions to apply the data quality control to the element of technical metadata for the given dataset.

[0009] In a fifth aspect, which can be combined with any of the first four aspects, the procedure includes updating the metadata model based on the one or more instructions of the executable file; recognizing that the one or more instructions of the executable file cause an update of the metadata model; traversing the metadata model to identify one or more relationships between (i) data added to the metadata model based on the update and (ii) other data in the metadata model; and, based on the one or more identified relationships, updating the one or more instructions according to the one or more relationships between (i) the data added to the metadata model based on the update and (ii) the other data in the metadata model.

[0010] In a sixth aspect, which can be combined with any of the first to fifth aspects, the procedure, based on determining no additional updates to the metadata model, includes outputting an executable file with updated instructions for execution; or saving the executable file for execution.

[0011] In a seventh aspect, which can be combined with any of the first to sixth aspects, the procedure includes receiving, from a metadata system, identifiers of data that are candidates for processing, according to the specified criteria; and causing the user interface to display graphical visualizations of the identifiers.

[0012] In an eighth aspect, which can be combined with any of the first to seventh aspects, the procedure includes: executing the updated instructions on the given data set.

[0013] In a ninth aspect, which can be combined with any of the first to eighth aspects, technical metadata includes metadata that describes one or more physical attributes of stored data, such as its technical name, structure and / or storage location.

[0014] In a tenth aspect, which can be combined with any of the first to ninth aspects, logical metadata includes metadata that provides meaning or context for data, such as its semantic or business name and / or its relationship to other data within an ontology.

[0015] In an eleventh aspect, which can be combined with any of the first to tenth aspects, the procedure includes executing the updated instructions to process the given data set according to the specification.

[0016] In a twelfth aspect, which can be combined with any of the first to eleventh aspects, executing the updated instructions involves performing the operations on the given data set.

[0017] In a thirteenth aspect, which can be combined with any of the first to twelfth aspects, the logical metadata are or refer to personally identifiable information.

[0018] In a fourteenth aspect, which can be combined with any of the first to thirteenth aspects, the technical metadata identifies a field in a dataset, such as the given dataset.

[0019] In a fifteenth aspect, which can be combined with any of the first to fourteenth aspects, the one or more operations include one or more data synthesis operations, such as masking, hashing, reducing, generalizing and / or obfuscation.

[0020] In a sixteenth aspect, which can be combined with any of the first to fifteenth aspects, the logical metadata of the logical metadata element in the metadata model is specified by the specification and relates to personally identifiable information, and the execution of the updated instructions involves performing one or more data synthesis operations specified by the specification on data specified by the technical metadata element for the given dataset, which corresponds to the technical metadata element in the metadata model that relates to the logical metadata element in the metadata model.

[0021] In a seventeenth aspect, which can be combined with any of the first to sixteenth aspects, the generation of one or more instructions to perform at least one of the one or more operations on the data of the given data set takes place without additional user interaction.

[0022] In general, in an eighteenth aspect, one or more machine-readable hardware storage devices store data for processing in accordance with specified criteria by providing a graphical user interface for selecting data to be processed, determining metadata of the selected data and based on the metadata, and automatically processing the selected data in accordance with the specified criteria, wherein the one or more machine-readable hardware storage devices store instructions that can be executed by one or more processing devices to perform the operations according to one of the first to seventeenth aspects.

[0023] In general, in a nineteenth aspect, a system for processing data in accordance with specified criteria by providing a graphical user interface for selecting data to be processed, determining metadata of the selected data, and, based on the metadata, automatically processing the selected data in accordance with the specified criteria, comprises: one or more processing devices; and one or more machine-readable hardware storage devices that store instructions executable by one or more processing devices to perform the operations according to any of the first through seventeenth aspects.

[0024] A system of one or more computers can be configured to perform specific operations or actions by installing software, firmware, hardware, or a combination thereof on the system that, during operation, causes the system to perform the actions. One or more computer programs can be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause the device to perform the actions.

[0025] One or more of the above aspects can provide one or more of the following benefits.

[0026] The techniques described herein enable efficient and reliable data processing with lower latency, fewer errors, and increased accuracy compared to previously known methods. By using a blueprint that specifies the processing requirements (e.g., requirements for cleaning, adapting, and transforming), the described system reliably and accurately applies these requirements to data being retrieved and / or data being stored in the system. The system achieves reduced latency because it can process this data in near real-time (in terms of when the request is sent), whereas it would otherwise have to wait for a lengthy code generation and debugging process. Additionally, these techniques improve the accuracy of applying criteria (e.g., masked PII) because a metadata model is provided.The metadata model allows the system to assign operations to be applied to the data, as well as to set data quality rules, data types, and data controls at a system-wide level or at a top-level metadata model. Lower-level nodes automatically inherit these data quality rules, data types, and data controls.

[0027] The metadata model contains nodes that represent data types or names. In the metadata model, the nodes are connected by edges that represent relationships between them. For example, a node in the metadata model might represent SSN data. This node is called the SSN node. The SSN node, in turn, can be related to nodes that specify the names of data fields for storing SSNs. These data field names might be hd73 and j343. The nodes representing these data fields are called data field nodes. The SSN node is a parent node to the data field nodes. A parent node is a node at a level of the metadata model that is higher than any level of other nodes. Thus, the data field nodes inherit from the SSN node.In this example, the system indicates that SSNs are personally identifiable information (PII) by creating a node in the metadata model, labeling this node as PII (e.g., PII node), and establishing an edge between the SSN node and the PII node. Now, an attribute of the SSN node is PII. The data field nodes inherit this attribute. Thus, each data field is now labeled as PII, effectively and reliably securing the data. This contributes to data security.

[0028] Inheritance refers to the attributes of a parent node that are associated with the child nodes of that parent node. A child node is a node located at a level in the metadata model lower than the level of any other node. In this example, if the blueprint includes a "masking PII" instruction, the blueprint does not need to specify which fields in the ingested dataset are PII. Rather, once the ingestion process begins, the system described here traverses the metadata model to identify nodes representing fields in the dataset and then traverses upwards to inherit attributes. If this dataset contains a field from hd73, then that field inherits the attributes of the SSN node, and the field from hd73 is marked as PII (e.g., associated with an attribute containing a PII value).Based on this iteration, the system updates instructions (based on blueprint content) to mask hd73. This inheritance process increases the accuracy of the ingestion process because a data type or attribute can be set for a parent node, and fields of a record (or the record itself) automatically inherit the attributes if the record (or field) is a child node of the parent node.

[0029] The details of one or more embodiments of the invention are set forth in the accompanying drawings and the following description. Other features, functions, and advantages of the invention will become apparent from the description, the drawings, and the claims. DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a block diagram of a system. Fig. 2A and Fig. 2B are diagrams of a system for self-service data pipelines. Fig. 3A to Fig. 3S are diagrams of the system of Fig. 2A in stages of data pipeline. Fig. 4A to Fig. 4G are diagrams that illustrate the addition of transformation rules to a data pipeline. Fig. 5A to Fig. 5D diagrams are diagrams that show cleaning rules and a preview of their output. Fig. 6A to Fig. 6C are diagrams that illustrate data quality rules and their application. Fig. Figure 7 is a flowchart of an exemplary process for self-service data pipeline. Fig. Figure 8 is a diagram showing details of a computer system, such as a data processing system. DETAILED DESCRIPTION

[0030] With reference to Fig. Figure 1 shows an inefficient data ingestion system. In this example, a business user can request a new record. In this example, a new record is referred to as an active customer. Additionally, a data quality and cleanup requirement setting system can specify data quality and cleanup requirements that must occur (e.g., for data security reasons) before data is ingested. The programmer receives the request for the new record. Using technical metadata, the programmer attempts to identify all fields and records in active customers to which the data quality and cleanup requirements apply. Code is then generated and sent to a quality assurance (QA) engineer.Due to the massive amount of technical metadata, data quality rules, and cleaning rules that must be considered in the code, the QA engineer inevitably finds errors and notifies the programmer. The programmer, in turn, writes more code to fix these errors, resulting in a massive amount of code to specify the relationships between the technical metadata and the data quality and cleaning requirements, ultimately to include the requested dataset in a clean, compliant state (e.g., to maintain data security). This cycle of the programmer fixing errors and the QA engineer finding new ones can continue for months. In this example, after three months, the programmer has finally produced code that the QA engineer determines to be error-free or nearly error-free.This code is sent to the ingestion machine, which executes the code to ingest the records. The records that are ultimately ingested will have missed cleaning and data quality rules, and there will be high latency, as it will often take weeks or months from the time a request to ingest the new record until it is actually ingested. Additionally, this process is incredibly inefficient because there is no metadata inheritance or mapping as described herein. Overall, this is in . Fig. The recording system shown is inefficient, inaccurate, and involves a high amount of latency.

[0031] With reference to Fig. Figure 2A shows a system 10 for efficiently and reliably ingesting data. In this example, the system 10 includes an executable pipeline generator 14, which generates an executable file (e.g., code or other logic) that, when executed, automatically ingests data records in a clean and compliant manner. The executable pipeline generator 14 includes a pipeline object generator 15. The pipeline object generator 15 generates a pipeline object that contains, for example, a data object or other data structure specifying actions to be performed when ingesting data. The executable pipeline generator 14 also includes a metadata inheritance engine 16, which retrieves data quality rules and controls from the metadata repository 24 that are associated with the data being ingested.The executable generator 17 creates an executable file to retrieve specified data records and perform actions on them as specified in the pipeline object. The metadata updater 18 updates the metadata repository 24 with information about the executable file being created, as well as with information specifying any new data records or fields that the executable generator 17 should create. The metadata modification analyzer 19 checks for metadata updates from the metadata updater 18 and, if a metadata update is found, the metadata modification analyzer 19 sends a request back to the metadata inheritance engine 16 to determine which data quality rules and / or controls are inherited by the new data specified or represented by the new metadata.The pipeline execution engine 21 executes an executable file generated by the executable generator 17, for example, if no additional metadata updates are available. The optimizer 20 can optionally optimize the executable file before execution, as described in US patent application number 15 / 993,284, entitled "Systems and Methods for Dataflow Graph Optimization," the entire contents of which are incorporated herein by reference.

[0032] System 10 also includes the developer device 11 for generating a blueprint. In general, a blueprint (sometimes called a specification) comprises logic that specifies how data is processed (e.g., cleaned and adjusted) before storage (e.g., ingestion). Because the blueprint is specified before any ingestion or processing, a data set to be processed can be easily requested and automatically processed in real time according to the blueprint. As described herein, a blueprint defines logic in the form of logical metadata rather than technical metadata—so that the logic can be described system-wide and independent of any specific data set. Technical metadata generally includes metadata that describes physical attributes of stored data, such as its technical name (e.g., record name, field name, etc.), its structure (e.g., record format), and its location.Logical metadata includes metadata that provides meaning or context for data, such as its semantic or business name and its relationship to other data within an ontology. System 10 also includes the blueprint engine 12, which transfers the blueprint or parts of the blueprint to various other devices. System 10 includes the client device 13 for specifying one or more datasets to be ingested. System 10 also includes the metadata manager 23 and the metadata repository 24, which can contain a data catalog. System 10 also includes storage systems 22a through 22n.

[0033] With reference to Fig. Figure 2B illustrates the environment 25, which automatically and efficiently captures records in response to a simple request 27 from a user, submitted via the client device 13. These records are represented as captured records 29. The metadata manager 23 includes the metadata model 28. The metadata model 28 specifies relationships between different types of data, including records, data elements, business data elements, data applications, controls, and PII data. In general, a control includes logic and / or instructions that specify one or more rules and one or more actions to be taken. The metadata model 28 enables metadata inheritance because certain data represented in the metadata model are linked or associated with controls or PII, for example.In this example, the blueprint developer uses the developer device 11 to generate a blueprint 26. A blueprint 26 includes generation rules and a template. Generation rules specify rules to be applied to the data being ingested. The template specifies which parts of the generation rules are exposed to a user, allowing that user to view and / or modify them. The executable pipeline generator 14 can read and process the metadata model 28 and / or receive information specifying attributes and / or rules to be associated with the dataset being processed. This is because the generation rules can be defined in terms of logical metadata (represented by nodes) in the metadata model 28. For example, a generation rule might specify masking PII.In this example, the generation rule does not actually specify which fields of a data record are to be masked. Thus, when applying generation rules, the metadata inheritance engine determines which data elements are associated with the node that represents PII in the metadata model 28.

[0034] With reference to Fig. 3A displays the graphical user interface 31 on the developer device. The graphical user interface 31 displays a rendering of a blueprint editor for a blueprint developer to specify generation rules and a template. In general, a blueprint editor includes logic for i) specifying and storing generation rules, ii) specifying and storing a template, and iii) specifying that for each of the one or more generated or stored generation rules, a part of the template corresponds to that generation rule (e.g., logic to expose the generation rules (or parts thereof) through the template—for example, for editing and / or viewing).

[0035] The graphical user interface 31 includes a section 32 for specifying and viewing generation rules. In general, a generation rule comprises a rule that specifies one or more operations to be performed on a data record being ingested or processed. The rule is defined in terms of logical metadata (e.g., data that provides semantic meaning for technical metadata). For example, PII is logical metadata. A rule might be defined as "mask PII." This rule is referred to as the PII rule. The system described herein determines which fields of a data record should apply the PII rule by traversing a metadata model to identify technical metadata (identifying the fields of the data record) associated with the logical PII metadata.In this way, the generation rules provide an abstraction and can be automatically applied to new data records that are added to the system (once the metadata model has been updated according to these new data records).

[0036] Part 32 includes a control 32a, the selection of which allows a user to add a new generation rule. The graphical user interface 31 includes a part 33, which represents the template. As previously described, the template specifies which parts of the generation rules are exposed to a user. Additionally, the blueprint editor includes logic for generating the one or more cells shown in Part 33. For example, if a user generates a raw generation rule, the blueprint editor includes logic for generating a raw zone with a raw data record. In this example, the contents of the graphical user interface 31 are a visual representation of the logic of the blueprint editor.

[0037] With reference to Fig. Figure 3B illustrates view 34, which shows blueprint 26 being transferred from the developer device 11 to the blueprint engine 12. The contents of blueprint 26 are shown in a visualization 35, which comprises part 35a, representing a generation rule, and part 35b, representing the template. Part 35a displays visual representations 32b-32h, each of which represents an associated generation rule.

[0038] With reference to Fig. Figure 3C shows the graphical user interface (Figure 36) as an alternative form of a blueprint. In this example, the blueprint includes generation rules (Figure 36a) and a template (Figure 36b). Generation rules are shown graphically and illustrate the logic of each generation rule. The template (Figure 36b) describes functionality that should be performed with respect to each generation rule or a part thereof.

[0039] In some examples, a single blueprint can have several different modes, each used to generate different executable files, thus avoiding the need to create new blueprints that perform variations of some processing. For example, a "Data Lake Ingestion" blueprint can support the following three modes (although different modes and / or a different number of modes may be supported without exceeding the scope of this disclosure): 1. Insertion (e.g., inserting a new record into a data lake) 2. Time variant (e.g., compares today's data with data from a previous period, performs change data capture (CDC), and outputs a time-variant dataset to a data lake) 3. Archive Reload (e.g., takes a user-specified data range and reloads a dataset from archived data using a more efficient single-pass algorithm)

[0040] In this way, each mode generates a completely different executable file (e.g., a data flow graph topology), while the logic for controlling executable generation resides within a single blueprint. Additionally, the selection between different modes can be controlled by metadata. For example, a user can be presented with a dropdown list in the "Pipeline Graph" column (see column 40g of [reference]). Fig. 3E) interact to select which generation mode and thus which executable file to generate.

[0041] With reference to Fig. 3D view 37 shows the communication between the metadata manager 23, the blueprint engine 12, the client device 13, the pipeline object generator 15, and the executable generator 17. The metadata manager 23 transmits data 38a to the blueprint engine 12, specifying the names of available data records, including, for example, data records available for ingestion. In this example, the metadata manager 23 initially determines the names of these available data records, or identifiers of these data records, based on technical metadata. In this example, the metadata manager 23 receives from storage systems 22a to 22n (as shown in Fig. (2A shown) technical metadata that specifies or otherwise identifies the field's records within these storage systems. Using this received technical metadata, the metadata manager 23 generates the names of the available records. The blueprint engine 12 uses the blueprint 26 with data 38a, which specifies the names of the available records, to generate instructions 38. Specifically, the instructions 38 are instructions for reproducing a visualization of the template with the available records. In generating the instructions 38, the blueprint engine 12 uses the template (as visually shown in part 35b ( Fig. 3B) shown) of Blueprint 26. The Blueprint Engine 12 also transfers Blueprint 26 to the Pipeline Object Generator 15 and the Executable Generator 17. In this example, the Blueprint Engine 12 is configured to use the template specified in Blueprint 26 and update it with a representation of the available records that can be selected for ingestion, as shown below with reference to Fig. 3F described.

[0042] With reference to Fig. Figure 3E illustrates the graphical user interface 40, which reproduces the instructions in Figure 38. Specifically, the graphical user interface 40 includes columns 40b, 40c, 40d, 40e, 40f, and 40g. In this example, column 40b specifies the name of a pipeline. Column 40c specifies the names of the source records to be ingested into the system. Column 40d specifies that each source record must be copied before any additional functionality is performed. Column 40e specifies that the records to be ingested must be cleansed. Column 40f specifies that the records to be ingested must be compliant. Column 40g includes controls, the selection of which enables the generation of the underlying logic to actually perform the ingestion. In this example, each of these columns is specified in the blueprint.In this example, the blueprint specifies the columns, and the user interface 40 allows a user to specify values ​​for these columns or view attributes of these columns. The graphical user interface 40 also includes the search field 40a, in which a user can search for a specific record to be included.

[0043] With reference to Fig. Figure 3F illustrates the transfer of the pipeline request 42 from the client device 13 to the pipeline object generator 15. The client device 13 represents the graphical user interface 40', which is a version of the graphical user interface 40 in which the active customer record for ingestion has been selected. In this example, the graphical user interface 40' includes a part 40", which displays the records that are candidates for ingestion. The blueprint engine 12 generates the data for part 40" based on the data 38a ( Fig. 3D), which specify the names of the datasets that are candidates for inclusion.

[0044] With reference to Fig. 3G illustrates in view 43 the creation of the pipeline object 44 and the transfer of the pipeline object 44 from the pipeline object generator 15 to the metadata inheritance engine 16. In response to requirement 42 ( Fig. 3F) The pipeline object generator 15 uses the blueprint 26 to create the pipeline object 44, which contains a data structure that specifies i) functionality to be performed by the pipeline execution engine 21 when processing data, and ii) attributes, properties, or data values ​​associated with that functionality. The pipeline object generator 15 creates the pipeline object 44, for example, as follows: The pipeline object generator 15 reads each creation rule (as in parts 32b-32h of Fig. (shown in 3B), for each generation rule, the pipeline object generator 15 assigns a part of the pipeline object to that rule. In this example, the pipeline object generator 15 generates based on the generation rules (shown in parts 32b-32h of Fig. 3B) Parts 44a-44g. Each part specifies a functionality and data related to that functionality. For example, part 44a specifies a functionality of the read source dataset (in Fig. 3F is shown as the source data set). The data associated with this functionality is “Active_cust.dat”, which is specified in requirement 42 ( Fig. 3F). As such, part 44a specifies to read Active_cust.dat. The functionality for each of parts 44a-44g corresponds to the function of the generation rule to which that part is assigned. For each part, the attributes of that part are determined from, among other things, input data, the generation rules themselves, metadata mapping, and / or inheritance. That is, blueprint 26 defines the functionality specified in pipeline object 44. Then, based on user input and metadata mapping and inheritance, it is determined how this functionality is applied to a particular data record to be ingested or processed. In determining how the functionality is applied, the executable pipeline generator 14 populates each of parts 44a-44g based on user input or metadata mapping and / or inheritance. For example, part 44a is populated with "active_cust.dat" based on user input in part 40 ( Fig. 3F) filled.

[0045] In this example, pipeline object 44 specifies source record functionality to be performed on that source record and resulting records. For example, in part 44a, pipeline object 44 specifies that in this example, the source record is active_cust.dat. Part 44b specifies that a raw record based on the one defined by part 32c ( Fig. 3B) is generated according to the specified raw generation rule. This raw record is also referred to as Active_cust.dat. Part 44b includes the word "generated" to specify that this raw record (or copied record) is generated because the generation rules specify that it must be generated. Thus, this raw record is generated. In general, a raw record is a copy of a source record.

[0046] Part 44c specifies the cleaning rules to be applied to the raw dataset. Part 44d specifies data quality rules to be applied to the raw dataset. Part 44e specifies the resulting dataset, which is the outcome of applying the cleaning and data quality rules, and is also referred to as Active_cust.dat. Part 44f specifies transformation rules to be applied to the cleaned dataset. Part 44g specifies a modified dataset, which is the outcome of applying the transformed rules to the cleaned dataset, and is also referred to as Active_cust.dat.

[0047] As previously described, it is the blueprint itself that specifies that—for a given source dataset—a raw dataset must first be created, and then cleaning and data quality rules can be applied as appropriate. It is the blueprint itself that specifies that, after the cleaning and data quality rules have been applied, a new, cleaned dataset is then created and stored. Additionally, it is the blueprint itself that specifies that transformed rules are applied when specified. It is the blueprint that specifies that a new, customized dataset is created based on the application of the transformed rules to the cleaned dataset. As described in the preceding and following figures, these parts of pipeline object 44 are populated as part of the data ingestion process.While this example is described in relation to pipeline ingestion of data, it will be understood by the average professional in the field that the pipeline object 44 can be applied equally to any system or functionality for modifying data, applying cleaning rules to it, and adapting it, even data that is already internal within a system.

[0048] With reference to Fig. Figure 3H shows the update of pipeline object 44 based on metadata inheritance. In this example, the metadata inheritance engine 16 receives pipeline object 44 and recognizes that parts 44b, 44e, and 44g specify that new records are being created. As such, the metadata inheritance engine 16 requests a record format for the Active_cust.dat record from the metadata manager 23. Based on this request, the metadata manager 23 traverses the metadata model 47 and identifies node 47a, which represents the source record. Based on this, the metadata manager 23 begins traversing the metadata model 47 and enters a layer to identify nodes 47b, 47c, and 47d, which together represent the record format (e.g., field names and order) for Active_cust.dat. The field names are cem, pc05, bdate14 - each of which is an element of technical metadata.Business data elements (BDE) – such as name – represent logical metadata, e.g., metadata that provides semantic meaning for technical metadata. When creating a metadata model, semantic discovery can be applied to previously stored or recorded fields, as described in U.S. Patent Application No. 16 / 794,361, entitled “Discovering a Semantic Meaning of Data Fields from Profile Data of the Data Fields,” the entire content of which is incorporated herein by reference.

[0049] In this example, the metadata model 47 comprises a number of layers with edges between the layers or nodes that represent relationships between or between the nodes and layers.

[0050] The Metadata Manager 23 transfers inherited metadata 46 to the Metadata Inheritance Engine 16. Using the inherited metadata 46, the Metadata Inheritance Engine 16 updates parts 44b, 44e, and 44g. These updated parts 44b, 44e, and 44g specify formats for these new records—a raw record, a cleaned record, and an adapted record are each created as the data is ingested into the system.

[0051] Additionally, the generated word in parts 44b, 44e, and 44g indicates that these records are generated based on the blueprint's generation rules. That is, the blueprint itself specifies that for each source record, this source record is copied, cleaned, and then adjusted with each resulting record that has landed in a raw zone (the raw record), a cleaned zone (the cleaned record), and an adjusted zone (the adjusted record).

[0052] With reference to Fig. Figure 3I illustrates the generation of executable logic from the pipeline object 44. In this example, the metadata inheritance engine 16 transfers the pipeline object 44 to the executable generator 17. Based on the pipeline object 44, the executable generator 17 creates the executable file 52. The executable generator 17 can do this using the techniques described in U.S. patent application serial number 15 / 795,917, entitled "Transforming a Specification into a Persistent Computer Program," the entirety of which is incorporated herein by reference. In one example, the executable generator 17 stores a template with a component for reading a record and a component for writing a record. The executable generator 17 also includes the logic required to add additional components to the template, for example, based on contents of the pipeline object 44.These additional components include "Apply" components. The executable generator 17 is configured to add appropriate parameter values ​​to each of the components (when generating a graph) based on values ​​in the pipeline object. For example, for a component read in the template, the executable generator 17 updates that component with a value from active_cust.dat, based on the value in part 44a. Fig. 3G) of pipeline object 44 ( Fig. 3G). Additionally, while parts 44c, 44d and 44f are populated, the executable generator 17 is configured to add components to the graph to perform the functionality specified in these parts and to update the added components with values ​​specified in these parts.

[0053] The executable generator 17 transfers the executable file 52 to the metadata updater 18. As described herein, the metadata updater 18 will see if additional updates need to be made to the executable file based on the fact that the executable file itself generates new records that in turn need to inherit attributes or rules based on the metadata model.

[0054] With reference to Fig. Figure 3J illustrates the traversal of a metadata model for metadata and attribute inheritance in Figure 53. In this example, the metadata updater 18 transfers metadata 54 for pipeline records to the metadata manager 23. The metadata updater 18 identifies that the metadata manager 23 needs to be updated with new metadata by calling the executable file 52 (located in Figure 3J). Fig. (Referring to 3I) the fact is identified that the read data is written to three new records – adapted active_cust.dat, copied active_cust.dat, and purified active_cust.dat. In response to receiving the metadata 54, the metadata manager 23 updates the metadata model 47 with nodes representing a copy, purge, and adaptation record, resulting in an updated metadata model 47'. This updated metadata model 47' includes the nodes 47a', which represent the source, copy, purge, and adaptation records. Additionally, the metadata model 47 is updated with fields for each of the new records. Each of these new fields is, in turn, linked or otherwise associated with the business data elements to which the corresponding field of the source record was associated.Specifically, the metadata manager creates 23 new nodes 47b', 47c', 47d' – each corresponding to nodes 47b, 47c, 47d for the source record. These new nodes 47b', 47c', 47d' are associated with nodes 47e, 47f, 47g in metadata model 47'. As such, new records (e.g., copy, clean, and adjustment records) are now reflected in metadata model 47', and fields for each of these new records are also reflected in metadata model 47'. The fields of these new records are also correctly linked to the corresponding business data elements. For example, the cem field in the cleansed record is now appropriately linked to the business data element of the email address.

[0055] With reference to Fig. 3K illustrates in View 55 the detection of new metadata, which in turn triggers an iterative process in metadata inheritance. In this example, the metadata updater 18 also sends metadata 54 to the metadata modification analyzer 19. When the metadata modification analyzer 19 receives metadata (such as metadata 54), it identifies that a metadata change has occurred in the metadata model 47. Once the metadata modification analyzer 19 identifies that a metadata change has occurred in the metadata model 47, it initiates an iterative process of identifying whether the new metadata in the metadata model needs to inherit any attributes, rules, or controls from parent nodes.In this example, the metadata modification analyzer 19 transmits a notification 56 of the modified metadata to the metadata inheritance engine 16. This notification 56 specifies the new data represented in the metadata model 47'. In this example, the new data are the copy, clean, and fit records 47a' and the nodes 47b', 47c, 47d' represented in the metadata model 47'.

[0056] With reference to Fig. 3L illustrates the metadata inheritance engine 16 in view 57, which initiates the process of identifying whether the nodes in the metadata model 47', representing the fields of the copy, clean, and match records, need to inherit any data quality controls or any PII levels. In this example, the metadata manager 23 traverses the metadata model 47' to identify that the cem field (of each of the copy, clean, and match records) is associated with the email address node, which in turn is associated with a PII-level node. Based on this identification of inherited metadata, the metadata manager 23 passes inherited metadata 58 to the metadata inheritance engine 16. The inherited metadata 58 specifies that cem = PII. The metadata inheritance engine 16, in turn, updates the pipeline object 44 by specifying "mask-cem" in part 44c.In this case, the metadata inheritance engine 16 identified that the field marked as 'cem' is PII. Additionally, part 44c specifies that it is generated to indicate that this cleanup rule is based on the application of the generation rules, which in turn are based on traversing the metadata model 47' to identify which fields are PII.

[0057] With reference to Fig. 3M illustrates in View 59 the additional traversal of the metadata model 47'. In addition to inheriting the metadata 58, the metadata inheritance engine 16 also inherits the metadata 60 based on the further traversal of the metadata model 47' by the metadata manager 23. In this example, the metadata manager 23 identifies that the node representing a business name of the email address is associated with a control specifying that the email address is a required field, which in turn is associated with the data application node, which is associated with two data quality controls specifying that fields with required = yes must be populated and that the email address must be valid. As such, the metadata manager 23 identifies that each of the cem fields in the copy, clean, and match records must be populated and must contain a valid email address.This information is specified in inherited metadata 60. The metadata inheritance engine 16 updates part 44d of pipeline object 44 with the information specified in inherited metadata 60.

[0058] With reference to Fig. 3N illustrates the additional traverse of the metadata model 47' in view 61. In this example, the metadata model specifies that the date field of each of the copy, clean, and match records is associated with the business data element of the date of birth, which in turn is associated with a control specifying that the date of birth must be before or equal to January 1, 1900. Based on this traverse and identification of a data quality control for the date field, the metadata manager 23 generates inherited metadata 62 and passes it to the metadata inheritance engine 16. In response to receiving the inherited metadata 62, the metadata inheritance engine 16 further updates part 44d.

[0059] With reference to Fig. Figure 3O illustrates the updating of the executable file 52 based on the updated pipeline object 44, which in turn is updated by identifying links in the metadata model 47' between newly added data in the metadata model and related nodes in the metadata model 47'. That is, for example, new data can be added to the metadata model 47 by adding a new node and then linking that node to a node in a layer above a layer of the newly added node. However, this link is not reflected in the executable file 52 until the pipeline object 44 is updated according to this link, which in turn causes an update of the executable file 52.In this example, the executable file 52 is updated with components that specify that the cleaning rules should be applied to "Masken-cem" and that the data quality rules should be applied (as in part 44d of ). Fig. 3M specified), resulting in executable file 52' - which is a version of executable file 52. Executable generator 17 transfers executable file 52' to metadata updater 18.

[0060] With reference to Fig. Figure 64 illustrates the receipt of executable file 52' by pipeline execution engine 21. In this example, metadata updater 18 detects in executable file 52' that there are no modifications to the metadata in metadata model 47'. As such, metadata updater 18 passes executable file 52' to metadata modification analyzer 19. Metadata modification analyzer 19 passes executable file 52' to pipeline execution engine 21. Metadata modification analyzer 19 does not receive any additional metadata for pipeline records from metadata updater 18. As such, metadata modification analyzer 19 determines that there are no modifications to the metadata model.

[0061] With reference to Fig. 3Q optimizer 20 generates executable file 52", which is a version of executable file 52'. In executable file 52", the component for applying data quality rules is moved to before the component for applying cleaning rules. Optimizer 20 performs this step for increased processing efficiency. This is because it is inefficient to mask an email address if that email address is not even valid.

[0062] With reference to Fig. 3R shows view 65, the real-time ingestion of data using the executable file 52. In this example, the storage system 22a transfers the source data set 66 to the pipeline execution engine 21. The pipeline execution engine 21 executes the executable file 52. View 68 shows the resulting data from the execution of each component in the executable file 52. View 68 illustrates that data 69a results from the execution of component 68a. Data 69b results from the execution of component 68b. Data 69c results from the execution of component 68c. Data 69d results from the execution of component 68d. Data 69e results from the execution of component 68e. Data 69f results from the execution of component 68f. After the execution of the executable file 52, the pipeline execution engine 21 outputs data set 67 to the storage system 22n.

[0063] With reference to Fig. 3S illustrates in view 70 the storage of metadata 70a resulting from the execution of the executable file 52" by the pipeline execution engine 21. In this example, the metadata 70a specifies, for example, which data records were created, which components were executed successfully, and it also specifies, for example, where the resulting actual adjusted and / or cleaned data is stored. In some examples, the metadata 70a specifies the data quality results from the execution of the executable file 52" by the pipeline execution engine 21.

[0064] With reference to Fig. 4A comprises the graphical user interface 71 cells 71a to 71e, each illustrating the name of a pipeline, the name of the source dataset, the name of the raw dataset, the name of the cleaned dataset, and a symbol for specifying transformation rules. Cell 71d also specifies the name of the adapted dataset.

[0065] With reference to Fig. 4B, the overlay 72 is displayed next to or above the graphical user interface 71, for example when selecting the plus symbol shown in part or cell 71e (in Fig. (4A referenced). Overlay 72 allows a user to enter and apply transformation rules to be applied during the capture process. In general, capture refers to retrieving data from one or more data sources and performing one or more actions on that data before either outputting or storing it.

[0066] With reference to Fig. Figure 4C illustrates in view 73 the updating of a pipeline object based on the transformation rules entered in overlay 72 or otherwise specified. In this example, client device 13 sends update 74 to pipeline object generator 15. Update 74 specifies the transformation rules entered in overlay 72 or otherwise specified. In response to receiving update 74, pipeline object generator 15 updates part 44f to indicate that there is a transformation rule to reformat a name to the format: last name, first name. In this example, part 44f includes the word "specified" to reflect the fact that this transformation rule is specified or otherwise entered into the system, rather than being based on the generation rules that are part of the blueprint.

[0067] With reference to Fig. 4D illustrates view 75, which shows the traversal of the metadata model 47' during the further generation of the transformation rules. In this example, the metadata inheritance engine 16 passes a request to the metadata manager 23, for which one or more fields are associated with a business data element of "Name". The metadata inheritance engine 16 updates part 44f to specify the field name that is actually reformatted.

[0068] With reference to Fig. Figure 4E illustrates view 77, which shows an update of the executable file 52" according to the transformation rule specified in part 44f of the pipeline object 44. The executable generator 17 creates the executable file 52''' and transfers the executable file 52''' to the metadata updater 18.

[0069] With reference to Fig. Figure 4F illustrates the transfer of executable file 52''' to pipeline execution engine 21. In this example, metadata updater 18 transfers executable file 52''' to metadata modification analyzer 19. Metadata modification analyzer 19 recognizes that there are no modifications to the metadata model, at least because metadata updater 18 did not provide any indication of these modifications. As such, metadata modification analyzer 19 transfers executable file 52''' to pipeline execution engine 21. In this example, metadata modification analyzer 19 determines that, because there are no further updates to the metadata model, there is no additional reason to determine whether attributes and / or data quality rules should be inherited from these new metadata elements.

[0070] With reference to Fig. 4G illustrates view 79, which shows the execution of executable file 52'''. In this example, executable file 52''' includes updating the application of the transformation rules. View 80 illustrates data resulting from the execution of each component in executable file 52'''.

[0071] With reference to Fig. 5A allows the graphical user interface 81 to view the cleaning rules applied to a specific dataset by selecting the icon in cell 81a. In this example, the pipeline name is US_customers_active. Generally, a pipeline refers to the final dataset that is cleaned, adjusted, and / or transformed and stored in a specified storage system. In another example, a pipeline generally refers to a name given or assigned to a set of actions or tasks for retrieving data and applying specified cleaning, adjustment, and / or transformation rules to that data. That is, the pipeline name refers to the name of the process that generates the data that is ultimately cleaned and adjusted.

[0072] With reference to Fig. 5B shows overlay 82 on the graphical user interface 81. Overlay 82 displays the cleaning rules that are applied to the source data record of active_cust.dat.

[0073] With reference to Fig. 5C, when selecting symbol 81b, leads to a preview of the data to be entered into the system according to the cleaning, adjustment, and transformation rule.

[0074] With reference to Fig. 5D illustrates the overlay 83, a preview of the datasets and the source dataset that have been cleaned and adjusted according to the blueprint and the specified rules.

[0075] With reference to Fig. 6A provides the graphical user interface 84 for viewing an underlying source dataset. In this example, the graphical user interface 84 includes column 84a, which represents source datasets in a raw zone. In this example, row 84b represents a specific pipeline. Cell 84c within row 84b contains a selectable part, the selection of which renders overlay 85. Overlay 85 contains details of the source dataset being received by the pipeline, as specified by row 84b. In this example, overlay 85 contains the selectable part 85a, the selection of which allows a user to view controls applied to the dataset.

[0076] With reference to Fig. 6B displays the graphical user interface 86 after selecting the selectable part 85a. In this example, cell 87 shows a description of one of the controls applied to this particular record. Selecting cell 87 displays the graphical user interface shown in Fig. 6C is described.

[0077] With reference to Fig. In cell 6C, the graphical user interface 88 is displayed. Graphical user interface 88 includes a section 88c that displays the business names of fields to which the control described in cell 87 is applied. In this example, section 88c also indicates whether a particular field failed the control. In this example, the field represented by the business name shown in cell 88b failed this control, as indicated in cell 88a. By providing this real-time verification of the data and which controls were successful or failed on which fields, the system described herein provides a very granular level of results verification.

[0078] With reference to Fig. Figure 7 shows a process 100 for generating an integrated on-demand profile. In operation, (102) a data processing system receives a specification that defines logical metadata, such as PII, and one or more operations, such as one or more data synthesis operations (e.g., masking, hashing, reducing, generalizing, and / or obfuscation), to be performed on data associated with the logical metadata. The data processing system provides (103) a user interface for specifying one or more data records to be retrieved and processed. The data processing system receives (104) from the user interface a user specification of a given data record.In response to at least the user input, (105) the data processing system generates one or more executable instructions to process the given data set according to the specification, such as maintaining data security by performing data synthesis operations on the data that includes PII. The data processing system identifies (106) technical metadata, such as fields, for the given data set. The data processing system accesses (107) a metadata model that specifies relationships between logical metadata and technical metadata. The data processing system traverses (108) the metadata model to identify a relationship between (i) an element of logical metadata (e.g., PII) in the metadata model and (ii) an element of technical metadata (e.g., a field) in the metadata model that corresponds to an element of technical metadata for the given data set.The data processing system updates (109) the one or more instructions according to the identified relationship between (i) the logical metadata element in the metadata model and (ii) the technical metadata element in the metadata model that corresponds to the technical metadata element for the given data set, such that, for example, the execution of the instructions for processing the given data set according to the specification results in maintaining data security by performing the data synthesis operations on the data that includes PII.

[0079] In some examples, traversing the metadata model involves identifying a relationship between (i) a logical metadata element in the metadata model that corresponds to logical metadata in the specification, and (ii) a technical metadata element in the metadata model that corresponds to a technical metadata element for the given dataset. Based on the identified relationship, the one or more instructions executable to process the given dataset can be modified or otherwise updated to specify that at least one of the operations associated with the logical metadata in the specification is to be performed on data of the given dataset represented by the technical metadata element that corresponds to the technical metadata element in the metadata model.

[0080] In some examples, traversing the metadata model involves identifying a data quality control or other control to be applied to the element of technical metadata in the metadata model that corresponds to an element of technical metadata for the given dataset, and updating the one or more instructions that are executable to process the given dataset, including modifying or otherwise updating the one or more instructions with additional instructions to apply the data quality control to data that corresponds to the element of technical metadata for the given dataset.

[0081] As described herein, data flow graph components comprise data processing components and / or datasets. A data flow graph can be represented by a directed graph comprising nodes or vertices representing the data flow graph components, which are connected by directed links or data flow links representing flows of work items (i.e., data) between the data flow graph components. The data processing components comprise code for processing data from at least one data input (e.g., a data source) and for supplying data to at least one data output (e.g., a data sink) of a system. The data flow graph can thus implement graph-based computation performed on data flowing from one or more input datasets through the graph components to one or more output datasets.

[0082] A system also includes a data processing system for executing one or more computer programs (such as data flow graphs) generated by transforming a specification into the computer program(s) using a transformation generator and the techniques described herein. The transformation generator transforms the specification into the computer program. In this example, the selections made by the user through the user interfaces described herein form a specification that specifies which data sources are to be included. Based on this specification, the transformations described herein are generated.

[0083] The data processing system can be hosted on one or more general-purpose computers under the control of a suitable operating system, such as the UNIX operating system. For example, the data processing system can comprise a multi-node parallel computing environment, which includes a configuration of computer systems using multiple central processing units (CPUs), either locally (e.g., multiprocessor systems such as SMP computers), locally distributed (e.g., multiple processors coupled as a cluster or MPP), remotely distributed (e.g., multiple processors coupled across LAN or WAN networks), or any combination thereof.

[0084] The graph configuration approach described above can be implemented using software to run on a computer. For example, the software forms procedures in one or more computer programs that run on one or more systems, such as computer-programmed or computer-programmable systems (which may be of various architectures, such as distributed, client / server, or grid), each comprising at least one processor, at least one data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device or port, and at least one output device or port. The software may form one or more modules of a larger computer program that, for example, provides other services related to the design and configuration of data flow graphs.The nodes and elements of the graph can be implemented as data structures stored in a computer-readable medium, or as other organized data corresponding to a data model stored in a data repository.

[0085] The software can be provided on a non-volatile storage medium, such as a hardware storage device, e.g., a CD-ROM, which is readable by a programmable general-purpose or specialized computer, or delivered to the computer (encoded in a disseminated signal) via a network communication medium, where it is executed. All functions can be performed on a specialized computer or using specialized hardware, such as coprocessors. The software can be implemented in a distributed manner, in which different parts of the data flow specified by the software are carried out by different computers. Each such computer program is preferably stored on a non-volatile storage medium or a hardware storage device (e.g., a CD-ROM).Solid-state storage media or magnetic or optical media) that are readable by a programmable general-purpose or specialized computer, are stored or downloaded to them for the purpose of configuring and operating the computer when the non-volatile storage medium or device is read by the system to perform the procedures described herein. The system can also be considered to be implemented as a computer-readable storage medium configured with a computer program, the storage medium so configured causing the system to operate in a specific and predefined manner to perform the functions described herein. Example computing environment

[0086] With reference to Fig.Figure 8 shows an exemplary operating environment for implementing embodiments of the present invention, generally referred to as a computing device 120. Essential elements of a computing device 120, or of a computer, data processing system, client, or server, are one or more programmable processors 122 for performing actions according to instructions and one or more storage devices 124 for storing instructions and data. In general, a computer also includes (via the bus 121, the structure, the network, etc.) I / O components 126, e.g., display devices, network / communication subsystems, etc. (not shown), and one or more mass storage devices 128 for storing data and instructions, etc., and a network communication subsystem 130, which is powered by, or operationally coupled to, a power supply (not shown).Memory 124 contains an operating system 124a and applications 124b for application programming.

[0087] Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or integrated into a specialized logic circuit.

[0088] To provide interaction with a user, embodiments of the subject matter described in this specification are implemented on a computer that includes a display device (monitor) for showing information to the user and a keyboard, a pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user (for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser).

[0089] Implementations of the subject matter described in this specification may be implemented in a computing system comprising a backend component (e.g., a data server), a middleware component (e.g., an application server), a frontend component (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such backend, middleware, or frontend components. The system components may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN) and a wide area network (WAN), an intermediate network (e.g., the Internet), and peer-to-peer networks (e.g.,Ad-hoc peer-to-peer networks).

[0090] The computing system can comprise clients and servers. A client and a server are generally located remotely and typically interact through a communication network. The client-server relationship arises from computer programs running on the respective computers, which establish a client-server relationship. In some embodiments, a server transmits data (e.g., an HTML page) to a client device (e.g., for the purpose of displaying data to and receiving user input from a user interacting with the device). Data generated at the client device (e.g., a result of user interaction) can be received by the client device at the server.

[0091] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or what can be claimed, but rather as descriptions of features specific to certain embodiments of certain inventions.

[0092] Similarly, although operations are depicted in a specific order in the drawings, this should not be interpreted as requiring that such operations be performed in the specific order shown or in sequential order, or that all depicted operations must be performed to achieve desired results. Furthermore, the separation of different system components in the embodiments described above should not be interpreted as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0093] Several embodiments have been described. However, it is understood that various modifications can be made without deviating from the spirit and scope of the techniques described herein. For example, some of the steps described above may be sequence-independent and thus performed in a sequence different from that described. Additionally, each of the preceding techniques described in relation to a data flow graph can also be implemented and executed in relation to a program. Accordingly, other embodiments are within the scope of the following claims. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 18 / 496,543

[0001] US 63 / 495,618

[0001] US 5,966,072

[0003] US 15 / 993,284

[0031] US 16 / 794,361

[0048] US 15 / 795,917

[0052]

Claims

[1] A method implemented by a data processing system for receiving a specification of a data set to be processed and for processing that data set according to one or more operations specified in a specification, comprising: Received by the data processing system, a specification that specifies logical metadata and one or more operations to be performed on data associated with the logical metadata; Provide, through the data processing system, a user interface for specifying one or more data records to be retrieved and processed; Received from the user interface, a specification of a given data set; and In response to at least the specification, generating one or more executable instructions to process the given data set according to the specification, including: Identifying technical metadata for the given dataset; Accessing, from a data source, a metadata model that specifies relationships between logical metadata and technical metadata; Traversing the metadata model to identify a relationship between (i) a logical metadata element in the metadata model that corresponds to the logical metadata of the specification, and (ii) a technical metadata element in the metadata model that corresponds to a technical metadata element for the given dataset; and Generating the one or more instructions to perform at least one of the one or more operations on data of the given dataset according to the identified relationship between (i) the element of logical metadata in the metadata model that corresponds to the logical metadata of the specification, and (ii) the element of technical metadata in the metadata model that corresponds to the element of technical metadata for the given dataset. [2] The method according to claim 1, wherein the generation of the one or more instructions to perform at least one of the one or more operations on the data of the given data set is carried out without additional user interaction. [3] The method according to claim 1, further comprising: based on traversing, identifying a data quality control to be applied to the element of technical metadata in the metadata model that corresponds to an element of technical metadata for the given dataset; and where the update includes: Updating one or more instructions with additional instructions to apply data quality control to the element of technical metadata for the given dataset. [4] The method according to claim 1, further comprising: Updating the metadata model based on one or more instructions from the executable file; Recognizing that one or more instructions in the executable file cause an update to the metadata model; Traversing the metadata model to identify one or more relationships between (i) data added to the metadata model based on the update and (ii) other data in the metadata model; and based on the one or more identified relationships, update the one or more instructions according to the one or more relationships between (i) the data added to the metadata model based on the update, and (ii) the other data in the metadata model. [5] The method according to claim 4, further comprising: based on the determination of no additional updates to the metadata model, Generating an executable file based on the updated instructions; and Save the executable file for execution. [6] The method according to claim 1, further comprising: Received from a metadata system, from identifiers of data that are candidates for processing; and To cause the user interface to display graphical visualizations of the identifiers. [7] The method according to claim 1, further comprising: Executing one or more instructions on the given data set. [8] One or more machine-readable hardware storage devices for receiving a specification of a data set to be processed and for processing that data set according to one or more operations specified in a specification, wherein the one or more machine-readable hardware storage devices store instructions that can be executed by one or more processing devices to perform operations, comprising: Received by the data processing system, a specification that specifies logical metadata and one or more operations to be performed on data associated with the logical metadata; Provide, through the data processing system, a user interface for specifying one or more data records to be retrieved and processed; Received from the user interface, a specification of a given data set; and In response to at least the specification, generating one or more executable instructions to process the given data set according to the specification, including: Identifying technical metadata for the given dataset; Accessing, from a data source, a metadata model that specifies relationships between logical metadata and technical metadata; Traversing the metadata model to identify a relationship between (i) a logical metadata element in the metadata model that corresponds to the logical metadata of the specification, and (ii) a technical metadata element in the metadata model that corresponds to a technical metadata element for the given dataset; and Generating the one or more instructions to perform at least one of the one or more operations on data of the given dataset according to the identified relationship between (i) the element of logical metadata in the metadata model that corresponds to the logical metadata of the specification, and (ii) the element of technical metadata in the metadata model that corresponds to the element of technical metadata for the given dataset. [9] The one or more machine-readable hardware storage devices according to claim 8, wherein the generation of the one or more instructions to perform at least one of the one or more operations on the data of the given data set is carried out without additional user interaction. [10] The one or more machine-readable hardware storage devices according to claim 8, wherein the operations further comprise: based on traversing, identifying a data quality control to be applied to the element of technical metadata in the metadata model that corresponds to an element of technical metadata for the given dataset; and where the update includes: Updating one or more instructions with additional instructions to apply data quality control to the element of technical metadata for the given dataset. [11] The one or more machine-readable hardware storage devices according to claim 8, wherein the operations further comprise: Updating the metadata model based on one or more instructions from the executable file; Recognizing that one or more instructions in the executable file cause an update to the metadata model; Traversing the metadata model to identify one or more relationships between (i) data added to the metadata model based on the update and (ii) other data in the metadata model; and based on the one or more identified relationships, update the one or more instructions according to the one or more relationships between (i) the data added to the metadata model based on the update, and (ii) the other data in the metadata model. [12] The one or more machine-readable hardware storage devices according to claim 11, wherein the operations further comprise: based on the determination of no additional updates to the metadata model, Generating an executable file based on the updated instructions; and Save the executable file for execution. [13] The one or more machine-readable hardware storage devices according to claim 8, wherein the operations further comprise: Received from a metadata system, from identifiers of data that are candidates for processing; and To cause the user interface to display graphical visualizations of the identifiers. [14] The one or more machine-readable hardware storage devices according to claim 8, wherein the operations further comprise: Executing one or more instructions on the given data set. [15] A system for receiving a specification of a data set to be processed and for processing that data set according to one or more operations specified in a specification, comprising: one or more processing devices; and one or more machine-readable hardware storage devices that store instructions executable by one or more processing devices to perform operations, comprising: Received by the data processing system, a specification that specifies logical metadata and one or more operations to be performed on data associated with the logical metadata; Provide, through the data processing system, a user interface for specifying one or more data records to be retrieved and processed; Received from the user interface, a specification of a given data set; and In response to at least the specification, generating one or more executable instructions to process the given data set according to the specification, including: Identifying technical metadata for the given dataset; Accessing, from a data source, a metadata model that specifies relationships between logical metadata and technical metadata; Traversing the metadata model to identify a relationship between (i) a logical metadata element in the metadata model that corresponds to the logical metadata of the specification, and (ii) a technical metadata element in the metadata model that corresponds to a technical metadata element for the given dataset; and Generating the one or more instructions to perform at least one of the one or more operations on data of the given dataset according to the identified relationship between (i) the element of logical metadata in the metadata model that corresponds to the logical metadata of the specification, and (ii) the element of technical metadata in the metadata model that corresponds to the element of technical metadata for the given dataset. [16] The system according to claim 15, wherein the generation of the one or more instructions to perform at least one of the one or more operations on the data of the given data set is carried out without additional user interaction. [17] The system according to claim 15, wherein the operations further comprise: based on traversing, identifying a data quality control to be applied to the element of technical metadata in the metadata model that corresponds to an element of technical metadata for the given dataset; and where the update includes: Updating one or more instructions with additional instructions to apply data quality control to the element of technical metadata for the given dataset. [18] The system according to claim 15, wherein the operations further comprise: Updating the metadata model based on one or more instructions from the executable file; Recognizing that one or more instructions in the executable file cause an update to the metadata model; Traversing the metadata model to identify one or more relationships between (i) data added to the metadata model based on the update and (ii) other data in the metadata model; and based on the one or more identified relationships, update the one or more instructions according to the one or more relationships between (i) the data added to the metadata model based on the update, and (ii) the other data in the metadata model. [19] The system according to claim 18, wherein the operations further comprise: based on the determination of no additional updates to the metadata model, Generating an executable file based on the updated instructions; and Save the executable file for execution. [20] The system according to claim 15, wherein the operations further comprise: Received from a metadata system, from identifiers of data that are candidates for processing; and To cause the user interface to display graphical visualizations of the identifiers. [21] The system according to claim 15, wherein the operations further comprise: Executing one or more instructions on the given data set.

Images