Operationalizing Metadata

Abstract

A method for using a metadata model to perform operations on a data item, the metadata model including parent nodes and child nodes connected by edges, the parent nodes specifying logical metadata, the child nodes specifying physical metadata representing the data item, and the edges specifying relationships between the nodes, the method includes identifying a given data item and physical metadata for the given data item, accessing the metadata model, identifying a child node in the metadata model that represents the physical metadata for the given data item, traversing one or more edges in the metadata model to identify a parent node of the child node, determining one or more operations to be performed on the given data item from the logical metadata associated with the identified parent node, applying the one or more operations to the given data item to transform the data item, and storing the transformed data item.

Description

[Technical Field]

[0001] Priority This application claims priority to and the benefit of U.S. Patent Application No. 18 / 104,066, filed January 31, 2023, and U.S. Provisional Patent Application No. 63 / 400,333, filed August 23, 2022, both of which are incorporated by reference in their entireties.

[0002] SUMMARY This disclosure relates to techniques for operationalizing metadata within a data enterprise. [Background technology]

[0003] Organizations maintain increasingly large and complex collections of data to run their businesses. Organizational data is often stored in ways that facilitate efficient use of computing resources rather than human comprehension. As a result, it can be difficult for users to understand what data an organization has or how to access the data for use in various tasks. Even when users are able to identify pieces of relevant data, generating code to perform desired operations on the data can be difficult and time-consuming, especially for non-technical users. Additionally, changes to organizational data can break existing data processing operations or otherwise render them obsolete. Summary of the Invention

[0004] Generally, in a first aspect, a method implemented by a metadata management system uses a metadata model to identify and perform one or more operations on one or more data items, the metadata model including one or more parent nodes and one or more child nodes, the one or more operations being defined with respect to the one or more parent nodes in the metadata model and applied to data represented by the one or more child nodes in the metadata model, the one or more parent nodes specifying logical metadata and the one or more child nodes specifying physical metadata representing the one or more data items. The method includes identifying a given data item and physical metadata for the given data item; accessing, by a metadata management system, a metadata model, where edges connect nodes and specify relationships between two nodes; identifying child nodes in the metadata model that represent the physical metadata for the given data item; traversing one or more edges in the metadata model to identify one or more parent nodes of the child nodes; determining one or more operations to be performed on the given data item from metadata associated with the identified one or more parent nodes; applying, by the metadata management system, the one or more operations to the given data item to transform the given data item; and storing the transformed data item in memory.

[0005] Generally, in a second aspect combinable with the first aspect, identifying child nodes representing physical metadata of the given data item includes matching the physical metadata of the given data item with physical metadata represented by the child nodes in the metadata model.

[0006] Generally, in a third aspect combinable with the first and second aspects, the method further includes accessing one or more metadata transforms to determine one or more operations to be performed on a given data item, each metadata transform specifying at least one operation to be performed on the data and at least one condition for performing the at least one operation.

[0007] Generally, in a fourth aspect that can be combined with any of the first to third aspects, determining one or more operations to be performed on the given data item includes selecting a metadata transformation from one or more metadata transformations; determining whether logical metadata associated with the one or more parent nodes satisfies at least one condition of the selected metadata transformation; and in response to determining that the logical metadata associated with the one or more parent nodes satisfies at least one condition of the selected metadata transformation, determining that the one or more operations to be performed on the given data item include at least one operation specified by the selected metadata transformation.

[0008] Generally, in a fifth aspect combinable with any of the first to fourth aspects, identifying the given data item includes identifying the given data item to be accessed in accordance with a processing specification, and applying the one or more operations to the given data item includes inserting the one or more operations into the processing specification and executing the processing specification to apply the one or more operations to the given data item.

[0009] Generally, in a sixth aspect combinable with any of the first to fifth aspects, the processing specification includes a specification of a dataflow graph, and applying one or more operations to the given data items includes executing the dataflow graph, and executing the dataflow graph applies the one or more operations to the given data items.

[0010] Generally, in a seventh aspect combinable with any of the first to sixth aspects, the method further comprises updating a metadata model based on the transformed data items.

[0011] Generally, in an eighth aspect combinable with any of the first to seventh aspects, updating the metadata model includes adding one or more nodes to the metadata model, the one or more added nodes representing metadata of the transformed data item.

[0012] Generally, in a ninth aspect combinable with any of the first to eighth aspects, updating the metadata model includes adding one or more edges to existing nodes in the metadata model to connect the existing nodes in the metadata model to the added nodes.

[0013] Generally, in a tenth aspect which can be combined with any of the first to ninth aspects, some of the added nodes are duplicates of some of the nodes that were already present in the metadata model before the update.

[0014] Generally, in an eleventh aspect combinable with any of the first to tenth aspects, the updated metadata model is output for further processing of the data items.

[0015] In general, in a twelfth aspect combinable with any of the first to eleventh aspects, the method further includes identifying physical metadata of the transformed data item; identifying child nodes in an updated metadata model that represent the physical metadata of the transformed data item; traversing one or more edges in the updated metadata model to identify one or more parent nodes of the child nodes that represent the physical metadata of the transformed data item; and determining from logical metadata associated with the one or more parent nodes identified by traversing the one or more edges in the updated metadata model that one or more operations performed on a given data item should not be performed on the transformed data item.

[0016] Generally, in a thirteenth aspect combinable with any of the first to twelfth aspects, determining that one or more operations performed on a given data item should not be performed on the transformed data item includes determining that logical metadata associated with one or more parent nodes identified by traversing one or more edges in the updated metadata model differs from logical metadata associated with one or more parent nodes identified by traversing one or more edges in the metadata model.

[0017] In a fourteenth aspect which may generally be combined with any of the first to thirteenth aspects, the method further includes identifying physical metadata of the transformed data item; identifying child nodes in an updated metadata model which represent the physical metadata of the transformed data item; traversing one or more edges in the updated metadata model to identify one or more parent nodes of the child nodes which represent the physical metadata of the transformed data item; determining one or more second operations to be performed on the transformed data item from logical metadata associated with the one or more parent nodes identified by traversing the one or more edges in the updated metadata model; applying, by a metadata management system, the one or more second operations to the transformed data item to further transform the transformed data item; and storing the transformed data item in a memory.

[0018] Generally, in a fifteenth aspect combinable with any of the first to fourteenth aspects, applying one or more operations to a given data item includes dropping a data field associated with the data item from further processing by the computer program.

[0019] Generally, in a sixteenth aspect which may be combined with any of the first to fifteenth aspects, applying one or more operations to a given data item includes adding a data field to the data item or a data set associated with the data item.

[0020] Generally, in a seventeenth aspect combinable with any of the first to sixteenth aspects, applying one or more operations to a given data item includes filtering data records associated with the data item.

[0021] Generally, in an 18th aspect combinable with any of the 1 to 17 aspects, the one or more operations to be performed on the given data item include a tokenization operation, and applying the one or more operations to the given data item to transform the given data item includes applying the tokenization operation to the given data item to tokenize one or more fields of the given data item.

[0022] Generally, in a 19th aspect which can be combined with any of the 1 to 18th aspects, the logical metadata associated with one or more parent nodes includes metadata received from a user through interaction with a metadata management system.

[0023] In a twentieth aspect combinable generally with any of the first to nineteenth aspects, the metadata received from the user is a personally identifiable information (PII) specification specified by the user via a graphical user interface, the graphical user interface including a visualization of a metadata model, and the PII classifications specified by the user interacting with the visualization of the metadata model to select items of metadata associated with one or more of the parent nodes in the model to which the PII specification is specified, the method further including updating the model after receiving the PII specification to associate logical metadata with the one or more parent nodes, the model including indicating in the model that one or more of the parent nodes associated with the selected items of metadata are associated with the PII specification, and the one or more operations applied to the given data item to transform the given data item include tokenizing one or more fields of the given data item.

[0024]

[0013] Generally, in a 21st aspect combinable with any of the 1st to 20th aspects, a system is configured to: access a plurality of data items stored in a hardware storage device; for each data item of the plurality of data items, identify physical metadata and logical metadata corresponding to the data item; generate a metadata model based on the identified physical metadata and logical metadata for each of the plurality of data items; provide access to the metadata model for a first application and a second application; access the metadata model by at least one of the first application or the second application;

[0025] Generally, in a 22nd aspect that can be combined with any of the 1st to 21st aspects, a metadata model includes a first data structure corresponding to a parent node, the data structure including logical metadata and at least a first pointer and a second pointer, the first pointer points to a second data structure corresponding to a child node representing physical metadata of a given data item, and the second pointer points to a third data structure corresponding to another child node representing physical metadata of another data item different from the given data item.

[0026] Generally, in a 23rd aspect, a system includes at least one processor and a memory storing instructions executable by the at least one processor to perform the operations of any of the 1st to 22nd aspects.

[0027] Generally, in a twenty-fourth aspect, a non-transitory computer-readable medium stores instructions executable by at least one processor to perform the operations of any of the first to twenty-second aspects.

[0028] Aspects can include one or more of the following advantages.

[0029] By using metadata to automate the identification and application of operations to data, operations can be automatically applied across multiple data items in a data enterprise without requiring users to define the operations for each data item. This not only improves the efficiency with which large volumes of continuously changing data are handled (e.g., because operations do not need to be defined for each individual data item), but also improves the robustness and completeness of data policies (e.g., data security policies, data governance policies, data quality policies, etc.) because operations can automatically adapt to account for changes in the underlying data, including the addition of new data. It also facilitates the incorporation of new operations, such as to account for new types of data or new data policies.

[0030] The techniques described herein also provide for more efficient processing of data with reduced memory. Traditional systems stored metadata in a read-only format, which was a major source of latency and inefficiency. This was because the metadata was only used by analysts to learn about the data, but analysts manually retrieved that information and utilized it in different projects as needed. Rather than metadata being read-only, the described techniques use metadata as the initial starting point in a connected chain for defining the required processing for selected data (e.g., datasets) and then operationalizing that metadata using other applications to perform that processing. For example, a metadata management system can connect to one or more other systems, such as a first system for writing data flow graphs (and / or other computer programs) that define data processing operations, and a second system that is a data orchestration system. In older systems, users could view the metadata for various datasets. However, if a user wants to access that data, for example, to define processing to be performed in a first system and / or to control processing to be performed by a second system, the data must be accessed and imported twice (once into each system) to access the metadata used in defining the data processing operations in the first system and specifying the control operations in the second system. Now, rather than accessing the data multiple times, i.e., once for each application that needs the metadata to define processing, control, etc., each data set only needs to be accessed once, thereby reducing the time and resources required to access the data sets. The metadata is identified and given semantic meaning. A metadata model is then generated and continuously updated as additional metadata is created.The data processing system makes the metadata in the metadata model accessible to each of the applications or systems, so that in the resulting system, the metadata for a data set need only be read once, and semantic discovery need only be performed once on that read metadata (or the data corresponding to that read metadata) for use in multiple applications or systems. The single read, access, and processing of the metadata (via semantic discovery) results in a metadata model that is continuously updated with new metadata (e.g., by adding new nodes to the metadata model) and made accessible to various systems and applications for use in defining processing.

[0031] According to preferred aspects, users can specify operations to be performed on data within a data enterprise without having to define (e.g., code) the means for accessing the data or performing the operations. For example, a user (e.g., a non-technical user) can specify that a particular item of logical metadata (e.g., an SSN) is a form of personally identifiable information (PII) without knowing which items of physical data within the enterprise correspond to SSNs or how to access those data items. Based on this metadata definition, data items corresponding to SSNs are automatically identified by the system (e.g., by traversing a metadata model) and processed to obfuscate the data (e.g., by performing operations that mask or tokenize the data or by inserting them into a computer program) without the user having to generate code to perform these operations.

[0032] According to a preferred embodiment, metadata generated as a result of applying an operation to data is propagated or replicated to a metadata model to reduce further consumption of computing resources during subsequent processing based on the metadata model. That is, the techniques described herein improve efficiency in processing data because the metadata model is continuously updated with the results of previous processing. For example, if a new data item is generated as a result of an operation (e.g., tokenized SSN data is generated as a result of a tokenization operation), the system can update the metadata model to include the metadata of the new data item. That is, nodes and edges are added to the metadata model, and the added nodes and edges represent the tokenized SSN data (e.g., the meaning of the tokenized data, relationships between the tokenized data and other data or metadata, storage location and other access parameters of the tokenized data, etc.). Thus, if a data processing system needs the tokenized data at a later point in time, the data processing system does not need to re-tokenize the original data. Rather, the data processing system uses the metadata model to identify and access the tokenized SSN data. This storage of metadata representing the tokenized SSN data in the metadata model saves computing resources because data processing systems can simply look up the tokenized data rather than having to recalculate the tokenized data based on the original data.

[0033] Previously discovered or generated metadata can also be propagated or replicated to new data items. That is, a new edge can be added to the metadata model to associate a node representing the new data item with an existing node in the model. For example, if a particular data item is associated with PII such as an SSN in the metadata model and a new data item is created based on that particular data item (e.g., via a copy operation), the system can automatically propagate the SSN association to the new data item in the metadata model (e.g., by adding an edge between the SSN node and the node representing the new data item). Thus, when the system accesses the new data item at a later point in time, it knows that the new data item is associated with an SSN and, if necessary, performs appropriate data quality and / or data security operations based on the association. By updating the metadata model in this manner, the system leverages existing work, such as work done to discover or generate metadata, to reduce the amount of computing resources (e.g., memory, processing cycles, etc.) required to perform subsequent operations on the data. Additionally, propagating metadata in the metadata model ensures that data policies (e.g., data security policies, data governance policies, data quality policies, etc.) are adhered to when data changes are made in the system.

[0034] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features, objects, and advantages set forth herein will be apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0035] [Figure 1] 1 illustrates an example of a metadata management system. [Figure 2A] 1 illustrates an exemplary metadata management system configured to operationalize metadata. [Figure 2B]1 illustrates an exemplary metadata management system configured to operationalize metadata. [Figure 3A] 1 illustrates an exemplary metadata management system configured to operationalize metadata. [Figure 3B] 1 illustrates an exemplary metadata management system configured to operationalize metadata. [Figure 3C] 1 illustrates an exemplary metadata management system configured to operationalize metadata. [Figure 3D] 1 illustrates an exemplary metadata management system configured to operationalize metadata. [Figure 4A] 1 illustrates an exemplary metadata management system configured to operationalize metadata for dropping fields. [Figure 4B] 1 illustrates an exemplary metadata management system configured to operationalize metadata for dropping fields. [Figure 4C] 1 illustrates an exemplary metadata management system configured to operationalize metadata for dropping fields. [Figure 4D] 1 illustrates an exemplary metadata management system configured to operationalize metadata for dropping fields. [Figure 5A] 1 illustrates an example of a metadata management system configured to enable metadata manipulation to add fields. [Figure 5B] 1 illustrates an example of a metadata management system configured to enable metadata manipulation to add fields. [Figure 6A] 1 illustrates an example metadata management system configured to operationalize metadata for row filtering. [Figure 6B] 1 illustrates an example metadata management system configured to operationalize metadata for row filtering. [Figure 7A]1 illustrates an exemplary metadata management system configured to operationalize metadata to adapt to changes in data. [Figure 7B] 1 illustrates an exemplary metadata management system configured to operationalize metadata to adapt to changes in data. [Figure 8] 1 is an exemplary process for using metadata to automate operations on data. DETAILED DESCRIPTION OF THE INVENTION

[0036] The present disclosure relates to the use of metadata to automate the identification and application of operations to data in a data enterprise. In some examples, a data processing system (sometimes referred to as a metadata management system) uses metadata of data stored in the data enterprise to generate a metadata model. The system can further enhance the metadata model with metadata specified by a user. When data in the data enterprise is accessed, the system uses the metadata model to automatically identify and apply operations to the data. In this manner, users can specify operations to be performed on data in the enterprise without having to understand the technical details of the enterprise or generate code to implement the operations. Additionally, because operations are defined at the metadata level, they can be automatically applied across multiple data items accessed or otherwise processed by multiple different applications, rather than requiring users to define operations for each data item in each application. Defining operations at the metadata level also allows the system to automatically adapt to changes in the underlying data, including the addition of new data. In some examples, the metadata model is updated as a result of the operations to improve the efficiency of subsequent data processing.

[0037] Generally, a metadata management system can implement various processes to obtain metadata for data stored in one or more data sources within a data enterprise. For example, the system can discover physical metadata that describes the attributes of the data (e.g., what system data exists, schemas, table and field names, data types, data formats, etc.) and the relationships between the data (e.g., primary-foreign key relationships, entity relationships, etc.). The data processing system can also generate logical metadata based on the content of the stored data (or a selected subset of the stored data). Logical metadata provides details about how the data is linked to each other to form a larger set. It also outlines how the data flows through systems and processes, from creation to storage, transformation, and consumption. Logical metadata can establish a roadmap for the data's path through the data supply chain, including its use and changes over time. For example, the system can determine that the stored data includes a dataset related to information about customers and that the dataset includes a data field that holds each customer's social security number (SSN).

[0038] Using the metadata, the system generates a metadata model that describes the physical and logical relationships and other attributes of the stored data. Generally, the metadata model can include nodes that represent items of physical and logical metadata, with edges that represent relationships between the nodes. The system can augment the metadata model with metadata specified by a user (sometimes referred to as user-specified metadata). For example, a user can specify that an SSN is a form of personally identifiable information (PII) that should be protected by the system. Based on this specification, the system can update the metadata model to indicate that the node representing the SSN is associated with PII.

[0039] The system then uses the metadata model to automatically perform operations on data within the data enterprise. For example, when a data item within the enterprise is accessed (e.g., by a computer program), the system identifies a node in the metadata model that corresponds to the metadata (e.g., physical metadata) of the data item being accessed. The system can then traverse the metadata model to find other nodes related to the identified node, as described in more detail below. Based on the metadata (e.g., logical metadata) associated with the related nodes, the system determines one or more operations to perform on the accessed data item. For example, if the accessed data item is related to an SSN node (which, as described above, is tagged as PII), the system can determine to automatically insert or otherwise perform an obfuscation or tokenization operation on the data item into a computer program. This improves data security. The system can also automatically perform operations (e.g., tokenization) on the data item based on a user's high-level metadata specification, without requiring the user to identify the specific data item to be processed or to generate code to perform the operation. This improves the efficiency of data processing.

[0040] In some examples, the system can update the metadata model based on operations performed on the data item. For example, if a new data item is generated as a result of the tokenization operation described above, the system can update the metadata model to include metadata for the new data item. In some examples, as described in more detail below, previously discovered or generated metadata may also be propagated to the new data item. By updating the metadata model in this manner, the system leverages existing work to reduce the amount of computing resources (e.g., memory, processing cycles, etc.) required to perform subsequent operations on the data.

[0041] Referring to FIG. 1 , a system 100 for operationalizing metadata is shown. In this example, system 100 includes a metadata management system 102, storage systems 104, 106, and client devices 108, 110. Generally, metadata management system 102 is a data processing system for automating the identification and application of operations to data stored in storage system 104, for example, using metadata. After processing the data, metadata management system 102 can store the data in storage system 106, provide the data to client device 110 for display, or both. Although shown as separate entities, in some examples, storage system 104 can be the same storage system as storage system 106. Similarly, in some examples, client device 108 can be the same client device as client device 110.

[0042] The metadata management system 102 includes a metadata discovery engine 112 and a metadata-based processing engine 114. The discovery engine 112 includes program instructions and / or executable logic for discovering or otherwise retrieving metadata for data stored in the storage system 104. For example, the discovery engine 112 may perform a discovery process to retrieve physical metadata describing attributes of the data (e.g., field names, data types, data formats, etc.) and relationships between the data (e.g., primary-foreign key relationships, entity relationships, etc.). In one example, data is prepared for processing by the discovery engine 112 using format information. Data including records having values ​​for fields is received on an input device or port. A target record format for processing the data is determined. A plurality of records is analyzed according to a validation test to determine whether the data matches a candidate record format. Each candidate record format specifies a format for each field, and each validation test corresponds to at least one candidate record format. In response to receiving the results of the validation tests, a target record format is associated with the data based on at least one of the candidate record formats determined to be at least a partial match according to the at least one validation test, the parsed record format selected according to a data type associated with the data, and the constructed record format generated from an analysis of the data characteristics. Other examples of such discovery processes are described in U.S. patent application Ser. No. 12 / 945,094, entitled "Managing Record Format Information," the entire contents of which are incorporated herein by reference.

[0043] The discovery engine 112 can also perform a semantic discovery process on the data (or a selected subset of the data) to generate logical metadata that, among other things, represents the semantic meaning of the data. For example, the discovery engine 112 can identify a field included in one or more datasets, where the field has an identifier. For that field, the discovery engine 112 profiles the data values ​​of the field to generate a data profile, accesses multiple label proposal tests, and generates a set of label proposals by applying the multiple label proposal tests to the data profile. The discovery engine 112 then determines the similarity between the label proposals and selects a classification. The discovery engine 112 identifies one of the label proposals as identifying the semantic meaning. The discovery engine 112 stores the identifier of the field along with the identified one of the label proposals that identifies the semantic meaning. Other examples of such semantic discovery are described in U.S. patent application Ser. No. 16 / 794,361, entitled "Discovering a semantic meaning of data fields from profile data of the data fields," the entire contents of which are incorporated herein by reference.

[0044] The discovery engine 112 passes the physical and logical metadata to a metadata-based processing engine 114, which uses the metadata to generate a metadata model 116 having a plurality of nodes 117 and edges 118 that represent relationships between the nodes. The metadata-based processing engine 114 also augments the metadata model 116 with user-defined metadata received, for example, from the client device 108. In some examples, the metadata model 116 may include a set of objects or data structures, each representing a node. Each object or data structure may include data elements that represent the physical, logical, and / or user-defined metadata of the respective node, as well as pointers to other objects or data structures that represent other nodes connected to the respective node by edges. In some examples, the metadata-based processing engine uses the metadata model and other data to generate a data catalog 120 of the data (or a selected subset of the stored data) stored in the storage system 104. The data catalog 120 may include one or more data objects that include metadata and other information that identify data or groups of data stored in the storage system 104. A user can interact with the data catalog 120 to, among other things, define object properties or select objects for use data processing. For example, a user can associate objects in the data catalog 120 with one or more metadata-driven transformations 122, as described below. As another example, a development environment (not shown) that is part of or in communication with the metadata management system 102 can include a user interface having a representation of the catalog 120, from which a user can select objects, for example, to use as input to a data flow graph or other computer program. Techniques for creating, maintaining, and using data catalogs are described in U.S. Patent No. 9,977,659, entitled "Managing Data Set Objects," the entire contents of which are incorporated herein by reference.

[0045] The metadata-based processing engine 114 can also store or otherwise access multiple process specifications 124. Generally, the process specifications 124 can define or include program instructions and / or executable logic for processing data stored in the storage system 104. In some examples, each of the process specifications 124 can be or otherwise define a computer program, such as a dataflow graph. A dataflow graph includes multiple vertices representing computational processes and multiple links, each vertex having an associated access method, each link connecting at least two vertices to each other and representing the flow of data between the connected vertices. When executed, a system executing the graph (e.g., the metadata management system 102 or another data processing system) can prepare the graph for execution by performing graph transformation steps until each vertex is executable and each link is associated with at least one communication method compatible with the access methods of the vertices connected by the link; activate each link by creating a combination of communication channels and / or data stores depending on the link's communication method; and launch each process on the executing system by invoking execution of the process. Additional details regarding implementing such graph-based computations are described in U.S. Patent No. 5,966,072, entitled "Executing Computations Expressed as Graphs," the entire contents of which are incorporated herein by reference. The process specification 124 may define or include operations for accessing data in the data catalog 120 (e.g., from the storage system 104) for any of a variety of reasons, such as to ingest the data into the storage system 106 or to generate views of the data for presentation on the client device 110.Regardless of the particular process defined by the processing specification 124, the metadata-based processing engine 104 can automatically perform operations (e.g., metadata-driven transformations 122) on data accessed from the storage system 104 using the metadata model 116 as described herein.

[0046] 2A and 2B illustrate a system 200 for operationalizing metadata according to one aspect of the present disclosure. Referring to FIG. 2A, the system 200 includes a metadata management system 202, storage systems 204, 206, and client devices 208, 210. Similar to the metadata management system 102 shown in FIG. 1, the metadata management system 202 includes a metadata discovery engine 212 and a metadata-based processing engine 214. In this example, the metadata discovery engine 212 receives a dataset 230 stored in the storage system 204 and processes the dataset 230 to obtain corresponding metadata. In particular, the metadata discovery engine 212 determines that the dataset 230a includes the following items of physical metadata: name "Cust_data," type "Table," and fields "cust_fnln" and "cust_ssn."

[0047] A user can access the metadata management system 202 (e.g., via an application executing on the client device 208) to view and interact with a graphical user interface 232 that includes physical metadata for the datasets 230. Within the user interface 232, the user can select 234 the datasets 230 to be added to a data cart 236 based on the physical metadata. The metadata discovery engine 212 can then perform semantic discovery or other processes on the datasets 230 in the cart 236 to generate logical metadata that represents the semantic meaning of the datasets 230. For example, the metadata discovery engine 212 can determine that dataset 230a contains information about customers, that the field "cust_fnln" contains the customer's name, and that the field "cust_ssn" contains the customer's SSN. In some examples, the user can approve dataset 230a and the meaning associated with its fields and submit the dataset for input into the data catalog 220 maintained by a system (not shown).

[0048] The metadata 238 discovered by the metadata discovery engine 212 is passed to the metadata-based processing engine 214. As described above, the discovered metadata 238 may include physical metadata and logical metadata (sometimes referred to as logical term associations) for the selected dataset 230. The metadata-based processing engine 214 uses the metadata 238 to generate a metadata model 216a for the dataset 230. In this example, the metadata model 216a includes oval nodes 217a, 217b, and 217c that represent the physical metadata for the dataset 230a, and rectangular nodes 217d, 217e, and 217f that represent the logical metadata for the dataset 230a. Metadata model 216a also includes edges 218a, 218b that represent the physical relationship between dataset 230a (e.g., node 217a) and its fields (e.g., nodes 217b, 217c), edge 218c that represents the logical relationship between dataset 230a and its corresponding logical entity (e.g., logical dataset "Customer"), and edges 218d, 218e that represent the logical relationship between each of the fields of dataset 230a and their corresponding logical entities (e.g., logical data elements "Name" and "SSN").

[0049] The metadata-based processing engine 214 also receives user-specified metadata 240 from a user of the client device 210. In some examples, the metadata 240 may be specified with respect to an item of physical or logical metadata in the metadata model 216a. For example, in this example, the user-specified metadata 240 indicates that an item of logical metadata representing an SSN (e.g., logical data element SSN) is associated with PII. Once received, the metadata-based processing engine 214 may incorporate the user-defined metadata 240 into the metadata model 216a. For example, the metadata-based processing engine 214 may update the metadata model 216a to indicate (219) that the node 217f representing the SSN is associated with PII. Such an association may be made, for example, by including a metadata definition or other indicator with the SSN logical metadata or by adding another node to the metadata model 216a linked by an edge to the node representing the SSN, among other things.

[0050] 2B , the metadata-based processing engine 214 can use the metadata model 216a to automatically perform operations on the datasets 230 stored in the storage system 204. In this example, the metadata-driven data transforms 222 associated with data in the catalog 220 (including the dataset 230) specify that the data should be tokenized if it contains PII to improve data security. Thus, when the metadata-based processing engine 214 executes the processing specifications 224, it uses the metadata model 216a (and the transforms 222) to determine whether some or all of the datasets 230 being accessed are associated with PII, and if so, performs a tokenization operation on the data. For example, in this example, one of the processing specifications 224 specifies a data pipeline configured to ingest the dataset 230a into the storage system 206 (e.g., a data pipeline configured to read the dataset 230a from the storage system 104 and write the dataset to the storage system 106). Thus, the metadata-based processing engine 214 determines the physical metadata of the dataset 230a and identifies nodes in the metadata model 216a that correspond to the physical metadata of the dataset 230a. For example, the metadata-based processing engine 214 may determine that the dataset 230a has a name "Cust_data" and may match this name to a node 217a in the metadata model 216a. The metadata-based processing engine 214 may then traverse the metadata model 216a to determine whether the dataset 230a contains PII. In this example, the metadata-based processing engine 214 determines that the field "cust_ssn" (represented by node 217c) of the dataset 230a is associated with the logical data element SSN (represented by node 217f), which is defined as PII.Thus, the metadata-based processing engine 214 tokenizes the data in the “cust_ssn” field of the data set 230 a to create a “Cust_data_tokenized” data set 242 for ingestion into the storage system 206 .

[0051] In this example, another one of the processing specifications 224 specifies operations configured to generate a catalog view of the dataset 230a for presentation to a user of the client device 210. The metadata-based processing engine 214 traverses the metadata model 216a as described above and determines that the field "cust_ssn" of the dataset 230a contains PII. As a result, the metadata-based processing engine 214 tokenizes the data in the "cust_ssn" field before sending the catalog view data 244 to the client device 210. Upon receipt, the client device 210 can present the data 244 in a graphical user interface 246, which allows a user to view the tokenized data records of the dataset 230a and use the tokenized data as a component in subsequent processing.

[0052] In some examples, the metadata-based processing engine 214 can update the metadata model 216a to generate a metadata model 216b that incorporates the metadata of the newly defined “cust_data_tokenized” dataset 242. For example, the metadata-based processing engine 214 can add nodes 217g, 217h, and 217i that represent the physical metadata of dataset 242, and edges 218f and 218g that represent the relationship between nodes 217g, 217h, and 217i. Because the “cust_ssn” field of dataset 242 has been tokenized and no longer represents an SSN, the metadata-based processing engine 214 can also add a logical node 217j (e.g., “tokenized SSN”) that accurately describes the contents of the tokenized field. Edge 218h can then be inserted to indicate the logical relationship between nodes 217i and 217j. The metadata-based processing engine 214 can also replicate existing metadata in the metadata model 216b (e.g., see nodes 217h, 217i) to populate the metadata for the new dataset 242. Specifically, the metadata-based processing engine 214 can insert edge 218i representing a logical relationship between dataset 242 and its corresponding logical entity (e.g., the logical dataset "Customer"), as well as edge 218j representing a logical relationship between the "cust_fnln" field of dataset 242 and its corresponding logical entity (e.g., the logical data element "Name"). Updating the metadata model in this way can reduce the computational resources required for subsequent processing of dataset 242 because the metadata for dataset 242 has already been populated into the model, replicating some of the metadata 238 discovered for dataset 230 (thereby eliminating the need to discover metadata on dataset 242), and because the model has been updated to reflect that the "cust_ssn" field of dataset 242 has been tokenized (and therefore does not need to be re-tokenized during subsequent processing).

[0053] 3A illustrates a system 300 for operationalizing metadata according to one aspect of the present disclosure. In this example, the system 300 includes a metadata management system 302, storage systems 304, 306, and client devices 308, 310. Similar to metadata management systems 102 and 202, the metadata management system 302 includes a metadata discovery engine 312 and a metadata-based processing engine 314. In this example, the metadata discovery engine 312 receives data sets 330a-330d stored in the storage system 304 and processes the data sets 330a-330d to obtain corresponding metadata 332. In particular, the metadata discovery engine 312 determines that the data set 330a has the name "Cust_data" and includes the fields "cust_id" (which serves as a primary key, as indicated by the key symbol) and "fname." Based on the semantic analysis of dataset 330a and its data, the metadata discovery engine 312 also determines that dataset 330a "Cust_data" represents information about "Customer," that the field "cust_id" represents "Customer ID," which is part of the broader group "DB Identifier," and that the field "fname" represents "Name," which is part of the broader group "Customer Identification Information." The metadata discovery engine 312 also determines that dataset 330a is related to dataset 330b (e.g., through a primary-foreign key relationship) and that dataset 330b has the name "SSN_table" and contains fields "I95" and "S14." Based on the semantic analysis of dataset 330b and its data, the metadata discovery engine 312 determines that dataset 330b represents "Customer SSN," that the field "I95" represents "Customer ID," which is part of the group "DB Identifier," and that the field "S14" represents the customer's "SSN," which is part of the group "Customer Identification."The metadata discovery engine 312 also determines that dataset 330b is related to dataset 330c (e.g., through a primary-foreign key relationship) and that dataset 330c has the name "Loan_data" and contains the fields "cust_ssn" and "amt." Based on a semantic analysis of dataset 330c, the metadata discovery engine 312 determines that dataset 330c represents a "customer loan," that the field "cust_ssn" represents the "SSN" of a customer that is part of the group "customer identification information," and that the field "amt" represents the "amount" within the group "loan information." The metadata discovery engine 312 also determines that dataset 330c is related to dataset 330d (e.g., through a primary-foreign key relationship) and that dataset 330d has the name "Cust_acct" and contains the fields "c_id" and "level." The metadata discovery engine 312 also determines, based on dataset 330d and semantic analysis of its data, that dataset 330d represents information about "customer accounts," that the field "c_id" represents a "customer ID" that is part of the group "DB identifier," and that the field "level" represents the customer's "account level."

[0054] The metadata discovery engine 312 passes the discovered metadata 332 to the metadata-based processing engine 314. The metadata-based processing engine 314 uses the metadata 332 to generate a metadata model 316 that includes multiple nodes 317 and edges 318. In this example, the oval-shaped nodes 317 represent items of physical metadata for the datasets 330a-330d, and the rectangular nodes 317 represent items of logical metadata for the datasets 330a-330d. In some examples, the metadata-based processing engine 314 also uses the metadata 332 to add the datasets 330a-330d to the data catalog 320. The datasets 330a-330d may also be associated with one or more metadata-driven data transformations 322 (e.g., within the data catalog 320 or otherwise). The metadata-based processing engine 314 may also store or otherwise have access to multiple processing specifications 324. In this example, the processing specification 324 includes data pipelines 324a-324d in the form of a data flow graph configured to access (e.g., read) data sets 330a-330d from the storage system 306 and store (e.g., write) them to the storage system 304.

[0055] Referring to FIG. 3B , the metadata-based processing engine 314 can receive a metadata specification 334 from a user of a client device 308. In particular, the user can use the client device 308 to access a graphical user interface 336 that includes a visualization 336a of the metadata model 316. The user can interact with the metadata model visualization 336a to select an item of metadata for which metadata will be specified. In this example, the user has selected an item of metadata corresponding to “customer identifying information” (336b). After selecting the item of metadata, the user can specify metadata for that item of metadata. For example, the user can specify a PII classification 336c, which is a form of PII designation, for the item of metadata. In this example, the user has specified that “customer identifying information” has a PII classification 336c of “Level 2 (Tokenized).” Upon completion, the user can choose to submit the metadata specification 334 to the metadata-based processing engine 314 (336d).

[0056] After receiving the metadata specification 334, the metadata-based processing engine 314 updates the model 316. For example, the metadata-based processing engine 314 can update the model 316 to indicate (319) that a node representing "customer identifying information" is associated with a level 2 PII classification. As shown in visualization 322a of metadata-driven transformation 322, the metadata-based processing engine 314 is configured to tokenize fields associated with a node when the associated node (e.g., a parent node) has a "level 2" PII classification. Incorporating the metadata specification 334 into the metadata model 316 in this manner reduces the amount of space required to store the metadata model 316. This is because the metadata specification 334 is defined only once in the model 316 (e.g., by adding an element having the specification 334 in the object or data structure representing "customer identifying information," or by adding a single new object or data structure having the specification 334 that points to the object or data structure representing "customer identifying information"), rather than defining the specification at each applicable node (which could amount to thousands or more definitions). As a result, only a single data element (or object or data structure) (as opposed to many) is added to the model 316, reducing storage and memory requirements for the model. A similar reduction in storage is achieved by linking multiple nodes representing physical metadata (e.g., "cust_ssn" and "S14" nodes) to a single node representing logical metadata (e.g., "SSN" node) in conjunction with defining and storing a separate logical node for each physical node.

[0057] 3C , the metadata-based processing engine 314 uses the metadata model 316 to apply the metadata-driven data transformations 322 to the processing specification 324 and, therefore, to the datasets 330a-330d. To do so, the metadata-based processing engine 314 can determine the physical metadata of the data items being accessed and can use the determined physical metadata to identify nodes in the metadata model 316 that correspond to the data items. For example, when executing the processing specification 324a, the metadata-based processing engine 314 can identify nodes in the metadata model 316 that correspond to the name of the dataset 330a (e.g., “Cust_data”) and / or the names of the fields of the dataset 330a (e.g., “cust_id”, “fname”). In some examples, the identified nodes are referred to as child nodes. The metadata-based processing engine 314 then traverses the metadata model 316 in accordance with the metadata-driven transformations 322 to determine the operations, if any, to perform on the dataset 330a. For example, the metadata-based processing engine 314 may traverse the metadata model 316 to identify one or more parent nodes of the identified child nodes, such as nodes corresponding to “customer,” “customer ID,” “DB identifier,” “name,” and “customer identity.” In this example, the metadata-based processing engine 314 determines, based on the metadata model 316, that the node representing “customer identity” associated with the “level 2” PII classification is a parent of the node representing “fname” (e.g., via the “name” node) but is not a parent of the node representing “cust_id.” As a result, the metadata-based processing engine 314 inserts operation 325a into processing specification 324a to tokenize “fname.” The metadata-based processing engine 314 may perform a similar process to insert tokenization operations 325b, 325c into processing specifications 324b, 324c.However, because there is no parent node associated with the "Level 2" PII classification for the dataset "Cust_acct" or its nodes corresponding to fields "c_id" and "level," no tokenization operation is added to processing specification 324d.

[0058] After applying the metadata-driven transformation 322, the metadata-based processing engine 314 can execute the processing specification to generate data sets 338a-338d having tokenized PII data (if applicable). Each of the data sets 338a-338d can be sent to the storage system 306 for storage. The metadata-based processing engine 314 can also apply similar processing to generate (among other things) tokenized catalog view data 340 of the data set 330a. The catalog view data 340 can be sent to the client device 310 for presentation in a graphical user interface 342 that allows a user to view each of the tokenized data records of the data set 330a.

[0059] By leveraging metadata in this manner, system 300 enables users to automatically implement operations, such as data security operations, across an entire data enterprise (e.g., across multiple data items, such as datasets 338a-338d) via a single global metadata specification (e.g., metadata specification 334), without requiring users to identify the specific data items to which the operations apply and without requiring users to generate code for each data item. As a result, system 300 provides more efficient implementation of enterprise-wide data policies, thereby reducing the latency of implementing such policies, compared to systems that do not leverage metadata in such a manner. Additionally, by automatically identifying and applying operations based on metadata, system 300 improves data security (as well as the implementation of other data policies) by reducing the likelihood that data subject to policy will be overlooked. Defining operations based on metadata also enables the system to automatically adapt to changes in the underlying data (e.g., changes in the name of the data, the storage location of the data, the keys of the data, etc.).

[0060] The metadata-based processing engine 314 can also update the metadata model 316 after applying the metadata-driven transformation 322, as shown in FIG. 3D . For example, the metadata-based processing engine 314 can update the metadata model 316 to incorporate the metadata of the newly defined “cust_data_tokenized” dataset 338a. For example, the metadata-based processing engine 314 can add nodes 317a, 317b, and 317c that represent the physical metadata of dataset 338a, and edges 318a, 318b that represent relationships between nodes 317a, 317b, and 317c. Because the “cust_ssn” field of dataset 338a has been tokenized and no longer represents an SSN, the metadata-based processing engine 314 can also add logical nodes 317d, 317e (e.g., “tokenized name” and “tokenized customer identity”) that accurately describe the contents of the tokenized field. Edges 318c, 318d can then be inserted to indicate the logical relationship between nodes 317c, 317d, and 317e.

[0061] The metadata-based processing engine 314 can also propagate or replicate existing metadata in the metadata model 316 to populate the metadata of the new dataset 338a. Specifically, the metadata-based processing engine 314 can insert edge 318e, which represents the logical relationship between dataset 338a and its corresponding logical entity (e.g., logical dataset "Customer"), as well as edge 318f, which represents the logical relationship between the "c_id" field of dataset 338a and its corresponding logical entity (e.g., logical data element "Customer ID"). Similar operations can be performed to update the metadata model 316 to incorporate the metadata of datasets 338b-338d (not shown).

[0062] Updating the metadata model in this manner can reduce the computational resources required for subsequent processing of datasets 338a-338d because the model is already populated with metadata for datasets 338a-338d, thereby eliminating the need to discover metadata on these datasets. Additionally, because the model has been updated to reflect that the "cust_ssn" field of dataset 338a (among other fields from other datasets) has been tokenized, the metadata-based processing engine 314 can use the previously tokenized data instead of re-tokenizing it during subsequent processing.

[0063] FIG. 4A illustrates a system 400 that is a variation of the system 300 illustrated in FIGS. 3A-3D. In this example, the system 400 includes a metadata management system 402, storage systems 404, 406, and client devices 408, 410. Similar to the metadata management systems 102, 202, and 302, the metadata management system 402 includes a metadata discovery engine 412 and a metadata-based processing engine 414. The metadata-based processing engine 414 stores or otherwise accesses multiple process specifications 424, including process specification 424a. In this example, process specification 424a instantiates a dataflow graph that defines a catalog view of multiple datasets (e.g., datasets 330a-330c) in the data catalog 420 from the perspective of the "Cust_data" dataset 330a. In other words, dataset 330a is the root node in the catalog view defined by process specification 424a, and other datasets are joined to dataset 330a based on discovered relationships between datasets 330a-330c.

[0064] The metadata-based processing engine 414 automatically applies the metadata-driven transform 322 (shown in FIG. 3B ) to the processing specification 424a using the techniques described herein. In particular, the metadata-based processing engine 414 uses the metadata model 316 (shown in FIG. 3B ) to determine that the fields “fname,” “S14,” and “cust_ssn” should be tokenized and inserts a tokenization operation 425a into the processing specification 424. After applying the metadata-driven transform 322, the metadata-based processing engine 414 can execute the processing specification 424a to generate tokenized catalog view data 430. The catalog view data 430 can be sent to the client device 410 for presentation in a graphical user interface 432 that allows a user to view the tokenized data records in a customer view. The metadata-based processing engine 414 can also generate a customer view dataset 434 (e.g., a customer view wide record) that is sent to the storage system 406 for storage.

[0065] In some examples, the metadata-based processing engine 414 can receive a new metadata-driven transformation 422a (or modifications to an existing metadata-driven transformation) from a user of a client device 408, as shown in FIG. 4B . In particular, the user can use the client device 408 to access a graphical user interface 436 for defining the metadata-driven transformation 422a. The user can interact with the user interface 436 to enter a name 436a for the metadata-driven transformation 422a, one or more conditions 436b for applying the transformation, and one or more expressions 436c that specify an operation to be performed when the condition 436b is met. In this example, the user has defined a metadata-driven transformation 422a named “Admin View” that causes a field to be dropped (e.g., removed from further processing) when the PII classification of the parent node is equal to “Level 2” and the PII view permission level of the parent node is greater than “User.” In some examples, the user can also define an item or group of items in the data catalog 420 to which the transformation 422a is to be applied (not shown). After defining the metadata-driven transformation 422a, the user can choose to submit 436d the metadata-driven transformation 422a to the metadata-based processing engine 414. The metadata-based processing engine 414 can then incorporate the metadata-driven transformation 422a into the set of transformations 422 maintained by the system 400, as shown in visualization 422b of the metadata-driven transformation 422.

[0066] Referring to FIG. 4C , the metadata-based processing engine 414 can receive a metadata specification 438 from a user of a client device 408. In particular, the user can use the client device 408 to access a graphical user interface 440 that includes a visualization 440a of the metadata model 316 (shown in FIG. 3B ). The user can interact with the metadata model visualization 440a to select an item of metadata for which metadata will be specified. In this example, the user has selected an item of metadata corresponding to “customer identifying information” (440b). After selecting the item of metadata, the user can specify metadata for that item of metadata. For example, the user can specify PII view permissions 440d, which are a form of PII specification, for the item of metadata. In this example, the user has specified that “customer identifying information” has “Admin level” PII view permissions 440d (in addition to the previously specified “Level 2 (Tokenized)” PII classification 440c). Upon completion, the user may choose to submit (440e) the metadata specification 438 to the metadata-based processing engine 414. After receiving the metadata specification 438, the metadata-based processing engine 414 updates the model 316 (shown in FIG. 3B ) to generate model 416. For example, the metadata-based processing engine 414 may update (419) the model 416 to indicate that the node representing “Customer Identifying Information” is also associated with “Admin Level” PII view permissions.

[0067] 4D , the metadata-based processing engine 414 can use the metadata model 416 to automatically apply metadata-driven transformations 422, including newly defined metadata-driven transformation 422a, to the processing specification 424 and underlying data. In this example, the metadata-based processing engine 414 identifies node 417a, which corresponds to the “Cust_data” dataset accessed in the processing specification 424a. The metadata-based processing engine 414 then traverses the metadata model 416, as indicated by the directional arrows, to identify nodes 417b, 417c, which correspond to fields in the “Cust_data” dataset, as well as nodes 417d-417h, which correspond to logical parent nodes of the physical nodes. In this example, the metadata-based processing engine 414 determines that the field “fname” corresponding to node 417c should be dropped (e.g., from further processing). 4D , the metadata model 416 is not populated with PII because one of its parent nodes (e.g., node 417f representing “Customer Identifying Information”) is associated with a “Level 2” PII classification (the first condition of metadata-driven transformation 422a) and is also associated with an “Admin-level” PII view permission that is greater than the “User-level” view permission (the second condition of metadata-driven transformation 422a). The decision to drop field “fname” is indicated in FIG. 4D by an “X” in node 417b, and the traversal of the metadata model 416 to make this decision is indicated by a dashed edge with a directional arrow. Because field “fname” is associated with PII (through its parent node 417f), the metadata-based processing engine 414 may also determine that the data contained within the field should be tokenized based on the “Tokenize” metadata-driven transformation 422 (shown in FIG. 4B ). However, the metadata-based processing engine 414 may determine that tokenizing the data in a field conflicts with dropping the field entirely, and may determine that a drop operation has priority based on rules (e.g., optimization rules) maintained by the system 400.

[0068] On the other hand, the metadata-based processing engine 414 determines that field “cust_id” should not be dropped because it does not have a parent node that meets both conditions specified by the metadata-driven transformation 422a. Similarly, the metadata-based processing engine 414 determines that the data in field “cust_id” should not be tokenized because it does not have a parent node with a “Level 2” PII classification. The decision to retain (not tokenize) field “cust_id” is indicated in FIG. 4D by the checkmark and bold outline of node 417c, and the traversal of the metadata model 416 to make this decision is indicated by the bold edge with a directional arrow. A similar analysis of the metadata model 416 results in the decision to retain fields “I95” and “amt” and to drop fields “cust_ssn” and “S14.” Note that the nodes of the dataset “Cust_acct” and associated metadata are faded in this example because they are not relevant to the processing specification 424a.

[0069] The metadata-based processing engine 414 may update the processing specification 424a using information determined by traversing the metadata model 416. For example, because the metadata-based processing engine 414 determined that fields “fname,” “S14,” and “cust_ssn” should be dropped according to the metadata-driven transformation 422a, the metadata-based processing engine 414 may update the processing specification to remove or modify operations that access those fields. In some examples, after applying these modifications, the metadata-based processing engine 414 may optimize the processing specification 424a to remove redundant or unnecessary operations. For example, in this example, the metadata-based processing engine 414 may determine that the “Cust_data read” operation 425b and the join operation 425c are no longer necessary, and may optimize these operations as a result (as indicated by the “X” through each operation).

[0070] After applying the metadata-driven transformation 422, the metadata-based processing engine 414 can execute the processing specification 424a to generate tokenized catalog view data 442. The catalog view data 442 can be sent to the client device 410 for presentation in a graphical user interface 444 that allows a user to view the tokenized data records in a customer view. The metadata-based processing engine 414 can also generate a customer view dataset 446 (e.g., a wide record of the customer view) that is sent to the storage system 406 for storage. As shown by the view data 442 and dataset 446, the "fname," "S14," and "cust_ssn" fields have been dropped (along with other redundant or unnecessary fields).

[0071] FIG. 5A illustrates a system 500 that is a variation of the system 400 illustrated in FIGS. 4A-4D. In this example, the system 500 includes a metadata management system 502, storage systems 504, 506, and client devices 508, 510. Similar to the other metadata management systems described herein, the metadata management system 502 includes a metadata discovery engine 512 and a metadata-based processing engine 514. In this example, the metadata-based processing engine 514 receives a new metadata-driven transformation 522a from a user of the client device 508. In particular, the user can use the client device 508 to access a graphical user interface 530 for defining the metadata-driven transformation 522a. The user can interact with the user interface 530 to input a name 530a for the metadata-driven transformation 522a, one or more conditions 530b for applying the transformation, and one or more expressions 530c that specify an operation to be performed when the condition 530b is met. In this example, the user has defined a metadata-driven transformation 522a named "Add Field" that causes a field to be inserted into the dataset when the PII classification of the parent node is equal to "Level 1" or "Level 2." After defining the metadata-driven transformation 522a, the user can choose to submit 530d the metadata-driven transformation 522a to the metadata-based processing engine 514. The metadata-based processing engine 514 can then incorporate the metadata-driven transformation 522a into the set of transformations 522 maintained by the system 500, as shown in visualization 522b of the metadata-driven transformation 522.

[0072] 5B , the metadata-based processing engine 514 can use the metadata model 516 to automatically apply a metadata-driven transformation 522 to the processing specification 524 and the underlying data. In this example, the metadata-based processing engine 514 identifies node 517a corresponding to the “Cust_data” dataset accessed in the processing specification 524a. The metadata-based processing engine 514 then traverses the metadata model 516, as indicated by the directional arrows, to identify nodes 517b and 517c corresponding to fields in the “Cust_data” dataset, as well as nodes 517d through 517h corresponding to logical parent nodes of the physical nodes. In this example, the metadata-based processing engine 514 determines that the field “fname” corresponding to node 517c has a parent node 517f (representing “Customer Identifying Information”) associated with a “Level 2” PII classification. As a result, the metadata-based processing engine 514 determines that a PII field should be added to the dataset including “fname” (e.g., “Cust_data”) according to the metadata-driven transformation 522a. Similarly, the metadata-based processing engine 514 also determines that the data in the "fname" field should be tokenized based on the "tokenize" metadata-driven transformation (shown in FIG. 5B). The decision to add a PII field and tokenize the data in field "fname" is indicated in FIG. 5B by the checkmark and bold outline of node 517b, and the traversal of the metadata model 516 to make this determination is indicated by the bold edge with directional arrows.

[0073] On the other hand, the metadata-based processing engine 514 determines, in accordance with the metadata-driven transformation 522a, that a PII field should not be added to the dataset containing the field “cust_id” because it has no parent node associated with PII. For this same reason, the metadata-based processing engine 514 determines that the data in “cust_id” does not need to be tokenized. The decision not to add a PII field and not to tokenize the data in the “cust_id” field is indicated in FIG. 5B by an “X” in node 517c, and the traversal of the metadata model 516 to make this decision is indicated by the dashed edge with a directional arrow. A similar analysis of the metadata model 516 results in the decision to add a PII field and tokenize the data in the “cust_ssn” and “S14” fields, but not to add a PII field and not to tokenize the data in the “I95” and “amt” fields. Note that the nodes for the dataset “Cust_acct” and associated metadata are faded in this example because they are not relevant to the processing specification 524a.

[0074] The metadata-based processing engine 514 can update the processing specification 524a using information determined by traversing the metadata model 516. For example, because the metadata-based processing engine 514 determined that a PII field should be added to a dataset containing “fname,” “S14,” and “cust_ssn” fields and that the data in these fields should be tokenized, the metadata-based processing engine 514 can update the processing specification to include a tokenize operation 525a that tokenizes these fields and an add PII operation 525b that adds a PII field having a value of “Yes” to each respective dataset. In some examples, the metadata-based processing engine 514 can optimize the processing specification 524a to remove redundant or unnecessary operations, such as by adding only a single PII field to the generated dataset (as shown in FIG. 5B ).

[0075] After applying the metadata-driven transformation 522, the metadata-based processing engine 514 can execute the processing specification 524a to generate tokenized catalog view data 532. The catalog view data 532 can be sent to the client device 510 for presentation in a graphical user interface 534 that allows a user to view the tokenized data records in a customer view. The metadata-based processing engine 514 can also generate a customer view dataset 536 (e.g., a customer view wide record) that is sent to the storage system 506 for storage. As shown by the view data 532 and the dataset 536, a PII field with a value of "Yes" has been added.

[0076] FIG. 6A illustrates a system 600 that is a variation of the system 500 illustrated in FIGS. 5A-5B. In this example, the system 600 includes a metadata management system 602, storage systems 604, 606, and client devices 608, 610. Similar to the other metadata management systems described herein, the metadata management system 602 includes a metadata discovery engine 612 and a metadata-based processing engine 614. In this example, the metadata-based processing engine 614 receives a new metadata-driven transformation 622a from a user of the client device 608. In particular, the user can use the client device 608 to access a graphical user interface 630 for defining the metadata-driven transformation 622a. The user can interact with the user interface 630 to enter a name 630a for the metadata-driven transformation 622a, one or more conditions 630b for applying the transformation, and one or more expressions 630c that specify an operation to be performed when the condition 630b is met. In this example, the user has defined a metadata-driven transformation 622a named "Allow PII View" that causes rows (e.g., data records) to be filtered from the dataset if the PII classification of the parent node is equal to "Level 2" and the customer's account level is greater than the user's level (e.g., the level of the user accessing the data). After defining the metadata-driven transformation 622a, the user can choose to submit 630d the metadata-driven transformation 622a to the metadata-based processing engine 614. The metadata-based processing engine 614 can then incorporate the metadata-driven transformation 622a into the set of transformations 622 maintained by the system 600, as shown in visualization 622b of the metadata-driven transformation 622.

[0077] Referring to FIG. 6B , the metadata-based processing engine 614 can use the metadata model 616 to automatically apply a metadata-driven transformation 622 to the processing specification 624 and the underlying data. In this example, a user with a non-admin user level requests access to the customer view defined by processing specification 624a. Because the metadata-driven transformation 622a requires a runtime comparison of the user's level with the customer's account level to determine whether to filter rows, the metadata-based processing engine 614 adds a read operation 625a to read the “Cust_acct” dataset and a join operation 625b to join the “Cust_acct” dataset to the “Loan_data” dataset. In some examples, the metadata-based processing engine 614 can use the metadata contained in the metadata model 616 (or data catalog) to determine how to access the “Cust_acct” dataset and how to join it with an existing dataset (e.g., the “Loan_data” dataset). The metadata-based processing engine 614 can also insert a filter rows operation 625c that filters rows when the condition "Cust_acct.level>user's level" is met. The metadata-based processing engine 614 also adds a tokenize operation 625d to tokenize the data according to the "tokenize" metadata-driven transform 622, as described herein.

[0078] After applying the metadata-driven transformation 622, the metadata-based processing engine 614 can execute the processing specification 624a to generate catalog view data 632. The catalog view data 632 can be sent to the client device 610 for presentation in a graphical user interface 634 that allows a user to view data records in a customer view when filtering conditions are met. The metadata-based processing engine 614 can also generate a customer view dataset 636 (e.g., a wide record of the customer view) that is sent to the storage system 606 for storage. As shown by dataset 636, when filtering conditions are met, rows are filtered out (as indicated by the strikethrough rows). Note that although the metadata-based processing engine 614 adds operation 625a to access the “Cust_acct” dataset for the filtering conditions, data associated with the “Cust_acct” dataset is not output according to the processing specification 624a.

[0079] Figure 7A shows a system 700 that is a variation of the system 600 shown in Figures 6A-6B. In this example, the system 700 includes a metadata management system 702, storage systems 704, 706, and client devices 708, 710. Similar to the other metadata management systems described herein, the metadata management system 702 includes a metadata discovery engine 712 and a metadata-based processing engine 714.

[0080] In this example, the metadata discovery engine 712 receives a new dataset 730 from the storage system 704. The metadata discovery engine 712 processes the dataset as described herein to discover or otherwise obtain metadata 732 for the new dataset 730. In particular, the metadata discovery engine 712 determines that the dataset 730 has the name “Cust_DOB” and contains fields “I95” (which serves as a primary key, as indicated by the key symbol) and “D55.” Based on a semantic analysis of the dataset 730 and its data, the metadata discovery engine 712 also determines that the dataset 730 “Cust_DOB” represents information about a “Customer DOB,” that the field “I95” represents a “Customer ID” that is part of a group “DB Identifier,” and that the field “D55” represents a “DOB” that is part of a group “Customer Identification Information.” The metadata discovery engine 712 also determines that the dataset 730 is related (e.g., through a primary-foreign key relationship) to the dataset 330a.

[0081] The metadata discovery engine 712 passes the discovered metadata 732 to the metadata-based processing engine 714. The metadata-based processing engine 714 uses the metadata 732 to update the metadata model 716. In particular, the metadata-based processing engine 714 may add nodes representing items of physical and logical metadata 732 for the dataset 730 and add edges representing relationships between the nodes. These additions are shown in bold in the metadata model 716 of FIG. 7A.

[0082] 7B , the metadata-based processing engine 714 uses the metadata model 716 to update the processing specification 724. In particular, the metadata-based processing engine 714 uses the metadata model 716 to update the processing specification 724a, which may specify the generation of a wide data record that includes all of the connected datasets associated with “Cust_data.” In this example, the metadata-based processing engine 714 updates the processing specification 724a to include an access operation 725a configured to read the new dataset 730 using the metadata in the metadata model 716 (e.g., metadata describing the dataset's access parameters). The metadata-based processing engine 714 also includes a join operation 725b configured to join the dataset 730 to the dataset “Cust_data” using the metadata in the metadata model 716 (e.g., metadata describing the primary-foreign key relationships). The metadata-based processing engine 714 can also use the metadata model 716 to apply the metadata-driven transformation 722 to the processing specification 724a, which now includes the new dataset 730. By leveraging metadata in this manner, system 700 can automatically adapt its processing of data to account for changes in the underlying data (including the addition of new data, such as dataset 730), without requiring the user to redefine or recode the underlying specifications.

[0083] 8 shows a flowchart of an example process 800 for using metadata to automate the identification and application of operations to data. Process 800 can be implemented by one or more of the systems and components described herein (e.g., a metadata management system and its components, such as a metadata discovery engine, a metadata-based processing engine, etc.), including one or more computing systems configured to implement the techniques described with reference to FIGS. 1-7.

[0084] The operations of process 800 include identifying 802 a given data item and physical metadata for the given data item. In some examples, identifying the given data item includes identifying the given data item to be accessed according to a processing specification. After identifying the given data item and the physical metadata for the data item, a metadata model is accessed 804. Generally, the metadata model can include parent nodes and child nodes connected by edges, where the parent node specifies logical metadata, the child node specifies physical metadata representing the data item, and the edges specify relationships between the nodes. In some examples, the metadata model includes a first data structure corresponding to the parent node, where the data structure includes logical metadata and at least a first pointer and a second pointer, where the first pointer points to a second data structure corresponding to a child node representing the physical metadata for the given data item, and the second pointer points to a third data structure corresponding to another child node representing the physical metadata of another data item different from the given data item.

[0085] Child nodes representing the physical metadata of the given data item are identified in the metadata model 806. In some examples, identifying the child nodes representing the physical metadata of the given data item includes matching the physical metadata of the given data item with the physical metadata represented by the child nodes in the metadata model.

[0086] One or more edges in the metadata model are traversed to identify one or more parent nodes of the child node (808). One or more operations to be performed on the given data item are determined based on logical metadata associated with the one or more identified parent nodes (810). In some examples, the metadata associated with the one or more parent nodes (e.g., logical metadata) includes metadata received from a user through interaction with a metadata management system. For example, a user can access a graphical user interface including a visualization of the metadata model and interact with the metadata model to select items of metadata associated with one or more nodes (e.g., parent nodes) and specify metadata for those nodes. For example, the metadata received from the user can be a personally identifiable information (PII) specification specified through the user's interaction with the visualization of the metadata model to select items of metadata associated with one or more of the parent nodes in the model for which a PII specification is to be specified. In some examples, after receiving the metadata specification (e.g., PII specification) to associate logical metadata with one or more parent nodes, the metadata model can be updated in the model, such as indicating that one or more of the parent nodes associated with the selected items of metadata are associated with the PII specification. The one or more operations applied to the given data item to transform the given data item may include, for example, tokenizing one or more fields of the given data item based on a PII specification.

[0087] In some examples, one or more metadata transforms are accessed to determine one or more operations to be performed on a given data item, with each metadata transform specifying at least one operation to be performed on the data and at least one condition for performing the at least one operation. The metadata transforms may then be used in conjunction with the metadata to determine the one or more operations to be performed on the given data item. For example, a metadata transform may be selected from one or more metadata transforms, and it is determined whether logical metadata associated with one or more parent nodes satisfies at least one condition of the selected metadata transform. If it is determined that the logical metadata associated with the one or more parent nodes satisfies at least one condition of the selected metadata transform, it may be determined that the one or more operations to be performed on the given data item include at least one operation specified by the selected metadata transform.

[0088] One or more operations are applied to the given data item to transform the data item (812). Generally, applying one or more operations to the given data item can include, among other things, transforming the data item, dropping the data item or a data field associated with the data item (e.g., from further processing by a computer program), adding a data field to the data item or a data set associated with the data item, and filtering data records associated with the data item. For example, the one or more operations to be performed on the given data item can include a tokenization operation, and applying one or more operations to the given data item to transform the given data item includes applying the tokenization operation to the given data item to tokenize one or more fields of the given data item. In some examples, applying one or more operations to the given data item includes inserting the one or more operations into a processing specification and executing the processing specification to apply the one or more operations to the given data item. In some examples, the process specification is a specification of a dataflow graph, and applying one or more operations to a given data item comprises executing the dataflow graph, where executing the dataflow graph applies the one or more operations to the given data item. After applying the one or more operations to the data item, the data item may be stored (814) (e.g., in memory or another hardware storage device), displayed to a user, or both.

[0089] In some examples, the metadata model is updated based on the transformed data item. For example, one or more nodes representing metadata of the transformed data item may be added to the metadata model, where the added one or more nodes represent the metadata of the transformed data item. Some of the added nodes may be duplicates of some of the nodes that were already present in the metadata model before the update. As another example, one or more edges may be added to the metadata model to propagate existing metadata to the transformed data item. For example, one or more edges may be added to existing nodes in the metadata model to propagate or connect existing nodes in the metadata model to the added nodes of the transformed data item. The updated metadata model may be output for further processing of the data item.

[0090] In some examples, operations 800 may further include identifying physical metadata for the transformed data item, identifying child nodes in the updated metadata model that represent the physical metadata of the transformed data item, traversing one or more edges in the updated metadata model to identify one or more parent nodes of the child nodes that represent the physical metadata of the transformed data item, and determining from logical metadata associated with the one or more parent nodes identified by traversing the one or more edges in the updated metadata model that one or more operations performed on the given data item should not be performed on the transformed data item. Determining that the one or more operations performed on the given data item should not be performed on the transformed data item may include determining that logical metadata associated with the one or more parent nodes identified by traversing the one or more edges in the updated metadata model differs from logical metadata associated with the one or more parent nodes identified by traversing the one or more edges in the metadata model.

[0091] In some examples, the operations 800 may further include identifying physical metadata for the transformed data item; identifying child nodes in the updated metadata model that represent the physical metadata of the transformed data item; traversing one or more edges in the updated metadata model to identify one or more parent nodes of the child nodes that represent the physical metadata of the transformed data item; determining one or more second operations to be performed on the transformed data item from logical metadata associated with the one or more parent nodes identified by traversing the one or more edges in the updated metadata model; applying the one or more second operations to the transformed data item to further transform the transformed data item; and storing the transformed data item.

[0092] In some examples, the operations 800 may further include accessing a plurality of data items stored on a hardware storage device, identifying, for each data item of the plurality of data items, physical metadata and logical metadata corresponding to the data item, generating a metadata model based on the identified physical metadata and logical metadata for each of the plurality of data items, and providing access to the metadata model for the first application and the second application. The first and / or second application may be, for example, a metadata management system or a data flow graph or other computer program executing on a client device, among other data processing systems. At least one of the first application or the second application may access the metadata model, and each of the first application and the second application may be configured to: access the given data item to identify physical metadata for the given data item; identify child nodes in the metadata model that represent the identified physical metadata for the given data item; traverse one or more edges in the metadata model to identify one or more parent nodes of the identified child nodes; determine at least one operation to be performed on the given data item from logical metadata associated with the identified one or more parent nodes; apply the at least one operation to the given data item to transform the given data item; and store the transformed data item.

[0093] Implementations of the subject matter and operations described herein, including the data capture system and its components, can be implemented in digital electronic circuitry, or computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Implementations of the subject matter described herein can be implemented as one or more computer programs (also referred to as data processing programs) (i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by or to control the operation of a data processing apparatus). A computer storage medium may be or be included in a computer-readable storage device, a computer-readable storage substrate, a random-access or serial-access memory array or device, or a combination of one or more of these. A computer storage medium may also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices). The subject matter may be implemented in computer program instructions stored on a non-transitory computer storage medium.

[0094] The operations described herein may be implemented as operations performed by a data processing system or apparatus on data stored in one or more computer-readable storage devices or received from other sources. The term “data processing system” encompasses all types of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, a system-on-a-chip, or a plurality or combination of the foregoing. A system may include special-purpose logic circuitry, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). In addition to hardware, a system may also include code that provides an execution environment for the computer program in question (e.g., code comprising processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof). The system and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0095] A computer program (also known as a program, software, software application, script, or code) can be written in any type of programming language, including compiled or interpreted, declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple cooperating files (e.g., files storing one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0096] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry (e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit)).

[0097] Processors suitable for executing a computer program include, by way of example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer include a processor for performing actions in accordance with the instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data, or be operatively coupled to receive data from, transfer data to, or both, although a computer need not have such devices. Furthermore, a computer can be incorporated into another device (e.g., a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, Global Positioning System (GPS) receiver, or portable storage device (e.g., a universal serial bus (USB) flash drive)). Suitable devices for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0098] Implementations of the subject matter described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a user computer having a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described herein), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0099] A computing system may include users and servers. Users and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server may arise by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some implementations, a server sends data (e.g., HTML pages) to a user device (e.g., for the purpose of displaying the data to and receiving user input from a user interacting with the user device). Data generated at the user device (e.g., results of user interaction) may be received from the user device at the server.

[0100] While this specification includes details of many specific implementations, these should not be construed as limitations on any implementation or the scope of what may be claimed, but rather as descriptions of features specific to particular implementations. Certain features described herein in the context of separate embodiments can also be implemented in combination in a single implementation. Conversely, various features described in the context of a single embodiment can also be implemented separately in multiple implementations or in any suitable subcombination. Furthermore, while features may be described above as acting in a particular combination and initially claimed as such, one or more features from a claimed combination can, in some cases, be removed from the combination, and the claimed combination may be directed to a subcombination or variations of the subcombination.

[0101] Similarly, although operations are depicted in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown, or in any sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above implementations should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0102] Other implementations are within the scope of the following claims.

Claims

1. 1. A method implemented by a metadata management system for using a metadata model to identify which one or more operations to perform when processing one or more data items, the metadata model including one or more parent nodes and one or more child nodes, the one or more operations being defined with respect to one or more parent nodes in the metadata model and applied to data represented by one or more child nodes in the metadata model, the one or more parent nodes specifying logical metadata and the one or more child nodes specifying physical metadata representing the one or more data items, the method comprising: identifying a given data item and physical metadata for said given data item; accessing the metadata model, wherein edges connect the nodes, each edge specifying a relationship between two nodes; identifying a child node in the metadata model that represents the physical metadata of the given data item; traversing one or more edges in the metadata model to identify one or more parent nodes of the child nodes; determining one or more operations to be performed on the given data item from logical metadata associated with the identified one or more parent nodes; applying said one or more operations to said given data item to transform said given data item; storing the transformed data items in a memory; A method comprising:

2. 2. The method of claim 1 , wherein identifying the child nodes representing the physical metadata of the given data item comprises matching the physical metadata of the given data item with physical metadata represented by the child nodes in the metadata model.

3. 10. The method of claim 1, further comprising: accessing one or more metadata transforms to determine the one or more operations to be performed on the given data item, each metadata transform specifying at least one operation to be performed on the data and at least one condition for performing the at least one operation.

4. Determining the one or more operations to be performed on the given data item comprises: selecting a metadata transformation from the one or more metadata transformations; determining whether the logical metadata associated with the one or more parent nodes satisfies the at least one condition of the selected metadata transformation; 4. The method of claim 3, further comprising: in response to determining that the logical metadata associated with the one or more parent nodes satisfies the at least one condition of the selected metadata transform, determining that the one or more operations to be performed on the given data item include the at least one operation specified by the selected metadata transform.

5. Identifying the given data item includes identifying the given data item to be accessed according to a processing specification, and applying the one or more operations to the given data item includes: inserting the one or more operations into the processing specification; executing the processing specification to apply the one or more operations to the given data item.

6. 6. The method of claim 5, wherein the process specification comprises a specification of a dataflow graph, and wherein applying the one or more operations to the given data items comprises executing the dataflow graph, and wherein executing the dataflow graph applies the one or more operations to the given data items.

7. The method of claim 1 , further comprising updating the metadata model based on the transformed data items.

8. 8. The method of claim 7, wherein updating the metadata model comprises adding one or more edges to existing nodes in the metadata model to connect the existing nodes in the metadata model to added nodes.

9. identifying physical metadata for the transformed data item; identifying a child node in the updated metadata model that represents the physical metadata of the transformed data item; traversing one or more edges in the updated metadata model to identify one or more parent nodes of the child nodes that represent the physical metadata of the transformed data item; determining, from logical metadata associated with the one or more parent nodes identified by traversing the one or more edges in the updated metadata model, that the one or more operations performed on the given data item should not be performed on the transformed data item; The method of claim 7 further comprising:

10. 10. The method of claim 9, wherein determining that the one or more operations performed on the given data item should not be performed on the transformed data item comprises determining that the logical metadata associated with the one or more parent nodes identified by traversing the one or more edges in the updated metadata model differs from the logical metadata associated with the one or more parent nodes identified by traversing the one or more edges in the metadata model.

11. 8. The method of claim 7, wherein updating the metadata model includes adding one or more nodes to the metadata model, the one or more added nodes representing metadata for the transformed data items.

12. The method of claim 11 , wherein some of the added nodes are duplicates of some of the nodes that were already present in the metadata model before the update.

13. The method of claim 12 , further comprising outputting the updated metadata model for further processing of data items.

14. identifying physical metadata for the transformed data item; identifying a child node in the updated metadata model that represents the physical metadata of the transformed data item; traversing one or more edges in the updated metadata model to identify one or more parent nodes of the child nodes that represent the physical metadata of the transformed data item; determining one or more second operations to be performed on the transformed data items from logical metadata associated with the one or more parent nodes identified by traversing the one or more edges in the updated metadata model; and applying, by the metadata management system, the one or more second operations to the transformed data items to further transform the transformed data items; storing said further transformed data items in a memory; The method of claim 12 further comprising:

15. 2. The method of claim 1, wherein applying the one or more operations to the given data item comprises dropping a data field associated with the data item from further processing by a computer program.

16. The method of claim 1 , wherein applying the one or more operations to the given data item comprises adding a data field to the data item or a data set associated with the data item.

17. The method of claim 1 , wherein applying the one or more operations to the given data item comprises filtering data records associated with the data item.

18. 2. The method of claim 1 , wherein the one or more operations to be performed on the given data item include a tokenization operation, and wherein applying the one or more operations to the given data item to transform the given data item includes applying the tokenization operation to the given data item to tokenize one or more fields of the given data item.

19. The method of claim 1 , wherein the logical metadata associated with the one or more parent nodes comprises metadata received from a user through interaction with the metadata management system.

20. the metadata received from the user is a personally identifiable information (PII) specification specified by the user via a graphical user interface, the graphical user interface including a visualization of the metadata model, and PII classifications specified by the user's interaction with the visualization of the metadata model to select items of metadata associated with one or more of the parent nodes in the model to which the PII specification is specified, the method comprising:

20. The method of claim 19, further comprising: updating the model after receiving the PII specification to associate the logical metadata with the one or more parent nodes, the updating including indicating in the model that the one or more of the parent nodes associated with the selected item of metadata are associated with the PII specification, and the one or more operations applied to the given data item to transform the given data item include tokenizing one or more fields of the given data item.

21. accessing a plurality of data items stored on a hardware storage device; for each data item of the plurality of data items, identifying physical metadata and logical metadata corresponding to the data item; generating the metadata model based on the physical metadata and the logical metadata identified for each of the plurality of data items; providing access to the metadata model for a first application and a second application; accessing the metadata model by at least one of the first application or the second application; wherein each of the first application and the second application further comprises: accessing the given data item to identify the physical metadata for the given data item; identifying the child node in the metadata model that represents the identified physical metadata of the given data item; traversing one or more edges in the metadata model to identify the one or more parent nodes of the identified child nodes; determining at least one operation to be performed on the given data item from the logical metadata associated with the identified one or more parent nodes; applying said at least one operation to said given data item to transform said given data item; and storing the transformed data items. The method of claim 1.

22. 2. The method of claim 1 , wherein the metadata model includes a first data structure corresponding to the parent node, the data structure including the logical metadata and at least a first pointer and a second pointer, the first pointer pointing to a second data structure corresponding to the child node representing the physical metadata of the given data item, and the second pointer pointing to a third data structure corresponding to another child node representing physical metadata of another data item different from the given data item.

23. 1. A system comprising: at least one processor; a memory storing instructions executable by the at least one processor to perform operations; and the operation comprises: identifying a given data item and physical metadata for said given data item; accessing a metadata model, the metadata model including one or more parent nodes specifying logical metadata and one or more child nodes specifying physical metadata, with edges connecting the nodes, the edges specifying relationships between two nodes; identifying a child node in the metadata model that represents the physical metadata of the given data item; traversing one or more edges in the metadata model to identify one or more parent nodes of the child nodes; determining one or more operations to be performed on the given data item from logical metadata associated with the identified one or more parent nodes; applying said one or more operations to said given data item to transform said given data item; and storing the transformed data items in a memory.

24. A non-transitory computer-readable medium storing instructions executable by at least one processor to perform operations, the operations comprising: identifying a given data item and physical metadata for said given data item; accessing a metadata model, the metadata model including one or more parent nodes specifying logical metadata and one or more child nodes specifying physical metadata, with edges connecting the nodes, the edges specifying relationships between two nodes; identifying a child node in the metadata model that represents the physical metadata of the given data item; traversing one or more edges in the metadata model to identify one or more parent nodes of the child nodes; determining one or more operations to be performed on the given data item from logical metadata associated with the identified one or more parent nodes; applying said one or more operations to said given data item to transform said given data item; and storing the transformed data item in a memory.

Images