Metadata management systems and their implementation methods, media, and program products

Abstract

A method for performing operations on data items using a metadata model, wherein the metadata model includes parent nodes and child nodes connected by edges, wherein the parent node specifies logical metadata and the child node specifies physical metadata representing the data item, and wherein the edges specify relationships between the nodes. The method includes: identifying a given data item and the physical metadata of the given data item; accessing the metadata model, identifying child nodes in the metadata model that represent the physical metadata of the given data item; traversing one or more edges in the metadata model to identify the parent node of the child node; determining one or more operations to be performed on the given data item from the logical metadata associated with the identified parent node; applying the one or more operations to the given data item to transform the data item; and storing the transformed data item.

Description

[0001] priority

[0002] This application claims priority and benefit to U.S. Patent Application No. 18 / 104,066, filed January 31, 2023, and U.S. Provisional Patent Application No. 63 / 400,333, filed August 23, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to techniques for making metadata operational within a data enterprise. Background Technology

[0004] To conduct business, organizations maintain increasingly large and complex datasets. In many cases, organizational data is stored in ways that facilitate efficient use of computing resources rather than in ways that are understandable to humans. Therefore, users may struggle to understand what data an organization possesses or how to access that data for various tasks. Even when users can identify a relevant piece of data, especially non-technical users, generating code to perform the desired operations on that data can be difficult and time-consuming. Furthermore, changes to organizational data may disrupt or otherwise render existing data processing operations obsolete. Summary of the Invention

[0005] Generally, in a first aspect, a method implemented by a metadata management system uses a metadata model to identify one or more data items and perform one or more operations on them, wherein the metadata model includes one or more parent nodes and one or more child nodes, wherein the one or more operations are defined relative to the one or more parent nodes in the metadata model and applied to data represented by the one or more child nodes in the metadata model, wherein the one or more parent nodes specify logical metadata, and wherein the one or more child nodes specify physical metadata representing one or more data items. The method includes: identifying a given data item and the physical metadata of the given data item; accessing the metadata model by the metadata management system, wherein edges connect nodes, wherein edges specify a relationship between two nodes; identifying child nodes in the metadata model that represent the physical metadata of the given data item; traversing one or more edges in the metadata model to identify one or more parent nodes of the child nodes; determining one or more operations to be performed on the given data item from the metadata associated with the identified one or more parent nodes; applying the one or more operations by the metadata management system to the given data item to transform the given data item; and storing the transformed data item in memory.

[0006] Generally speaking, in a second aspect that can be combined with the first aspect, identifying a child node representing the physical metadata of a given data item includes matching the physical metadata of the given data item with the physical metadata represented by the child node in the metadata model.

[0007] Generally speaking, in a third aspect that can be combined with the first and second aspects, the method further includes accessing one or more metadata transformations to determine one or more operations to be performed on a given data item, each metadata transformation specifying at least one operation to be performed on the data and at least one condition for performing the at least one operation.

[0008] Generally speaking, in a fourth aspect that can be combined with any of the first to third aspects, determining the one or more operations to be performed on a given data item includes: selecting a metadata transformation from the one or more metadata transformations; determining whether the logical metadata associated with the one or more parent nodes satisfies at least one condition of the selected metadata transformation; and in response to determining that the logical metadata associated with the one or more parent nodes satisfies at least one condition of the selected metadata transformation, determining that the one or more operations to be performed on a given data item includes at least one operation specified by the selected metadata transformation.

[0009] Generally speaking, in a fifth aspect that can be combined with any of the first to fourth aspects, identifying a given data item includes identifying a given data item accessed according to a processing specification, and applying one or more operations to the given data item includes inserting one or more operations into a processing specification, and executing the processing specification to apply one or more operations to the given data item.

[0010] Generally speaking, in the sixth aspect, which can be combined with any of the first to fifth aspects, the processing specification includes a specification for the data flow graph, and applying the one or more operations to a given data item includes executing the data flow graph, wherein the executing data flow graph applies one or more operations to the given data item.

[0011] Generally speaking, in the seventh aspect, which can be combined with any of the first to sixth aspects, the method also includes updating the metadata model based on the transformed data items.

[0012] Generally speaking, in the eighth aspect, which can be combined with any of the first to seventh aspects, updating the metadata model includes adding one or more nodes to the metadata model, wherein the added one or more nodes represent the metadata of the transformed data item.

[0013] Generally speaking, in the ninth aspect, which can be combined with any of the first to eighth aspects, updating the metadata model includes adding one or more edges to existing nodes in the metadata model to connect existing nodes in the metadata model to the added nodes.

[0014] Generally speaking, in the tenth aspect, which can be combined with any of the first to ninth aspects, some of the nodes added are copies of some of the nodes that already existed in the metadata model before the update.

[0015] Generally speaking, in the eleventh aspect, which can be combined with any of the first to tenth aspects, the updated metadata model is output for further processing of the data items.

[0016] Generally speaking, in a twelfth aspect that can be combined with any of the first to eleventh aspects, the method further includes: identifying physical metadata of the transformed data item; identifying child nodes representing the physical metadata of the transformed data item in the updated metadata model; traversing one or more edges in the updated metadata model to identify one or more parent nodes representing the child nodes of the physical metadata of the transformed data item; and determining from logical metadata associated with the one or more parent nodes identified by traversing the one or more edges in the updated metadata model that the one or more operations performed on a given data item will not be performed on the transformed data item.

[0017] Generally speaking, in the thirteenth aspect, which can be combined with any of the first to twelfth aspects, determining that one or more operations performed on a given data item will not be performed on the transformed data item includes determining that the logical metadata associated with one or more parent nodes identified by traversing one or more edges in the updated metadata model is different from the logical metadata associated with one or more parent nodes identified by traversing one or more edges in the metadata model.

[0018] Generally speaking, in a fourteenth aspect that can be combined with any of the first to thirteenth aspects, the method further includes: identifying physical metadata of the transformed data item; identifying child nodes representing the physical metadata of the transformed data item in the updated metadata model; traversing one or more edges in the updated metadata model to identify one or more parent nodes of the child nodes representing the physical metadata of the transformed data item; determining one or more second operations to be performed on the transformed data item from logical metadata associated with the one or more parent nodes identified by traversing the one or more edges in the updated metadata model; applying the one or more second operations to the transformed data item by the metadata management system to further transform the transformed data item; and storing the further transformed data item in memory.

[0019] Generally speaking, in the fifteenth aspect, which can be combined with any of the first to fourteenth aspects, applying one or more operations to a given data item includes discarding the data fields associated with the data item to prevent further processing by a computer program.

[0020] Generally speaking, in the sixteenth aspect, which can be combined with any of the first to fifteenth aspects, applying one or more operations to a given data item includes adding data fields to the data item or the dataset associated with the data item.

[0021] Generally speaking, in the seventeenth aspect, which can be combined with any of the first to sixteenth aspects, applying one or more operations to a given data item includes filtering the data records associated with that data item.

[0022] Generally speaking, in the eighteenth aspect, which can be combined with any of the first to seventeenth aspects, one or more operations performed on a given data item include a tokenization operation, and applying the one or more operations to the given data item to transform the given data item includes applying the tokenization operation to the given data item to tokenize one or more fields of the given data item.

[0023] Generally speaking, in the nineteenth aspect, which can be combined with any of the first to eighteenth aspects, the logical metadata associated with the one or more parent nodes includes metadata received from the user through interaction with the metadata management system.

[0024] Generally speaking, in the twentieth aspect, which can be combined with any of the first to nineteenth aspects, the metadata received from the user is a Personally Identifiable Information (PII) specification specified by the user through a graphical user interface, the graphical user interface including a visualization of the metadata model, and the user interacting with the visualization of the metadata model to select a metadata item associated with one or more parent nodes in the parent node for which the PII specification will be specified in the model to specify a PII classification, and the method further includes: updating the model after receiving the PII specification to associate logical metadata with the one or more parent nodes, including instructing in the model that the one or more parent nodes associated with the selected metadata item are associated with the PII specification, wherein the one or more operations applied to a given data item to transform the given data item include tokenizing one or more fields of the given data item.

[0025] Generally speaking, in the twenty-first aspect, which can be combined with any of the first to twentieth aspects, a plurality of data items stored in a hardware storage device are accessed; for each of the plurality of data items, physical metadata and logical metadata corresponding to that data item are identified; a metadata model is generated based on the physical metadata and logical metadata identified for each of the plurality of data items; access to the metadata model is provided for a first application and a second application; the metadata model is accessed by at least one of the first application or the second application, wherein each of the first application and the second application is configured to: access a given data item to identify the physical metadata of the given data item; identify child nodes in the metadata model that represent the identified physical metadata of the given data item; traverse one or more edges in the metadata model to identify the one or more parent nodes of the identified child nodes; determine at least one operation to be performed on the given data item from the logical metadata associated with the one or more identified parent nodes; apply the at least one operation to the given data item to transform the given data item; and store the transformed data item.

[0026] Generally speaking, in the twenty-second aspect, which can be combined with any of the first to twenty-first aspects, the metadata model includes a first data structure corresponding to a parent node, the data structure including logical metadata and at least a first pointer and a second pointer, wherein the first pointer points to a second data structure corresponding to a child node representing physical metadata of a given data item, and the second pointer points to a third data structure corresponding to another child node representing physical metadata of another data item different from the given data item.

[0027] Generally speaking, in the twenty-third aspect, a system includes at least one processor and a memory storing instructions that can be executed by the at least one processor to perform any of the operations of the first to twenty-second aspects.

[0028] Generally speaking, in the twenty-fourth aspect, there is a non-transitory computer-readable medium storage instruction that can be executed by at least one processor to perform the operations of any of the first to twenty-second aspects.

[0029] These aspects may include one or more of the following advantages.

[0030] By automating the identification of operations and their application to data through the use of metadata, operations can be automatically applied across multiple data items within a data enterprise without requiring users to define operations on a per-data-item basis. This not only improves efficiency in handling large volumes of continuously changing data (e.g., because operations don't need to be defined for each individual data item), but also enhances the robustness and integrity of data strategies (e.g., data security, data governance, data quality, etc.) because operations can automatically adapt to changes in the underlying data, including the addition of new data. Furthermore, it facilitates the integration of new operations, such as considering new data types or new data strategies.

[0031] The techniques described in this paper also provide for more efficient data processing with reduced memory. Existing systems store metadata in read-only format, which has become a major source of latency and inefficiency. This is because metadata is only used by analysts to understand the data, but they then manually retrieve that information and use it in different projects as needed. Instead of treating metadata as read-only, the techniques described use metadata as the initial starting point in a chain of connections for implementing that metadata using other applications to define the necessary processing of selected data (e.g., datasets), and then performing that processing. For example, a metadata management system can connect to one or more other systems, such as a first system for writing data flow diagrams (and / or other computer programs) that define data processing operations, and a second system as a data governance system. Under the legacy system, users could view metadata for various datasets. However, if a user wanted to access that data, for example, to define the processing to be performed in the first system and / or the control operations to be performed by the second system, the data would have to be accessed and imported twice (once for each system) to access the metadata, which was then used to define the data processing operations in the first system and specify the control operations in the second system. Now, each dataset only needs to be accessed once, instead of multiple accesses (once for each application that needs metadata to define processing, control, etc.), thus reducing the time and resources required to access the dataset. Metadata is identified and provided with semantic meaning. The metadata model is then generated and continuously updated as additional metadata is created. The data processing system makes the metadata in the metadata model accessible to everyone in the application or system, resulting in a system where the metadata of a dataset only needs to be read once, and semantic discovery only needs to be performed once on that read metadata (or the data corresponding to that read metadata) for use in multiple applications or systems. This single read, access, and processing of metadata (via semantic discovery) results in a metadata model that is continuously updated with new metadata (e.g., by adding new nodes to the metadata model), and this metadata model can be accessed by various systems and applications to define processing.

[0032] According to a preferred aspect, users can specify the operations to be performed on data within the data enterprise without needing to define (e.g., encode) the means for accessing the data or performing the operations. For example, a user (e.g., a non-technical user) can specify that a particular item in logical metadata (e.g., SSN) is in the form of Personally Identifiable Information (PII), without knowing which item(s) of the physical data within the enterprise corresponds to the SSN, or how to access those data items. Based on this metadata definition, the data item corresponding to the SSN is automatically identified by the system (e.g., by traversing the metadata model) and processed to obfuscate the data (e.g., by performing or inserting operations to mask or tokenize the data into a computer program), without the user having to generate code to perform these operations.

[0033] According to a preferred aspect, metadata generated from operations applied to data is propagated or copied to the metadata model to reduce further consumption of computational resources during subsequent processing based on the metadata model. In other words, the technique described herein provides increased efficiency in processing data. This is because the metadata model is continuously updated with the results of previous processing. For example, if a new data item is generated due to an operation (e.g., tokenized SSN data is generated due to a tokenization operation), the system can update the metadata model to include the metadata of the new data item. That is, nodes and edges are added to the metadata model, where the added nodes and edges represent the tokenized SSN data (e.g., the meaning of the tokenized data, the relationship between the tokenized data and other data or metadata, the storage location of the tokenized data, and other access parameters, etc.). Therefore, if the data processing system needs the tokenized data at a later point in time, the data processing system does not need to re-tokenize the original data. Instead, the data processing system uses the metadata model to identify and access the tokenized SSN data. Storing metadata representing the tokenized SSN data in the metadata model saves computational resources because the data processing system can simply look up the tokenized data instead of having to recompile it based on the original data.

[0034] Previously discovered or generated metadata can also be propagated or copied to new data items. That is, new edges can be added to the metadata model to associate the node representing the new data item with existing nodes in the model. For example, if a specific data item is associated with an SSN in the metadata model (such as a PII), and a new data item is created based on that specific data item (e.g., due to a copy operation), the system can automatically propagate the SSN association to the new data item in the metadata model (e.g., by adding an edge between the SSN node and the node representing the new data item). Therefore, if the system accesses the new data item at a later point in time, it will know that the new data item is associated with an SSN and, if necessary, perform appropriate data quality and / or data security operations based on that association. By updating the metadata model in this way, the system leverages existing work, such as the work done in discovering or generating metadata, to reduce the amount of computational resources (e.g., memory, processing loops, etc.) required to perform subsequent operations on the data. Furthermore, propagating metadata in the metadata model ensures adherence to data policies (e.g., data security policies, data governance policies, data quality policies, etc.) when data changes are created within the system.

[0035] Details of one or more specific embodiments are described in the accompanying drawings and the detailed description below. Other features, objectives, and advantages of the technology described herein will be apparent from the description, the drawings, and the claims. Attached Figure Description

[0036] Figure 1 An example of a metadata management system is shown.

[0037] Figures 2A to 2B An example metadata management system configured to implement metadata is shown.

[0038] Figures 3A to 3D An example metadata management system configured to implement metadata is shown.

[0039] Figures 4A to 4D This example demonstrates a metadata management system configured to implement metadata to discard fields.

[0040] Figure 5A and Figure 5B This example demonstrates a metadata management system configured to implement metadata to add fields.

[0041] Figure 6A and Figure 6B An example metadata management system configured to implement metadata to filter rows is shown.

[0042] Figure 7A and Figure 7B An example metadata management system configured to implement metadata to adapt to changes in the data is shown.

[0043] Figure 8 This is an example process for using metadata to automate operations on data. Detailed Implementation

[0044] This disclosure relates to using metadata to automate the identification of operations and their application to data within a data enterprise. In some examples, a data processing system (sometimes referred to as a metadata management system) uses metadata about the data stored within the data enterprise to generate a metadata model. The system can also enrich the metadata model with user-specified metadata. When data within the data enterprise is accessed, the system uses the metadata model to automatically identify operations and apply them to that data. In this way, users can specify operations to be performed on data within the enterprise without needing to understand the enterprise's technical details or generate code to perform the operations. Furthermore, because operations are defined at the metadata level, they can be automatically applied across multiple data items accessed or otherwise processed by multiple different applications, without requiring users to define operations item-by-item in each application. Defining operations at the metadata level also allows the system to automatically adapt to changes in the underlying data, including the addition of new data. In some examples, updating the metadata model due to operations will improve the efficiency of subsequent data processing.

[0045] Generally, a metadata management system can perform various processes to obtain metadata about data stored on one or more data sources within a data enterprise. For example, the system can discover physical metadata describing the attributes of the data (e.g., where the system data resides, schema, tables, field names, data types, data formats, etc.) and the relationships between the data (e.g., primary-foreign key relationships, entity relationships, etc.). Data processing systems can also generate logical metadata based on the content of the stored data (or a selected subset of the stored data). Logical metadata provides detailed information about how data is linked together to form a larger set. It also outlines how data flows through systems and processes from creation to storage, transformation, and consumption. Logical metadata can map the data path through the data supply chain, including its usage and changes over time. For example, the system might determine that the stored data includes a dataset related to information about customers, and that this dataset contains a data field storing each customer's Social Security Number (SSN).

[0046] Using metadata, the system generates a metadata model that describes the physical and logical relationships, as well as other attributes, of the stored data. Generally, a metadata model may include nodes representing physical and logical metadata items, where edges represent relationships between nodes. The system can enrich the metadata model using user-specified metadata (sometimes called user-specified metadata). For example, a user can specify that an SSN is a form of personally identifiable information (PII) that should be protected by the system. Based on this specification, the system can update the metadata model to indicate that nodes representing SSNs are associated with PIIs.

[0047] The system then uses the metadata model to automatically perform operations on data within the data enterprise. For example, when a data item within the enterprise is accessed (e.g., via a computer program), the system identifies a node in the metadata model corresponding to the metadata (e.g., physical metadata) of the data item being accessed. The system can then traverse the metadata model to find other nodes associated with the identified node, as described in detail below. Based on the metadata (e.g., logical metadata) associated with the relevant nodes, the system determines one or more operations to be performed on the data item being accessed. For example, if the accessed data item is associated with an SSN node (as described above, which has been marked as a PII), the system can determine to automatically insert or otherwise perform obfuscation or tokenization operations on the data item in the computer program. This enhances data security. Furthermore, the system can automatically perform operations (e.g., tokenization) on data items based on the user's high-level metadata specifications without requiring the user to identify the specific data item to be processed or generate code to perform the operation. This improves the efficiency of data processing.

[0048] In some examples, the system can update the metadata model based on operations performed on data items. For instance, if a new data item is generated due to the tokenization operation described above, the system can update the metadata model to include the metadata of the new data item. In some examples, previously discovered or generated metadata can also be propagated to new data items, as described in detail below. By updating the metadata model in this way, the system leverages existing work to reduce the amount of computational resources (e.g., memory, processing loops, etc.) required to perform subsequent operations on the data.

[0049] refer to Figure 1The diagram illustrates a system 100 for implementing metadata. In this example, system 100 includes a metadata management system 102, storage systems 104 and 106, and client devices 108 and 110. Generally, the metadata management system 102 is a data processing system that uses metadata to automate the identification of operations and their application to data, for example, stored in storage system 104. After processing the data, the metadata management system 102 may store the data in storage system 106, provide the data to client device 110 for display, or both. Although shown as separate entities, in some examples, storage system 104 may be the same storage system as storage system 106. Similarly, in some examples, client device 108 may be the same client device as client device 110.

[0050] Metadata management system 102 includes a metadata discovery engine 112 and a metadata-based processing engine 114. Discovery engine 112 includes program instructions and / or executable logic for discovering or otherwise obtaining metadata of data stored in storage system 104. For example, discovery engine 112 may perform a discovery process to obtain physical metadata describing the attributes of the data (e.g., field names, data types, data formats, etc.) and the relationships between the data (e.g., primary-foreign key relationships, entity relationships, etc.). In this example, data is prepared for processing by discovery engine 112 using format information. Data including records with field values ​​is received via an input device or port. A target record format for processing the data is determined. Multiple records are analyzed according to validation tests to determine whether the data matches candidate record formats. Each candidate record format specifies the format of each field, and each validation test corresponds to at least one candidate record format. In response to receiving the results of the validation tests, a target record format is associated with the data based on at least one of the following: candidate record formats that at least partially match are determined according to at least one validation test; a parsed record format is selected based on the data type associated with the data; and a constructed record format is generated from the analysis of data characteristics. Other examples of such discovery processes are described in U.S. Patent Application No. 12 / 945,094 entitled “Managing record format information,” the entire contents of which are incorporated herein by reference.

[0051] Discovery engine 112 may also perform a semantic discovery process on data (or a selected subset of data) to generate logical metadata, etc., representing the semantic meaning of the data. For example, discovery engine 112 may identify a field included in one or more datasets, that field having an identifier. For that field, discovery engine 112 profiles the data values ​​of that field to generate a data profile, accesses multiple label proposal tests, and generates a set of label proposals by applying the multiple label proposal tests to the data profile. Discovery engine 112 then determines the similarity between the label proposals and selects a category. Discovery engine 112 identifies one of the label proposals as identifying the semantic meaning. Discovery engine 112 stores the identifier of the field of the identified label proposal that has the semantic meaning identified in the label proposal. Other examples of such semantic discovery are described in U.S. Patent Application 16 / 794,361, entitled “Discovering a semanticmeaning of data fields from profile data of the data fields,” the entire contents of which are incorporated herein by reference.

[0052] Discovery engine 112 passes physical and logical metadata to metadata-based processing engine 114, which uses the metadata to generate a metadata model 116 with multiple nodes 117 and edges 118 representing relationships between nodes. Metadata-based processing engine 114 also enriches metadata model 116 using user-defined metadata received from, for example, client device 108. In some examples, metadata model 116 may include a set of objects or data structures, each representing a node. Each object or data structure may include data elements representing physical, logical, and / or user-defined metadata of the corresponding node, and pointers to other objects or data structures representing other nodes connected to the corresponding node via edges. In some examples, the metadata-based processing engine uses the metadata model and other data to generate a data catalog 120 of data (or a selected subset of the stored data) stored in storage system 104. Data catalog 120 may include one or more data objects containing metadata and other information identifying the data or groups of data stored in storage system 104. Users can interact with data catalog 120 to define object properties or select objects for data processing, etc. For example, a user can associate objects in data catalog 120 with one or more metadata-driven transformations 122, as discussed below. As another example, a development environment (not shown) that is part of or communicates with the metadata management system 102 may include a user interface having a representation of catalog 120, and the user can select objects from the catalog to use input, such as to a data flow graph or other computer program. Techniques for generating, maintaining, and using data catalogs are described in U.S. Patent No. 9,977,659, entitled “Managing Data Set Objects,” the entire contents of which are incorporated herein by reference.

[0053] The metadata-based processing engine 114 may also store or otherwise access multiple processing specifications 124. Generally, processing specifications 124 may define or include program instructions and / or executable logic for processing data stored in storage system 104. In some examples, each specification in processing specification 124 may be or otherwise define a computer program, such as a data flow graph. A data flow graph may include: multiple vertices representing computational processes, each vertex having an associated access method; and multiple links, each link connecting at least two vertices to each other and representing data flow between the connected vertices. Upon execution, a system executing the graph (e.g., metadata management system 102 or another data processing system) prepares the graph for execution by executing graph transformation steps until each vertex is runnable, and each link is associated with at least one communication method compatible with the access methods of the vertices connected by the link; each link is initiated by creating a combination of communication channels and / or data storage suitable for the link communication methods using the execution system; and each process is initiated by invoking the execution of a process on the execution system. Additional details regarding a specific implementation of such graph-based computation are described in U.S. Patent No. 5,966,072, entitled “Executing Computations Expressed as Graphs,” the entire contents of which are incorporated herein by reference. Processing specification 124 may define or include operations for accessing data in data catalog 120 for any of a variety of reasons (such as ingesting data into storage system 106 or generating a view of the data for presentation on client device 110) (e.g., from storage system 104). Regardless of the specific process defined by processing specification 124, metadata-based processing engine 114 may use metadata model 116 to automatically perform operations (e.g., metadata-driven transformations 122) on data accessed from storage system 104, as described herein.

[0054] Figure 2A and Figure 2B A system 200 for implementing metadata according to aspects of this disclosure is illustrated. (Reference) Figure 2A System 200 includes a metadata management system 202, storage systems 204 and 206, and client devices 208 and 210. Similar to... Figure 1The metadata management system 102 shown includes a metadata discovery engine 212 and a metadata-based processing engine 214. In this example, the metadata discovery engine 212 receives a dataset 230 stored in the storage system 204 and processes the dataset 230 to obtain the corresponding metadata. Specifically, the metadata discovery engine 212 determines that the dataset 230a includes the following physical metadata items: name "Cust_data", type "table", and fields "cust_fnln" and "cust_ssn".

[0055] Users can access and interact with the metadata management system 202 (e.g., via an application running on client device 208) to view and include the physical metadata of dataset 230 in a graphical user interface 232. Within the user interface 232, users can select 234 datasets 230 to add to a data cart 236 based on the physical metadata. The metadata discovery engine 212 can then perform semantic discovery or other processing on the datasets 230 within the cart 236 to generate logical metadata representing the semantic meaning of datasets 230. For example, the metadata discovery engine 212 may determine that dataset 230a includes information about customers, with the field "cust_fnln" including the customer's name and the field "cust_ssn" including the customer's SSN. In some examples, users can approve the meaning associated with dataset 230a and its fields and submit the dataset for input into a data catalog 220 maintained by a system (not shown).

[0056] Metadata 238 discovered by metadata discovery engine 212 is passed to metadata-based processing engine 214. As described above, the discovered metadata 238 may include physical metadata and logical metadata (sometimes referred to as logical item associations) of the selected dataset 230. Metadata-based processing engine 214 uses metadata 238 to generate metadata model 216a of dataset 230. In this example, metadata model 216a includes elliptical nodes 217a, 217b, and 217c representing the physical metadata of dataset 230a, and rectangular nodes 217d, 217e, and 217f representing the logical metadata of dataset 230a. Metadata model 216a also includes edges 218a and 218b representing the physical relationship between dataset 230a (e.g., node 217a) and its fields (e.g., nodes 217b and 217c), edge 218c representing the logical relationship between dataset 230a and its corresponding logical entity (e.g., logical dataset “customer”), and edges 218d and 218e representing the logical relationship between each field in dataset 230a and its corresponding logical entity (e.g., logical data elements “name” and “SSN”).

[0057] The metadata-based processing engine 214 also receives user-specified metadata 240 from the user of the client device 210. In some examples, metadata 240 may be specified relative to physical metadata items or logical metadata items in metadata model 216a. For example, in this example, user-specified metadata 240 indicates that a logical metadata item representing an SSN (e.g., a logical data element SSN) is associated with a PII. Once received, the metadata-based processing engine 214 may incorporate the user-specified metadata 240 into metadata model 216a. For example, the metadata-based processing engine 214 may update metadata model 216a to indicate that node 217f representing an SSN is associated with a PII. This association can be achieved, for example, by including metadata definitions or other indicative identifiers with the SSN logical metadata, or by adding another node linked by an edge to the node representing the SSN to metadata model 216a.

[0058] refer to Figure 2BThe metadata-based processing engine 214 can use metadata model 216a to automatically perform operations on dataset 230 stored in storage system 204. In this example, metadata-driven data transformation 222 associated with data in directory 220 (including dataset 230) specifies that if the data contains a PII, the data should be tokenized to enhance data security. Therefore, when the metadata-based processing engine 214 executes processing specification 224, it uses metadata model 216a (and transformation 222) to determine whether some or all of the datasets in the accessed dataset 230 are associated with a PII, and if so, performs a tokenization operation on that data. For example, in this example, a processing specification in processing specification 224 specifies a data pipeline configured to ingest dataset 230a into storage system 206 (e.g., a data pipeline configured to read dataset 230a from storage system 106 and write the dataset to storage system 104). Therefore, the metadata-based processing engine 214 determines the physical metadata of dataset 230a and identifies the node in metadata model 216a corresponding to the physical metadata of dataset 230a. For example, the metadata-based processing engine 214 can determine that dataset 230a has the name "Cust_data" and can match this name with node 217a in metadata model 216a. The metadata-based processing engine 214 can then traverse metadata model 216a to determine whether dataset 230a contains a PII. In this example, the metadata-based processing engine 214 determines that the field "cust_ssn" of dataset 230a (represented by node 217c) is associated with a logical data element SSN (represented by node 217f) that has already been defined as a PII. Therefore, the metadata-based processing engine 214 tokenizes the data in the "cust_ssn" field of dataset 230a to create a "Cust_data_tokenized" dataset 242 for ingestion into storage system 206.

[0059] In this example, another processing specification in processing specification 224 specifies that a catalog view of dataset 230a is configured to be presented to the user of client device 210 for operation. Metadata-based processing engine 214 traverses metadata model 216a as described above to determine that the field “cust_ssn” of dataset 230a includes a PII. Therefore, metadata-based processing engine 214 tokenizes the data in the “cust_ssn” field before sending catalog view data 244 to client device 210. Once received, client device 210 can present data 244 in graphical user interface 246, which allows the user to view the tokenized data records of dataset 230a and use the tokenized data as a component in subsequent processing.

[0060] In some examples, the metadata-based processing engine 214 can update the metadata model 216a to produce a metadata model 216b that incorporates the metadata of the newly defined “cust_data_tokenized” dataset 242. For example, the metadata-based processing engine 214 can add nodes 217g, 217h, and 217i representing the physical metadata of dataset 242, and edges 218f and 218g representing the relationships between nodes 217g, 217h, and 217i. Since the “cust_ssn” field of dataset 242 has been tokenized and no longer represents an SSN, the metadata-based processing engine 214 can also add a logical node 217j that accurately describes the contents of the tokenized field (e.g., “tokenized SSN”). Edge 218h can then be inserted to indicate the logical relationship between nodes 217i and 217j. The metadata-based processing engine 214 can also copy existing metadata (see, for example, nodes 217h and 217i) from metadata model 216b to populate the metadata of the new dataset 242. Specifically, the metadata-based processing engine 214 can insert edges 218i representing the logical relationship between dataset 242 and its corresponding logical entity (e.g., the logical dataset "customer"), and edges 218j representing the logical relationship between the "cust_fnln" field of dataset 242 and its corresponding logical entity (e.g., the logical data element "name"). By updating the metadata model in this way, the computational resources required for subsequent processing of dataset 242 can be reduced because the model has already been populated with the metadata of dataset 242 when copying some metadata from the metadata 238 discovered for dataset 242 (thus eliminating the need for metadata discovery for dataset 230), and because the model has been updated to reflect that the "cust_ssn" field of dataset 242 has been tokenized (and therefore does not need to be re-tokenized during subsequent processing).

[0061] Figure 3AA system 300 for implementing metadata according to aspects of this disclosure is illustrated. In this example, system 300 includes a metadata management system 302, storage systems 304 and 306, and client devices 308 and 310. Similar to metadata management systems 102 and 202, metadata management system 302 includes a metadata discovery engine 312 and a metadata-based processing engine 314. In this example, metadata discovery engine 312 receives datasets 330a to 330d stored in storage system 304 and processes datasets 330a to 330d to obtain corresponding physical metadata and logical metadata 332. Specifically, metadata discovery engine 312 determines that dataset 330a has the name "Cust_data" and includes the fields "cust_id" (which serves as a primary key, as indicated by the key symbol) and "fname". Metadata discovery engine 312 also determines, based on semantic analysis of dataset 330a and its data, that dataset 330a's "Cust_data" represents information about "customers," the field "cust_id" represents the "customer ID" as part of the broader group "DB identifier," and the field "fname" represents the customer's "name" as part of the broader group "customer identity." Metadata discovery engine 312 also determines that dataset 330a is related to dataset 330b (e.g., through a primary-foreign key relationship), and that dataset 330b has the name "SSN_table" and includes the fields "I95" and "S14." Based on semantic analysis of dataset 330b and its data, metadata discovery engine 312 determines that dataset 330b represents "customer SSN," the field "I95" represents the "customer ID" as part of the group "DB identifier," and the field "S14" represents the customer's "SSN" as part of the group "customer identity." Metadata discovery engine 312 also determines that dataset 330b is related to dataset 330c (e.g., through a primary-foreign key relationship), and that dataset 330c has the name "Loan_data" and includes the fields "cust_ssn" and "amt". Based on semantic analysis of dataset 330c, metadata discovery engine 312 determines that dataset 330c represents "customer loan", the field "cust_ssn" represents the "SSN" of a customer as part of the group "customer identity", and the field "amt" represents the "amount" within the group "loan information". Metadata discovery engine 312 determines that dataset 330c is related to dataset 330d (e.g., through a primary-foreign key relationship), and that dataset 330d has the name "Cust_acct" and the fields "c_id" and "level".The metadata discovery engine 312 also determines, based on semantic analysis of dataset 330d and its data, that dataset 330d represents information about “customer accounts”, with the field “c_id” representing the “customer ID” as part of the group “DB identifier”, and the field “level” representing the customer’s “account level”.

[0062] Metadata discovery engine 312 passes discovered metadata 332 to metadata-based processing engine 314. Metadata-based processing engine 314 uses metadata 332 to generate a metadata model 316 comprising multiple nodes 317 and edges 318. In this example, elliptical nodes 317 represent physical metadata items for datasets 330a to 330d, and rectangular nodes 317 represent logical metadata items for datasets 330a to 330d. In some examples, metadata-based processing engine 314 also uses metadata 332 to add datasets 330a to 330d to data catalog 320. Datasets 330a to 330d may also be associated (e.g., within data catalog 320 or otherwise) with one or more metadata-driven data transformations 322. Metadata-based processing engine 314 also stores or otherwise accesses multiple processing specifications 324. In this example, processing specification 324 includes data pipelines 324a to 324d in the form of a data flow graph, which are configured to access (e.g., read) datasets 330a to 330d from storage system 306 and store (e.g., write) them in storage system 304.

[0063] refer to Figure 3B The metadata-based processing engine 314 can receive metadata specifications 334 from a user on client device 308. Specifically, the user can use client device 308 to access a graphical user interface 336 that includes a visualization 336a of the metadata model 316. The user can interact with the visualization 336a of the metadata model to select metadata items for which metadata will be specified. In this example, the user has selected 336b corresponding to the metadata item “Customer Identity”. After selecting a metadata item, the user can specify metadata for that item. For example, the user can specify a PII category 336c for the metadata item, which is a form of PII specification. In this example, the user has specified a PII category 336c with “Level 2 (Tokenized)” for “Customer Identity”. Once completed, the user can choose to submit the metadata specifications 334 336d to the metadata-based processing engine 314.

[0064] Upon receiving metadata specification 334, the metadata-based processing engine 314 updates model 316. For example, the metadata-based processing engine 314 can update model 316 to indicate that the node 319 representing "Customer Identity" is associated with a Level 2 PII classification. As shown in visualization 322a of the metadata-driven transformation 322, the metadata-based processing engine 314 is configured to tokenize fields associated with a node when the relevant node (e.g., a parent node) has a "Level 2" PII classification. Incorporating metadata specification 334 into metadata model 316 in this way reduces the amount of space required to store metadata model 316. This is because metadata specification 334 is defined only once in model 316 (e.g., by adding an element with specification 334 to an object or data structure representing "Customer Identity," or by adding a single new object or data structure with specification 334 pointing to an object or data structure representing "Customer Identity"), rather than defining the specification at every applicable node (which could total thousands or more definitions). Therefore, only a single data element (or object or data structure) is added to model 316 (as opposed to many data elements), which reduces the model's storage and memory requirements. A similar reduction in storage is also achieved by linking multiple nodes representing physical metadata (e.g., the "cust_ssn" and "S14" nodes) to a single node representing logical metadata (e.g., the "SSN" node), rather than defining and storing separate logical nodes for each physical node.

[0065] refer to Figure 3CThe metadata-based processing engine 314 uses the metadata model 316 to apply metadata-driven data transformation 322 to processing specifications 324, and thus to datasets 330a to 330d. To this end, the metadata-based processing engine 314 can determine the physical metadata of the data item being accessed, and can use the determined physical metadata to identify the node in the metadata model 316 corresponding to that data item. For example, when processing specification 324a is executed, the metadata-based processing engine 314 can identify the node in the metadata model 316 corresponding to the name of dataset 330a (e.g., “Cust_data”) and / or the names of fields of dataset 330a (e.g., “cust_id”, “fname”). In some examples, the identified node is referred to as a child node. The metadata-based processing engine 314 then traverses the metadata model 316 according to the metadata-driven transformation 322 to determine the operation to be performed on dataset 330a (if any). For example, the metadata-based processing engine 314 can traverse the metadata model 316 to identify one or more parent nodes of the identified child nodes, such as nodes corresponding to "Customer," "Customer ID," "DB Identifier," "Name," and "Customer Identity." In this example, the metadata-based processing engine 314 determines, based on the metadata model 316, that the node representing "Customer Identity" associated with the "Level 2" PII classification is the parent node of the node representing "fname" (e.g., through the "Name" node), but not the parent node of the node representing "cust_id." Therefore, the metadata-based processing engine 314 inserts operation 325a into processing specification 324a to tokenize "fname." The metadata-based processing engine 314 can perform a similar process to insert tokenization operations 325b and 325c into processing specifications 324b and 324c. However, no tokenization operation is added to processing specification 324d because for the node corresponding to the dataset “Cust_acct” or its fields “c_id” and “level”, there is no parent node associated with the PII classification of “level 2”.

[0066] After applying metadata-driven transformation 322, the metadata-based processing engine 314 can execute processing specifications to generate datasets 338a to 338d (where applicable) with tokenized PII data. Each dataset in datasets 338a to 338d can be sent to storage system 306 for storage. The metadata-based processing engine 314 can also apply similar processing to generate tokenized catalog view data 340 of dataset 330a (and so on). Catalog view data 340 can be sent to client device 310 for presentation in a graphical user interface 342, which allows the user to view each tokenized data record in the tokenized data records of dataset 330a.

[0067] By utilizing metadata in this way, System 300 enables users to automatically implement operations such as data security operations across the entire data enterprise (e.g., across multiple data items, such as datasets 338a to 338d and other data items) through a single global metadata specification (e.g., metadata specification 334), without requiring users to identify the specific data items to which these operations apply and without requiring users to generate code on a per-data-item basis. Therefore, compared to systems that do not utilize metadata in this way, System 300 provides a more efficient implementation of cross-enterprise data policies, thereby reducing the latency of implementing such policies. Furthermore, by automatically identifying and applying operations based on metadata, System 300 enhances data security (and the implementation of other data policies) by reducing the likelihood that data that should conform to policies will be ignored. Defining operations based on metadata also allows the system to automatically adapt to changes in the underlying data (e.g., changes in data names, data storage locations, data keys, etc.).

[0068] The metadata-based processing engine 314 can also update the metadata model 316 after applying metadata-driven transformations 322, such as... Figure 3D As shown in the diagram. For example, the metadata-based processing engine 314 can update the metadata model 316 to incorporate metadata for the newly defined “cust_data_tokenized” dataset 338a. For example, the metadata-based processing engine 314 can add nodes 317a, 317b, and 317c representing the physical metadata of dataset 338a, and edges 318a and 318b representing the relationships between nodes 317a, 317b, and 317c. Since the “cust_ssn” field of dataset 338a has been tokenized and no longer represents an SSN, the metadata-based processing engine 314 can also add logical nodes 317d and 317e that accurately describe the content of the tokenized field (e.g., “tokenized name” and “tokenized customer identity”). Edges 318c and 318d can then be inserted to indicate the logical relationships between nodes 317c, 317d, and 317e.

[0069] The metadata-based processing engine 314 can also propagate or copy existing metadata in the metadata model 316 to populate the metadata of the new dataset 338a. Specifically, the metadata-based processing engine 314 can insert edges 318e representing the logical relationship between dataset 338a and its corresponding logical entity (e.g., the logical dataset "Customer"), and edges 318f representing the logical relationship between the "c_id" field of dataset 338a and its corresponding logical entity (e.g., the logical data element "Customer ID"). Similar operations can be performed to update the metadata model 316 to incorporate the metadata of datasets 338b to 338d (not shown).

[0070] By updating the metadata model in this way, the computational resources required for subsequent processing of datasets 338a to 338d can be reduced, as the model has already populated the metadata for datasets 338a to 338d, thus avoiding the need for metadata discovery for these datasets. Furthermore, because the model has been updated to reflect that the "cust_ssn" field of dataset 338a (and other fields of other datasets) has been tokenized, the metadata-based processing engine 314 can use the previously tokenized data instead of re-tokenizing it during subsequent processing.

[0071] Figure 4A The example is system 400, which is Figures 3A to 3D A variation of system 300 is shown. In this example, system 400 includes a metadata management system 402, storage systems 404 and 406, and client devices 408 and 410. Similar to metadata management systems 102, 202, and 302, metadata management system 402 includes a metadata discovery engine 412 and a metadata-based processing engine 414. The metadata-based processing engine 414 stores or otherwise accesses multiple processing specifications 424, including processing specification 424a. In this example, processing specification 424a exemplifies a data flow graph that defines a directory view of multiple datasets (e.g., datasets 330a to 330c) in data directory 420 from the perspective of “Cust_data” dataset 330a. In other words, dataset 330a is the root node in the directory view defined by processing specification 424a, where other datasets are connected to dataset 330a based on relationships discovered between datasets 330a to 330c.

[0072] The metadata-based processing engine 414 uses the techniques described herein to automatically transform metadata-driven transformations 322 (in... Figure 3B (As shown in the diagram) is applied to processing specification 424a. Specifically, the metadata-based processing engine 414 uses metadata model 316 (in... Figure 3B (As shown in the diagram) to determine that the fields “fname”, “S14”, and “cust_ssn” should be tokenized, and inserting the tokenization operation 425a into the processing specification 424. After applying the metadata-driven transformation 322, the metadata-based processing engine 414 can execute the processing specification 424a to generate tokenized catalog view data 430. The catalog view data 430 can be sent to the client device 410 for presentation in a graphical user interface 432, which allows the user to view the tokenized data records in the client view. The metadata-based processing engine 414 can also generate a client view dataset 434 (e.g., a wide record of the client view) that is sent to the storage system 406 for storage.

[0073] In some examples, the metadata-based processing engine 414 can receive new metadata-driven transformations 422a (or modifications to existing metadata-driven transformations) from the user of the client device 408, such as... Figure 4B As shown in the diagram. Specifically, a user can use client device 408 to access a graphical user interface 436 for defining a metadata-driven transformation 422a. The user can interact with the user interface 436 to input the name 436a of the metadata-driven transformation 422a, one or more conditions 436b for applying the transformation, and one or more expressions 436c specifying the action to be performed when condition 436b is met. In this example, the user has defined a metadata-driven transformation 422a named "Administrator View," which causes the field to be discarded (e.g., removed to avoid further processing) when the parent node's PII category is equal to "Level 2" and the parent node's PII view permission level is greater than "User." In some examples, the user can also define the items or groups of items (not shown) to which transformation 422a is applied in the data catalog 420. After defining the metadata-driven transformation 422a, the user can optionally submit the metadata-driven transformation 422a 436d to the metadata-based processing engine 414. The metadata-based processing engine 414 can then incorporate the metadata-driven transformations 422a into the transformation set 422 maintained by the system 400, as shown in the visualization 422b of the metadata-driven transformations 422.

[0074] refer to Figure 4C The metadata-based processing engine 414 can receive metadata specifications 438 from the user on the client device 408. Specifically, the user can use the client device 408 to access metadata models 316 (such as...). Figure 3B The graphical user interface 440 of the visualization 440a (shown) is a representation of the metadata model. Users can interact with the visualization 440a to select metadata items for which metadata will be specified. In this example, the user has selected the metadata item 440b corresponding to "Customer Identity". After selecting a metadata item, the user can specify metadata for that item. For example, the user can specify PII viewing permission 440d for the metadata item, which is a form of PII specification. In this example, the user has specified a PII viewing permission 440d with "Administrator Level" for "Customer Identity" (in addition to the previously specified "Level 2 (Tokenized)" PII category 440c). Once complete, the user can choose to submit the metadata specification 438 440e to the metadata-based processing engine 414. Upon receiving the metadata specification 438, the metadata-based processing engine 414 updates model 316 ( Figure 3B(as shown in the diagram) to generate model 416. For example, metadata-based processing engine 414 can update model 416 to indicate that node 419 representing "customer identity" is also associated with "administrator-level" PII viewing permission.

[0075] refer to Figure 4D The metadata-based processing engine 414 can use the metadata model 416 to automatically apply metadata-driven transformation 422, including the newly defined metadata-driven transformation 422a, to the processing specification 424 and the underlying data. In this example, the metadata-based processing engine 414 identifies node 417a corresponding to the “Cust_data” dataset accessed in the processing specification 424a. The metadata-based processing engine 414 then traverses the metadata model 416 as indicated by the directional arrows to identify nodes 417b, 417c corresponding to fields of the “Cust_data” dataset, and nodes 417d to 417h corresponding to the logical parent nodes of the physical nodes. In this example, the metadata-based processing engine 414 determines that the field “fname” corresponding to node 417c should be discarded (e.g., to avoid further processing) because one of its parent nodes (e.g., node 417f, representing “customer identity”) is associated with a “Level 2” PII classification (the first condition of metadata-driven transformation 422a) and also with an “Administrator Level” PII viewing permission greater than a “User Level” viewing permission (the second condition of metadata-driven transformation 422a). The determination to discard the field “fname” is in... Figure 4D The determination is indicated by the "X" in node 417b, and the traversal of the metadata model 416 to make this determination is shown by the dashed edge with a directional arrow. Because the field "fname" (via its parent node 417f) is associated with the PII, the metadata-based processing engine 414 can also determine that the data contained within this field should be based on the "tokenized" metadata-driven transformation 422. Figure 4B (as shown in the diagram) is tokenized. However, the metadata-based processing engine 414 can determine that the tokenized data within the field is inconsistent with the completely discarded field, and can determine the priority of the discard operation based on rules maintained by system 400, such as optimization rules.

[0076] On the other hand, the metadata-based processing engine 414 determines that the field "cust_id" should not be discarded because it does not have any parent node that satisfies the two conditions specified by the metadata-driven transformation 422a. Similarly, the metadata-based processing engine 414 determines that the data within the field "cust_id" should not be tokenized because the data does not have any parent node with a "Level 2" PII classification. The determination of the field "cust_id" is to retain (and not tokenize) it. Figure 4DThe determination is indicated by the checkmark and thick outline of node 417c, and by traversing metadata model 416 to make this determination, indicated by the thick edge with directional arrows. Similar analysis of metadata model 416 leads to the determination to retain the fields “I95” and “amt”, and to discard the fields “cust_ssn” and “S14”. Note that the nodes for the dataset “Cust_acct” and related metadata are diluted in this example because they are irrelevant to processing specification 424a.

[0077] The metadata-based processing engine 414 can update the processing specification 424a using information determined by traversing the metadata model 416. For example, since the metadata-based processing engine 414 determines, based on metadata-driven transformation 422a, that the fields “fname”, “S14”, and “cust_ssn” should be discarded, it can update the processing specification to remove or modify operations that access those fields. In some examples, after applying these modifications, the metadata-based processing engine 414 can optimize the processing specification 424a to remove redundant or unnecessary operations. For example, in this example, the metadata-based processing engine 414 determines that the “read Cust_data” operation 425b and the join operation 425c are no longer necessary and can optimize these operations as a result (as indicated by the “X” for the corresponding operations).

[0078] After applying metadata-driven transformation 422, the metadata-based processing engine 414 can execute processing specification 424a to generate tokenized catalog view data 442. Catalog view data 442 can be sent to client device 410 for presentation in a graphical user interface 444, which allows the user to view tokenized data records in a client view. The metadata-based processing engine 414 can also generate a client view dataset 446 (e.g., a wide record of the client view) that is sent to storage system 406 for storage. As shown in view data 442 and dataset 446, the fields “fname”, “S14”, and “cust_ssn” (along with other redundant or unnecessary fields) have been discarded.

[0079] Figure 5A The example is system 500, which is Figures 4A to 4DThis is a variation of system 400 shown. In this example, system 500 includes a metadata management system 502, storage systems 504 and 506, and client devices 508 and 510. Similar to other metadata management systems described herein, metadata management system 502 includes a metadata discovery engine 512 and a metadata-based processing engine 514. In this example, the metadata-based processing engine 514 receives a new metadata-driven transformation 522a from a user on client device 508. Specifically, the user can use client device 508 to access a graphical user interface 530 for defining the metadata-driven transformation 522a. The user can interact with user interface 530 to input a name 530a for the metadata-driven transformation 522a, one or more conditions 530b for applying the transformation, and one or more expressions 530c specifying the operation to be performed when condition 530b is met. In this example, the user has defined a metadata-driven transformation 522a named “Add Field”, which causes a field to be inserted into the dataset when the parent node’s PII classification is equal to “Level 1” or “Level 2”. After defining the metadata-driven transformation 522a, the user can choose to submit the metadata-driven transformation 522a to the metadata-based processing engine 514. The metadata-based processing engine 514 can then incorporate the metadata-driven transformation 522a into a transformation set 522 maintained by the system 500, as shown in the visualization 522b of the metadata-driven transformation 522.

[0080] refer to Figure 5B The metadata-based processing engine 514 can use the metadata model 516 to automatically apply metadata-driven transformations 522 to the processing specification 524 and the underlying data. In this example, the metadata-based processing engine 514 identifies the node 517a corresponding to the “Cust_data” dataset accessed in the processing specification 524a. The metadata-based processing engine 514 then traverses the metadata model 516 as indicated by the directional arrows to identify nodes 517b, 517c corresponding to fields in the “Cust_data” dataset, and nodes 517d to 517h corresponding to the logical parent nodes of the physical nodes. In this example, the metadata-based processing engine 514 determines that the field “fname” corresponding to node 517c has a parent node 517f (representing “customer identity”) associated with the “Level 2” PII classification. Therefore, the metadata-based processing engine 514 determines, based on the metadata-driven transformation 522a, that the PII field should be added to the dataset containing “fname” (e.g., “Cust_data”). Similarly, the metadata-based processing engine 514 also determines that the data within the "fname" field should be based on a transformation driven by "tokenized" metadata (in... Figure 5B(As shown in the image) is tokenized. The determination of adding a PII field and tokenizing the data within the "fname" field is in... Figure 5B The determination is indicated by the checkmark and thick outline of node 517b, and the metadata model 516 is traversed to make this determination by the thick edge indicated by the directional arrow.

[0081] On the other hand, the metadata-based processing engine 514 determines, based on metadata-driven transformation 522a, that the PII field should not be added to the dataset containing the "cust_id" field because the dataset does not have any parent node associated with the PII. For the same reason, the metadata-based processing engine 514 determines that the data within "cust_id" does not need to be tokenized. The determination not to add the PII field and not to tokenize the data within the "cust_id" field is based on… Figure 5B The determination is indicated by the "X" in node 517c, and the traversal of metadata model 516 to make this determination is indicated by the dashed edge with a directional arrow. Similar analysis of metadata model 516 results in the determination to add a PII field and tokenize the data within the "cust_ssn" and "S14" fields, and not to add a PII field and not to tokenize the data within the "I95" and "amt" fields. Note that the nodes for the dataset "Cust_acct" and related metadata are downplayed in this example because they are irrelevant to processing specification 524a.

[0082] The metadata-based processing engine 514 can update the processing specification 524a using information determined by traversing the metadata model 516. For example, since the metadata-based processing engine 514 determines that PII fields should be added to a dataset containing the fields “fname”, “S14”, and “cust_ssn”, and that the data within these fields should be tokenized, the metadata-based processing engine 514 can update the processing specification to include a tokenization operation 525a for tokenizing these fields and an add PII operation 525b for adding PII fields with the value “yes” to each corresponding dataset. In some examples, the metadata-based processing engine 514 can optimize the processing specification 524a to remove redundant or unnecessary operations, such as by adding only a single PII field to the generated dataset (e.g., ...). Figure 5B (as shown in the image).

[0083] After applying the metadata-driven transformation 522, the metadata-based processing engine 514 can execute processing specification 524a to generate tokenized catalog view data 532. Catalog view data 532 can be sent to client device 510 for presentation in a graphical user interface 534, which allows the user to view tokenized data records in a client view. The metadata-based processing engine 514 can also generate a client view dataset 536 (e.g., a wide record of the client view) that is sent to storage system 506 for storage. As shown in view data 532 and dataset 536, a PII field with the value "Yes" has been added.

[0084] Figure 6A The example is system 600, which is Figures 5A to 5B A variation of system 500 is shown. In this example, system 600 includes a metadata management system 602, storage systems 604 and 606, and client devices 608 and 610. Similar to other metadata management systems described herein, metadata management system 602 includes a metadata discovery engine 612 and a metadata-based processing engine 614. In this example, the metadata-based processing engine 614 receives a new metadata-driven transformation 622a from a user on client device 608. Specifically, the user can use client device 608 to access a graphical user interface 630 for defining the metadata-driven transformation 622a. The user can interact with user interface 630 to input a name 630a for the metadata-driven transformation 622a, one or more conditions 630b for applying the transformation, and one or more expressions 630c specifying the operation to be performed when condition 630b is met. In this example, the user has defined a metadata-driven transformation 622a named “PII Viewing Permission”, which filters rows (e.g., data records) from the dataset when the parent node’s PII category is equal to “Level 2” and the customer’s account level is greater than the user’s level (e.g., the level of the user accessing the data). After defining the metadata-driven transformation 622a, the user can optionally submit the metadata-driven transformation 622a 630d to the metadata-based processing engine 614. The metadata-based processing engine 614 can then incorporate the metadata-driven transformation 622a into a transformation set 622 maintained by the system 600, as shown in the visualization 622b of the metadata-driven transformation 622.

[0085] refer to Figure 6BThe metadata-based processing engine 614 can use the metadata model 616 to automatically apply metadata-driven transformations 622 to the processing specification 624 and the underlying data. In this example, a user with a non-administrator user level has requested access to the customer view defined by the processing specification 624a. Because the metadata-driven transformation 622a requires a runtime comparison of the user's level with the customer's account level to determine whether to filter rows, the metadata-based processing engine 614 adds a read operation 625a for reading the "Cust_acct" dataset and a join operation 625b for joining the "Cust_acct" dataset to the "Loan_data" dataset. In some examples, the metadata-based processing engine 614 can use metadata included in the metadata model 616 (or data catalog) to determine how to access the "Cust_acct" dataset and join it to an existing dataset (e.g., the "Loan_data" dataset). The metadata-based processing engine 614 can also insert a filter row operation 625c when the condition "Cust_acct.level > user level" is met. The metadata-based processing engine 614 also adds a tokenization operation 625d to tokenize data according to a transformation 622 driven by “tokenization” metadata, as described herein.

[0086] After applying the metadata-driven transformation 622, the metadata-based processing engine 614 can execute processing specification 624a to generate catalog view data 632. Catalog view data 632 can be sent to client device 610 for presentation in a graphical user interface 634, which allows the user to view data records in the client view when filter conditions are met. The metadata-based processing engine 614 can also generate a client view dataset 636 (e.g., wide records of the client view) that is sent to storage system 606 for storage. As shown in dataset 636, rows are filtered out when filter conditions are met (as indicated by rows that run through the data). Note that although the metadata-based processing engine 614 has added operation 625a to access the “Cust_acct” dataset for filtering purposes, data associated with the “Cust_acct” dataset is not output according to processing specification 624a.

[0087] Figure 7A The example is system 700, which is Figures 6A to 6B A variation of system 600 is shown. In this example, system 700 includes a metadata management system 702, storage systems 704 and 706, and client devices 708 and 710. Similar to other metadata management systems described herein, metadata management system 702 includes a metadata discovery engine 712 and a metadata-based processing engine 714.

[0088] In this example, metadata discovery engine 712 receives a new dataset 730 from storage system 704. Metadata discovery engine 712 processes the dataset as described herein to discover or otherwise obtain metadata 732 of the new dataset 730. Specifically, metadata discovery engine 712 determines that dataset 730 has the name “Cust_DOB” and includes the fields “I95” (which serves as a primary key, as indicated by the key symbol) and “D55”. Metadata discovery engine 712 also determines, based on semantic analysis of dataset 730 and its data, that dataset 730 “Cust_DOB” represents information about “customer DOB”, the field “I95” represents “customer ID” as part of the group “DB identifier”, and the field “D55” represents customer “DOB” as part of the group “customer identity”. Metadata discovery engine 712 also determines that dataset 730 (e.g., through a primary-foreign key relationship) is related to dataset 330a.

[0089] The metadata discovery engine 712 passes the discovered metadata 732 to the metadata-based processing engine 714. The metadata-based processing engine 714 uses the metadata 732 to update the metadata model 716. Specifically, the metadata-based processing engine 714 can add nodes representing items of the physical metadata and logical metadata 732 of the dataset 730, and add edges representing the relationships between nodes. These additions... Figure 7A The metadata model 716 is shown in bold.

[0090] refer to Figure 7B The metadata-based processing engine 714 uses the metadata model 716 to update the processing specification 724. Specifically, the metadata-based processing engine 714 uses the metadata model 716 to update the processing specification 724a, which can specify the generation of wide data records including all join datasets associated with "Cust_data". In this example, the metadata-based processing engine 714 uses metadata in the metadata model 716 (e.g., metadata describing the access parameters of the datasets) to update the processing specification 724a to include an access operation 725a configured to read the new dataset 730. The metadata-based processing engine 714 also uses metadata in the metadata model 716 (e.g., metadata describing primary-foreign key relationships) to include a join operation 725b configured to join dataset 730 to dataset "Cust_data". The metadata-based processing engine 714 can also use the metadata model 716 to apply a metadata-driven transformation 722 to the processing specification 724a, which now includes the new dataset 730. By utilizing metadata in this way, system 700 can automatically adjust data processing to take into account changes in the underlying data (including the addition of new data, such as dataset 730), without requiring users to redefine or recode the underlying specifications.

[0091] Figure 8 A flowchart illustrating an example process 800 for automating the identification of operations and their application to data using metadata is provided. Process 800 can be implemented by one or more of the systems and components described herein (e.g., a metadata management system or its components, such as a metadata discovery engine, a metadata-based processing engine, etc.), including those configured to implement the reference. Figure 1 One or more computing systems to the technology described in Figure 7.

[0092] The operation of process 800 includes identifying 802 a given data item and its physical metadata. In some examples, identifying a given data item includes identifying the given data item accessed according to the processing specification. After identifying the given data item and its physical metadata, 804 the metadata model is accessed. Generally, the metadata model may include parent and child nodes connected by edges, where the parent node specifies logical metadata and the child node specifies physical metadata representing the data item, and where the edges specify the relationships between the nodes. In some examples, the metadata model includes a first data structure corresponding to the parent node, which includes logical metadata and at least a first and a second pointer, where the first pointer points to a second data structure corresponding to a child node representing the physical metadata of the given data item, and the second pointer points to a third data structure corresponding to another child node representing physical metadata of a different data item than the given data item.

[0093] In the metadata model, the identifier 806 represents a child node that represents the physical metadata of a given data item. In some examples, identifying a child node that represents the physical metadata of a given data item includes matching the physical metadata of the given data item with the physical metadata represented by the child node in the metadata model.

[0094] Traverse one or more edges in the metadata model to identify one or more parent nodes of a child node. Based on the logical metadata associated with the identified parent nodes, determine one or more operations to be performed on a given data item. In some examples, the metadata associated with the parent nodes (e.g., logical metadata) includes metadata received from a user through interaction with a metadata management system. For example, a user may access a graphical user interface that includes a visualization of the metadata model and interact with the metadata model to select metadata items associated with one or more nodes (e.g., parent nodes) and specify metadata for those nodes. For example, the metadata received from the user may be a Personally Identifiable Information (PII) specification, which is specified by the user through interaction with a visualization of the metadata model to select metadata items associated with one or more parent nodes for which a PII specification will be specified in the model. In some examples, after receiving a metadata specification (e.g., a PII specification), the metadata model may be updated to associate logical metadata with the parent nodes, such as indicating in the model that one or more parent nodes associated with the selected metadata item are associated with a PII specification. In some examples, the one or more operations applied to a given data item to transform the given data item include tokenizing one or more fields of the given data item based on, for example, the PII specification.

[0095] In some examples, one or more metadata transformations are accessed to determine one or more operations to be performed on a given data item, wherein each metadata transformation specifies at least one operation to be performed on the data and at least one condition for performing that at least one operation. The metadata transformations can then be used in conjunction with metadata to determine the one or more operations to be performed on the given data item. For example, a metadata transformation can be selected from the one or more metadata transformations, and it can be determined whether the logical metadata associated with the one or more parent nodes satisfies at least one condition of the selected metadata transformation. When it is determined that the logical metadata associated with the one or more parent nodes satisfies the at least one condition of the selected metadata transformation, it can be determined that the one or more operations to be performed on the given data item include at least one operation specified by the selected metadata transformation.

[0096] The one or more operations are applied to a given data item to transform it. Generally, applying one or more operations to a given data item may include transforming the data item, discarding the data item or data fields associated with it (e.g., to prevent further processing by a computer program), adding data fields to the data item or a dataset associated with it, or filtering data records associated with it, etc. For example, performing one or more operations on a given data item includes tokenization, and applying one or more operations to a given data item to transform it includes applying a tokenization operation to the given data item to tokenize one or more fields of the given data item. In some examples, applying one or more operations to a given data item includes inserting the one or more operations into a processing specification and executing the processing specification to apply the one or more operations to the given data item. In some examples, the processing specification is a specification for a data flow graph, and applying one or more operations to a given data item includes executing the data flow graph, where executing the data flow graph applies one or more operations to the given data item. After the one or more operations are applied to the data item, the data item may be stored 814 (e.g., in memory or another hardware storage device), displayed to a user, or both.

[0097] In some examples, the metadata model is updated based on the transformed data item. For instance, one or more nodes representing the metadata of the transformed data item may be added to the metadata model, where the added nodes represent the metadata of the transformed data item. Some of the added nodes may be copies of nodes that already existed in the metadata model before the update. As another example, one or more edges may be added to the metadata model to propagate existing metadata to the transformed data item. For instance, one or more edges may be added to existing nodes in the metadata model to propagate or connect existing nodes in the metadata model to the added nodes of the transformed data item. The updated metadata model can be output for further processing of the data item.

[0098] In some examples, the operation of process 800 may further include identifying physical metadata of the transformed data item; identifying child nodes representing the physical metadata of the transformed data item in the updated metadata model; traversing one or more edges in the updated metadata model to identify one or more parent nodes representing the child nodes of the physical metadata of the transformed data item; and determining from the logical metadata associated with the one or more parent nodes identified by traversing the one or more edges in the updated metadata model that the one or more operations performed on a given data item will not be performed on the transformed data item. Determining that the one or more operations performed on a given data item will not be performed on the transformed data item may include determining that the logical metadata associated with the one or more parent nodes identified by traversing one or more edges in the updated metadata model is different from the logical metadata associated with the one or more parent nodes identified by traversing one or more edges in the metadata model.

[0099] In some examples, the operation of process 800 may further include identifying physical metadata of the data item being transformed; identifying child nodes representing the physical metadata of the data item being transformed in the updated metadata model; traversing one or more edges in the updated metadata model to identify one or more parent nodes representing the child nodes of the physical metadata of the data item being transformed; determining one or more second operations to be performed on the data item being transformed from logical metadata associated with the one or more parent nodes identified by traversing the one or more edges in the updated metadata model; applying the one or more second operations to the data item being transformed to further transform the data item being transformed; and storing the further transformed data item.

[0100] In some examples, the operation of process 800 may further include accessing a plurality of data items stored in a hardware storage device; identifying physical metadata and logical metadata corresponding to each of the plurality of data items; generating a metadata model based on the physical metadata and logical metadata identified for each of the plurality of data items; and providing access to the metadata model for a first application and a second application. The first application and / or the second application may be a data flow graph or other computer program executed on, for example, a metadata management system or client device and other data processing systems. At least one of the first application or the second application may access the metadata model, and each of the first application and the second application may be configured to: access a given data item to identify the physical metadata of the given data item; identify child nodes in the metadata model that represent the identified physical metadata of the given data item; traverse one or more edges in the metadata model to identify the one or more parent nodes of the identified child nodes; determine at least one operation to be performed on the given data item from the logical metadata associated with the one or more identified parent nodes; apply the at least one operation to the given data item to transform the given data item; and store the transformed data item.

[0101] Specific implementations of the subject matter and operations described in this specification, including data acquisition systems and their components, may be implemented in digital electronic circuits, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination thereof. Specific implementations of the subject matter described in this specification may be implemented as one or more computer programs (also referred to as data processing programs) (i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by a data processing device or for controlling the operation of said data processing device). The computer storage medium may be, or may be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof. The computer storage medium may also be, or may be included in one or more separate physical components or media (e.g., multiple CDs, discs, or other storage devices). The subject matter may be implemented on computer program instructions stored on a non-transitory computer storage medium.

[0102] The operations described in this specification can be implemented as operations performed by a data processing system or apparatus on data stored on one or more computer-readable storage devices or received from other sources. The term "data processing system" encompasses all kinds of apparatus, devices, and machines for processing data, including, for example, programmable processors, computers, systems-on-a-chip, or a combination of the foregoing. The system may include special-purpose logic circuitry (e.g., FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits)). In addition to hardware, the system may also include code that provides an execution environment for associated computer programs (e.g., code constituting processor firmware, protocol stacks, database governance systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof). The system and execution environment can implement various computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0103] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, objects, or other units suitable for a computing environment. A computer program may, but is not required to, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinating files (e.g., a file storing one or more modules, subroutines, or portions of code). Computer programs can be deployed to execute on one computer or on multiple computers located at a site or distributed across multiple sites and interconnected via a communication network.

[0104] The processes and logic flows described in this specification can be executed by one or more programmable processors that execute one or more computer programs to perform actions by manipulating input data and generating outputs. These processes and logic flows can also be executed by special-purpose logic circuits (such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits)), and the devices can also be implemented as special-purpose logic circuits.

[0105] Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or operatively coupled thereto to receive data or transfer data to or both; however, a computer does not need to have such devices. Furthermore, a computer may be embedded in another device (e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive)). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM discs. The processor and memory can be supplemented or incorporated into dedicated logic circuitry.

[0106] Specific implementations of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with specific implementations of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. Components of the system can be interconnected via any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., autonomous peer-to-peer networks).

[0107] A computing system may include users and servers. Users and servers are typically geographically separated and usually interact via a communication network. The user-server relationship arises from computer programs running on the respective computers, and a user-server relationship exists between them. In some specific implementations, the server transmits data (e.g., HTML pages) to the user device (e.g., for the purpose of displaying data to a user interacting with the user device and receiving user input from that user). Data generated at the user device (e.g., the result of user interaction) may be received at the server from the user device.

[0108] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any particular implementation or potentially claimed content, but rather as descriptions of features specific to a particular implementation. Certain features described in this specification within the context of individual implementations may also be implemented in combination within a single implementation. Conversely, individual features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations. Furthermore, although the features described above may function in certain combinations, even initially claimed accordingly, in some cases one or more features of the claimed combination may be removed from this combination, and the claimed combination may be for a sub-combination or a variation thereof.

[0109] Similarly, although the accompanying drawings show the operations in a specific order, this should not be construed as meaning that such operations must be performed in the specific or sequential order shown, or that performing all shown operations must yield the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components in the above embodiments should not be interpreted as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0110] All other specific embodiments are within the scope of the following claims.

Claims

1. A method implemented by a metadata management system for using a metadata model to identify which one or more operations to be performed when processing one or more data items, wherein the metadata model includes one or more parent nodes and one or more child nodes, wherein the one or more operations are defined relative to the one or more parent nodes in the metadata model and applied to data represented by the one or more child nodes in the metadata model, wherein the one or more parent nodes specify logical metadata, and wherein the one or more child nodes specify physical metadata representing the one or more data items, the method comprising: Access a given data item and its physical metadata from memory using one or more processors; Access the metadata model from memory, the metadata model comprising nodes and specifying one or more relationships between at least two nodes; The one or more processors are used to access child nodes of the metadata model from memory, the child nodes being associated with the physical metadata of the given data item; Based on the one or more relationships, the one or more processors access one or more parent nodes of the child node from memory, wherein the one or more parent nodes specify logical metadata including information related to the physical metadata of the child node; The processors access data specifying one or more operations to be performed on a given data item from memory and from logical metadata that includes information related to the physical metadata of the child nodes and associated with the identified parent nodes. as well as The transformed data items are stored in the memory. The transformed data items are generated by applying one or more operations specified by the data to one or more values ​​of a field of the given data item or one or more fields of the given data item to transform the given data item, wherein the transformed data items can be accessed for retrieval.

2. The method according to claim 1, further comprising: The child node representing the physical metadata of the given data item is identified by matching the physical metadata of the given data item with the physical metadata represented by the child node in the metadata model.

3. The method according to claim 1, further comprising: Access one or more metadata transformations to determine the one or more operations to be performed on the given data item, each metadata transformation specifying at least one operation to be performed on the data and at least one condition for performing the at least one operation.

4. The method according to claim 3, further comprising: The one or more operations to be performed on the given data item are determined by the following: Select a metadata transformation from the one or more metadata transformations; Determine whether the logical metadata associated with the one or more parent nodes satisfies at least one of the conditions of the selected metadata transformation; as well as In response to determining that the logical metadata associated with the one or more parent nodes satisfies at least one condition of the selected metadata transformation, it is determined that the one or more operations to be performed on the given data item include at least one operation specified by the selected metadata transformation.

5. The method according to claim 1, further comprising: The method further includes identifying the given data item by identifying the given data item accessed according to the processing specification, and applying the one or more operations to the given data item by: Insert the data specifying the one or more operations into the processing specification; and The processing specification is executed to apply the one or more operations to the given data item.

6. The method of claim 5, wherein the processing specification includes a specification for a data flow graph, and wherein applying the one or more operations to the given data item includes executing the data flow graph, wherein executing the data flow graph applies the one or more operations to the given data item.

7. The method according to claim 1, further comprising: The metadata model is updated based on the transformed data items.

8. The method of claim 7, wherein updating the metadata model comprises adding one or more edges to existing nodes in the metadata model to connect the existing nodes in the metadata model to the added nodes.

9. The method according to claim 7, further comprising: Physical metadata that identifies the transformed data item; In the updated metadata model, identify the child nodes representing the physical metadata of the transformed data items; Traverse one or more edges in the updated metadata model to identify one or more parent nodes of the child nodes representing the physical metadata of the transformed data item; as well as From the logical metadata associated with the one or more parent nodes identified by traversing the one or more edges in the updated metadata model, it is determined that the one or more operations performed on the given data item will not be performed on the transformed data item.

10. The method of claim 9, further comprising determining that the one or more operations performed on the given data item will not be performed on the transformed data item, including: The logical metadata associated with the one or more parent nodes identified by traversing one or more edges in the updated metadata model is determined to be different from the logical metadata associated with the one or more parent nodes identified by traversing one or more edges in the metadata model.

11. The method of claim 7, wherein updating the metadata model comprises adding one or more nodes to the metadata model, the added one or more nodes representing metadata of the transformed data item.

12. The method of claim 11, wherein some of the added nodes are copies of some of the nodes that already existed in the metadata model prior to the update.

13. The method according to claim 12, further comprising: Output the updated metadata model for further processing of data items.

14. The method according to claim 12, further comprising: Physical metadata that identifies the transformed data item; In the updated metadata model, identify the child nodes representing the physical metadata of the transformed data items; Traverse one or more edges in the updated metadata model to identify one or more parent nodes of the child nodes representing the physical metadata of the transformed data item; From the logical metadata associated with the one or more parent nodes identified by traversing the one or more edges in the updated metadata model, determine one or more second operations to be performed on the transformed data item; The metadata management system applies the one or more second operations to the transformed data item to further transform the transformed data item; and The data items for further transformation are stored in memory.

15. The method according to claim 1, further comprising: The one or more operations are applied to the given data item by discarding the data fields associated with the data item to prevent further processing by the computer program.

16. The method according to claim 1, further comprising: The one or more operations are applied to the given data item by adding data fields to the data item or the dataset associated with the data item.

17. The method according to claim 1, further comprising: The one or more operations are applied to the given data item by filtering the data records associated with the data item.

18. The method of claim 1, wherein the one or more operations performed on the given data item include a tokenization operation, and wherein the method further comprises: The tokenization operation is applied to the given data item to tokenize one or more fields of the given data item.

19. The method of claim 1, wherein the logical metadata associated with the one or more parent nodes includes metadata received from a user through interaction with the metadata management system.

20. The method of claim 19, wherein the metadata received from the user is a Personally Identifiable Information (PII) specification specified by the user through a graphical user interface, the graphical user interface including a visualization of the metadata model, and wherein the PII specification is specified by the user interacting with the visualization of the metadata model to select a metadata item associated with one or more parent nodes in the model for which the PII specification will be specified, and the method further includes: Upon receiving the PII specification, the metadata model is updated to associate the logical metadata with the one or more parent nodes, including indicating in the metadata model that the one or more parent nodes associated with the selected metadata item are associated with the PII specification, wherein the one or more operations applied to the given data item to transform the given data item include tokenizing one or more fields of the given data item.

21. The method according to claim 1, further comprising: Access multiple data items stored in a hardware storage device; For each of the plurality of data items, identify the physical metadata and logical metadata corresponding to that data item; The metadata model is generated based on the physical metadata and the logical metadata that identify each of the plurality of data items; Provide access to the metadata model for both the first and second applications; The metadata model is accessed by at least one of the first application and the second application, wherein each of the first application and the second application is configured to: Access the given data item to identify the physical metadata of the given data item; In the metadata model, identify the child node that represents the identified physical metadata of the given data item; Traverse one or more edges in the metadata model to identify the one or more parent nodes of the identified child node; Determine at least one operation to be performed on the given data item from the logical metadata associated with the identified one or more parent nodes; Apply the at least one operation to the given data item to transform the given data item; as well as Store the transformed data items.

22. The method of claim 1, wherein the metadata model includes a first data structure corresponding to the parent node, the data structure including the logical metadata and at least a first pointer and a second pointer, wherein the first pointer points to a second data structure corresponding to a child node representing the physical metadata of the given data item, and the second pointer points to a third data structure corresponding to another child node representing physical metadata of another data item different from the given data item.

23. The method according to claim 1, further comprising: This enables the rendering of one or more visualizations of the transformed data items on the display of a display device.

24. A metadata management system, the system comprising: At least one processor; and The memory stores instructions that can be executed by the at least one processor to perform operations including: Access a given data item and its physical metadata from memory; Accessing a metadata model from memory, the metadata model comprising nodes and specifying one or more relationships between at least two nodes; Accessing child nodes of the metadata model from memory, the child nodes being associated with the physical metadata of the given data item; Based on the one or more relationships, access one or more parent nodes of the child node from memory, wherein the one or more parent nodes specify logical metadata including information related to the physical metadata of the child node; Access data specifying one or more operations to be performed on the given data item from memory and from logical metadata that includes information related to the physical metadata of the child nodes and associated with the identified one or more parent nodes; as well as The transformed data items are stored in the memory. The transformed data items are generated by applying one or more operations specified by the data to one or more values ​​of a field of the given data item or one or more fields of the given data item to transform the given data item, wherein the transformed data items can be accessed for retrieval.

25. The system of claim 24, wherein the operation further comprises: Access one or more metadata transformations to determine the one or more operations to be performed on the given data item, each metadata transformation specifying at least one operation to be performed on the data and at least one condition for performing the at least one operation.

26. The system of claim 24, wherein the operation further comprises: The metadata model is updated based on the transformed data items.

27. The system of claim 26, wherein updating the metadata model comprises adding one or more nodes to the metadata model, the added one or more nodes representing metadata of the transformed data item.

28. The system of claim 24, wherein the one or more operations performed on the given data item include a tokenization operation, and wherein the operations further include: The tokenization operation is applied to the given data item to tokenize one or more fields of the given data item.

29. The system of claim 24, wherein the logical metadata associated with the one or more parent nodes includes metadata received from a user through interaction with the metadata management system.

30. A non-transitory computer-readable medium storing instructions executable by at least one processor to perform operations including: Access a given data item and its physical metadata from memory; Accessing a metadata model from memory, the metadata model comprising nodes and specifying one or more relationships between at least two nodes; Accessing child nodes of the metadata model from memory, the child nodes being associated with the physical metadata of the given data item; Based on the one or more relationships, access one or more parent nodes of the child node from memory, wherein the one or more parent nodes specify logical metadata including information related to the physical metadata of the child node; Access data specifying one or more operations to be performed on the given data item from memory and from logical metadata that includes information related to the physical metadata of the child nodes and associated with the identified one or more parent nodes; as well as The transformed data items are stored in the memory. The transformed data items are generated by applying one or more operations specified by the data to one or more values ​​of a field of the given data item or one or more fields of the given data item to transform the given data item, wherein the transformed data items can be accessed for retrieval.

31. The non-transitory computer-readable medium of claim 30, wherein the operation further comprises: Access one or more metadata transformations to determine the one or more operations to be performed on the given data item, each metadata transformation specifying at least one operation to be performed on the data and at least one condition for performing the at least one operation.

32. The non-transitory computer-readable medium of claim 30, wherein the operation further comprises: The metadata model is updated based on the transformed data items.

33. The non-transitory computer-readable medium of claim 32, wherein updating the metadata model comprises adding one or more nodes to the metadata model, the added one or more nodes representing metadata of the transformed data item.

34. The non-transitory computer-readable medium of claim 30, wherein the one or more operations performed on the given data item include a tokenization operation, and wherein the operations further include: The tokenization operation is applied to the given data item to tokenize one or more fields of the given data item.

35. The non-transitory computer-readable medium of claim 30, wherein the logical metadata associated with the one or more parent nodes includes metadata received from a user through interaction with a metadata management system.

36. A computer program product comprising computer instructions that, when executed by one or more processors, implement the steps of the method of any one of claims 1 to 23.

Images