Conversion of data lineages

Abstract

The present disclosure relates to a computer-implemented method for conversion of a first data lineage to a second data lineage, the method comprising: obtaining a first data lineage specifying relationships among physical components of a plurality of physical components; receiving an identification of a portion of the first data lineage; generating a second data lineage from the identified portion of the first data lineage, the second data lineage specifying relationships among second components of a plurality of second components, wherein the second components of the plurality of second components are associated with at least some of the physical components of the identified portion of the first data lineage. A corresponding computer-readable medium, a corresponding a data processing system, and a corresponding computer program are also described.

Description

[0001] CONVERSION OF DATA LINEAGES

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to a computer-implemented method for conversion of a first data lineage to a second data lineage, as well as to a corresponding computer-readable medium, a corresponding data processing apparatus, and a corresponding computer program.

[0004] BACKGROUND

[0005] Different environments, such as different groups, organizations or systems, may have different standards for tracking the origin of data. When exchanging data between two environments, the lineage of the data produced in one of the environments is often lost when that data is imported into the other environment.

[0006] Conventional approaches to the maintenance of the lineage of data across environments are not reliable.

[0007] SUMMARY

[0008] It is thus an objective of the present disclosure to provide means for a more reliable maintenance of a data lineage.

[0009] This objective is achieved by a computer-implemented method as defined by independent claim 1, by a computer-readable medium as defined by independent claim 20, by a data processing system as defined by independent claim 21, and by a computer program as defined by independent claim 22. Some of the preferred embodiments are described in the dependent claims, in the description and in the figures.

[0010] The inventors have recognized that conventional approaches to the maintenance of the lineage of data across source and target environments (which may be defined by a group of people, an organization, or a computing system or network etc.) involve manual specification of a target lineage for the target environment that is hoped to be in line with a source data lineage of the source environment. This not only places a burden on the user, but creates the risk that the source and target lineages are inconsistent as stated or become inconsistent over time. For this reason, the inventors have developed techniques for a computer-implemented (automated) generation of a target data lineage with minimal input from the user. The inventors have also recognized that generating the target data lineage from the source data lineage allows for more accurate and reliable generation of the target data lineage by eliminating discrepancies between the target data lineage and the source data lineage. This is especially useful when working on data in source and target environments and / or when establishing data exchange between the source environment having the source data lineage and the target environment having the target data lineage. However, the technologies described herein may also be used for other purposes. Accordingly, according to a general aspect of the present disclosure, a computer-implemented method is provided for conversion of a first data lineage to a second data lineage, comprising: obtaining a first data lineage specifying relationships among physical components of a plurality of physical components; receiving an identification of a portion of the first data lineage; generating a second data lineage from the identified portion of the first data lineage, the second data lineage specifying relationships among second components of a plurality of second components, wherein the second components of the plurality of second components are associated with at least some of the physical components of the identified portion of the first data lineage. Overall, this eliminates the burden of manually specifying the second data lineage and eliminates discrepancies between the second data lineage and the first data lineage, and can be used to accurately and reliably maintain data lineage.

[0011] Generally, each physical component includes a data set and / or a transformation. Each data set may include one or more data elements. Each data set may be a table, wherein each row or column is a data element of the data set.

[0012] According to a preferred aspect of the present disclosure, the identification of the portion of the first data lineage includes an identification of a first physical component of the plurality of physical components that indicates a start point for the second data lineage and an identification of a second physical component of the plurality of physical components that indicates an end point for the second data lineage. This allows to perform an accurate selection of a portion of the first data lineage for the generation of the second data lineage. This also allows that portions of the first data lineage that are not of interest can be excluded when generating the second data lineage, which reduces the consumption of computational resources for the generation of the second data lineage.

[0013] According to a preferred aspect of the present disclosure, the generating of the second data lineage from the identified portion of the first data lineage includes generating the second data lineage by traversing the first data lineage from the start point to the end point or vice versa. The inventors have recognized that the traversing is a particularly accurate and computationally efficient way of generating the second data lineage while reliably keeping the relationships among the components.

[0014] According to a preferred aspect of the present disclosure, the method may include: in response to detecting a change in the first data lineage, updating the generated second data lineage, or generating a new second data lineage, in accordance with the detected change. Preferably, each second component of the plurality of second components is associated by an identifying link with a corresponding physical component of the at least some of the physically components of the portion of the first data lineage, which identifying link may keep proof where a data set originated from. The identifying link allows to effectively eliminate discrepancies between the second data lineage and the first data lineage and to automatically update / re -generate the second data lineage should the underlying first data lineage change.

[0015] According to a preferred aspect of the present disclosure, the graphical representation of the portion of the generated second data lineage includes graphical indicators of second components of the plurality of second components such that the graphical indicators are arranged within the graphical representation of the portion of the generated second data lineage in a manner that is in accordance with the relationships among the plurality of second components. The graphical representation of the portion of the generated second data lineage may include links connecting the graphical indicators of the second components of the plurality of second components, wherein the method may further comprise: receiving a user selection of one of the links; and based on the selected link, generating a third data lineage specifying relationships among those physical components of the plurality of physical components that are associated with the second components whose graphical indicators are connected by the selected link. Accordingly, the data lineage information specified by the first data lineage is directly accessible via the second data lineage even when using the second data lineage instead of the first data lineage. In other words, one can deeply dive into the physical components and their relationship via the second data lineage. The third data lineage may be a portion of the first data lineage, so that one can deeply dive into the first data lineage via the second data lineage.

[0016] According to a preferred aspect of the present disclosure, the method includes repeatedly executing the generation of the second data lineage from the identified portion of the first data lineage; comparing a newly generated second data lineage with a previously generated second data lineage of the repeated execution, including determining whether there is a difference between the newly generated second data lineage and the previously generated second data lineage; and outputting whether there is a difference between the newly generated second data lineage and the previously generated second data lineage. Accordingly, a discrepancy is indicated to a user, as an indication of a potential error condition within the collection of first and second data lineages. In particular, a change in the first data lineage over time can be identified by tracking changes of the second data lineage over time.

[0017] According to a preferred aspect of the present disclosure, the method includes obtaining a fourth data lineage specifying relationships among physical components of a plurality of physical components; receiving an identification of a portion of the fourth data lineage; generating a fifth data lineage from the identified portion of the fourth data lineage, the fifth data lineage specifying relationships among fifth components of a plurality of fifth components, wherein the fifth components of the plurality of fifth components are associated with at least some of the physical components of the identified portion of the fourth data lineage; comparing the fifth data lineage with the second data lineage, including determining whether there is a difference between the fifth data lineage and the second data lineage; and outputing whether there is a difference between the fifth data lineage and the second data lineage. Accordingly, a discrepancy is indicated to a user, as an indication of a potential error condition within the collection of first and second data lineages. In particular, a discrepancy between the first data lineage and the fourth data lineage can be identified by comparing the second data lineage with the fifth data lineage.

[0018] Generally, each second and / or fifth component is a representation of a data set and / or of a transformation of the corresponding physical component. The second and / or fifth component may be associated with a semantic meaning of the corresponding physical component in accordance with a dictionary associating physical components with corresponding semantic meanings.

[0019] According to a preferred aspect of the present disclosure, the first and / or third data lineage is generated at least in part by performing (a) analyzing source code of at least one computer program configured to access at least some of the plurality of physical components to identify one or more physical components accessed by the computer program; and / or (b) analyzing information obtained during runtime of the at least one computer program including analyzing one or more logs generated during runtime of the computer program to identify one or more physical components accessed by the computer program. By these means, the first and / or third data lineage can be generated efficiently and comprehensively.

[0020] According to a preferred aspect of the present disclosure, before the generating of the second and / or fifth data lineage from the identified portion of the first and / or third data lineage, the method includes pre-processing the physical components of the identified portion of the first and / or third data lineage, wherein the generating of the second and / or fifth data lineage from the identified portion of the first and / or third data lineage is executed based on a result of the pre-processing. Preferably, the preprocessing includes: identifying a control flow and / or an internal data lineage associated with one or more of the physical components of the portion of the first and / or third data lineage and excluding the control flow and / or the internal data lineage when generating the second and / or fifth data lineage. Control flow may be represented by the data that do not belong to the first and / or third data lineage but which play a role in the selection process of a data record associated with a physical component: join keys in relational joins, where clause in SQL sentence, and / or conditions in programming languages. The control flow may be the sum of all data participants that will influence the result indirectly. The internal data lineage one intermediate steps within a component. The dropping of the control flow and / or of internal data lineage will reduce the number of interactions and thereby the time and resources needed for the generation of the second data lineage. Preferably, when generating the second and / or fifth data lineage, a filter may be added to drop parts of the first data lineage that are not of interest, to reduce the amount of data to be processed and stored in association with the second and / or fifth data lineage. The second and or fifth data lineage may be a summary of only the relevant parts of the first data lineage from the start point to the end point.

[0021] According to a preferred aspect of the present disclosure, the method includes receiving a user selection of a portion of a generated second data lineage, and view more detailed information about the selected portion, such as number of passes through a component, number of transforms involved up to a generation of a particular data set, number of inputs and / or outputs of a component, or the like.

[0022] According to a preferred aspect of the present disclosure, the plurality of second components of the generated second data lineage may include second components that are manually created in addition to second components that are generated by the computer.

[0023] Computer (storage) media, preferably of non-transitory kind, and data processing apparatuses or computing systems can be provided that correspond to the methods described herein.

[0024] In other words, the inventors have developed techniques that efficiently and reliably maintain data lineage when generating a target data lineage. Means are provided that allow for more accurate and reliable generation of the target data lineage while keeping consumption of computational resources low. Discrepancies between the target data lineage and a source data lineage are eliminated and changes among the source and target data lineages are identified. This is especially useful when establishing data exchange between a (dynamic) source environment having the source data lineage and the target environment having the target data lineage. Moreover, means are provided that allow to deeply drive from the target data lineage back into the physical components associated with the source data lineage.

[0025] Other features and advantages will become apparent from the following description and the above summary is not meant to limit the scope of the invention.

[0026] BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Various aspects and embodiments will be described with reference to the following figures. It should be appreciated that the figures are not necessarily drawn to scale. Items appearing in multiple figures are indicated by the same reference number in all the figures in which they appear.

[0028] Figure 1 illustrates an illustrative graphical representation of an illustrative data lineage, in accordance with some embodiments of the technology described herein.

[0029] Figure 2 illustrates an illustrative graphical representation of an illustrative data lineage, in accordance with some embodiments of the technology described herein. Figure 3 is a block diagram of an illustrative computing environment, in which some embodiments of the technology described herein may operate.

[0030] Figure 4 illustrates a diagram illustrating an association between an example second data lineage 420 and an example first data lineage 400, in accordance with some embodiments of the technology described herein.

[0031] Figure 5 illustrates a graphical user interface graphically illustrating an association between an example second data lineage 520 and an example first data lineage 500, in accordance with some embodiments of the technology described herein.

[0032] Figure 6 illustrates exemplary relationships between data sets, data elements, transforms and transform data elements, in accordance with some embodiments of the technology described herein.

[0033] Figure 7 illustrates exemplary relationships between data sets, data elements, transforms, transform data elements, application and system, in accordance with some embodiments of the technology described herein.

[0034] Figure 8A-8E illustrate an exemplary computer-implemented method of generating a second data lineage from the first data lineage and diving into the first data lineage by generating a third data lineage that is identical to a portion of the first data lineage, in accordance with some embodiments of the technology described herein.

[0035] Figure 9 illustrates a graphical user interface, in accordance with some embodiments of the technology described herein.

[0036] DETAILED DESCRIPTION

[0037] According to the disclosure, a first data lineage is used to generate a second data lineage from the first data lineage.

[0038] Data lineage is a data structure, which may be composed of multiple data structures (e.g. a nested data structure), that describes relationships among components of a plurality of components, wherein each component includes or represents one or more data sets and / or one or more transformations. Each data set may include one or more data elements. In particular, data lineage describes the life cycle of data sets and their data elements to which the transformations are applied when being processed by a data processing system. Data lineage information for a given data set includes an identifier of one or more upstream data sets on which the given data set depends, one or more downstream data sets that depend on the given data set, and one or more transformations that process data to generate the given data set. By a particular component being “downstream” to a given component, we mean that the particular component is directly or indirectly dependent on the given data set. By a particular component being “upstream” to a given component, we mean that the given component is directly or indirectly dependent on the particular component. For example, by a downstream data set depending on an upstream data set, we mean that the processing of (one or more data elements of) the upstream data set by the data processing system directly or indirectly results in the generation of (one or more data elements of) the downstream data set. The generated downstream data set can be a data set that is output from the data processing system (sometimes referred to as an output data set) or can be a data set that is to be processed further by the data processing system (sometimes referred to as an intermediate data set). The upstream data set can be a data set input into the data processing system (sometimes referred to as an input data set or a reference data set) or a data set that has already undergone processing by the data processing system (sometimes referred to as an intermediate data set). A transformation is a data processing operation that is applied to an upstream data set to produce a downstream data set that is provided to a data sink. A data lineage diagram is a graphical depiction of the data lineage including components as well as relationships among the components in a data processing system.

[0039] Figure 1 is an example data lineage diagram 100 illustrating a plurality of components 102-122, wherein the relationships among the components are indicated by arrows. For illustration purposes, the components 102-122 are physical components. Each physical component includes a data set and / or a transformation, wherein each data set may include one or more data elements. The data lineage diagram 100 is illustrating the data lineage for output data 110 generated by a data processing system. In the example of Fig. 1, the data processing system receives two sets of source data 102, 104. The source data can be data sets stored in or received from, for instance, a file such as a flat file, a database such as a relational database or an object database, a queue, or another repository for storing data in a computing system. For instance, a data set may generally be a (database) table, wherein each row or column is a data element of the data set. For example, the source data 102 can be data sets of credit card transactions in the U.S. stored in a file “US_feed.dat.” Each data set can include a value for each of one or more fields, such as attributes defined within a record structure or columns in a database table. The source data 102, 104 can be received and processed in batches, for instance, data from a file or database that is processed hourly, daily, weekly, monthly, quarterly, yearly, or at another interval. The source data 102, 104 can be received as a stream and processed continuously, for instance, buffered by a queue and processed as data is available and system resources allow.

[0040] The data sets and / or data elements may be associated with transformations. The source data 102 is processed by a transform element 106, which operates on the source data 102, for instance, to change the source data 102 in some way. The transform element can be an executable program that can manipulate data, such as a Java program executed within a virtual machine, an executable, a data flow graph, or another type of executable program. For instance, the transform element 106 can be an executable named “TransformA.exe”. In a specific example, the transform elements 106 can be a filter component that filters out unwanted data sets or data elements from the source data 102, such as data sets or data elements having an incorrect format. Further examples of transform elements may also include data base operations such as join, select, rollup, merge and union operations or mathematical calculations or algorithmic transformations. It is understood, that a transform element may, as such, entail any processing of the data in the data sets. The transform element 106 processes the source data 102 in view of reference data 120 to produce intermediate data 112. Reference data is data that is used by a transform element to enable the transform element to process of data. For instance, reference data that enables a mapping operation includes one or more fields having values that correspond to values in one or more fields in the data being processed. The intermediate data 112 can be stored in a file, a database, a queue, or another repository for storing data in a computing system.

[0041] A transform element 108 processes the set of source data 104 in view of reference data 122 to produce intermediate data 114. The intermediate data 114 can be stored in a file, a database, a queue, or another repository for storing data in a computing system.

[0042] The intermediate data 112, 114 are processed together by a transform element 116, which makes use of reference data 118. In an example, the transform element 116 is a mapping operation and the reference data 118 includes data sets that indicate state values and corresponding region values. When the intermediate data 112, 114 are processed by the transform element 116, the value in the state field in each data set in the intermediate data 112, 114 is mapped to the corresponding region as indicated in the reference data 118. In an example, the reference data 118 include data that indicate corporate units and corresponding department identifiers, manager names, and locations. When the intermediate data 112, 114 are processed by the transform element 116, each data set is assigned to a corporate unit based on the mapping enabled by the set of reference data. Reference data 118 can be used to process multiple sets of data and is unchanged by the processing. Reference data 118 can be updated by a user periodically or as needed.

[0043] The transform element 116 outputs the output data 110, which is stored in a file, a database, a queue, or another repository for storing data in a computing system. The output data 110 can be further processed, e.g., by other transform elements in the same data processing system or in a different data processing system, or can be stored for future analysis.

[0044] In the example of Fig. 1, the data lineage of the output data 110 is shown for data lineage components in a single data processing system. In some examples, the data lineage of a set of data can be tracked through multiple data processing systems. For instance, source data can be initially processed by a first data processing system to produce output data X. A second data processing system reads the output data X from the first data processing system and processes the output data X to generate output data Y. The output data Y is processed by a third data processing system to generate output data Z. The data lineage of the output data Z includes the initial source data, the transforms included in each of the three data processing systems, and any reference data used during processing by any of the three data processing systems.

[0045] As illustrated in Figure 1, a user may select (here via a mouse cursor in a graphical user interface) physical components 104 and 110 as start point and end point, respectively, of a portion 124 of the first data lineage 100, which selection identifies the portion 124 to be converted (see arrow in the bottom left comer of Figure 1) to a second data lineage 100A. That is, all physical components of the first lineage 100 between the component 104 according to the start point and the component 110 according to the end point are incorporated in the identified portion 124 and participate at the conversion to the second data lineage 100A. The first data lineage 100 may be hosted in one (computing) environment and the second data lineage may be hosted in another (computing) environment.

[0046] Accordingly, a computer-implemented method for conversion of a first data lineage 100 to a second data lineage 100A in accordance with embodiments of the claimed subject matter comprises: obtaining a first data lineage 100 specifying relationships among physical components of a plurality of physical components 102-122; receiving an identification of aportion 124 ofthe first data lineage 100; generating a second data lineage 100A from the identified portion 124 of the first data lineage 100, the second data lineage 100A specifying relationships (see arrows between the second components in Fig. 1) among second components of a plurality of second components 103, 105, 107, 109, 111, wherein the second components of the plurality of second components 103, 105, 107, 109, 111 are associated (cf. associations 113, 115, 117, 119, 121 in Fig. 1) with at least some of the physical components, here components 104, 108, 114, 116, 110, ofthe identified portion 124 ofthe first data lineage 100. In this way, data lineage is maintained when converting a portion of one data lineage to another data lineage.

[0047] The generation of the second data lineage 100A may include traversing the first data lineage 100 from start to end point, here 104 and 110 respectively, or vice versa. The traversing may incorporate investigation of all interactions between the start and end point. In one aspect the traversing comprises selecting those data elements of the start 104 and end point 110, which are associated with data elements of second components in the plurality of second components 103, 105, 107, 109, 111. The first data lineage 100 is traced such that all target and / or source components of the start and end points, 104 and 110, are found. The traversing may comprise recursively listing identifiers that identify all such source and / or target components. The resulting list may contain the data elements related to the selected data elements in the start and / or end point components and the comprising physical components themselves, the physical components that are transformations of the data elements, the resulting data elements and comprising physical components and the distance from the start or end point. For example, in a first step, when traversing from end point 110 to start point 104, the recursively generated list may contain identifiers relating to a selected data element contained in physical component 110, physical component 110, the source component 116, a data element in physical component 116 that is transformed and the distance between end point 110 and physical component 116. The distance may quantify how many transformations are applied between the target or source component and the start or end point, respectively, i.e. in this case 1. This process is repeated recursively until the process arrives at the start or end point.

[0048] The traversal may further comprise storing a record of listed source-target pairs of components and any coupling transformations coupling the source and target components in a respective source-target pair. When a data lineage contains circular dependencies, traversal of said data lineage can cause infinite looping. In order to circumvent such scenarios, the stored record of listed source-target pairs and coupling transformations can be cross-referenced when traversing the data lineage. For each traversed transformation and its source-target pair it can be checked, whether a record exists corresponding to it so that a loop can be averted.

[0049] Physical components in the first data lineage 100 may be related to applications. Such applications may be configured to perform operations associated with a portion of the physical components in the first data lineage 100. It may be beneficial to restrict the second data lineage to such second components, which are associated with one or more of such applications. Therefore, the generation of the second data lineage may be based on such entries in the recursively generated list of identifiers, which relate to physical components that are linked to one or more predetermined applications as described above. This limits the second data lineage to those components which are relevant for further processing.

[0050] Figure 1 is an example data lineage diagram. In some examples, output data can be generated by a more complex data processing system, such as shown in the example end-to-end data lineage diagram 200A of Fig. 2 for a target element 206A.

[0051] Figure 2 illustrates an example data lineage diagram 200A that is illustrating a plurality of physical components 202A-206A and their relationships (see arrows). Each physical component includes a data set and / or a transformation, wherein each data set may include one or more data elements. In the data lineage diagram 200A, connections or links (see arrows) are shown between data sets 202A and transform elements 204A. Data sets 202A can represent tables, rows within tables, columns in tables, fields in tables or files, or other data. An example of a transform element is an element of an executable that describes how a single output of a data element is produced. The root cause of a potential error in a target component 206A (or in another intermediate component) can be tracked in the data processing system of Fig. 2.

[0052] The information shown in a data lineage diagram, such as the data lineage diagram of Fig. 1 or Fig. 2, illustrates which upstream data sources, data sets, data sinks, or transformations affect a downstream data set. For instance, the data lineage diagram 100 of Fig. 1 reveals that the output data 110 is affected by the source data 102, 104, the reference data 118, 120, 122, and the transform elements 106, 108, 116.

[0053] Understanding the lineage of a downstream data set (such as the output data set 110) can be useful in identifying the root cause of an error in a data set that may occur in the downstream data set. By root cause, we mean an identification of an upstream component that is at least partially a cause of the error in the downstream component. An error in a downstream data set, such as in the output data set 110, can be due to erroneous source data, erroneous reference data, or an error in a transform element in the upstream lineage of the output data set 110, or a combination of any two or more of them. Tracking the quality or status of data lineage components can provide information that can be used to evaluate a possible root cause of errors in data sets.

[0054] By an error in a data set, we mean generally whether the data set has the expected characteristics, such as to avoid an error in data processing. An error in a data set can manifest in a set of data not behaving characteristics as expected, for instance, falling outside of statistical norms, returning a lookup failure in response to a standard query, not satisfying a condition or rule, compliance with which is required for further computation, or another type of behavior. As another example, data in a data set may cause errors in further processing when it is in the wrong format (e.g. incorrect array shape) or has the wrong type (e.g. integer instead of floating point numbers). As described above, errors in a data set may also manifest as missing values or compensatory values (e.g. NaN). A parameter to be evaluated may be a measure for a quality of data or program code associated with the data set. The parameter evaluation or quality of a set of data can be characterized based on a profde of some or all of the data sets in the set of data or based on the value in each of one or more fields of a specific data set or both.

[0055] An error in a set of downstream data (e.g., the output data 110) can be traced to any of various factors in the upstream data lineage of the output data set. One possible cause of erroneous output data can be erroneous source data, erroneous reference data, or erroneous transform logic, or a combination of these factors. For instance, a set of source data may have been corrupted or cut off during transmission, may be the wrong set of data, may have missing data, or may have another problem. A set of reference data may have been exposed to an error in a recent update to the set of reference data, may be corrupted, may be the wrong set of data, or may have another problem. Another possible cause of poor quality output data can be an issue with a transform element in the upstream data lineage of the output data. For instance, if the software implementing a transform element was recently updated to a new version, the transform element may no longer perform the desired processing if, for example, the updated software has an error or has been corrupted. Source data, reference data, and transform elements in the data lineage of the set of output data 110 can be monitored to facilitate preemptive identification of a potential errors like data quality issue that may occur in the set of output data, subsequent tracking of the root cause of an error that occurred in the set of output data, or both.

[0056] Monitoring and analysis of the source data and reference data can help a user or system to diagnose one or more possible causes of erroneous output data. For instance, if a set of erroneous output data is generated, analysis of the source data or reference data in the data lineage of the set of poor quality output data can indicate whether a given set of source data or reference data is itself erroneous and thus a possible contributor to the erroneous output data. Monitoring of the source data and reference data can also preemptively identify erroneous source data or reference data that, if processed, may cause an error like a data quality issue in downstream output data.

[0057] Figure 3 is a block diagram of an illustrative computing environment 300, in which some embodiments of the technology described herein may operate. Computing environment 300 includes data processing system 305, which is configured to operate on data stored in data store 304.

[0058] The data store 304 may include one or multiple storage devices storing data sets in one or more formats of any suitable type. For example, the storage device(s) part of data store 304 may store data sets in form of one or more database tables, spreadsheet files, flat text files, and / or files in any other suitable format (e.g., a native format of a mainframe). The storage device(s) may be of any suitable type and may include one or more servers, one or more database systems, one or more portable storage devices, one or more non-volatile storage devices, one or more volatile storage devices, and / or any other device(s) configured to store data electronically. The data store 304 may include one or more online data streams of data sets in addition to or instead of storage device(s). Accordingly, for example, data processing system 305 may have access to data sets provided over one more data streams in any suitable format.

[0059] In aspects where data store 304 includes multiple storage devices, the storage devices may be co-located in one physical location (e.g., in one building) or distributed across multiple physical locations (e.g., in multiple buildings, in different cities, states, or countries). The storage devices may be configured to communicate with one another using one or more networks such as, for example, network 306 shown in Fig. 3.

[0060] The data sets stored by the storage device(s) may include one or multiple data entities such as one or more files, tables, data in rows and / or columns of tables, spreadsheets, data sets, data records (e.g., credit card transaction records, phone call records, and bank transaction records), fields, variables, messages, and / or reports. The storage device(s) may store thousands, millions, tens of millions, or hundreds of millions of data entities. Each data set may include one or multiple data elements.

[0061] A data element may be any data element stored and / or processed by a data processing system. For example, a data element may be a field in a data record, and the value of the data element may be the value stored in the field of the data record or may be a column or row of a (database) table. As a specific non-limiting example, a data element may be a field storing a caller’s name in a data record storing information about a phone call (which data record may be part of multiple data records about phone calls made by customers of a telecommunication’s company) and the value of the data element may be the value stored in the field. As another example, a data element may be a cell in a table (e.g., a cell occurring at a particular row and column of the table) and the value of the data element may be the value in the cell of the table. As another example, a data element may be a variable (e.g., in a report) and the value of the element may be value of the variable (e.g., in a particular instance of the report). As a specific non-limiting example, a data element may be a variable in a report about a bank loan applicant representing the applicant’s credit score, and the value of the data element may be the numeric value of the credit score (e.g., a numeric value between 300 and 850). The value of the data element representing the applicant’ s credit score may change depending on the data used to generate the report about the bank loan applicant.

[0062] A data element may take on a value of any suitable type. For example, a data element may take on a numeric value, an alphabetic value, a value from a discrete set of options (e.g., a finite set of categories), or any other suitable type of value, as aspects of the technology described herein are not limited in this respect.

[0063] Data processing system 305 may include one or multiple computer programs 309 configured to operate on data in data store 304. The computer programs 309 may be of any suitable type and written in any suitable programming language(s). For example, computer programs 309 may include one or more computer programs written at least in part using the structured query language (SQL) and configured to access data in one or more databases part of data store 304. As another example, data processing system 305 is configured to execute programs in the form of graphs and computer programs 309 may comprise one or more computer programs developed as dataflow graphs. A dataflow graph may include components, termed “nodes” or “vertices,” representing data processing operations to be performed on input data and links between the components representing flows of data. Techniques for executing computations encoded by dataflow graphs are described in U.S. Patent No.: 5,966,072, titled “Executing Computations Expressed as Graphs”. In the example of Fig. 3, data processing system 305 further includes development environment 308 that may be used by a person (e.g., a developer) to develop one or more of computer programs 309 for operating on data in data store 304. For example, user 302 may use computing device 303 to interact with development environment 308 to specify a computer program 309, such as a dataflow graph, and save the computer program as part of computer programs 309. For example, user 302 may use computing device 303 to interact, via network 306, with other users 330 of other computing devices 334. An environment for developing computer programs as data flow graphs is described in U.S. Pat. Pub. No.: 2007 / 0031668, titled “Managing Parameters for Graph-Based Applications”.

[0064] One or more of computer programs 309 may be configured to perform any suitable operations on data in data store 304. For example, one or more of computer programs 309 may be configured to access data from one or more sources, transform the accessed data (e.g., by changing data values, filtering data records, changing data formats, sorting the data, combining data from multiple sources, splitting data into multiple portions, and / or in any other suitable way), calculate one or more new values from accessed data, and / or write the data to one or multiple destinations.

[0065] One or more of computer programs 309 may be configured to perform computations on and / or generate reports from data in data store 309. The computations performed and / or reports generated may be related to one or more quantities relevant to a business. For example, a computer program may be configured to access credit history data for a person and determine a credit score for the person based on the credit history. As another example, a computer program may access telephone call logs of multiple customers of a telephone company and generate a report indicating how many of the customers use more data than allowed for in their data plans. As yet another example, a computer program may access data indicating the types of loans made by a bank and generate a report indicating the overall risk of loans made by the bank. These examples are illustrative and non-limiting, as a computer program may be configured to generate any suitable information (e.g., for any suitable purpose business or technology) from data stored in data store 304.

[0066] In the example of Fig. 3, data processing system 305 may also include a data governance module 310 that supports the performance of various data governance tasks. For example, in the illustrated embodiment, data governance module 310 includes data dictionary module 312, role management module 314, parameter evaluation module 316, data lineage module 318, derived lineage module 320, and lineage association module 322, each of which comprises processor-executable instructions that, when executed, perform functionality supporting the performance of one or more data governance tasks, as described in greater detail below. The data dictionary module 312 may be configured to store information about data in data store 304. That is, data dictionary 312 may be configured to store metadata associated with data in data store 304. For example, data dictionary 312 may store one or more alternative names for data elements in data store 304. In this way, rather than referring to a data element by the name of the variable to which it corresponds (which variable name may have been created by a programmer and is not “user-friendly” in that it does not immediately convey to a user what information the variable represents), the data dictionary may include one or more alternative terms for the data element (e.g. second data elements of the second data lineage) such as, for example, a natural language term or phrase that business people would use to refer to the data element. As a specific example, the data dictionary 312 may store the name “Bank Customer Credit Score” or “Bank Customer FICO Credit Score” as an alternative name for a data element corresponding to a variable named “cstCrdScr,” which stores the value of a FICO credit score for a particular bank customer. As another specific example, the data dictionary 312 may store the name “Order Amount” as language that may be used for referring to the data element corresponding to a field named “order amt.”

[0067] The role management module 314 may manage information indicating which party or parties are responsible for various data elements stored in data store 304. Managing such role information may include storing the role information, allowing one or more users to modify such information (e.g., by removing, adding, or changing parties and / or their responsibilities), and / or displaying the role information.

[0068] The role management module 314 may specify responsible parties for one or more data elements of the first and / or second data lineage. For example, role management module 314 may be configured to manage information used for generating (and may be configured to generate) a graphical interface indicating parties accountable for management of a data element.

[0069] The parameter evaluation module 316 may be configured to evaluate one or more parameters indicating a potential error within a data set and / or its one or more data elements. The parameter may be defined in any suitable way that indicates a potential error within a data set. The parameter may a measure for a quality of data or program code associated with the data set. The parameter may be evaluated using predefined parameter rules, which may define criteria for evaluating the values of data sets and / or their data elements, such as by identifying characteristics (e.g., accuracy, precision, completeness, and validity) of the values according to the criteria. The extent to which the values exhibit these characteristics may thereby produce a measure of data quality for the data set and / or data elements. Aspects of evaluating the quality of data using data quality rules are described in U.S. Pat. Pub. No.: 2014 / 0308357, “Specifying and Applying Rules to Data”. The data lineage module 318 may be configured to generate a data lineage for at least some of the data in data store 304. A data lineage may include information about the lineage of one or physical components including data sets and optionally transformations. For example, a derived data linage, such as the second and fifth data lineages described herein, may include upstream lineage information indicating how the value of an upstream component was obtained and / or downstream lineage information indicating one or more other downstream components that depend on the value of the upstream component.

[0070] The data lineage module 318 may be configured to generate a data lineage, such as the data lineage 100, 200A of Figures 1 and 2, respectively, by analyzing the source code of at least one computer program configured to access (e.g., read, write, and modify) at least some of the plurality of physical components managed by a data processing system. The source code analysis may be used to identify inputs to a computer program (e.g., identify one or more data sets accessed by the computer program), identify one or more transformations applied to the inputs and / or computations performed using the inputs as part of the computer program, and / or identify one or more outputs of the computer program. The computer program may comprise a dataflow graph.

[0071] The data lineage module 318 may be configured to generate a data lineage, such as the data lineage 100, 200A of Figures 1 and 2, respectively, by analyzing information obtained during runtime of the at least one computer program. For example, one or more logs generated during runtime of a computer program may be analyzed to identify inputs to the computer program, one or more transformations applied to the inputs and / or computations performed using the inputs as part of the computer program, and / or one or more outputs of the computer program.

[0072] The data lineage module 318 may be configured to generate a data lineage, such as the data lineage 100, 200A of Figures 1 and 2, respectively, by using one or more data discovery processes. For example, a computer program implementing a data discovery may be configured to identify different data sets containing the same data values and, based on that identification, determine that these data sets are related. For example, the computer program may be configured to determine that a same table of data is stored in multiple different databases and, on that basis, determine that the data elements in these tables are related. It should be appreciated that the data lineage module 318 may be configured to generate a data lineage using any of the above-described ways or any combination of two or more of the abovedescribed or other ways, as aspects of the technology described herein are not limited in this respect.

[0073] The derived lineage module 320 may be configured to facilitate the generation of a derived lineage by a system (e.g., a computer system). The derived lineage module 320 may indicate lineage relationships among components of the derived lineage. The association module 322 may be configured to facilitate the generation of an association between the source data lineage (e.g. first, third and fourth data lineages) and a (target) lineage derived from the source data lineage (e.g. second and fifth data lineages). To this end, association module 322 may generate, for each of one or more components of the derived lineage, an association between the component of the derived lineage and one or more corresponding physical components of the (source) data lineage.

[0074] Coming back to Figure 2, the computer-implemented method according to the disclosure comprises: obtaining a first data lineage 200A specifying relationships among physical components of a plurality of physical components 202A, 204A; receiving an identification of a portion of the first data lineage (see, e.g. the components in the dashed rectangle in Figure 2); generating a second data lineage from the identified portion of the first data lineage, the second data lineage specifying relationships among second components of a plurality of second components, wherein the second components of the plurality of second components are associated with at least some of the physical components of the identified portion of the first data lineage.

[0075] Figure 4 is a diagram illustrating an association between an example second data lineage 420 and an example first data lineage 400, in accordance with some embodiments of the technology described herein. Each of second data lineage 420 and first data lineage 400 may be obtained in any of the ways described herein involving a computer-implemented method, the method comprising: obtaining a first data lineage 400 specifying relationships among physical components of a plurality of physical components 440, 442, 444, 446, 448, and 450; receiving an identification of a portion of the first data lineage; generating a second data lineage 420 from the identified portion of the first data lineage, the second data lineage specifying relationships among second components of a plurality of second components 403, 405, 406, 409, wherein the second components of the plurality of second components are associated (cf. associations or links 452, 454, 456, 458) with at least some of the physical components of the identified portion of the first data lineage.

[0076] The first data lineage 400, may include a plurality of physical components of one or more (physical) data processing systems, wherein each physical component may include a data set and / or transformation. The second, derived data lineage 420 may include a plurality of second components, wherein each second component is associated with and / or represents a corresponding physical component of the first data lineage 400.

[0077] Each of the physical components may include one or more physical data elements 422, 424, 426, 428, 430, 432, 434, 436. Each of the second components may include one or more second data elements 302, 304, 307, 308. First data lineage 400 includes data sets 440, 442, 444, 446, 448, and 450. Each of the data sets may be stored in different systems and / or computing devices. Alternatively two or more (or all) of the data sets may be stored in one system and / or computing device. Examples of data sets are provided herein. Each data set may include one or multiple physical data elements. Data set 440 contains one or more physical data elements including physical data element 422. Data set 442 contains one or more physical data elements including physical data element 424. Data set 444 contains multiple physical data elements including physical data elements 426, 428, and 430. Data set 446 includes one or more physical data elements including physical data element 432. Data set 448 includes one or more physical data elements including physical data element 434. Data set 450 includes one or more physical data elements including physical data element 436.

[0078] It should be appreciated that second and first data lineages may be more complex than the lineages shown in Fig. 4 and, for example, may include many more second data elements, second components, physical data elements and physical components. The examples of lineages shown in Fig. 4 are being used for ease of exposition and not by way of limitation.

[0079] In some embodiments, a first data lineage may include upstream data lineage information for one or more physical data elements, which provides information about how the physical data element(s) were obtained and / or generated. For example, in the illustrative example of Fig. 4, first data lineage 400 includes upstream data lineage information for physical data element 422. As indicated by the shading shown in Fig. 4, physical data element 422 was obtained from physical data element 424, which was obtained from multiple physical data elements including physical data element 426, which was obtained from physical data element 432. Accordingly, physical data element 422 was obtained based, at least in part, on physical data element 432 in data set 446.

[0080] Second data lineage 420 includes second components 403, 405, 407, and 409. A second components may be any suitable container for encapsulating a representation of a corresponding physical component. The second component may be used to present the corresponding physical component to a user or system in terms the user or system can understand or process. For example, a second component may be a report, a spreadsheet, a presentation having one or more slides, a text file, a Word document, and / or a PDF file. In some embodiments, the content in the second component may be generated by a computer, for example, by performing a database query (e.g., a SQL query) and placing the results of the database query into the second component. As a specific non-limiting example, a computer creating a second data lineage may perform a database query and insert a table returned as a result of the query into a spreadsheet file.

[0081] As shown in Fig. 4, second component 403 includes one or more second data elements including second data element 402. Second component 405 includes one or more second data elements including second data element 404. Second component 407 includes one or more second data elements including second data element 406. Second component 409 includes one or more second data elements including second data element 408.

[0082] In some embodiments, a second data lineage may include upstream data lineage information, which provides information about how the second data element(s) were obtained and / or generated, and / or downstream lineage information for one or more second data elements, which provides information indicating which other second data element(s) depend on the second data element(s). For example, in the illustrative example of Fig. 4, user specified lineage 400 includes upstream data lineage information for second data element 402. As shown in Fig. 4, the second data lineage 420 indicates that second data element 402 was obtained from second data element 404, which was obtained from second data element 406, which was obtained from second data element 408.

[0083] As discussed herein, in some embodiments, an association may be generated between a second data lineage and a first data lineage by generating an association or link between one or more physical data elements in the first data lineage and one or more corresponding second data elements in the second data lineage. An illustrative example of such an association is shown in Fig. 4, which shows that: (1) second data element 402 is associated with physical data element 422 via association link 452; (2) second data element 404 is associated with physical data element 424 via association link 454; (3) second data element 406 is associated with physical data element 426 via association link 456; and (4) second data element 408 is associated with physical data element 432 via association link 458. As may be appreciated from the example of Fig. 4, an association between a second data lineage and a first data lineage may comprise information specifying one or more association links between data elements in the lineages. For example, the association shown in Fig. 4 indicates that there is no discrepancy between the second data lineage for the second data element 402 and first data lineage for the physical data element 422, which is associated with the second data element 402. In this example, every physical data element in the upstream first data lineage of physical data element 422 is associated with a corresponding second data element in the upstream second data lineage for the second data element 402. For example, physical data element 432, which is used to obtain physical data element 422, according to the first data lineage 400, is associated with second data element 408, which is used to obtain second data element 402, according to the second data lineage 420.

[0084] Figure 5 is a graphical user interface graphically illustrating an association between an example second data lineage 520 and an example first data lineage 500, both obtained in accordance with some embodiments of the technology described herein. Each of the physically components of the first data lineage 500 includes a data set and / or a transformation. Physical component 530 having one data element is associated by association 515 with second data element 510 of second component 510 of the second data lineage 520. Several physical components, like physical components 550, 560, 570, may be associated with the same application or process 540, as indicated in the first data lineage 500 by associating physically components 550, 560, 570 to the same application or process 540. This is further explained in context of Figures 6 and 7.

[0085] Figures 6 and 7 illustrate exemplary relationships between data sets 601, 701, data elements 602, 702, transforms 603, 703, transform data elements 604, 704, application 706 and system 705.

[0086] As mentioned, a data lineage, such as the first and / or third data lineage, describing relationships among physical components of a plurality of physical components is a data structure. Each physical component includes one or more data sets 601 and / or one or more transformations 603 (cf. Figs. 2 and 6). In particular, the data lineage describes the life cycle of data sets 601, 701 and their data elements 602 to which the transformations 603 are applied when being processed by a data processing system (cf. Figs. 2 and 6). A data element 602 is a unit of data in a data set 601. A data element 602 that is nested (recursive) can contain other data elements 602. The data element 602 can be one of the following subtypes of data elements (the base object type): File field,

[0087] Table column or row, Message field etc. A data element 602 is the most detailed physical element that identifies and describes data within an automated system. For example, in a file, it is a field, in a database table, it is a column. A data set 601 is a named collection of input or output data elements 602 that are used by transforms 603, or that have values produced by transforms. The data set 601 can be one of the following subtypes: File, Table, View, Interface, etc. A data set 601 is a collection of data elements 602. It represents a storage unit in the automated system or an interchange unit between systems.

[0088] As illustrated in Fig. 7, the data set 701 and the transform 703, which is an executable, can be associated to the same application 706, which is executed by a system 705. The transform 703 may be applied to the data set 701 when running the application 705 on the system 706.

[0089] Figures 8A to 8E illustrate an exemplary computer-implemented method of generating a second data lineage from the first data lineage and diving into the first data lineage by generating a third data lineage that is identical to a portion of the first data lineage. Diving into the first data lineage entails generating the third data lineage, which specifies specific relationships between particular physical components, based on a received selection. Thus, a more detailed analysis of the first data lineage is enabled.

[0090] Figure 8A is a graphical user interface graphically illustrating an example first data lineage 800, in accordance with some embodiments of the technology described herein. First data lineage 800 indicates relationships among a plurality of physical components such as physical components 802, 804, 807, 809, 810 and 811. The relationship is indicated by links, such as links 803, 805, 808. The links indicate which upstream component generates output that is an input of a downstream component. For example, physical component 804 is downstream of physical component 802 and depends on physical component 802, because the output of physical component 802 flows via connection indicated by link 803 to input of physical component 804. Both physical components 802, 804 are associated with the same application or process 801. Each of the physical components of the first data lineage 800 includes a data set and / or a transformation and data set may include one or more data elements. Data set 811 includes a collection 812 of data elements. External data, like reference data, may be incorporated into the first data lineage 800, as indicated by Figure 8A. The external data as well as data sets 807, 809, 810 and 811 are all associated with same application 806.

[0091] As derivable from Figures 8A to 8C the computer-implemented method is for conversion of the first data lineage 800 to a second data lineage 820 and comprises: obtaining a first data lineage 800 specifying relationships among physical components of a plurality of physical components 802, 804, 807, 809, 810 and 811; receiving an identification of a portion 813 - 817 of the first data lineage; generating a second data lineage 820 from the identified portion of the first data lineage, the second data lineage specifying relationships among second components of a plurality of second components 822, 824, 827 wherein the second components of the plurality of second components are associated with at least some of the physical components of the identified portion of the first data lineage.

[0092] As indicated by exemplary components 813 to 817 in Figure 8B, a user may select the portion of the first data lineage 800 for which the second data lineage 820 is generated by selecting start and / or endpoints within the first data lineage 800, such as specific data elements among data elements 813 to 817. That is, the user can make the selection of the start and end points. The selection of the portion, such as the start and end points, may be made in various ways, such as via a graphical user interface, via one or more configuration files, via commands (short cuts, voice commands etc.), command line arguments or any one of a variety of other suitable methods. Furthermore, the user may specify that only certain types of components are to be included in the generated second data lineage. Such restrictions may, for example, relate to components from specific applications, components hosted in particular systems, certain types of transformations etc. or any combination of such restrictions. It is clear that any other suitable restriction may also be specified.

[0093] As indicated by Figure 8C, the graphical representation of the generated second data lineage 820 includes graphical indicators of second components 822, 824, 827 of the plurality of second components such that the graphical indicators are arranged within the graphical representation of the portion of the generated second data lineage in a manner that is in accordance with the relationships among the plurality of second components of the first data lineage 800. As derivable from Figure 8C, the algorithm filtered out a lot of generally irrelevant information from a perspective of the target environment. Using the upwards direction, the algorithm kept the lineage starting from the data elements enriched with associated terms, all the way up to their sources; all while only keeping the data elements that belong to the selected portion of the first data lineage 813 to 817 (e.g. marked applications or processes) in a previous step.

[0094] As indicated by Figure 8D, user may select a link 823 associating two second components 822, 824 of the second data lineage 820. For example, upon the user selection of link 823, a pop-up window can be generated in which the user can activate an icon 829 (“drill down point-to-point technical lineage”) to submit instructions to dive into the first data lineage for the portion of the second data lineage 820 associated with the selected link 823.

[0095] Based on the selected link, the third data lineage 830 illustrated in Figure 8E is generated, the first data lineage 830 specifying relationships among those physical components 802, 804, 807, 809, 810 of the plurality of physical components that are associated with the second components 822, 824 whose graphical indicators are connected by the selected link 823. The third data lineage 830 may be a portion of the first data lineage 800. In this way, the user can dive into a portion of the first data lineage 800 by of the generated second data lineage 820.

[0096] Figure 9 is a graphical user interface 900 graphically illustrating an example second data lineage, in accordance with some embodiments of the technology described herein. As indicated by Figure 9, the method may further include: receiving a user selection of a portion of a generated second data lineage, and generating a view for more detailed information about the selected portion, such as number of passes through a component, number of transforms involved up to a generation of a particular data set, number of inputs and / or outputs of a component, or the like. The GUI 900 is displaying annotated information for a selected portion of the second data lineage.

[0097] In other words, the inventors have developed techniques that efficiently and reliably maintain data lineage when generating a target data lineage. Means are provided that allow for more accurate, reliable and comprehensive generation of the target data lineage while keeping consumption of computational resources low. Discrepancies between the target data lineage and a source data lineage are eliminated and changes among the source and target data lineages are identified. This is especially useful when establishing data exchange between a (dynamic) source environment having the source data lineage and the target environment having the target data lineage. Moreover, means are provided that allow to deeply drive from the target data lineage back into the physical components associated with the source data lineage.

[0098] The above-described aspects and embodiments of the technology described herein can be implemented in any of numerous ways. For example, the embodiments may be implemented using hardware, software or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers. Such processors may be implemented as integrated circuits, with one or more processors in an integrated circuit component, including commercially available integrated circuit components known in the art by names such as CPU chips, GPU chips, microprocessor, microcontroller, or co-processor. Alternatively, a processor may be implemented in custom circuitry, such as an ASIC, or semicustom circuitry resulting from configuring a programmable logic device. As yet a further alternative, a processor may be a portion of a larger circuit or semiconductor device, whether commercially available, semi-custom or custom. As a specific example, some commercially available microprocessors have multiple cores such that one or a subset of those cores may constitute a processor. However, a processor may be implemented using circuitry in any suitable format. Instead of the means to carry out the method, a processor and a memory can be used to carry out the method.

[0099] Further, it should be appreciated that a “computer”, such as client 1, may be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. Additionally, a computer may be embedded in a device not generally regarded as a computer but with suitable processing capabilities, including a Personal Digital Assistant (PDA), a smart phone or any other suitable portable or fixed electronic device.

[0100] Also, a computer may have one or more input and output devices. These devices can be used, among other things, to present a user interface. Examples of output devices that can be used to provide a user interface include printers or display screens for visual presentation of output and speakers or other sound generating devices for audible presentation of output. Examples of input devices that can be used for a user interface include keyboards, and pointing devices, such as mice, touch pads, and digitizing tablets. As another example, a computer may receive input information through speech recognition or in other audible format.

[0101] Such computers may be interconnected by one or more networks in any suitable form, including as a local area network or a wide area network, such as an enterprise network or the Internet 306. Such networks may be based on any suitable technology and may operate according to any suitable protocol and may include wireless networks, wired networks or fiber optic networks.

[0102] Also, the various methods or processes outlined herein may be coded as software that is executable on one or more processors that employ any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages and / or programming or scripting tools, and also may be compiled as executable machine language code or intermediate code that is executed on a framework or virtual machine. In this respect, aspects of the present disclosure may be embodied as a computer-readable storage medium (or multiple computer readable media) (e.g., a computer memory, one or more floppy discs, compact discs (CD), optical discs, digital video disks (DVD), magnetic tapes, flash memories, circuit configurations in Field Programmable Gate Arrays or other semiconductor devices, or other tangible computer storage medium) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods that implement the various embodiments of the disclosure discussed above. As is apparent from the foregoing examples, a computer readable storage medium may retain information for a sufficient time to provide computer-executable instructions in a non-transitory form. Such a computer readable storage medium or media can be transportable, such that the program or programs stored thereon can be loaded onto one or more different computers or other processors to implement various aspects of the present disclosure as discussed above. As used herein, the term "computer-readable storage medium" as a preferred example of “computer-readable medium” encompasses only a non-transitory computer-readable medium that can be considered to be a manufacture (i.e., article of manufacture) or a machine. Alternatively or additionally, the disclosure may be embodied as a computer readable medium other than a computer-readable storage medium, such as a propagating signal.

[0103] The terms “application”, “algorithm”, “service”, “program code”, “computer program” or “software” are used herein in a generic sense to refer to any type of computer code or set of computer-executable instructions that can be employed to program a computer or other processor to implement various aspects of the present disclosure as discussed above. The data storage, receiver, subscriber, dispatcher, transformer, and synthesizer may be implemented by computer software and / or computer hardware. Additionally, it should be appreciated that according to one aspect of this embodiment, one or more computer programs that when executed perform methods of the present disclosure need not reside on a single computer or processor, but may be distributed in a modular fashion amongst a number of different computers or processors to implement various aspects of the present disclosure.

[0104] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0105] Also, the data set may be stored in computer-readable media in any suitable form. For simplicity of illustration, data sets may be shown to have data set entries. However, any suitable mechanism may be used to establish locations within the data set to store data, such as parameter values. Various aspects of the present disclosure may be used alone, in combination, or in a variety of arrangements not specifically discussed in the embodiments described in the foregoing and is therefore not limited in its application to the details and arrangement of components set forth in the foregoing description or illustrated in the drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

[0106] Also, aspects of the disclosure may be embodied as a method, of which an example has been provided. The acts performed as part of the method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.

[0107] Further, some actions are described as taken by a “user” . It should be appreciated that a “user” need not be a single individual, and that for example, actions attributable to a “user” may be performed by a team of individuals and / or an individual in combination with computer-assisted tools or other mechanisms. It should be appreciated that a “user” need not be an individual and could be a machine, such as in form of a bot.

[0108] Use of ordinal terms such as “first,” “second,” “third,” etc., in the claims to modify a claim element does not by itself connote any priority, precedence, or order of one claim element over another or the temporal order in which acts of a method are performed, but are used merely as labels to distinguish one claim element having a certain name from another element having a same name (but for use of the ordinal term) to distinguish the claim elements. Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," or "having," “containing,” “involving,” and variations thereof herein, is meant to encompass the items listed thereafter and equivalents thereof as well as additional items.

[0109] Further, though advantages of the present disclosure are indicated, it should be appreciated that not every embodiment of the disclosure described herein will include every described advantage. Some aspects and embodiments may not implement any features described as advantageous herein and in some instances one or more of the described features may be implemented to achieve further embodiments. Accordingly, this description and these drawings are by way of example only.

[0110] The present disclosure can (alternatively) be described by one or more of the following numbered aspects:

[0111] ASPECTS 1. A computer-implemented method for conversion of a first data lineage to a second data lineage, comprising: obtaining a first data lineage specifying relationships among physical components of a plurality of physical components; receiving an identification of a portion of the first data lineage; generating a second data lineage from the identified portion of the first data lineage, the second data lineage specifying relationships among second components of a plurality of second components, wherein the second components of the plurality of second components are associated with at least some of the physical components of the identified portion of the first data lineage.

[0112] 2. The method of aspect 1, wherein the identification of the portion of the first data lineage includes an identification of a first physical component of the plurality of physical components that indicates a start point for the second data lineage and an identification of a second physical component of the plurality of physical components that indicates an end point for the second data lineage.

[0113] 3. The method of aspect 2, wherein the generating of the second data lineage from the identified portion of the first data lineage includes generating the second data lineage by traversing the first data lineage from the start point to the end point.

[0114] 4. The method of any one preceding aspects, wherein the identification of the portion of the first data lineage is based on user input, preferably wherein the user input includes a user selection of one or more physical components of the plurality of physical components.

[0115] 5. The method of any one of aspects 2 to 4, wherein the identification of the first physical component and / or second physical component is based on user input, preferably wherein the user input includes a user selection of the first physical component and / or a user selection of the second physical component by the user.

[0116] 6. The method of any one preceding aspects, wherein each second component of the plurality of second components is associated by an identifying link with a corresponding physical component of the at least some of the physically components of the portion of the first data lineage.

[0117] 7. The method of any one of the preceding aspects, further comprising: outputting the generated second data lineage. 8. The method of aspect 7, wherein the outputting of the generated second data lineage includes displaying, in a graphical user interface, a graphical representation of a portion of the generated second data lineage.

[0118] 9. The method of aspect 8, wherein the graphical representation of the portion of the generated second data lineage includes graphical indicators of second components of the plurality of second components such that the graphical indicators are arranged within the graphical representation of the portion of the generated second data lineage in a manner that is in accordance with the relationships among the plurality of second components.

[0119] 10. The method of any one preceding aspects, wherein the generating of the second data lineage is executed automatically by the computer.

[0120] 11. The method of any one of the preceding aspects, further comprising: repeatedly executing the generation of the second data lineage from the identified portion of the first data lineage; comparing a newly generated second data lineage with a previously generated second data lineage of the repeated execution, including determining whether there is a difference between the newly generated second data lineage and the previously generated second data lineage; and outputting whether there is a difference between the newly generated second data lineage and the previously generated second data lineage.

[0121] 12. The method of aspect 11, further comprising: based on the comparing, determining, based on one or more criteria, whether to keep or drop determined one or more differences between the newly generated second data lineage and the previously generated second data lineage for a second data lineage to be maintained, preferably wherein the one or more criteria include whether a number of the differences exceeds a threshold value.

[0122] 13. The method of aspect 11 or 12, further comprising: based on the comparing, determining whether to select the newly generated second data lineage or the previously generated second data lineage as a second data lineage to be maintained.

[0123] 14. The method of any one of the preceding aspects, further comprising: in response to detecting a change in the first data lineage, updating the generated second data lineage, or generating a new second data lineage, in accordance with the detected change. The method of any one of the preceding aspects, wherein each physical component of the plurality of physical components includes a data set and / or a transformation. The method of any one of the preceding aspects, wherein the first data lineage is generated at least in part by performing

[0124] (a) analyzing source code of at least one computer program configured to access at least some of the plurality of physical components to identify one or more physical components accessed by the computer program; and / or

[0125] (b) analyzing information obtained during runtime of the at least one computer program including analyzing one or more logs generated during runtime of the computer program to identify one or more physical components accessed by the computer program. The method of aspect 16, wherein the one or more physical components accessed by the computer program include one or more inputs of the computer program, and / or one or more transformations applied to inputs of the computer program, and / or one or more outputs of the computer program. The method of any one of the preceding aspects, further comprising: before the generating of the second data lineage from the identified portion of the first data lineage, pre-processing the physical components of the identified portion of the first data lineage, wherein the generating of the second data lineage from the identified portion of the first data lineage is executed based on a result of the pre-processing. The method of aspect 18, wherein the pre-processing includes: identifying a control flow and / or an internal data lineage associated with one or more of the physical components of the portion of the first data lineage and excluding the control flow and / or the internal data lineage when generating the second data lineage. The method of any one of aspects 9 to 19, wherein the graphical representation of the portion of the generated second data lineage includes links connecting the graphical indicators of the second components of the plurality of second components, wherein the method further comprises: receiving a user selection of one of the links; and based on the selected link, generating a third data lineage specifying relationships among those physical components of the plurality of physical components that are associated with the second components whose graphical indicators are connected by the selected link. The method of any one of the preceding aspects, further comprising: obtaining a fourth data lineage specifying relationships among physical components of a plurality of physical components; receiving an identification of a portion of the fourth data lineage; generating a fifth data lineage from the identified portion of the fourth data lineage, the fifth data lineage specifying relationships among fifth components of a plurality of fifth components, wherein the fifth components of the plurality of fifth components are associated with at least some of the physical components of the identified portion of the fourth data lineage; comparing the fifth data lineage with the second data lineage, including determining whether there is a difference between the fifth data lineage and the second data lineage; and outputting whether there is a difference between the fifth data lineage and the second data lineage.

[0126] 22. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one preceding aspects.

[0127] 23. A data processing system comprising means for carrying out the method of any one of aspects 1 to 21.

[0128] 24. A computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method of any one of aspects 1 to 21.

[0129] The present invention is defined by the following independent claims. Some of the preferred embodiments are described by the following dependent claims as well as by the above description and the enclosed figures. Having described several aspects of embodiments of this invention, it is to be appreciated that various alterations, modifications, and improvements of these aspects are possible within the scope of the invention as defined by the appended claims.

[0130] The various aspects and embodiments described above can be combined to create yet further embodiments. These and other changes can be made to the embodiments in light of the above detailed description. In general, the terms used in the following claims should not be construed as limiting the claims to the specific aspects and embodiments disclosed in the description and the claims, but should be construed as including all possible embodiments along with the full scope of equivalents to which these claims are entitled.

Claims

CLAIMS1. A computer-implemented method for conversion of a first data lineage to a second data lineage, comprising: obtaining a first data lineage specifying relationships among physical components of a plurality of physical components; receiving an identification of a portion of the first data lineage; and generating a second data lineage from the identified portion of the first data lineage, the second data lineage specifying relationships among second components of a plurality of second components, wherein the second components of the plurality of second components are associated with at least some of the physical components of the identified portion of the first data lineage.

2. The method of claim 1, wherein the identification of the portion of the first data lineage includes an identification of a first physical component of the plurality of physical components that indicates a start point for the second data lineage and an identification of a second physical component of the plurality of physical components that indicates an end point for the second data lineage.

3. The method of claim 2, wherein the generating of the second data lineage from the identified portion of the first data lineage includes generating the second data lineage by traversing the first data lineage from the start point to the end point or vice versa.

4. The method of claim 3, wherein the traversing comprises recursively listing identifiers of all source or target components of the start or end point respectively.

5. The method of claim 4, further comprising: storing a record of listed source-target pairs of components and any coupling transformations coupling the source and target components in a respective source-target pair.

6. The method of claim 4 or 5, wherein the generating of the second data lineage is based on selecting, from the recursively generated list of identifiers, those list entries, which relate to physical components that are linked to one or more predetermined applications that are configured to perform operations associated with a portion of said physical components of the first data lineage.

7. The method of any one preceding claims, wherein the identification of the portion of the first data lineage is based on user input, preferably wherein the user input includes a user selection of the first physical component and / or a user selection of the second physical component by the user.

8. The method of any one preceding claims, wherein each second component of the plurality of second components is associated by an identifying link with a corresponding physical component of the at least some of the physically components of the portion of the first data lineage.

9. The method of any one of the preceding claims, further comprising: outputting the generated second data lineage.

10. The method of claim 9, wherein the outputting of the generated second data lineage includes displaying, in a graphical user interface, a graphical representation of a portion of the generated second data lineage, wherein the graphical representation of the portion of the generated second data lineage includes graphical indicators of second components of the plurality of second components such that the graphical indicators are arranged within the graphical representation of the portion of the generated second data lineage in a manner that is in accordance with the relationships among the plurality of second components.

11. The method of claim 10, wherein the graphical representation of the portion of the generated second data lineage includes links connecting the graphical indicators of the second components of the plurality of second components, wherein the method further comprises: receiving a user selection of one of the links; and based on the selected link, generating a third data lineage specifying relationships among those physical components of the plurality of physical components that are associated with the second components whose graphical indicators are connected by the selected link.

12. The method of any one preceding claims, wherein the generating of the second data lineage is executed automatically by the computer.

13. The method of any one of the preceding claims, further comprising: repeatedly executing the generation of the second data lineage from the identified portion of the first data lineage; comparing a newly generated second data lineage with a previously generated second data lineage of the repeated execution, including determining whether there is a difference between the newly generated second data lineage and the previously generated second data lineage; and outputting whether there is a difference between the newly generated second data lineage and the previously generated second data lineage.

14. The method of claim 13, further comprising:based on the comparing, determining, based on one or more criteria, whether to keep or drop determined one or more differences between the newly generated second data lineage and the previously generated second data lineage for a second data lineage to be maintained, preferably wherein the one or more criteria include whether a number of the differences exceeds a threshold value.

15. The method of claim 13 or 14, further comprising: based on the comparing, determining whether to select the newly generated second data lineage or the previously generated second data lineage as a second data lineage to be maintained.

16. The method of any one of the preceding claims, further comprising: in response to detecting a change in the first data lineage, updating the generated second data lineage, or generating a new second data lineage, in accordance with the detected change.

17. The method of any one of the preceding claims, wherein the first data lineage is generated at least in part by performing:(a) analyzing source code of at least one computer program configured to access at least some of the plurality of physical components to identify one or more physical components accessed by the computer program; and / or(b) analyzing information obtained during runtime of the at least one computer program including analyzing one or more logs generated during runtime of the computer program to identify one or more physical components accessed by the computer program, preferably wherein the one or more physical components accessed by the computer program include one or more inputs of the computer program, and / or one or more transformations applied to inputs of the computer program, and / or one or more outputs of the computer program.

18. The method of any one of the preceding claims, further comprising: before the generating of the second data lineage from the identified portion of the first data lineage, pre-processing the physical components of the identified portion of the first data lineage, wherein the generating of the second data lineage from the identified portion of the first data lineage is executed based on a result of the pre-processing, preferably wherein the pre-processing includes: identifying a control flow and / or an internal data lineage associated with one or more of the physical components of the portion of the first data lineage and excluding the control flow and / or the internal data lineage when generating the second data lineage.

19. The method of any one of the preceding claims, further comprising: obtaining a fourth data lineage specifying relationships among physical components of a plurality of physical components;receiving an identification of a portion of the fourth data lineage; generating a fifth data lineage from the identified portion of the fourth data lineage, the fifth data lineage specifying relationships among fifth components of a plurality of fifth components, wherein the fifth components of the plurality of fifth components are associated with at least some of the physical components of the identified portion of the fourth data lineage; comparing the fifth data lineage with the second data lineage, including determining whether there is a difference between the fifth data lineage and the second data lineage; and outputting whether there is a difference between the fifth data lineage and the second data lineage.

20. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one preceding claims.

21. A data processing system comprising means for carrying out the method of any one of claims 1 to 19.

22. A computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method of any one of claims 1 to 19.

Images