DATA TRACEABILITY CONVERSION

Abstract

This disclosure relates to a computer-implemented method for converting a first data traceability into a second data traceability, the method comprising: obtaining a first data traceability specifying relationships between physical components of a plurality of physical components; receiving an identification of a portion of the first data traceability; generating a second data traceability from the identified portion of the first data traceability, the second data traceability specifying relationships between second components of a plurality of second components, the second components of the plurality of second components being associated with at least some of the physical components of the identified portion of the first data traceability.A corresponding computer-readable medium, a corresponding data processing system, and a corresponding computer program are also described. Figure for the abstract: Fig. 1.

Description

Title of the invention: CONVERSION OF DATA TRACEABILITIES Technical field

[0001] The present invention relates to a computer-implemented method for converting a first data traceability into a second data traceability, as well as to a corresponding computer-readable medium, a corresponding data processing apparatus and a corresponding computer program. Background

[0002] Different environments, such as different groups, organizations, or systems, may have different standards for tracing the origin of data. When exchanging data between two environments, the traceability of data produced in one of the environments is often lost when importing the data into the other environment.

[0003] Conventional approaches to maintaining data traceability across environments are generally unreliable. Summary of the invention

[0004] The present invention therefore aims to provide means allowing more reliable maintenance of data traceability.

[0005] This object is reached by a computer-implemented method for converting a first data traceability into a second data traceability, comprising: obtaining a first traceability of data specifying relationships between the physical components of a plurality of physical components; receipt of an identification of a part of the first data traceability; generating a second data trace from the identified portion of the first data trace, the second data trace specifying relationships between second components of a plurality of second components, the second components of the plurality of second components being associated with at least some of the physical components of the identified portion of the first data trace, by a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to implement the method according to the invention, by a data processing system comprising means for implementing the method according to the invention, and by a computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to implement the method according to the invention. Some of the preferred embodiments are described in the description and in the figures.

[0006] The inventors have recognized that conventional approaches to maintaining data traceability between source and target environments involve manually specifying a target traceability for the target environment, which is hopefully consistent with a source data traceability of the source environment. This not only represents a burden for the user, but creates the risk that the source and target traceabilities are inconsistent or become inconsistent over time. Therefore, the inventors have developed techniques for computer-generating (in an automated manner) a target data traceability with minimal user input.The inventors have also recognized that generating target data traceability from source data traceability enables more accurate and reliable generation of target data traceability by eliminating discrepancies between target data traceability and source data traceability. This is particularly useful when working with data in source and target environments and / or when establishing data exchange between the source environment having source data traceability and the target environment having target data traceability. An environment may be defined by a group of people, an organization, a computer system, or a network. However, the technologies described herein may also be used for other purposes.

[0007] Accordingly, according to a general aspect of the invention, a computer-implemented method is provided for converting a first data traceability into a second data traceability, comprising: obtaining a first data traceability specifying relationships between physical components of a plurality of physical components; receiving an identification of a portion of the first data traceability; generating a second data traceability from the identified portion of the first data traceability, the second data traceability specifying relationships between second components of a plurality of second components, the second components of the plurality of second components being associated with at least some of the physical components of the identified portion of the first data traceability.Overall, this eliminates the burden of manually specifying second data lineage and eliminates discrepancies between second data lineage and first data lineage, and can be used to maintain data lineage accurately and reliably.

[0008] In general, each physical component comprises a data set and / or a transformation. Each data set may comprise one or more data elements. Each data set may be a table, with each row or column being a data element of the data set.

[0009] According to a preferred aspect of the present invention, identifying the portion of the first data traceability comprises identifying a first physical component of the plurality of physical components that indicates a starting point for the second data traceability and identifying a second physical component of the plurality of physical components that indicates an ending point for the second data traceability. This allows for accurate selection of a portion of the first data traceability for generating the second data traceability. This also allows for excluding portions of the first data traceability that are not of interest when generating the second data traceability, thereby reducing the consumption of computing resources for generating the second data traceability.

[0010] According to a preferred aspect of the present invention, generating the second data traceability from the identified portion of the first data traceability comprises generating the second data traceability by traversing the first data traceability from the starting point to the ending point. The inventors have recognized that traversing is a particularly accurate and computationally efficient means of generating the second data traceability while reliably maintaining the relationships between the components.

[0011] According to a preferred aspect of the present invention, the method may comprise: in response to detecting a change in the first data traceability, updating the generated second data traceability, or generating a new second data traceability, based on the detected change. Preferably, each second component of the plurality of second components is associated by an identification link with a corresponding physical component of the at least part of the physical components of the part of the first data traceability, this identification link being able to preserve the proof of the origin of a data set. The identification link makes it possible to efficiently eliminate discrepancies between the second data traceability and the first data traceability and to automatically update or generate the second data traceability in the event of a modification of the underlying first data traceability.

[0012] According to a preferred aspect of the present invention, the graphical representation of the portion of the generated second data traceability comprises graphical indicators of second components of the plurality of second components, such that the graphical indicators are arranged in the graphical representation of the portion of the second data traceability generated in a manner consistent with relationships between the plurality of second components. The graphical representation of the portion of the second data traceability generated may include links connecting the graphical indicators of the second components of the plurality of second components, the method may further comprise: receiving a user selection of one of the links; and, based on the selected link, generating a third data traceability specifying relationships between the physical components of the plurality of physical components that are associated with the second components whose graphical indicators are connected by the selected link.Therefore, the data traceability information specified by the first data traceability is directly accessible through the second data traceability, even if the second data traceability is used instead of the first data traceability. In other words, it is possible to delve deeper into the physical components and their relationships through the second data traceability. The third data traceability can be a part of the first data traceability, so that the first data traceability can be delved deeper through the second data traceability.

[0013] According to a preferred aspect of the present invention, the method comprises repeatedly performing the generation of the second data traceability from the identified portion of the first data traceability; comparing a newly generated second data traceability with a previously generated second data traceability during the repeated execution, comprising determining the existence of a difference between the newly generated second data traceability and the previously generated second data traceability; and issuing a message indicating the existence of a difference between the newly generated second data traceability and the previously generated second data traceability. Accordingly, a discrepancy is reported to the user, as an indication of a potential error condition in the collection of the first and second data traceabilities.In particular, a change in the first data lineage over time can be identified by tracking changes in the second data lineage over time.

[0014] According to a preferred aspect of the present invention, the method comprises obtaining a fourth data traceability specifying relationships between physical components of a plurality of physical components; receiving an identification of a portion of the fourth data traceability; generating a fifth data traceability from the identified portion of the fourth data traceability, the fifth data traceability specifying relationships between fifth components of a plurality of fifth components, the fifth components of the plurality of fifth components being associated with at least some of the physical components of the identified portion of the fourth data traceability; comparing the fifth data traceability to the second data traceability, including determining whether there is a difference between the fifth data traceability and the second data traceability; and outputting whether there is a difference between the fifth data traceability and the second data traceability. Accordingly, a discrepancy is reported to the user, as an indication of a potential error condition in the collection of the first and second data traceabilities. In particular, a discrepancy between the first data traceability and the fourth data traceability may be identified by comparing the second data traceability with the fifth data traceability.

[0015] In general, each second and / or fifth component is a representation of a data set and / or a transformation of the corresponding physical component. The second and / or fifth component may be associated with a semantic meaning of the corresponding physical component, according to a dictionary associating physical components with corresponding semantic meanings.

[0016] According to a preferred aspect of the present invention, the first and / or third data traceability is generated at least in part by performing (a) an analysis of the source code of at least one computer program configured to access at least a portion of the plurality of physical components to identify one or more physical components accessed by the computer program; and / or (b) an analysis of information obtained during execution of the at least one computer program, comprising analyzing one or more logs generated during execution of the computer program to identify one or more physical components accessed by the computer program. By these means, the first and / or third data traceability can be generated efficiently and completely.

[0017] According to a preferred aspect of the present invention, before generating the second and / or fifth data traceability from the identified portion of the first and / or third data traceability, the method comprises preprocessing the physical components of the identified portion of the first and / or third data traceability, wherein the generation of the second and / or fifth data traceability from the identified portion of the first and / or third data traceability is performed based on a result of the preprocessing. Preferably, the preprocessing comprises: identifying a control flow and / or an internal data traceability associated with one or more physical components of the part of the first and / or third data lineage and the exclusion of the control flow and / or internal data lineage when generating the second and / or fifth data lineage. The control flow can be represented by the data that does not belong to the first and / or third data lineage but plays a role in the process of selecting a data record associated with a physical component: input keys in relational joins, WHERE clause in SQL queries and / or conditions in programming languages. The control flow can be the sum of all data participants that will indirectly influence the result. Internal data lineage, an intermediate step within a component.Abandoning the internal control flow and / or data trace will reduce the number of interactions and thus the time and resources required to generate the second data trace. Preferably, when generating the second and / or fifth data trace, a filter may be added to eliminate portions of the first data trace that are not of interest, in order to reduce the amount of data to be processed and stored in association with the second and / or fifth data trace. The second or fifth data trace may be a summary of only the relevant portions of the first data trace, from the starting point to the ending point.

[0018] According to a preferred aspect of the present invention, the method comprises receiving a user selection of a portion of a generated second data trace, and displaying more detailed information about the selected portion, such as the number of passes through a component, the number of transformations involved until a particular data set is generated, the number of inputs and / or outputs of a component, or other such information.

[0019] According to a preferred aspect of the present invention, the plurality of second components of the generated second data traceability may include manually created second components in addition to the computer-generated second components.

[0020] Computer (storage) media, preferably of non-transitory type, and data processing apparatus or computer systems may be provided which correspond to the methods described herein.

[0021] In other words, the inventors have developed techniques that allow data traceability to be efficiently and reliably maintained when generating a target data traceability. Means are provided to enable more accurate and reliable generation of the target data traceability while maintaining low consumption of computing resources. Discrepancies between the target data traceability and a source data traceability are eliminated and changes between source and target data traceability are identified. This is particularly useful for establishing data exchange between a (dynamic) source environment with source data traceability and the target environment with target data traceability. In addition, means are provided to enable deepening of target data traceability to the physical components associated with source data traceability.

[0022] Other features and advantages will become apparent from the following description and the above summary is not intended to limit the scope of the invention. Brief description of the drawings

[0023] Various aspects and embodiments will be described with reference to the following figures. It should be noted that the figures are not necessarily drawn to scale. Elements appearing in multiple figures are indicated by the same reference numeral in all figures in which they appear.

[0024] [Fig. 1] [Fig. 1] is a block diagram of an illustrative computing environment in which certain embodiments of the technology described herein may operate.

[0025] [Fig.2] [Fig.2] illustrates a graphical representation of data traceability, in accordance with certain embodiments of the technology described herein.

[0026] [Fig.3] [Fig.3] illustrates a graphical representation of data traceability, in accordance with certain embodiments of the technology described herein.

[0027] [Fig.4] [Fig.4] is a diagram illustrating an association between a second data traceability example 420 and a first data traceability example 400, in accordance with certain embodiments of the technology described herein.

[0028] [Fig.5] [Fig.5] shows a graphical user interface graphically illustrating an association between a second data traceability example 520 and a first data traceability example 500, in accordance with certain embodiments of the technology described herein.

[0029] [Fig.6] [Fig.6] illustrates exemplary relationships between data sets, data elements, transformations, and transformation data elements, in accordance with certain embodiments of the technology described herein.

[0030] [Fig.7] [Fig.7] illustrates exemplary relationships among data sets, data elements, transformations, transformation data elements, the application, and the system, in accordance with certain embodiments of the technology described herein.

[0031] [Fig.8A] to [Fig.8E] Figures 8A-8E illustrate an exemplary computer-implemented method for generating a second data trace from the first data trace and furthering the first data trace by generating a third data trace identical to a portion of the first data trace, in accordance with certain embodiments of the technology described herein.

[0032] [Fig.9] illustrates a graphical user interface in accordance with certain embodiments of the technology described herein. Description of the embodiments

[0033] According to the invention, a first data traceability is used to generate a second data traceability from the first data traceability.

[0034] Data traceability is a data structure that describes relationships between components of a plurality of components, where each component includes or represents one or more data sets and / or one or more transformations. Each data set may include one or more data elements. In particular, data traceability describes the lifecycle of data sets and their data elements to which transformations are applied as they are processed by a data processing system. Data traceability information for a given data set includes an identifier of one or more upstream data sets on which the given data set depends, one or more downstream data sets that depend on the given data set, and one or more transformations that process data to generate the given data set.When a particular component is "downstream" of a given component, it means that the particular component depends directly or indirectly on the given data set. By "upstream" of a given component, it is meant that the given component is directly or indirectly dependent on the given component. For example, by a downstream data set depending on an upstream data set, it is meant that the processing (of one or more data elements) of the upstream data set by the data processing system directly or indirectly results in the generation (of one or more data elements) of the downstream data set. The downstream-generated data set may be a data set that comes out of the data processing system (sometimes called an output data set) or a data set that is to be further processed by the data processing system (sometimes called an intermediate data set).The upstream dataset can be a dataset introduced into the data processing system (sometimes called an input dataset or reference dataset) or a dataset that has already been processed by the data processing system (sometimes called an intermediate dataset). A transformation is a data processing operation applied to an upstream dataset to . produce a downstream data set that is provided to a data collector. A data lineage diagram is a graphical representation of data lineage, including components and relationships between components of a data processing system.

[0035] [Fig. 1] is an exemplary data traceability diagram 100 illustrating a plurality of components 102-122, with relationships between the components indicated by arrows. For illustration purposes, the components 102-122 are physical components. Each physical component includes a data set and / or a transformation, each data set may include one or more data elements. The data traceability diagram 100 illustrates the traceability of output data 110 generated by a data processing system. In the example of [Fig. 1], the data processing system receives two source data sets 102, 104.The source data may be data sets stored or received, for example, in a file such as a flat file, a database such as a relational database or an object database, a queue, or other data storage repository in a computer system. For example, a data set may generally be a (database) table, with each row or column being a data element of the data set. For example, the source data 102 may be data sets of credit card transactions in the United States, stored in a file "US_feed.dat." Each data set may include a value for each of one or more fields, such as attributes defined in a record structure or columns of a database table.The source data 102, 104 may be received and processed in batches, e.g., data from a file or database that is processed hourly, daily, weekly, monthly, quarterly, annually, or at other intervals. The source data 102, 104 may be received as a stream and processed continuously, e.g., buffered in a queue and processed as data becomes available and system resources permit.

[0036] The data sets and / or data elements may be associated with transformations. The source data 102 is processed by a transformation element 106, which acts on the source data 102, for example to modify the source data 102 in some way. The transformation element may be an executable program capable of manipulating data, such as a Java program running in a virtual machine, an executable, a data flow graph, or another type of executable program. For example, the transformation element 106 may be an executable named “TransformA.exe”. In a specific example, the Transformation elements 106 may be a filtering component that removes unwanted data sets or data elements from the source data 102, such as data sets or data elements having an incorrect format. Transformation element 106 processes the source data 102 in light of reference data 120 to produce intermediate data 112. Reference data is data used by a transformation element to enable the transformation element to process data. For example, reference data that enables a matching operation includes one or more fields whose values ​​match those of one or more fields in the processed data. Intermediate data 112 may be stored in a file, database, queue, or other repository for storing data in a computer system.

[0037] A transformation element 108 processes all of the source data 104 in light of the reference data 122 to produce intermediate data 114. The intermediate data 114 may be stored in a file, a database, a queue, or any other directory for storing data in a computer system.

[0038] The intermediate data 112, 114 are processed together by a transformation element 116, which uses reference data 118. In one example, the transformation element 116 is a mapping operation and the reference data 118 includes data sets that indicate state values ​​and corresponding region values. When the intermediate data 112, 114 is processed by the transformation element 116, the value of the state field of each data set of the intermediate data 112, 114 is mapped to the corresponding region indicated in the reference data 118. In one example, the reference data 118 includes data that indicates business units and corresponding department IDs, manager names, and locations.When the intermediate data 112, 114 are processed by the transformation element 116, each data set is assigned to a business unit based on the mapping enabled by the reference data set. The reference data 118 may be used to process multiple data sets and is not modified by the processing. The reference data 118 may be updated by a user periodically or as needed.

[0039] The transformation element 116 produces the output data 110, which is stored in a file, database, queue, or other data storage repository in a computer system. The output data 110 may be further processed, for example, by other transformation elements in the same or a different data processing system, or may be stored for later analysis.

[0040] In the example of [Fig. 1], data traceability of the output data 110 is shown for data traceability components in a single data processing system. In some examples, data traceability of a data set may be tracked by multiple data processing systems. For example, source data may be initially processed by a first data processing system to produce output data X. A second data processing system reads output data X from the first data processing system and processes the output data X to generate output data Y. The output data Y is processed by a third data processing system to generate output data Z.Data traceability of the Z output data includes the initial source data, transformations included in each of the three data processing systems, and all reference data used during processing by any of the three data processing systems.

[0041] As illustrated in [Fig.l], a user can select (here using a mouse cursor in a graphical user interface) physical components 104 and 110 as the starting point and end point, respectively, of a portion 124 of the first data traceability 100, this selection identifies the portion 124 to be converted (see the arrow in the lower left corner of [Fig.l]) into a second data traceability 100A. In other words, all physical components of the first traceability 100 between the component 104 according to the starting point and the component 110 according to the end point are incorporated into the identified portion 124 and participate in the conversion to the second data traceability 100A. The first data traceability 100 can be hosted in one (computing) environment and the second data traceability can be hosted in another (computing) environment.

[0042] Accordingly, a computer-implemented method for converting a first data traceability 100 to a second data traceability 100A in accordance with embodiments of the claimed invention comprises: obtaining a first data traceability 100 specifying relationships between physical components of a plurality of physical components 102-122; receiving an identification of a portion 124 of the first data traceability 100; generating a second data traceability 100A from the identified portion 124 of the first data traceability 100, the second data traceability 100A specifying relationships (see the arrows between the second components in [Fig.l]) between second components of a plurality of second components 103, 105, 107, 109, 111, the second components of the plurality of second components 103, 105, 107, 109, 111 being associated (cf. associations 113, 115, 117, 119, 121 in [Fig.l]) with at least some of the physical components, here the components 104, 108, 114, 116, 110, of the identified part 124 of the first data traceability 100. In this way, the data traceability is maintained when converting a part of a data traceability into another data traceability.

[0043] [Fig. 1] is an exemplary data traceability diagram. In some examples, output data may be generated by a more complex data processing system, as shown in the exemplary end-to-end data traceability diagram 200A of [Fig. 2] for a target element 206A.

[0044] [Fig. 2] illustrates an exemplary data traceability diagram 200A that shows a plurality of physical components 202A-206A and their relationships (see arrows). Each physical component includes a data set and / or a transformation, each data set may include one or more data elements. In the data traceability diagram 200A, connections or links (see arrows) are indicated between data sets 202A and transformation elements 204A. Data sets 202A may represent tables, rows in tables, columns in tables, fields in tables or files, or other data. An example of a transformation element is an element of an executable that describes how a single output of a data element is produced.The root cause of a potential error in a target component 206A (or other intermediate component) can be tracked in the data processing system of [Fig.2].

[0045] The information in a data traceability diagram, such as the data traceability diagram of [Fig. 1] or [Fig. 2], illustrates which upstream data sources, data sets, data collectors, or transformations affect a downstream data set. For example, the data traceability diagram 100 of [Fig. 1] shows that the output data 110 is affected by the source data 102, 104, the reference data 118, 120, 122, and the transformation elements 106, 108, 116.

[0046] Understanding the traceability of a downstream dataset (such as the output dataset 110) may be useful in identifying the root cause of an error in a dataset that may occur in the downstream dataset. By root cause, we mean identifying an upstream component that is at least partially causing the error in the downstream component. An error in a downstream dataset, such as in the output dataset 110, may be due to erroneous source data, erroneous reference data, or an error in a transformation element in the traceability upstream of the output dataset 110, or a combination of two or more of these elements. Monitoring the quality or status of data traceability components can provide information that can be used to assess a possible root cause of errors in data sets.

[0047] By error in a dataset, we generally mean whether the dataset exhibits expected characteristics, for example to avoid an error in data processing. An error in a dataset may manifest itself by a dataset not exhibiting expected characteristics, for example by falling outside statistical norms, returning a lookup failure in response to a standard query, or any other type of behavior. A parameter to be evaluated may be a measure of the quality of the data or the program code associated with the dataset. The evaluation of the parameters or the quality of a dataset may be characterized based on a profile of some or all of the datasets in the dataset or based on the value of each of one or more fields in a specific dataset, or both.

[0048] An error in a downstream data set (e.g., output data 110) may be attributed to various factors in the traceability of data upstream of the output data set. A possible cause of erroneous output data may be erroneous source data, erroneous reference data, erroneous transformation logic, or a combination of these factors. For example, a source data set may have been corrupted or cut off during transmission, it may be a bad data set, there may be missing data, or there may be some other problem. A reference data set may have been exposed to an error during a recent update of the reference data set, may be corrupted, may be a bad data set, or may have some other problem.Another possible cause of poor output data quality may be a problem with a transformation element in the data lineage upstream of the output data. For example, if the software implementing a transformation element has been recently updated, the transformation element may no longer perform the desired processing if, for example, the updated software has an error or has been corrupted. Source data, reference data, and transformation elements in the data lineage of the output data set 110 may be monitored to facilitate preemptive identification of a potential error, such as a data quality issue that may occur in the output data set, subsequent tracking of the root cause of an error that has occurred in the output data set, or both.

[0049] Control and analysis of source data and reference data may help a user or system diagnose one or more possible causes of erroneous output data. For example, if a set of erroneous output data is generated, analyzing the source data or reference data in the data lineage of the poor-quality output data set can indicate whether a given set of source data or reference data is itself erroneous and therefore likely to contribute to the erroneous output data. Monitoring the source data and reference data also helps identify in advance erroneous source data or reference data that, if processed, may cause an error, such as a data quality issue, in the downstream output data.

[0050] [Fig. 3] is a block diagram of an illustrative computing environment 300, in which certain embodiments of the technology described herein may operate. The computing environment 300 includes the data processing system 305, which is configured to operate on data stored in the data storage 304.

[0051] The data storage 304 may include one or more storage devices storing data sets in one or more formats of any suitable type. For example, the one or more storage devices forming part of the data storage 304 may store data sets as one or more database tables, spreadsheet files, flat text files, and / or files in any other suitable format (e.g., a format native to a mainframe computer). The one or more storage devices may be of any suitable type and may include one or more servers, one or more database systems, one or more portable storage devices, one or more non-volatile storage devices, one or more volatile storage devices, and / or any other device configured to store data electronically.The data storage 304 may include one or more online data streams of data sets in addition to or instead of the one or more storage devices. Thus, for example, the data processing system 305 may have access to data sets provided by one or more data streams in any suitable format.

[0052] In cases where the data storage 304 comprises multiple storage devices, the storage devices may be located in a single physical location (e.g., in a building) or distributed across multiple physical locations (e.g., in multiple buildings, in different cities, different states, or different countries). The storage devices may be configured to communicate with each other using one or more networks such as, for example, the network 306 illustrated in [Fig.l].

[0053] The data sets stored by the storage device(s) may include one or more data entities such as one or more files, tables, data in rows and / or columns of tables, spreadsheets, data sets, data records (e.g., credit card transaction records, telephone call records, and banking transaction records), fields, variables, messages, and / or reports. The storage device(s) may store thousands, millions, tens of millions, or hundreds of millions of data entities. Each data set may include one or more data elements.

[0054] A data item may be any data item stored and / or processed by a data processing system. For example, a data item may be a field of a data record, and the value of the data item may be the value stored in the field of the data record or may be a column or row of a (database) table. As a specific, non-limiting example, a data item may be a field storing the name of a caller in a data record storing information about a telephone call (this data record may be one of multiple data records about telephone calls made by customers of a telecommunications company) and the value of the data item may be the value stored in the field.As another example, a data element may be a cell in a table (e.g., a cell in a particular row and column of the table), and the value of the data element may be the value of the table cell. As another example, a data element may be a variable (e.g., in a report), and the value of the element may be the value of the variable (e.g., in a particular instance of the report). As a specific, non-limiting example, a data element may be a variable in a bank loan applicant report representing the applicant's credit score, and the value of the data element may be the numeric value of the credit score (e.g., a numeric value between 300 and 850). The value of the data element representing the applicant's credit score may change depending on the data used to generate the bank loan applicant report.

[0055] A data element may take a value of any suitable type. For example, a data element may take a numeric value, an alphabetic value, a value from a discrete set of options (e.g., a finite set of categories), or any other suitable value type, and aspects of the technology described herein are not limited in this regard.

[0056] The data processing system 305 may include one or more computer programs 309 configured to operate on data in the storage of data 304. Computer programs 309 may be of any type and written in any suitable programming language(s). For example, computer programs 309 may include one or more computer programs written at least in part using Structured Query Language (SQL) and configured to access data in one or more databases forming part of data storage 304. As another example, data processing system 305 is configured to execute programs in the form of graphs and computer programs 309 may include one or more computer programs developed as data flow graphs. A data flow graph may include components, called "nodes" or "vertices," representing data processing operations to be performed on input data and links between the components representing the data flows.Techniques for executing computations encoded by data flow graphs are described in U.S. Patent No. 5,966,072, entitled “Executing Computations Expressed as Graphs.”

[0057] In the example of [Fig. 3], the data processing system 305 further includes a development environment 308 that may be used by a person (e.g., a developer) to develop one or more computer programs 309 to operate on data in the data storage 304. For example, the user 302 may use the computing device 303 to interact with the development environment 308 to specify a computer program 309, such as a data flow graph, and save the computer program as part of the computer programs 309. For example, the user 302 may use the computing device 303 to interact, via the network 306, with other users 330 of other computing devices 334. An environment for developing computer programs in the form of data flow graphs is described in U.S. Pat. Pub.No.: 2007 / 0031668, entitled “Managing Parameters for Graph-Based Applications”.

[0058] One or more computer programs 309 may be configured to perform any appropriate operation on data in the data storage 304. For example, one or more computer programs 309 may be configured to access data from one or more sources, transform the accessed data (e.g., by changing values ​​of the data, filtering data records, changing data formats, sorting the data, combining data from multiple sources, dividing data into multiple parts, and / or in any other appropriate manner), calculate one or more new values ​​from the accessed data, and / or write the data to one or more destinations.

[0059] One or more computer programs 309 may be configured to perform calculations and / or generate reports from the data contained in the data storage 309. The calculations performed and / or reports generated may be related to one or more quantities relevant to a business. For example, a computer program may be configured to access data relating to a person's credit history and determine a credit score for the person based on the credit history. As another example, a computer program may access the telephone call logs of multiple customers of a telephone company and generate a report indicating how many customers are using more data than is allowed in their data plan.As another example, a computer program may access data indicating the types of loans made by a bank and generate a report indicating the overall risk of the loans made by the bank. These examples are illustrative and not limiting, as a computer program may be configured to generate any appropriate information (e.g., for any appropriate business or technological purpose) from the data stored in data storage 304.

[0060] In the example of [Fig. 3], the data processing system 305 may also include a data governance module 310 that supports the execution of various data governance tasks. For example, in the illustrated embodiment, the data governance module 310 includes the data dictionary module 312, the role management module 314, the parameter evaluation module 316, the data traceability module 318, the derived traceability module 320, and the traceability association module 322, each including processor-executable instructions that, when executed, accomplish one or more data governance tasks, as described in more detail below.

[0061] The data dictionary module 312 may be configured to store information about the data in the data storage 304. In other words, the data dictionary 312 may be configured to store metadata associated with the data in the data storage 304. For example, the data dictionary 312 may store one or more alternative names for data elements in the data storage 304. Thus, rather than referring to a data element by the name of the variable to which it corresponds (this variable name may have been created by a programmer and may not be “user-friendly” in that it does not immediately indicate to the user what information the variable represents), the data dictionary may include one or more alternative terms for the data element (e.g., second data elements of the second data traceability) such as, for example, a natural language term or phrase that professionals would use to refer to the data element. As an example, the data dictionary 312 may store the name “Bank Customer Credit Score” or “Bank Customer FICO Credit Score” as an alternative name for a data element corresponding to a variable named “cstCrdScr,” which stores the value of a FICO credit score for a particular bank customer. As another specific example, the data dictionary 312 may store the name “Order Amount” as a language that can be used to refer to the data element corresponding to a field named “order_amt.”

[0062] The role management module 314 may manage information indicating which party(ies) is / are responsible for various data items stored in the data storage 304. Managing this role information may include storing the role information, allowing one or more users to modify this information (e.g., by deleting, adding, or modifying parties and / or their responsibilities), and / or displaying the role information.

[0063] The role management module 314 may specify parties responsible for one or more data elements of the first and / or second data traceability. For example, the role management module 314 may be configured to manage information used to generate (and may be configured to generate) a graphical interface indicating parties responsible for managing a data element.

[0064] The parameter evaluation module 316 may be configured to evaluate one or more parameters indicating a potential error in a data set and / or its one or more data elements. The parameter may be defined in any suitable manner indicating a potential error in a data set. The parameter may be a measure of the quality of the data or program code associated with the data set. The parameter may be evaluated using predefined rules, which may define criteria for evaluating the values ​​of the data sets and / or their data elements, for example, by identifying characteristics (e.g., accuracy, precision, completeness, and validity) of the values ​​based on the criteria. The manner in which the values ​​exhibit these characteristics may thus produce a measure of data quality for the data set and / or data elements.Some aspects of data quality assessment using data quality rules are described in US Pat. Pub. No.: 2014 / 0308357, “Specifying and Applying Rules to Data.”

[0065] The data traceability module 318 may be configured to generate data traceability for at least a portion of the data in the data storage. 304. A data traceability may include traceability information for one or more physical components, including data sets and, where applicable, transformations. For example, a derived data traceability, such as the second and fifth data traces described herein, may include backward traceability information indicating how the value of an upstream component was obtained and / or forward traceability information indicating one or more other downstream components that depend on the value of the upstream component.

[0066] The data traceability module 318 may be configured to generate data traceability, such as the data traceability 100, 200A of Figures 1 and 2, respectively, by analyzing the source code of at least one computer program configured to access (e.g., read, write, and modify) at least a portion of the plurality of physical components managed by a data processing system. The analysis of the source code may be used to identify inputs to a computer program (e.g., identify one or more data sets accessed by the computer program), identify one or more transformations applied to the inputs and / or computations performed using the inputs as part of the computer program, and / or identify one or more outputs of the computer program. The computer program may include a data flow graph.

[0067] The data traceability module 318 may be configured to generate data traceability, such as the data traceability 100, 200A of Figures 1 and 2, respectively, by analyzing information obtained during execution of the at least one computer program. For example, one or more logs generated during execution of a computer program may be analyzed to identify inputs to the computer program, one or more transformations applied to the inputs and / or computations performed using the inputs within the computer program, and / or one or more outputs of the computer program.

[0068] The data traceability module 318 may be configured to generate a data traceability, such as the data traceability 100, 200A of Figures 1 and 2, respectively, using one or more data discovery processes. For example, a computer program implementing a data search may be configured to identify different data sets containing the same data values ​​and, based on this identification, determine that these data sets are related. For example, the computer program may be configured to determine that the same data table is stored in multiple different databases and, based on this, determine that the data elements of these tables are related. It should be noted that the data traceability module 318 may be configured to generate data traceability using any of the methods described above or any combination of two or more of the methods described above or other methods, aspects of the technology described herein not being limited in this regard.

[0069] The derived traceability module 320 may be configured to facilitate generation of a derived traceability by a system (e.g., a computer system). The derived traceability module 320 may indicate traceability relationships between components of the derived traceability.

[0070] The association module 322 may be configured to facilitate the generation of an association between the source data traceability (e.g., first, third, and fourth data traceabilities) and a (target) traceability derived from the source data traceability (e.g., second and fifth data traceabilities). To this end, the association module 322 may generate, for each of one or more components of the derived traceability, an association between the component of the derived traceability and one or more corresponding physical components of the (source) data traceability.

[0071] Returning to [Fig. 2], the computer-implemented method of the invention comprises: obtaining a first data traceability 200A specifying relationships between physical components of a plurality of physical components 202A, 204A; receiving an identification of a portion of the first data traceability (see, for example, the components in the dotted rectangle of [Fig. 2]); generating a second data traceability from the identified portion of the first data traceability, the second data traceability specifying relationships between second components of a plurality of second components, the second components of the plurality of second components being associated with at least some of the physical components of the identified portion of the first data traceability.

[0072] [Fig. 4] is a diagram illustrating an association between an example of a second data traceability 420 and an example of a first data traceability 400, in accordance with certain embodiments of the technology described herein. Each of the second data traceability 420 and the first data traceability 400 may be obtained in either of the ways described herein, using a computer-implemented method: obtaining a first data traceability 400 specifying relationships between physical components of a plurality of physical components 440, 442, 444, 446, 448, and 450; receiving an identification of a portion of the first data traceability; generating a second data traceability 420 from the identified portion of the first data traceability, the second data traceability specifying relationships between second components of a plurality of second components 403, 405, 406, 409, the second components of the plurality of second components being associated (cf. associations or links 452, 454, 456, 458) with at least some of the physical components of the identified part of the first data traceability.

[0073] The first data traceability 400 may comprise a plurality of physical components of one or more (physical) data processing systems, each physical component may comprise a data set and / or a transformation. The derived second data traceability 420 may comprise a plurality of second components, each second component being associated with and / or representing a corresponding physical component of the first data traceability 400.

[0074] Each of the physical components may include one or more physical data elements 422, 424, 426, 428, 430, 432, 434, 436. Each of the second components may include one or more second data elements 302, 304, 307, 308. The first data traceability 400 includes data sets 440, 442, 444, 446, 448, and 450. Each of the data sets may be stored in different computing systems and / or devices. Alternatively, two or more (or all) data sets may be stored in a single computing system and / or device. Examples of data sets are provided herein. Each data set may include one or more physical data elements. Data set 440 contains one or more physical data elements, including physical data element 422.Dataset 442 contains one or more physical data elements, including physical data element 424. Dataset 444 contains multiple physical data elements, including physical data elements 426, 428, and 430. Dataset 446 includes one or more physical data elements, including physical data element 432. Dataset 448 includes one or more physical data elements, including physical data element 434. Dataset 450 includes one or more physical data elements, including physical data element 436.

[0075] It should be appreciated that second and first data traceabilities may be more complex than the traceabilities illustrated in [Fig. 4] and, for example, may include many more second data elements, second components, physical data elements, and physical components. The traceability examples shown in [Fig. 4] are used for ease of discussion without being limiting.

[0076] In some embodiments, a first data traceability may include backward data traceability information for one or more physical data elements, which provides information about how the one or more physical data elements were obtained and / or generated. For example, in the illustrative example of [Fig. 4], the first data traceability 400 includes backward data traceability information for physical data element 422. As indicated by the shading of [Fig. 4], physical data element 422 was obtained from physical data element 424, which was obtained from multiple physical data elements, including physical data element 426, which was obtained from physical data element 432. Accordingly, physical data element 422 was obtained based, at least in part, on physical data element 432 of data set 446.

[0077] The second data traceability 420 includes second components 403, 405, 407, and 409. A second component may be any suitable container for encapsulating a representation of a corresponding physical component. The second component may be used to present the corresponding physical component to a user or system in terms that the user or system can understand or process. For example, a second component may be a report, a spreadsheet, a presentation comprising one or more slides, a text file, a Word document, and / or a PDF file. In some embodiments, the content of the second component may be generated by a computer, for example, by performing a query against a database (e.g., an SQL query) and placing the results of the query into the second component.As a specific, non-limiting example, a computer creating a second data lineage may perform a query against a database and insert a table resulting from the query into a spreadsheet file.

[0078] As shown in [Fig.4], the second component 403 comprises one or more second data elements, including the second data element 402. The second component 405 comprises one or more second data elements, including the second data element 404. The second component 407 comprises one or more second data elements, including the second data element 406. The second component 409 comprises one or more second data elements, including the second data element 408.

[0079] In some embodiments, a second data traceability may include upstream data traceability information, which provides information about how the one or more second data elements were obtained and / or generated, and / or downstream traceability information for one or more second data elements, which provides information indicating which other second data elements depend on the one or more second data elements. For example, in the illustrative example of [Fig. 4], the user-specified traceability 400 includes information about the backward data traceability for the second data element 402. As shown in [Fig. 4], the second data traceability 420 indicates that the second data element 402 was obtained from the second data element 404, which was obtained from the second data element 406, which was obtained from the second data element 408.

[0080] As indicated herein, in some embodiments, an association may be generated between a second data traceability and a first data traceability by generating an association or link between one or more physical data elements in the first data traceability and one or more corresponding second data elements in the second data traceability. An illustrative example of such an association is shown in [Fig.4], which shows that: (1) the second data element 402 is associated with the physical data element 422 via the association link 452; (2) the second data element 404 is associated with the physical data element 424 via the association link 454; (3) the second data element 406 is associated with the physical data element 426 via the association link 456; and (4) the second data element 408 is associated with the physical data element 432 via the association link 458. As shown in the example of [Fig. 4], an association between a second data traceability and a first data traceability may include information specifying one or more association links between data elements in the traceabilities. For example, the association shown in [Fig.4] indicates that there is no discrepancy between the second data traceability for the second data element 402 and the first data traceability for the physical data element 422, which is associated with the second data element 402. In this example, each physical data element in the first data traceability upstream of the physical data element 422 is associated with a corresponding second data element in the second data traceability upstream for the second data element 402. For example, the physical data element 432, which is used to obtain the physical data element 422, according to the first data traceability 400, is associated with the second data element 408, which is used to obtain the second data element 402, according to the second data traceability 420.

[0081] [Fig.5] is a graphical user interface graphically illustrating an association between an example of second data traceability 520 and an example of first data traceability 500, both obtained in accordance with certain embodiments of the technology described herein. Each of the physical components of the first data traceability 500 comprises a data set and / or a transformation. The physical component 530 comprising a data element is associated by association 515 with the second data element 510 of the second component 505 of the second data traceability 520. Multiple physical components, such as physical components 550, 560, 570, may be associated with the same application or process 540, as shown in the first data traceability 500 by associating physical components 550, 560, 570 with the same application or process 540. This is explained in more detail in the context of Figures 6 and 7.

[0082] Figures 6 and 7 illustrate exemplary relationships between data sets, data elements, transformations, transformation data elements, the application, and the system.

[0083] As indicated, a data traceability, such as the first and / or third data traceability, describing relationships between physical components of a plurality of physical components is a data structure. Each physical component comprises one or more data sets and / or one or more transformations (see [Fig. 2] and 6). In particular, the data traceability describes the lifecycle of the data sets and their data elements to which the transformations are applied when they are processed by a data processing system (see [Fig. 2] and 6). A data element is a unit of data in a data set. A nested (recursive) data element may contain other data elements. The data element may be one of the following subtypes of data elements (the basic object type): file field, column or row of a table, field of a message, etc.A data element is the most detailed physical element that identifies and describes data in an automated system. For example, in a file it is a field, in a database table it is a column. A dataset is a named collection of input or output data elements that are used by transformations or whose values ​​are produced by transformations. The dataset can be one of the following subtypes: File, Table, View, Interface, etc. A dataset is a collection of data elements. It represents a unit of storage in the automated system or a unit of exchange between systems.

[0084] As shown in [Fig.7], the dataset and the transformation, which is an executable, can be associated with the same application, which is executed by a system. The transformation can be applied to the dataset when the application is executed on the system.

[0085] Figures 8A and 8B illustrate an exemplary computer-implemented method and consisting of generating a second data traceability from the first data traceability and deepening the first data traceability by generating a third data traceability identical to a part of the first data traceability.

[0086] [Fig. 8A] is a graphical user interface graphically illustrating an example of a first data traceability 800, in accordance with certain embodiments of the technology described herein. The first data traceability 800 indicates relationships between a plurality of physical components such as physical components 802, 804, 807, 809, 810, and 811. The relationship is indicated by links, such as links 803, 805, 808. The links indicate which upstream component generates an output that is an input of a downstream component. For example, physical component 804 is downstream of physical component 802 and dependent on physical component 802, because the output of physical component 802 flows through the connection indicated by link 803 to the input of physical component 804. Both physical components 802 and 804 are associated with the same application or process 801.Each of the physical components of the first data traceability 800 includes a data set and / or a transformation, and the data set may include one or more data elements. The data set 811 includes a collection 812 of data elements. External data, such as reference data, may be incorporated into the first data traceability 800, as shown in [Fig. 8A]. The external data as well as data sets 807, 809, 810 and 811 are all associated with the same application 806.

[0087] As shown in Figures 8A-8C, the computer-implemented method includes converting the first data traceability 800 into a second data traceability 820: obtaining a first data traceability 800 specifying relationships between physical components of a plurality of physical components 802, 804, 807, 809, 810, and 811; receiving an identification of a portion 813-817 of the first data traceability; generating a second data trace 820 from the identified portion of the first data trace, the second data trace specifying relationships between second components of a plurality of second components 822, 824, 827, the second components of the plurality of second components being associated with at least some of the physical components of the identified portion of the first data trace.

[0088] As indicated by exemplary elements 813-817 of [Fig.8B], a user may select the portion of the first data trace 800 for which the second data trace 820 is generated by selecting starting and / or ending points in the first data trace 800, such as elements of specific data among data elements 813 to 817.

[0089] As shown in [Fig.8C], the graphical representation of the generated second data traceability 820 includes graphical indicators of second components 822, 824, 827 of the plurality of second components, such that the graphical indicators are arranged in the graphical representation of the portion of the generated second data traceability in a manner that conforms to the relationships among the plurality of second components of the first data traceability 800. As shown in [Fig.8C], the algorithm filtered out a large amount of information that is generally irrelevant from the perspective of the target environment. Using the bottom-up direction, the algorithm preserved the traceability from the data elements enriched with associated terms, to their sources, while retaining only the data elements that belong to the applications or processes 821, 825 and 826 marked in a previous step.

[0090] As shown in [Fig.8D], the user may select a link 823 associating two second components 822, 824 of the second data lineage 820. For example, when the user selects link 823, a pop-up window may be generated in which the user may activate an icon 829 (“drill down point-to-point technical lineage”) to submit instructions to drill down the first data lineage for the portion of the second data lineage 820 associated with the selected link 823.

[0091] Based on the selected link, the third data traceability 830 illustrated in [Fig.8E] is generated, the first data traceability 830 specifying relationships between the physical components 802, 804, 807, 809, 810 of the plurality of physical components that are associated with the second components 822, 824 whose graphical indicators are connected by the selected link 823. The third data traceability 830 may be a part of the first data traceability 800. In this way, the user can further explore a part of the first data traceability 800 through the generated second data traceability 820.

[0092] [Fig. 9] is a graphical user interface 900 graphically illustrating an example of a second data traceability, in accordance with certain embodiments of the technology described herein. As indicated in [Fig. 9], the method may also include: receiving a user selection of a portion of a generated second data traceability, and generating a view for more detailed information about the selected portion, such as the number of passes through a component, the number of transformations involved until a particular data set is generated, the number of inputs and / or outputs of a component, or other such information. The graphical interface 900 displays annotated information for a selected portion of the second data traceability. data.

[0093] In other terms, the inventors have developed techniques that efficiently and reliably maintain data traceability when generating a target data traceability. Means are provided to enable more accurate, reliable, and complete generation of the target data traceability while maintaining low consumption of computing resources. Discrepancies between the target data traceability and a source data traceability are eliminated, and changes between the source and target data traceabilities are identified. This is particularly useful for establishing data exchange between a (dynamic) source environment having the source data traceability and the target environment having the target data traceability. Further, means are provided to enable deepening the target data traceability to the physical components associated with the source data traceability.

[0094] The aspects and embodiments of the technology described above may be implemented in numerous ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. When implemented as software, the software code may be executed on any suitable processor or set of processors, whether in a single computer or distributed among multiple computers. These processors may be implemented as integrated circuits, with one or more processors in an integrated circuit component, including commercially available integrated circuit components known in the art by names such as CPU chips, GPU chips, microprocessor, microcontroller, or coprocessor.A processor may also be implemented in a custom circuit, such as an ASIC, or in a semi-custom circuit resulting from the configuration of a programmable logic device. Alternatively, a processor may be part of a larger circuit or semiconductor device, whether commercially available, semi-custom, or custom. For example, some commercially available microprocessors have multiple cores, so that a single or subset of these cores may constitute a processor. However, a processor may be implemented using circuitry in any suitable form factor. A processor and memory may be used instead of the means for implementing the method.

[0095] Further, it should be noted that a "computer," such as client 1, may take various forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. Furthermore, a computer may be integrated into a device that is not generally considered a computer but has appropriate processing capabilities, including a personal digital assistant (PDA), smartphone or other suitable portable or fixed electronic device.

[0096] A computer may also have one or more input and output devices. These devices may be used, among other things, to present a user interface. Examples of output devices that may be used to provide a user interface include printers or display screens for visual presentation of results and speakers or other sound-generating devices for audible presentation of results. Examples of input devices that may be used for a user interface include keyboards and pointing devices, such as mice, touch pads, and digitizing tablets. As another example, a computer may receive input information by voice recognition or in another audible format.

[0097] These computers may be interconnected by one or more networks in any suitable form, including a local area network or a wide area network, such as a corporate network or the Internet 306. These networks may be based on any suitable technology and may operate according to any suitable protocol and may include wireless networks, wired networks, or fiber optic networks.

[0098] Further, the various methods or processes described herein may be coded as software executable on one or more processors using one of a variety of operating systems or platforms. Further, such software may be written using a number of programming languages ​​and / or suitable programming or scripting tools, and may also be compiled into executable machine language code or byte code that is executed on a framework or virtual machine.

[0099] In this regard, the present invention may take the form of a computer-readable storage medium (or multiple computer-readable media) (e.g., computer memory, one or more floppy disks, compact discs (CDs), optical discs, digital video discs (DVDs), magnetic tapes, flash memories, circuit configurations in programmable gate arrays or other semiconductor devices, or any other tangible computer storage medium) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods that implement the various embodiments of the invention set forth above.As evident from the preceding examples, a computer-readable storage medium may retain information for a period of time sufficient to provide computer-executable instructions in a non-transitory form. Such a computer-readable storage medium or media may be trans . portable, such that the program(s) stored therein may be loaded onto one or more different computers or other processors to implement various aspects of the present invention, as discussed above. As used herein, the term "computer-readable storage medium," which is a preferred example of "computer-readable medium," encompasses only a non-transitory computer-readable medium that may be considered a manufactured product (i.e., an article of manufacture) or a machine. Alternatively or additionally, the invention may be in the form of a computer-readable medium other than a computer-readable storage medium, such as a propagating signal.

[0100] The terms "application", "algorithm", "service", "program code", "computer program" or "software" are used herein in a generic sense to refer to any type of computer code or computer-executable set of instructions that can be used to program a computer or other processor to implement various aspects of the present invention, as noted above. The data storage, receiver, subscriber, splitter, transformer and synthesizer may be implemented by computer software and / or hardware.Further, it should be appreciated that, in accordance with one aspect of this embodiment, one or more computer programs which, when executed, carry out methods of the present invention, need not reside on a single computer or processor, but may be modularly distributed among a number of different computers or processors to implement various aspects of the present invention.

[0101] Computer-executable instructions can take many forms, such as program modules, executed by one or more computers or other devices. In general, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules can be combined or distributed as desired in various embodiments.

[0102] Further, the data set may be stored on a computer-readable medium in any suitable form. For simplicity of illustration, data sets may be shown as having data set entries. However, any suitable mechanism may be used to establish locations within the data set for storing data, such as parameter values.

[0103] Various aspects of the present invention may be used alone, in combination, or in a variety of arrangements not specifically discussed in the embodiments described herein and are therefore not limited in their application. to the details and arrangement of the components set forth in the foregoing description or illustrated in the drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

[0104] The invention may also take the form of a method, an example of which has been provided. The acts performed in the method may be ordered in any suitable manner. Accordingly, it is possible to construct embodiments in which acts are performed in a different order than illustrated, which may include performing certain acts simultaneously, even if they are shown as sequential acts in illustrative embodiments.

[0105] Furthermore, certain actions are described as being performed by a “user” or “player”. It should be noted that a “user” or “player” is not necessarily a single person and, for example, actions attributable to a “user” or “player” may be performed by a team of people and / or a person in combination with computer-assisted tools or other mechanisms. It should be noted that a “user” or “player” is not necessarily an individual and may be a machine, for example in the form of a robot.

[0106] The use of ordinal terms such as "first", "second", "third", etc., to modify a claimed element does not in itself connote priority, anticipation or order of one claimed element over another or the temporal order in which acts of a method are performed, but are used merely as labels to distinguish a claimed element having a certain name from another element having the same name (without the use of the ordinal term) to distinguish the claimed elements. Furthermore, the phraseology and terminology used herein are for the purpose of description and should not be construed as limiting. The use of the terms "including", "comprising" or "having", "containing", "involving" and variations thereof herein is intended to encompass the elements listed below and their equivalents, as well as other elements.

[0107] Further, although advantages of the present invention are set forth, it should be appreciated that not all embodiments of the invention described herein include all of the described advantages. Some aspects and embodiments may not implement all of the features described as advantageous herein, and in some instances, one or more of the described features may be implemented to achieve other embodiments. Accordingly, this description and drawings are provided by way of example only.

[0108] The present invention may (alternatively) be described by one or more of the aspects numbered as follows: Aspects

[0109] 1. Computer-implemented method for converting a first traceability of data into a second data traceability, comprising: obtaining a first data traceability specifying relationships between physical components of a plurality of physical components; receiving an identification of a portion of the first data traceability; generating a second data traceability from the identified portion of the first data traceability, the second data traceability specifying relationships between second components of a plurality of second components, the second components of the plurality of second components being associated with at least some of the physical components of the identified portion of the first data traceability.

[0110] 2. Method according to aspect 1, identifying the part of the first traceability of data comprising an identification of a first physical component of the plurality of physical components that indicates a starting point for the second data traceability and an identification of a second physical component of the plurality of physical components that indicates an end point for the second data traceability.

[0111] 3. Method according to aspect 2, the generation of the second data traceability to starting from the identified portion of the first data traceability including generating the second data traceability by traversing the first data traceability from the starting point to the ending point.

[0112] 4. A method according to any one of the preceding aspects, identifying the part wherein the first data traceability is based on user input, preferably the user input comprising a user selection of one or more physical components of the plurality of physical components.

[0113] 5. A method according to any one of aspects 2 to 4, identifying the first physical component and / or the second physical component being based on user input, preferably the user input comprising a user selection of the first physical component and / or a user selection of the second physical component by the user.

[0114] 6. A method according to any one of the preceding aspects, each second component of the plurality of second components being associated by an identification link to a corresponding physical component of the at least one part of the physical components of the part of the first data traceability.

[0115] 7. A method according to any one of the preceding aspects, further comprising: the output of the second generated data traceability.

[0116] 8. Method according to aspect 7, the output of the second generated data traceability comprising displaying, in a graphical user interface, a graphical representation of a portion of the second generated data traceability.

[0117] 9. Method according to aspect 8, the graphical representation of the part of the second generated data traceability comprising graphical indicators of second components of the plurality of second components such that the graphical indicators are arranged in the graphical representation of the portion of the generated second data traceability in a manner that is consistent with the relationships among the plurality of second components.

[0118] 10. A method according to any one of the preceding aspects, generating the second data traceability being performed automatically by the computer.

[0119] 11. A method according to any one of the preceding aspects, further comprising: repeatedly performing the generation of the second data trace from the identified portion of the first data trace; comparing a newly generated second data trace with a previously generated second data trace of the repeated execution, including determining whether there is a difference between the newly generated second data trace and the previously generated second data trace; and output if there is a difference between the newly generated second data traceability and the previously generated second data traceability.

[0120] 12. The method of aspect 11, further comprising: based on the comparison, determining, based on one or more criteria, whether to retain or discard one or more determined differences between the newly generated second data traceability and the previously generated second data traceability for a second data traceability to be retained, preferably the one or more criteria including whether a number of differences exceeds a threshold value.

[0121] 13. A method according to aspect 11 or 12, further comprising: based on the comparison, determining whether to select the newly generated second data trace or the previously generated second data trace as the second data trace to be retained.

[0122] 14. Method according to one of the preceding aspects, further comprising: in response to detecting a change in the first data traceability, updating the generated second data traceability, or generating a new second data traceability, based on the detected change.

[0123] 15. A method according to any one of the preceding aspects, each component physics of the plurality of physical components comprising a data set and / or a transformation.

[0124] 16. Method according to any one of the preceding aspects, the first traceability of data being generated at least in part by performing: (a) analyzing the source code of at least one computer program configured to access at least a portion of the plurality of physical components to identify one or more physical components accessed by the computer program; and / or (b) analyzing information obtained during the execution of the at least one computer program, including analyzing one or more logs generated during the execution of the computer program to identify one or more physical components accessed by the computer program.

[0125] 17. Method according to aspect 16, the physical component(s) to which the computer program accesses comprising one or more inputs of the computer program, and / or one or more transformations applied to the inputs of the computer program, and / or one or more outputs of the computer program.

[0126] 18. Method according to one of the preceding aspects, further comprising: before generating the second data traceability from the identified portion of the first data traceability, preprocessing the physical components of the identified portion of the first data traceability, the generation of the second data traceability from the identified portion of the first data traceability being performed based on a result of the preprocessing.

[0127] 19. A method according to aspect 18, the pretreatment comprising: identifying a control flow and / or internal data trace associated with one or more of the physical components of the part of the first data trace and excluding the control flow and / or internal data trace when generating the second data trace.

[0128] 20. A method according to any one of aspects 9 to 19, the graphical representation of the portion of the generated second data traceability comprising links connecting the graphical indicators of the second components of the plurality of second components, the method further comprising: receiving a user selection for one of the links; and based on the selected link, generating a third data traceability specifying relationships between the physical components of the plurality of physical components that are associated with the second components whose graphical indicators are connected by the selected link.

[0129] 21. A method according to any one of the preceding aspects, further comprising: obtaining a fourth data traceability specifying relationships between physical components of a plurality of physical components; receiving an identification of a portion of the fourth data trace; generating a fifth data trace from the identified portion of the fourth data trace, the fifth data trace specifying relationships between fifth components of a plurality of fifth components, the fifth components of the plurality of fifth components being associated with at least some of the physical components of the identified portion of the fourth data trace; comparing the fifth data traceability with the second data traceability, including determining whether there is a difference between the fifth data traceability and the second data traceability; and the output emission if there is a difference between the fifth data traceability and the second data traceability.

[0130] 22. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to implement the method according to any of the preceding aspects.

[0131] 23. Data processing system comprising means for implementing the method according to any one of aspects 1 to 21.

[0132] 24. Computer program comprising instructions which, when the program computer is executed by a computer, causing the computer to implement the method according to any one of aspects 1 to 21.

[0133] The present invention is defined by the following embodiments. Some of the preferred embodiments are described in the following dependent claims as well as in the above description and in the accompanying figures. Having described several aspects of the embodiments of the present invention, it should be appreciated that various alterations, modifications and improvements of these aspects are possible within the scope of the invention as defined by the appended claims.

Claims

Claims

1. A computer-implemented method for converting a first data traceability into a second data traceability, comprising: obtaining a first data traceability specifying relationships between physical components of a plurality of physical components; receiving an identification of a portion of the first data traceability; generating a second data traceability from the identified portion of the first data traceability, the second data traceability specifying relationships between second components of a plurality of second components, the second components of the plurality of second components being associated with at least some of the physical components of the identified portion of the first data traceability.

2. The method of claim 1, wherein the identification of the portion of the first data traceability comprises an identification of a first physical component of the plurality of physical components that indicates a starting point for the second data traceability and an identification of a second physical component of the plurality of physical components that indicates an ending point for the second data traceability.

3. The method of claim 2, wherein generating the second data trace from the identified portion of the first data trace comprises generating the second data trace by traversing the first data trace from the starting point to the ending point.

4. A method according to any preceding claim, wherein the identification of the part of the first data traceability is based on user input, preferably the user input comprising a user selection of the first physical component and / or a user selection of the second physical component by the user.

5. A method according to any preceding claim, each second component of the plurality of second components being associated by an identification link with a corresponding physical component. responsible for at least part of the physical components of the part of the first data traceability.

6. A method according to any preceding claim, further comprising: outputting the generated second data traceability.

7. The method of claim 6, outputting the generated second data traceability comprising displaying, in a graphical user interface, a graphical representation of a portion of the generated second data traceability, the graphical representation of the portion of the generated second data traceability comprising graphical indicators of second components of the plurality of second components, such that the graphical indicators are arranged in the graphical representation of the portion of the generated second data traceability in a manner that is consistent with relationships among the plurality of second components.

8. The method of claim 7, wherein the graphical representation of the portion of the generated second data traceability comprises links connecting the graphical indicators of the second components of the plurality of second components, the method further comprising: receiving a user selection for one of the links; and based on the selected link, generating a third data traceability specifying relationships between the physical components of the plurality of physical components that are associated with the second components whose graphical indicators are connected by the selected link.

9. A method according to any preceding claim, wherein the generation of the second data traceability is performed automatically by the computer.

10. A method according to any preceding claim, further comprising: repeatedly performing the generation of the second data traceability from the identified portion of the first data traceability; comparing a newly generated second data traceability with a previously generated second data traceability generated from the repeated execution, including determining whether there is a difference between the newly generated second data trace and the previously generated second data trace; and outputting whether there is a difference between the newly generated second data trace and the previously generated second data trace.

11. The method of claim 10, further comprising: based on the comparison, determining, based on one or more criteria, whether to retain or discard one or more determined differences between the newly generated second data traceability and the previously generated second data traceability for a second data traceability to be retained, preferably the one or more criteria comprising a number of differences exceeding a threshold value.

12. The method of claim 10 or 11, further comprising: based on the comparison, determining whether to select the newly generated second data traceability or the previously generated second data traceability as the second data traceability to be retained.

13. The method of any preceding claim, further comprising: in response to detecting a change in the first data traceability, updating the generated second data traceability, or generating a new second data traceability, based on the detected change.

14. A method according to any preceding claim, wherein the first data traceability is generated at least in part by performing: (a) analyzing the source code of at least one computer program configured to access at least a portion of the plurality of physical components to identify one or more physical components accessed by the computer program; and / or (b) analyzing information obtained during execution of the at least one computer program, comprising analyzing one or more logs generated during execution of the computer program to identify one or more physical components accessed by the computer program, preferably the physical component(s) accessed by the computer program comprising one or more inputs to the computer program, and / or one or more transformations applied to the inputs to the computer program, and / or one or more outputs of the computer program.

15. A method according to any preceding claim, further comprising: before generating the second data traceability from the identified part of the first data traceability, pre-processing the physical components of the identified part of the first data traceability, the generation of the second data traceability from the identified part of the first data traceability being performed based on a result of the pre-processing, preferably pretreatment including: identifying a control flow and / or internal data trace associated with one or more of the physical components of the part of the first data trace and excluding the control flow and / or internal data trace when generating the second data trace.

16. A method according to any preceding claim, further comprising: obtaining a fourth data traceability specifying relationships between physical components of a plurality of physical components; receipt of an identification of a part of the fourth data traceability; generating a fifth data trace from the identified portion of the fourth data trace, the fifth data trace specifying relationships between fifth components of a plurality of fifth components, the fifth components of the plurality of fifth components being associated with at least some of the physical components of the identified portion of the fourth data trace; comparing the fifth data traceability with the second data traceability, including determining whether there is a difference between the fifth data traceability and the second data traceability; and the output emission if there is a difference between the fifth data traceability and the second data traceability.

17. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of any preceding claim.

18. Data processing system comprising means for implementing the method according to any one of claims 1 to 16.

19. A computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method according to any one of claims 1 to 16.

Images