Data Quality Analysis

Abstract

We propose data quality analysis techniques. [Solution] The method includes receiving information indicative of an output dataset to be generated by a data processing system, identifying one or more upstream datasets on which the output dataset relies based on data lineage information related to the output dataset, and analyzing one or more of the identified one or more upstream datasets on which the output dataset relies.

Description

[Technical field]

[0001] background This specification relates to data quality analysis. The data quality of a dataset is an indication of whether the data records in the dataset have errors. Often, if errors occur during the processing of a dataset, the data quality of the dataset is low. Summary of the Invention [Means for solving the problem]

[0002] summary In a general aspect, a method includes receiving information indicative of an output dataset generated by a data processing system, identifying one or more upstream datasets on which the output dataset relies based on data lineage information related to the output dataset, and analyzing one or more of the identified one or more upstream datasets on which the output dataset relies. The analyzing includes, for each particular upstream dataset of the one or more upstream datasets, applying one or more of (i) a first rule indicating an acceptable deviation between a profile of the particular upstream dataset and a reference profile for the particular upstream dataset, and (ii) a second rule indicating one or more acceptable or prohibited values ​​for each of one or more data elements in the particular upstream dataset, and selecting one or more of the upstream datasets based on a result of applying the one or more rules. The method includes outputting information related to the selected one or more upstream datasets.

[0003] Implementations may include one or more of the following features.

[0004] One or more of the first rule and the second rule are automatically generated. The first rule is automatically generated based on an automated analysis of a historical profile of the particular upstream data set. The reference profile is based on a historical average profile for the particular upstream data set. The second rule is automatically generated based on an automated analysis of historical values ​​for one or more data elements in the particular upstream data set. The allowed or prohibited values ​​are determined based on the automated analysis.

[0005] One or more of the first rule and the second rule are specified by a user.

[0006] The method includes receiving, through a user interface, one or more designations of a first rule and a second rule.

[0007] The data lineage information indicates one or more datasets on which the output dataset is based, one or more datasets on which the output dataset is based, or both.

[0008] Analyzing each of the one or more datasets to identify a subset of the dataset includes determining which of the one or more datasets have errors or possible errors, and the method includes selecting the datasets that have the errors or possible errors for the subset.

[0009] Analyzing each of the one or more datasets to identify a subset of the datasets includes identifying a particular dataset where a deviation between a profile of the particular dataset and a reference profile for the particular dataset exceeds an allowable deviation indicated by a corresponding first rule, and the method includes selecting the particular dataset for the subset.

[0010] Analyzing each of the one or more datasets to identify a subset of the datasets includes identifying particular datasets having data elements having values ​​that do not meet the allowed or prohibited values ​​indicated by the corresponding second rule, and the method includes selecting the particular datasets for the subset.

[0011] The method includes identifying a data element in the output dataset, where identifying one or more datasets on which the output dataset is based includes identifying datasets that influence the identified data element in the output dataset, where identifying a data element in the output dataset includes identifying a data element having an error or a possible error.

[0012] The method includes generating one or more profiles of the upstream dataset. Generating a profile for a particular dataset includes generating a new profile for the particular dataset when a new version of the particular dataset is received.

[0013] A baseline profile for a particular data set is derived from one or more past profiles for the particular data set.

[0014] Outputting information relating to the subset of the datasets includes outputting an identifier for each of the datasets in the subset.

[0015] Outputting information associated with the subset of the data sets includes outputting an indication of an error or possible error associated with each of the subset data sets.

[0016] The method includes displaying a representation of the data processing system on a user interface, and outputting information related to the subset of the datasets includes displaying information related to the particular dataset of the subset near a representation of the particular dataset of the subset of the datasets. The displayed information related to the particular dataset of the subset includes a value indicative of a deviation between a profile of the particular dataset and a reference profile for the particular dataset. The displayed information related to the particular dataset of the subset includes a value representing a number of data elements in the particular dataset that do not meet an acceptable or prohibited value indicated by a corresponding second rule. The method includes displaying an information bubble or pop-up window showing information about the subset of the dataset.

[0017] The method includes providing a user interface to allow a user to add a rule, modify a rule, or remove a rule.

[0018] The datasets include one or more source datasets and one or more reference datasets, where the source datasets include data elements to be processed by the data processing system and the reference datasets include reference values ​​referenced by the data processing system when processing data elements in the source datasets, the reference datasets include data related to a business entity associated with the data processing system, and the source datasets include data related to customers of the business entity.

[0019] The data processing system includes transformation elements, and the method includes identifying one or more transformation elements that affect the output data set based on the data lineage information. The method includes determining whether any one or more of the transformation elements have an error or a possible error. The method includes determining whether a particular data processing element has an error or a possible error based on an implementation date associated with the particular transformation element.

[0020] In a general aspect, a non-transitory computer-readable medium stores instructions for causing a computing system to receive information indicative of an output dataset generated by a data processing system, identify one or more upstream datasets on which the output dataset is based based on data lineage information relating to the output dataset, and analyze one or more of the identified one or more upstream datasets on which the output dataset is based. The analyzing includes applying, for each particular upstream dataset of the one or more upstream datasets, one or more of (i) a first rule indicating an acceptable deviation between a profile of the particular upstream dataset and a reference profile for the particular upstream dataset, and (ii) a second rule indicating one or more acceptable or prohibited values ​​for each of one or more data elements in the particular upstream dataset, and selecting one or more of the upstream datasets based on a result of applying the one or more rules. The instructions cause the computing system to output information related to the selected one or more upstream datasets.

[0021] In a general aspect, a computing system includes a processor coupled to a memory. The processor and memory are configured to receive information indicative of an output dataset generated by the data processing system, identify one or more upstream datasets on which the output dataset is based based on data lineage information related to the output dataset, and analyze one or more of the identified one or more upstream datasets on which the output dataset is based. The analyzing includes applying, for each particular upstream dataset of the one or more upstream datasets, one or more of (i) a first rule indicating an acceptable deviation between a profile of the particular upstream dataset and a reference profile for the particular upstream dataset, and (ii) a second rule indicating one or more acceptable or prohibited values ​​for each of the one or more data elements in the particular upstream dataset, and selecting one or more of the upstream datasets based on a result of applying the one or more rules. The processor and memory are configured to output information related to the selected one or more upstream datasets.

[0022] In a general aspect, a computing system includes means for receiving information indicative of an output dataset generated by a data processing system, means for identifying one or more upstream datasets on which the output dataset is based based on data lineage information relating to the output dataset, and means for analyzing one or more of the identified one or more upstream datasets on which the output dataset is based. The analyzing includes applying, for each particular upstream dataset of the one or more upstream datasets, one or more of (i) a first rule indicating an acceptable deviation between a profile of the particular upstream dataset and a reference profile for the particular upstream dataset, and (ii) a second rule indicating one or more acceptable or prohibited values ​​for each of the one or more data elements in the particular upstream dataset, and selecting one or more of the upstream datasets based on a result of applying the one or more rules. The computing system includes means for outputting information related to the selected one or more upstream datasets.

[0023] In a general aspect, a method includes, upon identifying an error or potential error in a data element of a downstream dataset of a data processing system, automatically identifying one or more upstream datasets that affect the data element based on data lineage information related to the downstream dataset, determining which of the upstream datasets have the error or potential error including analyzing current and reference profiles of each of the identified upstream datasets, and outputting information associated with each upstream dataset determined to have an error or be likely to have an error.

[0024] Aspects may include one or more of the following advantages.

[0025] The techniques described herein may help users, such as data analysts or application developers, to quickly identify the root cause of data quality issues. For example, reference data in a data processing system may be frequently updated but not necessarily thoroughly inspected before deployment. Errors in the reference data may cause data quality issues in downstream data that is processed using the reference data. Analyzing the root cause of data quality issues in downstream datasets may help identify reference data or other upstream data that have data quality issues that may be affecting the data quality of downstream datasets. Notifying users of potential data quality issues may help users to proactively manage data processing.

[0026] Other features and advantages of the invention will become apparent from the following description and from the claims. [Brief description of the drawings]

[0027] [Figure 1] FIG. [Diagram 2] FIG. [Figure 3A] FIG. [Figure 3B] FIG. [Figure 4] FIG. [Diagram 5] FIG. [Figure 6] FIG. [Figure 7] FIG. 1 is a diagram of a data processing system. [Figure 8A] FIG. 1 is a diagram of a data processing system. [Figure 8B] FIG. 1 is a diagram of a data processing system. [Figure 8C] This is an example of a record. [Figure 9A] FIG. 1 is a diagram of a data processing system. [Figure 9B] FIG. 1 is a diagram of a data processing system. [Figure 10A] FIG. 1 is a diagram of a data processing system. [Figure 10B] This is an example of a record. [Figure 11] 1 is a flowchart. [Figure 12] 1 is a flowchart. [Figure 13] 1 is a flowchart. [Figure 14] 1 is a flowchart. [Figure 15] 1 is a flowchart. [Figure 16] FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] explanation Described herein is an approach for identifying the root cause of a data quality problem based on an analysis of a data series. When a data quality problem is identified in a downstream dataset, the upstream dataset and the upstream transformation elements (sometimes referred to as upstream data series elements) from which the downstream dataset is derived are identified. The quality of each upstream data series element is evaluated to identify one or more of the upstream data series elements that may themselves have data quality problems that contributed to the data quality problem in the downstream dataset. In some examples, a profile characterizing each upstream dataset is compared to a reference profile, such as a historical average profile for that dataset, to determine whether the dataset has a data quality problem. In some examples, a value in a field of the upstream dataset is compared to one or more allowed or prohibited values ​​for that field to determine whether the dataset has a data quality problem.

[0029] Data lineage is information describing the life cycle of a data record processed by a data processing system. Data lineage information for a given dataset includes identifiers of one or more upstream datasets on which the given dataset relies, one or more downstream datasets on which the given dataset relies, and one or more transformations that process data to generate the given dataset. A downstream dataset that relies on an upstream dataset means that the processing of the upstream dataset by the data processing system directly or indirectly results in the generation of the downstream dataset. The generated downstream dataset can be a dataset that is output from the data processing system (sometimes called an output dataset) or a dataset that is further processed by the data processing system (sometimes called an intermediate dataset). An upstream dataset can be a dataset that is input into the data processing system (sometimes called an input dataset or a reference dataset) or a dataset that has already been processed by the data processing system (sometimes called an intermediate dataset). A transformation is a data processing operation that is applied to an upstream dataset to result in a downstream dataset that is provided to a data sink. A data lineage diagram is a graphical representation of data lineage elements in a data processing system.

[0030] FIG. 1 is an example of a data lineage diagram 100 for output data 110 generated by a data processing system. In the example of FIG. 1, the data processing system receives two sets of source data 102, 104. The source data can be data records stored in or received from a file, such as a flat file for storing data in a computing system, a database, such as a relational or object database, a queue, or another repository. For example, the source data 102 can be data records of credit card transactions in the United States stored in a file "US_feed.dat". Each data record can include values ​​for each of one or more fields, such as attributes defined in a record structure or columns in a database table. The source data 102, 104 can be received and processed in batches (e.g., data from a file or database processed hourly, daily, weekly, monthly, quarterly, yearly, or at another interval). The source data 102, 104 can be received as a stream and processed continuously, or can be buffered, for example, by a queue, and processed when data is available and system resources permit.

[0031] The source data 102 is processed by the transformation element 106, which acts on the source data 102, for example, to change the source data 102 in some way. The transformation element can be an executable program capable of manipulating data, such as a java program running in a virtual machine, an executable file, a data flow graph, or another type of executable program. For example, the transformation element 106 can be an executable file named "TransformA.exe". In one embodiment, the transformation element 106 can be a filter component that filters out undesired data records from the source data 102, such as data records having an incorrect format. The transformation element 106 processes the source data 102 taking into account the reference data 120 to result in intermediate data 112. The reference data is data used by the transformation element to enable the transformation element to process the data. For example, the reference data that enables a mapping operation includes one or more fields having values ​​that correspond to values ​​in one or more fields in the data being processed. The intermediate data 112 can be stored in a file, database, queue, or another repository for storing data in a computing system.

[0032] The transformation component 108 processes the set of source data 104 taking into account the reference data 122 to result in intermediate data 114. The intermediate data 114 may be stored in a file, database, queue, or another repository for storing data in a computing system.

[0033] The intermediate data 112, 114 are processed together by a transformation element 116 using reference data 118. In one example, the transformation element 116 is a mapping operation, where the reference data 118 includes data records indicating state values ​​and corresponding region values. When the intermediate data 112, 114 are processed by the transformation element 116, the values ​​in the state field in each data record in the intermediate data 112, 114 are mapped to the corresponding region indicated in the reference data 118. In one example, the reference data 118 includes business data indicating a corporate entity, a corresponding department identifier, an owner name, and a location. When the intermediate data 112, 114 are processed by the transformation element 116, each data record is assigned to a corporate entity based on the mapping enabled by the reference data set. The reference data 118 can be used to process multiple data sets and is not altered by the processing. The reference data 118 can be updated periodically or as needed by a user.

[0034] The transformation element 116 outputs output data 110 that is stored in a file, database, queue, or another repository for storing data in a computing system. The output data 110 may be further processed by other transformation elements, for example in the same data processing system or in a different data processing system, or may be stored for future analysis.

[0035] In the example of Figure 1, the data lineage of the output data 110 is illustrated with respect to data lineage elements within a single data processing system. In some examples, the data lineage of a data set may be tracked by multiple data processing systems. For example, source data may be initially processed by a first data processing system to result in output data X. A second data processing system reads the output data X from the first data processing system and processes the output data X to generate output data Y. The output data Y is processed by a third data processing system to generate output data Z. The data lineage of the output data Z includes the initial source data, the transformations included in each of the three data processing systems, and any reference data used during processing by any of the three data processing systems.

[0036] In some cases, the output data may be generated by a more complex data processing system, as illustrated in the example end-to-end data lineage diagram 200A for the target element 206A. In the data lineage diagram 200A, the connection between the data element 202A and the transformation element 204A is illustrated. The data element 202A may represent a data set, a table in a data set, a column in a table, a field in a file, or other data. An example of a transformation element is an executable file element that describes how a single output of the data element is produced. The root cause of a potential data quality problem in the target element 206A (or in another data element 202A) may be traced within the data processing system of FIG. 2. Further description of FIG. 2 may be found in U.S. Patent Application Publication No. 2010 / 0138431, the entire contents of which are incorporated herein by reference.

[0037] Information shown in a data lineage diagram, such as the data lineage diagram of Figure 1 or Figure 2, indicates which upstream data sources, data sinks, or transformations affect downstream data. For example, the data lineage diagram 100 of Figure 1 reveals that output data 110 is influenced by source data 102, 104, reference data 118, 120, 122, and transformation elements 106, 108, 116.

[0038] Understanding the lineage of downstream datasets (such as output data 110) can be useful in identifying root causes of possible data quality issues in the downstream data. Root causes of data quality issues refer to identifying upstream systems, operations, or datasets that are at least partially responsible for data quality issues in the downstream data. The cause of data quality issues in downstream datasets, such as in output data 110, can be poor quality source data, poor quality reference data, errors in transformation elements in the upstream lineage of the set of output data 110, or a combination of any two or more of these. Tracking the quality or state of data lineage elements can provide information that can be used to evaluate possible root causes of poor quality output data.

[0039] The data quality of a dataset generally refers to whether the dataset has expected characteristics. Poor data quality may be manifested in a dataset that does not behave as expected, e.g., falls outside statistical norms, returns lookup failures or another type of behavior in response to a standard query. The quality of a dataset may be characterized based on a profile of some or all of the data records in the dataset, based on the values ​​in each of one or more fields of a particular data record, or both, as described below.

[0040] Poor data quality in a downstream data set (e.g., output data 110) can be traced back to any of a variety of factors in the upstream data lineage of the output data. One possible cause of low-quality output data can be low-quality source data, low-quality reference data, or both. For example, a source data set may have been corrupted or interrupted during transmission, may be the wrong data set, may have missing data, or may have another problem. A reference data set may have been subject to errors in recent updates to the reference data set, may be corrupted, may be the wrong data set, or may have another problem. Another possible cause of low-quality output data can be a problem with a transformation element in the upstream data lineage of the output data. For example, if software implementing the transformation element has recently been updated to a new version, the transformation element may no longer perform the desired processing, for example if the updated software has errors or is corrupted. The source data, reference data, and transformation elements in the data lineage of the set of output data 110 may be monitored to facilitate preemptive identification of potential data quality issues that may arise in the output data set, subsequent tracing of the root cause of data quality issues that arise in the output data set, or both.

[0041] Monitoring and analyzing the source data and reference data may assist a user in diagnosing one or more possible causes of low-quality output data. For example, if a low-quality output data set is generated, analyzing the source data or reference data in the data lineage of the low-quality output data set may indicate whether a given source data set or reference data set is itself of low quality and therefore a possible contributor to the low-quality output data. Monitoring the source data and reference data may also preemptively identify low-quality source data or reference data that, if processed, may cause data quality issues in downstream output data.

[0042] Figures 3A and 3B illustrate an approach for tracing the root cause of unknown or potential data quality problems in a set of output data 110 having the data sequence shown in Figure 1. With reference to Figure 3A, prior to processing input data (e.g., source data 102, 104 of Figure 1), the quality of reference data 118, 120, 122 is characterized by quality factors 154, 156, 158, respectively. In some examples, the quality of the reference data may be characterized when the reference data sets are updated, at scheduled times (e.g., periodically or when a reference data update is scheduled), prior to processing each input data set, or at other times.

[0043] To characterize the quality of a data set, a quality element calculates a profile (sometimes called a statistical study) of the fields in the data set. A profile of a set of data records is a summary, e.g., field-by-field, of the data values ​​in the data records. The profile may include statistics characterizing the data values ​​in each of one or more fields in each of at least a portion of the data records in the set, a histogram of values, maximum values, minimum values, average (e.g., mean or median) values, standard deviation from the mean, number of distinct values, a sample of the most and least frequent values ​​in one or more fields (e.g., for significant data elements per data set), or other statistics. In some examples, a profile may include processed information characterizing the data values ​​in each of one or more fields in the data records. For example, a profile may include classifications of values ​​within a field (e.g., categorizing data within an income data field into high, medium, or low categories), indications of relationships between data fields within individual data records (e.g., an indication that a state data field and a ZIP data field are not unrelated), relationships between data records (e.g., an indication that data records having a common value in a customer_identifier field are related), or other information that characterizes the data within a set of data records.

[0044] The quality element then applies one or more rules to identify any actual or potential data quality issues in the data set. As discussed further below, the rules may be specified by a user and may indicate acceptable or prohibited characteristics of a profile. In one specific example, if the reference data set includes a field that lists abbreviations for U.S. states, an example rule may indicate that if the number of distinct values ​​in the field is greater than 50, a data quality issue should be identified. In some examples, the rules may be based on a historical profile of the data set, for example, a historical average value. If a data quality issue is not identified in the data set, the profile of the data set may be used to update the rules, for example, to update the historical average value. If the reference data set is identified as having an actual or potential data quality issue, processing may be paused until the data quality issue is addressed.

[0045] 3B, the quality of the source data 102, 104 is characterized by quality elements 150, 152, respectively. The quality elements 150, 152 may characterize the data quality of each of the source data 102, 104 as the data is received into the data processing system, prior to scheduled processing of the respective source data, or at other times. If a source data set is identified as having known or potential data quality issues, information about the data quality issues may be output, for example, to warn a user or to store in a data store for future reference. For example, as each quality element 150, 152 reads data from the corresponding data set, the quality element 150, 152 calculates a profile of the data set.

[0046] In one example, to calculate a profile of source data 102, the quality element 150 may calculate the sum of all of the values ​​in a trade_volume field in the source data 102. A rule for the source data 102 may compare all of the sums of the values ​​in the trade_volume field to the average and standard deviation of that sum over the past 30 runs, and may indicate that if all of the sums of the values ​​in the trade_volume field of the source data 102 are outside one standard deviation from the average value of the sums, a data quality issue should be identified.

[0047] In some examples, rules used to characterize the quality of a dataset may indicate acceptable or prohibited features of a profile of data records in the dataset. A profile feature may be a value or a range of values. A rule indicating an acceptable feature of a profile is satisfied if the profile contains an acceptable feature. An example of an acceptable feature for a field may be the maximum and minimum acceptable values ​​for that field, and the rule is satisfied if the average value of the field is between the maximum and minimum acceptable values. A rule indicating a prohibited feature of a profile is satisfied as long as the profile does not contain the prohibited feature. An example of a prohibited feature for a field may be a list of prohibited values ​​for that field, and the rule is not satisfied if the field contains any of the prohibited values.

[0048] The rules characterizing the profile may indicate an acceptable deviation between a profile of a field of a particular dataset and a reference profile for the field of the dataset. A deviation between the profile of a dataset and a reference profile for the dataset that exceeds the acceptable deviation indicated by the corresponding rule may be an indication of a data quality problem in the dataset and therefore that the dataset is a possible root cause of existing or potential data quality problems in downstream datasets. In some examples, the acceptable deviation may be specified as a range of values, such as a maximum acceptable value and a minimum acceptable value. In some examples, the acceptable deviation may be specified as a standard deviation from a single value, which may be an average value (e.g., the mean or median of values ​​in past datasets).

[0049] In some examples, rules used to characterize the quality of a data set may indicate allowable or forbidden characteristics of values ​​in each of one or more fields of a data record, such as based on the validity of values ​​in the field. A rule indicating allowable characteristics for a field is satisfied if a value in the field conforms to the allowable characteristics. A rule indicating forbidden characteristics for a field is satisfied as long as a value in the field does not conform to the forbidden characteristics. Values ​​that satisfy the rule may be referred to as valid values, and values ​​that do not satisfy the rule may be referred to as invalid values. Various characteristics of values ​​in a field may be indicated as allowable or forbidden characteristics by rules. An example of a rule may indicate allowable or forbidden characteristics of the contents of a field, such as an allowable or forbidden range of values, a maximum allowable value, a minimum allowable value, or a list of one or more specific values ​​that are allowed or forbidden. For example, a birth_year field with a value below 1900 or above 2016 may be considered invalid. An example of a rule may indicate allowable or forbidden characteristics of the data type of the field. An example of a rule may indicate whether the absence of a value (or a null) in a particular field is allowed or forbidden. For example, a last_name field that contains a string value (e.g., "Smith") may be considered valid, while a last_name field that is blank or contains a numeric value may be considered invalid. An example of a rule may indicate allowed or prohibited relationships between two or more fields within the same data record. For example, a rule may specify a list of values ​​for the ZIP field that correspond to each possible value of the state field, and may specify that any combination of values ​​of the ZIP field and values ​​of the state field that are not supported by the list is invalid.

[0050] In some examples, rules can be generated based on an automatic analysis of historical data. This type of rule is referred to as an automatically generated rule. The automatically generated rule can indicate acceptable or prohibited characteristics of the profile of the data records in the dataset. For example, the automatically generated rule for a profile can indicate an acceptable deviation between a profile of a field of a particular dataset and an automatically determined historical reference profile of the field of the dataset. The historical reference profile of a dataset can be based on historical data, for example, the historical reference profile can be a profile of the same dataset on the previous day, an average profile of the same dataset over the past days (e.g., over the last week or month), an average profile of the same dataset over its lifetime. More broadly, the reference profile can hold a wide variety of reference information for utilizing various types of statistical analysis. For example, the reference profile can include information regarding the standard deviation or other indication of the value distribution. In the following examples, and without limiting the general concept of the present application, it is assumed that the reference profile includes the numerical average and possibly also the standard deviation of the past dataset.

[0051] An automated generation rule may indicate automatically determined allowable or forbidden characteristics of values ​​in a field of a data record. In one example, an automated generation rule for a field may indicate a maximum or minimum allowable value for the field based on an analysis of historical maximum or minimum values ​​for the field. In one example, an automated generation rule for a field may indicate a list of allowable values ​​for the field based on an analysis of past values ​​for the field. In some examples, an automated generation rule is specified for all fields of a data set. In some examples, rules are specified for a subset of fields. Fields for which rules are specified may be identified automatically, for example, based on an analysis of the data records. For example, any field in a set of data records that generally has a small number of distinct values ​​(sometimes referred to as a low cardinality field) may be identified as a field for which an automated generation rule can be generated.

[0052] In some examples, machine learning techniques are used to generate the auto-generation rules. For example, data can be analyzed over a learning period to identify historical averages or expectations before generating the rules. The learning period can be a specified period of time or can be the time until the averages or expectations converge to a stable value.

[0053] In some cases, rules may be specified by a user. This type of rule is referred to as a user-specified rule. A user-specified rule may specify allowed or prohibited characteristics of a profile of a field of a particular dataset, allowed or prohibited characteristics of values ​​in each of one or more fields of data records in the dataset, or both. A user may specify rules based on, for example, his or her own understanding of the expected characteristics of data records to be processed by the system. In some cases, a user-specified rule may be assigned a default value that may be modified by the user.

[0054] In one specific example, the source data is credit card transaction records for transactions occurring within the United States. The source data is streaming data that is processed in hourly increments. Based on their knowledge of the source data and the operations to perform when processing the credit card transaction records, a user may identify a transaction identifier field, a card identifier field, a state field, a date field, and an amount field as important data elements to profile.

[0055] In a specific example where the source data are credit card transaction records, the user may know that there are only 50 allowable values ​​for the state field. The user may create a rule that triggers the use of a warning flag if the source data set profile identifies more than 50 values ​​in the state field, regardless of the standard deviation of the source data set profile relative to the norm. The user may also know that only credit card transaction records in the source data set are for transactions completed on the same day as the processing. The user may create a rule that triggers the sending of a warning message if any source data records have a date that does not match the processing date.

[0056] 4, in some examples, a user can specify one or more rules through a user interface 400. One example of a user interface 400 includes a number of rows 402 and a number of columns 404. Each row 402 is associated with a field 406 of a data record in a data set, and each column 404 is associated with a rule 408. The user interface 400 allows a user to specify a rule for one or more fields 406 or to accept pre-populated default rules for the fields. Further description of the user interface 400 can be found in U.S. Patent Application No. 13 / 653,995, filed October 17, 2012, the entire contents of which are incorporated herein by reference. Other implementations of the user interface 400 are possible.

[0057] In some examples, when a potential data quality problem is detected in a dataset, such as in a new version of a reference dataset or in a source dataset, an identifier for the dataset having the potential data quality problem is placed on a list of root cause datasets stored in the database. If a data quality problem is later detected in a set of output data 110, the database can be queried to identify the upstream data series elements of the set of output data 110 and determine which, if any, of those upstream data series elements are included in the list of root cause datasets.

[0058] In some examples, if a possible data quality issue is detected in a dataset, such as in a new version of a reference dataset or in a source dataset, a user notification may be enabled. In some examples, a warning flag may be stored to indicate the data quality issue. For example, if a possible data quality issue is detected in a new version of a reference dataset, a warning flag may be stored with the profile data for the new version of the reference dataset. If a possible data quality issue is detected in a source dataset, a warning flag may be stored with the profile data for the source dataset. In some examples, a warning message may be communicated to a user to indicate the presence of a possible data quality issue. The warning message may be, for example, a message on a user interface, an icon, a pop-up window, an email, a Short Message Service (SMS) message, or another format.

[0059] In some examples, rules may specify one or more threshold deviations from the reference profile at which a warning flag or message is used. For example, a warning flag may be stored if the deviation between the profile of the current data set and the reference profile for that data set is less than 1-2 standard deviations, etc., and a warning message may be communicated if the deviation is greater than 2 standard deviations. The threshold deviation may be specific to each source data set and the reference data set.

[0060] In some cases, such as when the deviation is extreme, e.g., three standard deviations above the reference profile, further processing by the data processing system may be halted until a user intervenes. For example, any further processing affected by source data or reference data with extreme deviations is paused. Transformations to be paused may be identified by data referencing data series elements downstream of the affected source data or reference data.

[0061] In some examples, the reference profile data is determined automatically. For example, the reference profile data for a given dataset may be automatically updated as a historical running average of past profile data for that dataset (e.g., by recalculating the reference profile data each time new profile data for that dataset is determined). In some examples, a user may provide the initial reference profile data, for example, by profiling a dataset having desired characteristics.

[0062] The update status of the transformation elements 106, 108, 116 in the data sequence of the output data may be tracked, such as the time or date of the most recent update for each of the transformation elements 106, 108, 116. The timing of the most recent updates to the transformation elements may be accessed, allowing a user to evaluate whether one or more of the transformation elements, for example an incorrect or corrupted transformation element, is a possible root cause of an existing or potential data quality problem in the output data 110. For example, if the transformation element 116 is updated immediately before the output data 110 is output from the transformation element 116, then the transformation element 116 may be identified as a possible root cause of an existing or potential data quality problem in the output data 110.

[0063] Referring to FIG. 5, the tracking engine 500 monitors profiles of data series elements, such as source data and reference data, as well as updates to data series elements, such as reference data and transformations, in upstream data series of a given data set, such as output data generated by a data processing system.

[0064] The tracking engine 500 includes a data lineage repository 502 that stores data 504 that references data lineage elements that are upstream of a given data set, such as output data generated by a data processing system. For example, the data lineage repository 502 may store an identifier for each data lineage element and data indicating relationships between the data lineage elements. The data lineage repository 502 may be a file, a database, or another data storage mechanism.

[0065] The tracking engine 500 includes an update monitor 506. The update monitor 506 monitors when transformation elements and reference data sets in the data processing system are updated. For each transformation element referenced by the data lineage repository 502, the update monitor 506 monitors when the software implementing the transformation element is updated. When an update occurs, the update monitor 506 stores an entry 510 in an update repository 508, such as a file, database, another data storage mechanism, etc. The entry 510 indicates the timing of the update, such as the date, time, or both, that the software was updated. In some examples, the entry 510 may also include an indication of the nature of the update, such as a manually entered description of the update, the text of the lines of code that were changed by the update, another indication of the nature of the update, etc. The update repository 508 may be indexed by an identifier of the transformation element, by the timing of the update, or both.

[0066] For each reference dataset referenced by data lineage repository 502, update monitor 506 monitors when the reference dataset is updated. When an update occurs, update monitor 506 stores an entry 514 in profile repository 516, such as a file, database, another data storage mechanism, etc. The entry 514 indicates the timing of the update, such as the date, time, or both, that the reference dataset was updated. Profile repository 516 may be indexed by an identifier for the reference dataset, by the timing of the update, or both.

[0067] When a reference data set is updated, the quality elements for that reference data set generate a profile for the updated reference data, sometimes referred to as a new version of the reference data. The quality elements may generate a profile according to a list of important data elements 520 stored in a rules repository 522, such as a file, database, another storage mechanism, etc. Important data elements are fields in a data record that are known to be important to a user or the system, e.g., fields specified by a user or automatically identified. A profile is generated for each important data element for the new version of the reference data. For example, the profile generated for a given important data element may be statistical survey data indicating how many distinct values ​​for the important data element are in the reference data set and the number of times each distinct value occurs. Reference profile data 524 indicating the generated profile for each important data element is stored in the profile repository 516, e.g., in association with the entry 514 indicating the update to the reference data.

[0068] As source data is provided to a data processing application, a profile is generated for each source data set referenced by the data lineage repository 502 with corresponding quality elements. A profile is generated for each significant data element in the source data, the significant data elements being specified in a list of significant data elements 520 stored in a rules repository 522. Source profile data 526 indicative of the generated profile for each profiled source data set is stored in a profile repository 516, such as a file, database, another data storage mechanism, etc.

[0069] In some examples, the reference profile data 524 and the source profile data 526 are accessed only if a data quality problem occurs in the downstream output data. In some examples, the reference profile data 524, the source profile data 526, or both are analyzed by the profile module to determine if the data indicates a potential data quality problem with a new version of the reference data or the received source data, respectively. The profile data 524, 526 can be analyzed immediately after the profile is generated, or can be analyzed at a later point in time, such as any time the tracking engine has free computational resources for analysis.

[0070] To analyze the reference profile data 524 or the source profile data 526, an analysis module 530 applies rules 536, such as automatically generated or user-specified rules stored in the rules repository 522. The rules may indicate, for example, one or more important data elements for each data set, marginal deviations that may cause data quality issues, or another type of rule.

[0071] In some examples, when a potential data quality issue is detected in a new version of the reference data or in a source dataset, an identifier for the dataset having the potential data quality issue is placed on a list of root cause datasets 550 stored in the data lineage repository 502. If a user later detects a data quality issue in a downstream dataset, the user can query the data lineage repository 502 to identify data lineage elements that are upstream of the output dataset and which (if any) of those upstream data lineage elements are included on the list of root cause datasets 550.

[0072] In some examples, the output data 110 is automatically analyzed to determine whether there are possible data quality issues. For example, each batch or time interval of the output data 110 can be profiled, and profiling and validation rules can be applied to the output data 110, to compare the profile of the current output data 110 to a baseline profile of a previous version of the output data 110. If the profile of the current output data 110 deviates from the baseline profile by more than a threshold amount, as specified in the output data profiling rules, the current output data 110 can be identified as having a potential data quality issue. If a particular data element in the current output data 110 has a value that deviates from an expected value range by more than a threshold amount, as specified in the output data validation rules, the current output data 110 can be identified as having a potential data quality issue. A warning flag can be stored with the output data 110 in the data warehouse, or a user can be notified, for example, by a user interface or a message.

[0073] In some cases, a user identifies a set of output data 110 as having a potential data quality issue. For example, a business analyst creating a report that summarizes multiple sets of output data 110 may realize that a particular set of output data 110 makes little sense compared to other output data sets that the analyst is analyzing. The analyst can flag that particular set of output data 110 as having a potential data quality issue.

[0074] If the output data has data quality issues, information stored within the tracking engine 500 may be accessed to identify the root cause of the data quality issue. For example, an identifier of the output data, such as a file name or timestamp, may be provided to the query module 548, for example, automatically or by a user. The query module 548 queries each of the relevant repositories for information that may be related to the identified output data. In particular, the query module 548 queries the data lineage repository 502 to identify the transformations, source data, and reference data on which the identified output data is based. The query module 548 may then query the update repository for any entries 510 that indicate updates to any of the identified transformation elements that occurred immediately prior to processing the output data. The query module 548 may query the profile repository 516 for any entries 514 that indicate updates to the identified reference data along with associated reference profile data 524 and any associated warning flags. The query module 548 may query the profile repository 516 for source profile data 526 for any identified source datasets.

[0075] Results returned in response to queries by the query module 548 are displayed on a user interface. The display allows a user to view and manipulate the data to gain an understanding of potential root causes of data quality issues in the output data. For example, if there was a software update to a transformation element just before the output data was processed, the user can view information related to the update, such as a description of the update or the lines of code that were changed. If there were warning flags associated with the reference profile data or the source profile data, the user can view the profile data.

[0076] In some examples, the results returned by the query module 548 may indicate that an update to a transformation element occurred immediately before the transformation element performed a process on the output data having a potential data quality issue. This may be referred to as a recently updated transformation element. Immediately means within a set time, such as within 10 minutes, within an hour, within a day, or within another time, of the process. The update monitor 506 may obtain additional information about the recently updated transformation elements that may indicate whether one or more of the recently updated transformation elements are a potential root cause of a data quality issue in the output data. For example, the update monitor 506 may identify any processing artifacts associated with the recently updated transformation element. The presence of processing artifacts may indicate a potential problem with the recently updated transformation element. To ensure that the update log reflects the updates to the recently updated transformation element, the update monitor 506 may check the update log associated with the recently updated transformation element. A discrepancy between the update log and the data 510 indicating updates to the recently updated transformation element may indicate a potential problem with the transformation element. The update monitor 506 can check checksums or other system data to identify potential errors that may have been introduced during updates of recently updated transformations.

[0077] In some examples, user notification may be enabled if a potential problem with a recently updated transformation element is detected. In some examples, a warning flag may be stored in the update repository 508 along with, for example, data indicating the update 510 to indicate a potential problem. In some examples, a warning message may be communicated to a user by the communication module 546 to indicate the existence of a potential problem with a recently updated transformation element. For example, the warning message may be a message on a user interface, an icon, a pop-up window, an email, an SMS message, or another format. In some examples, the analysis of data lineage and data quality may be at the level of a dataset, which may be referred to as a coarse-grained data lineage. The coarse-grained data lineage looks at the data lineage of a downstream dataset. The upstream dataset and the upstream transformation elements used to generate the downstream dataset are considered to be in the data lineage of the downstream dataset. In some examples, the analysis of data lineage and data quality may be at the level of an individual field, which may be referred to as a fine-grained data lineage. The fine-grained data lineage looks at the data lineage of a particular field in a downstream dataset. The upstream transformation elements and the fields in the upstream dataset used to generate a particular field in a downstream dataset are considered to be in the data lineage of the downstream dataset. The techniques described herein for analyzing data quality may be applied in connection with both coarse-grained and fine-grained data series.

[0078] Further information regarding profiling can be found in U.S. Patent No. 8,868,580, entitled "Data Profiling," which is incorporated herein by reference in its entirety. Typically, a data record is associated with a set of data fields, with each field having a particular value (possibly including a null value) for each record. In some examples, data records in a dataset have a fixed record structure, in which each data record has the same fields. In some examples, data records in a dataset have a variable record structure, including, for example, variable length vectors or conditional fields. In some examples, the profile module 218 can provide initial format information for data records in a dataset to the profile elements 150, 152, 154. The initial format information can include, for example, the number of bits to represent a distinct value (e.g., 16 bits), an order of values, including values ​​associated with record fields and values ​​associated with tags or delimiters, the type of value represented by the bits (e.g., string, signed / unsigned integer, or other type), or other format information. The format information can be specified in a data manipulation language (DML) file stored in the rules repository 522. The profile elements 150, 152, 154 can use pre-defined DML files to automatically interpret data in a variety of common data system formats, such as SQL tables, XML files, CSV files, etc., or can use DML files obtained from the rules repository 222 that describe specialized data system formats.

[0079] FIG. 6 illustrates an example of a user interface 300 that allows a user to investigate the root cause of potential data quality issues in an output dataset. The user interface 300 allows a user to input an identifier 302 of an output dataset or an identifier 304 of a particular data element in the output data. For example, the identifier 302 or 304 can identify an output dataset or a particular data element that has a potential data quality issue. In the example of FIG. 6, the user inputs the output dataset "Billing_records.dat.". An interactive data lineage diagram 310 is displayed on the user interface 300 to graphically show the identified set of output data or the data lineage elements upstream of the identified data element. In the example of the data lineage diagram 310, the identified data lineage elements upstream of the output dataset include two sets of source data 312, 314, two transformation elements 316, 318, and one set of reference data 320.

[0080] Upstream data lineage elements with potential data quality issues, such as source data 312, transformation element 318, and reference data 320 in this example, are marked with warning flags 324a, 324b, and 324c, respectively. A user can select a warning flag, such as by clicking or tapping on the warning flag, hovering over the warning flag, or otherwise selecting the warning flag, to access information about the associated potential data quality issue. Information about the potential data quality issue associated with a data set may include profile data, reference profile data for one or more data elements, results of a statistical analysis of the profile data (such as deviations of the profile data from the reference profile data), values ​​that do not meet tolerances specified by validation rules, or other information. Information about the potential data quality issue associated with a transformation element may include the date of the most recent update to the transformation element, a description of the update, an excerpt of the code from the update, or other information. In some examples, an information bubble may be overlaid on the data lineage diagram in response to a user selecting one of the warning flags. In some examples, a new screen may be displayed in response to a user selecting one of the warning flags. In some cases, the information displayed in an information bubble or new screen may be interactive, allowing the user to access further details by selecting a piece of information.

[0081] The user interface 300 also provides the user with access to a rules editor 328, which allows the user to add, remove, or modify profiling rules, validation rules, or both. For example, the user can add, remove, or modify important data elements for each data set, update threshold deviations that trigger the identification of potential data quality issues, specify whether a profiling or validation rule should be applied automatically upon receipt of a new data set or only upon detection of a downstream data quality issue, or make other changes to a profiling or validation rule.

[0082] In one specific example, a data processing system processes call records to generate billing records. Each source data record represents a call and includes fields that store data such as the date of the call, the time, the duration of the call, the dialed phone number, and the called phone number. The source data records are processed monthly in a batch process to generate bills. In this example, bills were not generated for 95% of customer accounts during the month of May 2015. A user requested information regarding the profile of data series elements in the upstream data series of output data used to generate the May 2015 bill and updates to such data series elements. While the baseline source profile data indicated an expected range of 1.5 to 2.4 million unique values ​​in the dialed phone number field, the source profile data indicated that the dialed phone number field in the source data record used to generate the May 2015 bill had only 10 unique values. Based on this inspection of the source profile data, the user determined that the source data record was corrupted. The source data record was retrieved from compressed storage and processed again to properly generate the May 2015 bill.

[0083] In another specific example, a data processing system processes corporate financial records and allocates each financial record to a corporate division. The allocation of each financial record to a corporate division is made by mapping a division identifier in each record to one of six corporate divisions provided by a corporate baseline data set. Baseline profile data for the corporate baseline data indicated that the number of corporate divisions has been consistently six over the past ten years. The baseline data is updated quarterly. After the most recent update, the baseline data was profiled and indicated that the number of corporate divisions in the baseline data had increased to 60. The deviation of the profile of the updated baseline data from the six division baseline was large enough to trigger the sending of a warning message to a system administrator. In addition, further processing by the data processing system was suspended until the baseline data could be investigated and corrected if necessary.

[0084] Referring to FIG. 7, in one embodiment, a data processing system 50 includes multiple transformation elements 52, 54, 56 that process input data 58 including records of online purchases made at thebostonshop.com on April 1, 2016. Each record of the input data 58 includes multiple fields, including a state field. In this example, component 56 is a splitting component that sends each data record to one of eight files 60a-60h based on the value in the state field of the input data. For example, records having a value of MA in the state field are sent to file 60a, records having a value of TX are sent to file 60b, records having a value of CA are sent to file 60c, records having a value of DE are sent to file 60d, records having a value of NY are sent to file 60e, records having a value of IL are sent to file 60f, records having a value of RI are sent to file 60g, and records having any other value are sent to file 60h. The number of records sent to each file is shown in FIG. 7. In the example of FIG. 7, the number of records sent to each file is within the expected range, and therefore no data quality warning is generated. This is because the input data 58 falls within the expected range.

[0085] The quality of the input data 58 is characterized by a quality element 62, which generates a profile for the state field of the input data 58 and applies automated generation rules indicating an acceptable deviation between the profile of the state field and a reference profile for the state field of the input data. The reference profile represents an average profile of data processed by the data processing system 50 over the past year and indicates an acceptable deviation above which a potential data quality issue will be identified. In this example, the automated generation rules indicate that the input data 58 should be identified as having a potential data quality issue if the value distribution in the state field in the profile of the input data 58 differs from the value distribution in the reference profile by more than 10%, and the reference profile for the state field is: MA: 6% TX:25% CA: 33% DE:3% New York: 17% IL:11% RI:4% Any other value: 1% As can be seen from Fig. 7, the actual profile of the state field is within the 10% allowable deviation of the reference profile, therefore there is no data quality problem with the input data.

[0086] Referring to FIG. 8A, in one example of abnormal operation of data processing system 50, input data 55 includes a record of an online purchase made at bostonshop.com on April 2, 2016. In this example, the record has not been sent to file 60g. An operator of data processing system 50 may notice that file 60g is empty, or an empty file may cause errors in further processing by downstream data processing systems. An operator of data processing system 50 may track down the root cause of the record not being sent to file 60g by investigating the quality of upstream data elements in the data series of files 60a-60h. Specifically, input data 55 belongs to the upstream data series of files 60a-60h.

[0087] Referring also to FIG. 8B, the quality element 62 generates the following actual profile for the state field of the input data 55: MA: 6% TX: 25.1% CA: 32.7% DE: 2.9% New York: 17.1% IL: 11.1% RI:0% Any other value: 5.1%.

[0088] A deviation between the state field profile of the input data 55 and the reference profile for the state field identifies the input data 55 as having a potential data quality problem, and a warning flag is stored to indicate the potential data quality problem. When the operator tracks down the root cause of the empty file 60g, the operator can easily recognize that a potential data quality problem existed in the input data 55. The operator can use that knowledge to investigate the cause of the deviation to determine, for example, whether the input data 55 was corrupted, whether previous processing of the input data 55 in an upstream data processing system caused the deviation, or another cause. For example, referring also to FIG. 8C, by looking at a portion of the actual input data 55 in this example, the operator can recognize that the letters in the value "RI" have been reversed to read "IR," causing those records to be stored in file 60h instead of file 60g.

[0089] 9A, in another example of abnormal operation of data processing system 50, input data 64 includes a record of an online purchase made at thebostonshop.com on April 3, 2016. In this example, the record was sent only to file 60a and not to any of the other files 60b-60h. An operator of data processing system 50 may notice that files 60b-60h are empty, or the empty files may cause errors in further processing by downstream data processing systems.

[0090] 9B, an operator of the data processing system can trace the root cause of all of the records sent to file 60a by examining the quality of the upstream data elements in the data series of files 60a-60h. In this example, quality element 62 generates the following profile of the state field of input data 64: MA: 6.1% TX: 25.2% CA: 32.6% DE: 2.9% New York: 17.0% IL: 11.1% RI: 4.1% Any other value: 1%.

[0091] The profile of the state field in the input data 64 is consistent with the baseline profile for the state field, and therefore no potential data quality issues are identified. The operator may then investigate the update status of transformation elements 52, 54, 56 in the data series of files 60a-60h. For example, the operator may determine that transformation element 56 was updated immediately prior to processing input data 64, and therefore transformation element 56 may be the root cause of empty files 60b-60h.

[0092] 10A, in one embodiment, a data processing system 80 includes multiple conversion elements 82, 84 that process a stream of input data 86 that includes call records for mobile phone calls handled by a particular tower. Each record of the input data 86 includes multiple fields, including a phone_number field. The input data 86 is formatted by conversion element 82, then sorted by conversion element 84 by the value in the phone_number field, and output into a queue 88 from which it is fed into a second data processing system 90 for further processing. In this example, 25% of the records fed from queue 88 into second data processing system 90 cause processing errors. An operator of data processing system 80 can track down the root cause of these processing errors by examining the quality of the upstream data elements in the data sequence of queue 88.

[0093] The quality of the input data 86 is characterized by a quality element 90, and the quality of the data 94 output from the format conversion element 82 is characterized by a quality element 92. Both quality elements 90, 92 apply a user-generated rule that specifies that the value in the phone_number field should be a 10-digit integer, and that if more than 3% of the records do not satisfy the rule, a potential data quality issue should be identified. In this example, the quality element 90 determines that 0.1% of the records in the data 86 have an 11-digit integer in the phone_number field. Because this percentage of records is below the 3% threshold, the quality element 90 does not identify any potential data quality issues with the input data 86. The quality element 92 characterizes 25% of the records in the data 94 as having an alphanumeric value in the phone_number field. An example of a portion of the data 94 is shown in FIG. 10B. A warning flag is stored to indicate a potential data quality issue with the data 94. When an operator traces the root cause of a processing error, the operator can readily recognize that no data quality issues have been identified in the input data 86, but that potential data quality issues exist in the data 94.

[0094] Referring to FIG. 11, in one example of a process for determining the quality of a source dataset, a source dataset is received in a data processing application (400). A profile of the source dataset is generated and stored (402). One or more rules for the source dataset are obtained (404). The source data or the profile of the source data is analyzed according to the one or more rules (406). If one or more rules are not satisfied by the source dataset (408), a warning indicating a potential data quality problem is stored with the profile data and / or communicated to a user (410) and the source data is added to a list of datasets with possible data quality problems. If one or more rules are satisfied by the source data (408), the source data is processed by the data processing application (412). In some cases, such as for extreme deviations from thresholds or tolerances specified by the rules, processing is paused until user intervention allows processing to resume. During or after processing, a user is authorized to access the stored profile data, for example, to investigate potential root causes of downstream data quality problems.

[0095] Referring to FIG. 12, in one example of a process for monitoring the quality of reference data in a data processing system, a reference data set is monitored (500). When the reference data set is updated, a profile of the new version of the reference data is generated and stored (502). For example, the generation of the profile may occur after each scheduled update to the reference data. One or more rules for the reference data set are obtained (504). The new version of the reference data or the profile of the new version of the reference data is analyzed (506) according to the one or more rules. If the one or more rules are not satisfied by the new version of the reference data (508), a warning indicating a possible data quality problem is stored with the profile data, communicated to a user, or both (510). If the one or more rules are satisfied by the new version of the reference data (508), further processing by the data processing system is allowed to begin or continue (512). In some cases, such as for extreme deviations from thresholds or tolerances specified by the rules, processing is paused until user intervention allows processing to begin or continue. During or after processing, users may be empowered to access the stored profile data, for example, to investigate potential root causes of downstream data quality problems.

[0096] In some examples, rules are analyzed before being applied, for example to determine an update date for each rule. If a rule is older than the critical validity period, the rule may not be applied, or the user may be alerted that the rule may be ready to be updated.

[0097] Referring to FIG. 13, in one example process for analyzing updates to a transformation element, the time of the most recent update to the transformation element is identified (600). For example, a timestamp of the most recent update may be stored in a data repository. If the transformation element does not have a recent update (602), no further updates to the transformation element are analyzed (604). A recent update may be an update within a certain time limit, such as within 10 minutes, within an hour, within a day, within another time period, etc. If the transformation element has been recently updated (602), any processing artifacts are identified (606). An update log associated with the transformation element is checked (608) to identify any inconsistencies between the update log and the timestamp of the most recent update stored in the data repository. A checksum or other system data associated with the transformation element is checked (610) for an indication of any potential errors that may have been introduced during the update of the transformation element. If no potential problems are identified (612), the system is allowed to begin or continue processing (614). If one or more potential problems are identified 612, a warning indicating the potential problem with the transformation element is stored in the data repository, communicated to a user, or both 616. Processing by the data processing system may be allowed to begin or continue, or may be paused until user intervention allows processing to begin or continue.

[0098] FIG. 14 is a flow chart of an example process. Information indicative of an output dataset to be generated by a data processing system is received (700). One or more upstream datasets on which the output dataset relies are identified (702) based on data lineage information related to the output dataset. The data lineage information indicates one or more datasets on which the output dataset relies, one or more datasets on which the output dataset relies, or both. Each of the identified upstream datasets on which the output dataset relies is analyzed to identify a subset of the datasets, including determining which of the one or more datasets have errors or possible errors (704). For each particular upstream dataset, a first rule is applied (706) indicating an acceptable deviation between a profile of the particular upstream dataset and a reference profile for the particular upstream dataset, and a second rule is applied (708) indicating an acceptable or prohibited value of one or more data elements in the particular upstream dataset. In some examples, only the first rule or the second rule is applied. The first rule, the second rule, or both may be automatically generated or specified by a user. Based on the results of applying the first rule, the second rule, or both, one or more of the upstream datasets are selected for subsetting 710. Information related to the subset of the upstream datasets is output 712.

[0099] FIG. 15 is a flow chart of an example process. An error or possible error in a data element of a downstream dataset of a data processing system is identified (900), for example, automatically or based on user input. One or more upstream datasets that affect the data element are automatically identified (902) based on data lineage information related to the downstream dataset. It is determined (904) which upstream datasets have errors or are likely to have errors, including analyzing the current profile and the reference profile of each of the identified upstream datasets. For example, each upstream dataset may be analyzed by applying one or more rules to each of the current profiles. The rules may indicate an acceptable deviation between the current profile of the particular upstream dataset and the corresponding reference profile of the particular upstream dataset. The rules may indicate an acceptable value for the data element in the particular upstream dataset. Information related to each upstream dataset that has errors or is likely to have errors is output (906).

[0100] The techniques for monitoring and tracking data quality described herein are based on computer technology and can be used to address problems that arise during the execution of a computer-implemented process. For example, using the monitoring and tracking techniques described herein, the processing of a data set by a computer-implemented data processing system can be monitored and made more efficient, effective, or accurate. In addition, the techniques described herein can be applied to assist users, such as system administrators, in managing the operation of a data processing system.

[0101] 16 illustrates an example of a data processing system 1000 in which the techniques for monitoring and tracking may be used. The system 1000 includes a data source 1002, which may include one or more data sources, such as a connection to a storage device or an online data stream, each of which may store or provide data in any of a variety of formats (e.g., a database table, a spreadsheet file, a flat text file, or a native format used by a mainframe computer). The data may be logistic data, analytical data, or machine data. The execution environment 1004 includes a pre-processing module 1006 and an execution module 1012. The execution environment 1004 may be hosted on one or more general-purpose computers under the control of a suitable operating system, such as a version of the UNIX operating system. For example, the execution environment 1004 may include a multi-node parallel computing environment including configurations of computer systems using multiple central processing units (CPUs) or processor cores, either local (e.g., a multiprocessor system such as a symmetric multiprocessing (SMP) computer), locally distributed (e.g., multiple processors coupled as a cluster or a massively parallel processing (MPP) system), or remote or remotely distributed (e.g., multiple processors coupled by a local area network (LAN) and / or wide area network (WAN)), or any combination thereof.

[0102] The storage device providing the data source 1002 may be local to the execution environment 1004, for example stored on a storage medium (e.g., hard drive 1008) connected to the computer hosting the execution environment 1004, or may be remote to the execution environment 1004, for example hosted on a remote system (e.g., mainframe computer 1010) that communicates with the computer hosting the execution environment 1004 over a remote connection (e.g., provided by a cloud computing infrastructure).

[0103] The pre-processing module 1006 reads data from the data source 1002 and prepares a data processing application for execution. For example, the pre-processing module 1006 may compile the data processing application, store and / or load the compiled data processing application to / from a data storage system 1016 accessible to the execution environment 1004, and perform other tasks to prepare the data processing application for execution.

[0104] The execution module 1012 executes the data processing application created by the pre-processing module 1006 to process the data set and generate output data 1014 resulting from the processing. The output data 1014 may be stored back in the data source 1002, may be stored in a data storage system 1016 accessible to the execution environment 1004, or may be used in other ways. The data storage system 1016 is also accessible to a development environment 1018 in which a developer 1020 can design and edit the data processing application executed by the execution module 1012. In some implementations, the development environment 1018 is a system for developing applications as data flow graphs that include vertices (representing data processing components or data sets) connected by directed edges (representing the flow of work elements, i.e., data) between the vertices. For example, such an environment is described in more detail in U.S. Patent Application Publication No. 2007 / 0011668, entitled "Managing Parameters for Graph-Based Applications," which is incorporated herein by reference. A system for performing such graph-based computations is described in U.S. Patent No. 5,966,072, entitled "EXECUTING COMPUTATIONS EXPRESSED AS GRAPHS," the entire contents of which are incorporated herein by reference. Dataflow graphs created in accordance with this system provide methods for moving information in and out of the individual processes represented by the graph components, for moving information between processes, and for sequencing the execution of processes. The system includes algorithms for selecting inter-process communication methods from any available methods (e.g., communication paths through the links in the graph can use TCP / IP or UNIX domain sockets to pass data between processes, or use shared memory).

[0105] The pre-processing module 1006 can receive data from various kinds of systems that may embody data sources 1002, including various forms of database systems. The data may be organized as records having values ​​for each field (also called "attributes" or "columns"), possibly including null values. When initially reading data from a data source, the pre-processing module 1006 typically starts with some initial formatting information about the records in that data source. In some situations, the record structure of a data source may not be known initially, but instead may be determined after analysis of the data source or data. The initial information about the records may include, for example, the number of bits that represent distinct values, the order of the fields in the record, and the type of value represented by the bits (e.g., string, signed / unsigned integer).

[0106] The monitoring and tracking techniques described above may be implemented using a computing system running appropriate software. For example, the software may include procedures in one or more computer programs running on one or more programmed or programmable computing systems (which may be of various architectures such as distributed, client / server, grid, etc.), each of which includes at least one processor, at least one data storage system (including volatile and / or non-volatile memory and / or storage elements), and at least one user interface (for accepting input using at least one input device or port, and for providing output using at least one output device or port). The software may include one or more modules of a larger program that provides services related to, for example, graph design, configuration, and execution. The modules of the program (e.g., elements of the graph) may be implemented as data structures or other organized data conforming to a data model stored in a data repository.

[0107] The software can be provided on a tangible, non-transitory medium, such as a CD-ROM or other computer readable medium (e.g., readable by a general purpose or special purpose computing system or device), or can be carried over a communication medium of a network (e.g., encoded in a propagating signal) to a tangible, non-transitory medium of a computing system where the software executes. Some or all of the processing can be performed on a special purpose computer, or using special purpose hardware, such as a co-processor, field programmable gate array (FPGA), special purpose application specific integrated circuit (ASIC), etc. The processing may also be implemented in a distributed manner, where different parts of the computation specified by the software are performed by different computing elements. Each such computer program is stored on or downloaded to a computer readable storage medium (e.g., solid state memory or medium, or magnetic or optical medium) of a storage device accessible by a general purpose or special purpose programmable computer, to configure and operate the computer when the storage medium is read by the computer to perform the processes described herein. The systems of the present invention may also be considered to be implemented as a tangible, non-transitory medium configured with a computer program, the medium so configured causing a computer to operate in a specific and predetermined manner to perform one or more of the process steps described herein.

[0108] Several embodiments of the present invention have been described. Nevertheless, it should be understood that the above description is intended to be illustrative rather than limiting the scope of the present invention, which is defined by the appended claims. Accordingly, other embodiments are within the scope of the appended claims. For example, various modifications may be made without departing from the scope of the present invention. In addition, some of the steps described above may be order-insensitive and thus may be performed in a different order than described.

Claims

1. 1. A computer-implemented method for analyzing updates to a transformation, comprising: Identifying (600) a time of a most recent update for a particular transform element of one or more transform elements, the most recent update being an update within a time limit; In response to determining (602) that the particular transform has been recently updated, inspecting (608) the update log to identify one or more inconsistencies between the update log associated with the particular translation element and the time of the most recent update stored in a data repository, the presence of an inconsistency may indicate an error or potential error associated with the particular translation element; If an error or potential error associated with the particular conversion element is identified, outputting (616) a warning to indicate the identified error or potential error associated with the particular conversion element; 4. A computer-implemented method comprising:

2. 2. The method of claim 1 , In response to determining (602) that the particular transform has been recently updated, The method further includes checking (610) a checksum associated with the particular translation element for an indication of potential errors that may have been introduced during an update of the particular translation element.

3. 3. The method according to claim 1 or 2, The method further comprising, in response to identifying an error or potential error associated with the particular conversion element, pausing the processing until user intervention allows the data processing system to continue processing.

4. 4. The method of claim 1, 2 or 3, monitoring when the one or more transformation elements are updated; storing an entry (510) in the update log in response to determining that a transformation element of the one or more transformation elements has been updated, the entry (510) indicating a timing of the update of the updated transformation element; The method further comprises:

5. 5. The method of claim 4, wherein the entry (510) also includes an indication of the nature of the update, such as a manually entered description of the update or the text of lines of code that are changed by the update.

6. 6. A method according to claim 4 or 5, wherein the update log is indexed by identifiers of the updated transformation elements, or by the timing of the updates, or both.

7. A method according to any preceding claim, wherein the warning comprises a warning flag stored in the update log together with a time of the most recent update to indicate the error or potential error.

8. The method of any one of claims 1 to 7, wherein the warning comprises a warning message communicated to a user to indicate the presence of the error or potential error.

9. A non-transitory computer readable medium storing instructions for causing a computing system to perform the operations of the method according to any one of claims 1 to 8.

10. A computing system including a processor coupled to a memory, the processor and memory configured to perform the operations of the method of any one of claims 1 to 8.

Images