Data recommender using lineage to propagate value indicators

By deriveing ​​value indicators and implicit metadata in the data analysis system, identifying the metadata characteristics of high-value impacts, and spreading value indicators, the problem that data scientists have difficulty recommending valuable data sets and processing steps is solved, and data is realized Efficiency and accuracy of analysis.

CN117312650BActive Publication Date: 2024-11-15HEWLETT PACKARD ENTERPRISE DEV LP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211321001.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2022-06-17
Filing Date
2022-10-26
Publication Date
2024-11-15
Estimated Expiration
2042-10-26

AI Technical Summary

Technical Problem

In the existing data analysis field, data scientists have difficulty effectively using and recommending valuable data sets and processing steps, resulting in waste of resources and underutilization of data secondary use, and the lack of explicit metadata makes it difficult for automated systems to recommend suitable ones. Dataset.

Method used

Through a data analysis system, collect information from the data processing pipeline, derive value indicators and implicit metadata, use valuation algorithms to quantify the value of data-related products and processing steps, identify the metadata characteristics of high-value influence, and disseminate through data lineage representation Value indicator to automatically recommend appropriate data sets and processing steps.

Benefits of technology

It improves the efficiency and accuracy of data analysis, reduces resource waste, realizes efficient secondary use of data, and provides intelligent recommendations that are consistent with human value judgments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117312650B_ABST
    Figure CN117312650B_ABST
Patent Text Reader

Abstract

Embodiments of the present disclosure relate to a data recommender that uses a pedigree to propagate value indicators. Systems and methods provide a system that collects information about data as it advances through a data processing pipeline of a data analysis project. The data analysis system derives value indicators and implicit metadata from the data processing pipeline. For example, the data analysis system can derive value indicators and implicit metadata from the data-related products themselves, semantic analysis of the code / processing steps used to process the data-related products, the structure of the data processing pipeline, and human behavior related to the production and use of the data-related products. Once a new data analysis project is initiated, the data analysis system collects parameters and characteristics about the new data analysis project, and references the value indicators and implicit metadata to recommend useful processing steps, data sets, and / or other data-related products for the new data analysis project.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] In order to extract valuable insights from the data, people can use data processing pipelines to analyze the data. The data processing pipeline may include a series of processing steps to transform the data (as used herein, data may generally refer to various types of data-related products, such as raw data, data sets, databases, analytical models derived from the data, dashboards displaying one or more of the aforementioned data-related products, etc.). In other words, through a series of processing steps, the data processing pipeline may iteratively transform upstream / ancestor data-related products (e.g., raw data, data sets, databases, etc.) into downstream / descendant data-related products (e.g., new data sets derived from upstream data-related products, analytical models derived from upstream data-related products, dashboards displaying one or more of the aforementioned data-related products, etc.). For example, using a first processing step, the data processing pipeline may transform a first data-related product (e.g., an original data source) into a second data-related product (e.g., a data set derived from the original data source). Using the second processing step, the data processing pipeline can transform the second data-related product into a third data-related product (e.g., an analytical model derived from the data set), and so on (herein, the third data-related product can be referred to as a "descendant" of the first data-related product and the second data-related product; conversely, the first data-related product and the second data-related product can be referred to as ancestors of the third data-related product. In this way, the data processing pipeline can transform data as it progresses through the data processing pipeline.

[0002] In real-world applications, the size and amount of data available for analysis can be enormous. Not only are individual datasets very large, but there are a large number of available datasets, which can be an organizational and logistical challenge. In theory, care can be taken to flexibly organize and label new data. In practice, however, following such a system is hit or miss. Therefore, there is a high demand for data scientists in the field of data analysis. Among other responsibilities, these data scientists become domain experts in certain areas and accumulate empirical knowledge about available relevant data (e.g., original datasets / data sources and other data-related products that have been transformed for or through previous analysis). For example, a data scientist who is an expert in the field of handwriting analysis can understand the differences in characteristics between the MNIST dataset and the SVHN dataset and which one is best suited for certain analyses. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The present disclosure according to one or more various examples is described in detail with reference to the following drawings. The drawings are provided for illustration purposes only and depict only typical or examples.

[0004] Figure 1 Depicted are example data lineage representations according to various examples of the disclosed technology.

[0005] Figure 2 An example data analysis system in accordance with various examples of the disclosed technology is depicted.

[0006] Figure 3 Depicted are example metadata lineage representations according to various examples of the disclosed technology.

[0007] Figure 4 Depicted are example data lineage representations according to various examples of the disclosed technology.

[0008] Figure 5 An example computing system is depicted that can be used to recommend at least one of an existing data set and an existing processing step for producing a new / prospective data-related product having desired metadata characteristics in accordance with various examples of the disclosed technology.

[0009] Figure 6 An example computing system is depicted that may be used to generate metadata indicator representations in accordance with various examples of the disclosed technology.

[0010] Figure 7 An example computing system that can be used to recommend valuable metadata properties for new / prospective data sets in accordance with various examples of the disclosed technology is depicted.

[0011] Figure 8 An example computing system is depicted that can be used to recommend one or more existing data sets and processing steps for creating valuable analytical models based on propagating value indicators through lineage representations for one or more existing analytical models in accordance with various examples of the disclosed technology.

[0012] Fig. 9 Depicted are example data lineage representations that may be used to illustrate examples of value indicator propagation in accordance with various examples of the disclosed technology.

[0013] Fig.10 An example computing system that can be used to recommend data-related products for a target data analysis project in accordance with various examples of the disclosed technology is depicted.

[0014] Fig.11 Depicted is a block diagram of an example computer system in which various examples described herein may be implemented.

[0015] The drawings are not exhaustive and do not limit the disclosure to the precise forms disclosed. DETAILED DESCRIPTION

[0016] While hiring data scientists to handle large amounts of data is a viable option for some data analytics projects, the sheer volume and variety of potential analyses can overwhelm the budgets, timelines, and capabilities available to most organizations. Without pervasive knowledge of all activity within the domain, even a highly experienced data scientist cannot know every available data-related product and every analysis that has been done in the past. While this is true globally, it is often true within the scope of an organization as well. Many organizations have experienced inefficient duplication: laboriously gathering data for a dataset, only to later discover that another department or business unit had already gathered the same dataset for a previous analytics project.

[0017] Another difficulty with relying on data scientists and their accumulated knowledge is the rate of turnover. Often, many of the most valuable insights useful in selecting data for analysis are not written down or are not publicly known. Conversely, the actual experience and employment of data scientists provides them with insights that can be used to find highly relevant data. For example, a long-term hired data scientist may be able to find data from a different project undertaken by a different department many years ago, while a more recently hired data scientist may not even know that such data exists.

[0018] Data consumers (e.g., business users) can derive substantial value from data. Although much of this value comes from initial analysis, secondary uses of data are an underutilized way to add value. However, discovering secondary uses of data can be challenging due to a lack of documentation (i.e., explicit metadata) and a lack of information describing potential secondary uses of the data. Even automated systems that attempt to recommend or select datasets are flawed because the explicit metadata they rely on is not universally or sometimes even typically applied to the data. When automated systems are used to recommend datasets that are actually useful, blind spots and poor recommendations are often the norm. The lack of explicit metadata in the data processing pipeline makes it difficult for data consumers and existing automated systems to mine data for secondary uses. As a result, many valuable uses of existing datasets and other dataset-related products remain unrealized.

[0019] In an example consistent with the present disclosure, a data analysis system collects information from previous data analysis projects / data processing pipelines and automatically recommends processing steps, data sets, and / or other data-related products that are most likely to be used to implement a target data analysis project for a data scientist or other data consumer.

[0020] As will be described in more detail below, the data analysis system collects information about the data as it progresses through the data processing pipeline of a previous data analysis project. The data analysis system derives value indicators and implicit metadata from the data processing pipeline. For example, the data analysis system can derive value indicators and implicit metadata from the data-related products themselves, semantic analysis of the code / processing steps used to process the data-related products, the structure of the data processing pipeline, and human behavior related to the production and use of the data-related products. Once a new data analysis project is initiated, the data analysis system collects parameters and characteristics about the new data analysis project and references the value indicators and implicit metadata to recommend useful processing steps, data sets, and / or other data-related products for the new data analysis project.

[0021] The examples may use a valuation algorithm to derive a value indicator that quantifies the value level of a given data-related product or processing step based on perceived human behavior associated with the use of the given data-related product / processing step (as used herein, a value indicator may refer to a quantifiable measurement of the value of a data-related product or processing step based on perceived human behavior associated with the use of a given data-related product / processing step). For example, the examples may utilize a simple valuation algorithm that quantifies the value level of an analytical model (i.e., an example of a data-related product) based on how long humans have deployed the analytical model in a production environment. Using this simple valuation algorithm, the examples may attribute a value of, for example, v=3 to an analytical model that has been deployed in production for three years, and a value of, for example, v=2 to an analytical model that has been deployed in production for two years. In various instances, the examples may utilize a more complex valuation algorithm that quantifies two or more categories of human behavioral actions associated with the use of a data-related product or processing step to a common numerical scale. For example, the examples may utilize more complex valuation algorithms to quantify the value level of an analytical model based on a common numerical scale based on any one or combination of the following factors: (a) the duration that humans / users have deployed the analytical model in production; (b) the number of independent users who have deployed the analytical model into production; and (c) the quality scores / accuracy scores attributed to the analytical model (these quality scores / accuracy scores can be attributed to a given analytical model by humans, artificial intelligence, etc.). As described above, the examples may utilize valuation algorithms to quantify the value level of two or more analytical models using a common numerical scale. Therefore, the examples may utilize these valuation algorithms to compare / rank the relative value of multiple analytical models based on multiple categories of human behavioral actions. Such valuation algorithms can provide an enhanced approximation of actual human value judgments.

[0022] Examples can utilize value indicators in various ways. For example, examples can enhance data analysis systems by helping them evaluate and rank processing steps, data sets, and other data-related products in a manner that more accurately reflects human judgment.

[0023] Examples may also utilize value indicators to derive implicit metadata from a data processing pipeline. Certain examples of this implicit metadata may be referred to as metadata indicators. As will be described below, metadata indicators may represent statistically significant / interesting correspondences identified / perceived by examples of the disclosed technology on a metadata pedigree. Examples may utilize derived metadata indicators to identify "metadata characteristics of high value impact". As used herein, "metadata characteristics of high value impact" may refer to (a) metadata characteristics of upstream / ancestor data-related products or processing steps that may increase / affect the value of downstream / descendant data-related products derived from upstream / ancestor data-related products / processing steps having "metadata characteristics of high value impact"; and / or (b) metadata characteristics of a given data-related product that may increase / affect the value of the given data-related product itself. As indicated by the metadata indicators, these "metadata characteristics of high value impact" may be metadata characteristics that appear with unusual / statistically significant frequencies on the metadata pedigree of "high-value" data-related products. Examples may enhance data analytics systems by identifying and recommending novel combinations of existing data-related products and processing steps (with these “high-value-impact metadata features”) for producing new high-value data-related products.

[0024] As a simplified analogy to illustrate this concept, the top 100 most "high value" online cake recipes in the United States may all include the word "cake" in the recipe title (here, the example can judge / quantify the value based on value indicators (such as 5-star ratings, social media shares, etc.). However, more "low value" online cake recipes may also include the word "cake". As an example of the technology disclosed herein, having the word "cake" in the recipe title may not be a (metadata) characteristic of high value impact of cake recipes. However, the example can find that 18 of the top 100 most "high value" online cake recipes have the word "Grandma / Grandma" in the recipe title. In contrast, the word "Grandma / Grandma" may appear in the titles of "low value" online cake recipes, but with a much lower frequency. Therefore, the use of the word "Grandma / Grandma" in the title of an online cake recipe can be a "high value impact (metadata) characteristic" because it appears with an unusual / statistically significant frequency among the top 100 most "high value" online cake recipes. Therefore, the example can recommend that new / expected online cake recipes include the word "Grandma / Grandma" in the title.

[0025] The example may derive metadata indicators by: (1) extracting and characterizing (explicitly or implicitly) metadata from a high-value producing data processing pipeline (i.e., a data processing pipeline that has produced a high-value data-related product as quantified by the value indicator) using a metadata extraction and characterization algorithm; (2) constructing a metadata lineage representation for the high-value producing data processing pipeline, the metadata lineage representation comprising lineage associations between processing steps of the high-value producing data processing pipeline and the extracted metadata characteristics of the data-related product; and (3) utilizing the constructed metadata lineage representation to perceive statistically interesting / significant correspondences (i.e., metadata indicators) between processing steps of the high-value producing data processing pipeline and metadata of the data-related product. As described above, the example may use the derived metadata indicators to identify and recommend high-value-impacting metadata characteristics of upstream / ancestor data-related products and processing steps that may produce useful downstream / descendant data-related products for a given data consumer / data scientist's target data analysis project.

[0026] In various instances, the examples may derive metadata indicators by "propagating" the value indicators through a metadata lineage representation of high-value data-related products. In some instances, the examples may achieve such value propagation by characterizing the value indicators as metadata characteristics associated with high-value data-related products. The examples may then derive metadata indicators using a cross-occurrence calculation that effectively causes the value indicators to propagate through a metadata lineage of high-value data-related products. As described above, these calculated / derived metadata indicators may represent statistically interesting / significant correspondences perceived across a metadata lineage of a data processing pipeline that consistently result in high-value downstream data-related products.

[0027] In various instances, an example may propagate a value indicator through a data lineage representation for multiple downstream / descendant data-related products. For a given downstream / descendant data-related product, the data lineage representation may include an association of a lineage between the given downstream / descendant data-related product and its ancestor data sets and ancestor processing steps (i.e., the data sets and processing steps used to produce the downstream / descendant data-related product). In many cases, multiple descendant / downstream data-related products that are not ancestors / descendants of each other may share ancestors. An example may propagate a value indicator through a data lineage representation by: (1) attributing a value indicator of a given downstream / descendant data-related product to each ancestor data set and ancestor processing step of the given downstream / descendant data-related product; and (2) for a given ancestor data set or ancestor processing step, aggregating one or more of the value indicators attributed to the given ancestor data set or ancestor processing step (as described above, in some cases, multiple downstream / descendant data-related products may share ancestors). In some instances, an example may reduce an aggregate value indicator for a given ancestor based on the cost of implementing the aggregate value indicator for the given ancestor. Similarly, examples can reduce the aggregate value indicator for a given ancestor based on the remoteness between the given ancestor and one or more of its associated downstream / descendant data-related products. Here, the remoteness between a given ancestor and a given downstream / descendant data-related product can be measured by the number of ancestors between the given ancestor and the given downstream / descendant data-related product along a representation of the pedigree of the given ancestor and the given downstream / descendant data-related product (e.g., mother to grandmother to great-grandmother, etc.). Examples can utilize various value propagation algorithms to perform / calculate the above-mentioned attributes, aggregation, and reduction.

[0028] By propagating value indicators via data lineage representations, examples can assign passed-value credits to (upstream / ancestor) data-related products and processing steps that have been used to produce high-value (downstream / descendant) data-related products. Thus, examples can enhance data analysis systems by enabling them to rank data-related products and processing steps according to perceived value (which, as described above, is based on actual human behavior and can therefore be commensurate with actual human value judgments). Such rankings can be used to inform recommendations, such as novel combinations of data sets and processing steps for producing new data-related products that may be useful / valuable to a data consumer / scientist's target data analysis project.

[0029] Examples of the disclosed technology can provide many advantages over existing automatic data recommendation technologies. For example, examples of the disclosed technology do not rely on data generators and processors that create explicit metadata related to the potential use of their data. Instead, the example provides a data analysis system that analyzes human behavior and combines data pedigrees with semantic analysis of data processing steps to derive their own value indicators and implicit metadata (e.g., metadata indicators) that can drive intelligent recommendations. As another example, examples of the disclosed technology consider human judgment when making recommendations. By utilizing value indicators, examples can provide recommendations that are commensurate with actual human value judgments. As a third example, examples of the disclosed technology can extend the data processing pipeline by recommending novel data-related products (i.e., data-related products that do not yet exist). In other words, the example can not only recommend existing data-related products, but also recommend processing steps for transforming existing data-related products into brand-new products. As will be described below, these novel / new data-related products can be intelligently customized for target data analysis projects of data consumers.

[0030] The benefits provided by the examples of the disclosed technology can be extended to far beyond the recommended data-related products and processing steps. For example, a common problem in many enterprises is that valuable data (e.g., data sets and other data-related products that can be used to produce new data-related products that may be valued by the data community) are not even shared at the beginning because the producer lacks the time and expertise to (1) make the data useful, or (2) ensure that the negative consequences associated with the use of the data are mitigated. Relatedly, even when the relevant data is identified by the data consumer, it may be unclear how to access and / or utilize it. Ensuring that the insights derived from the data are credible (unbiased, robust, interpretable, etc.) may also be an important challenge. The examples of the disclosed technology can free data producers and consumers from these burdens by providing a data analysis system that automatically recommends appropriate trustworthy data sets and processing steps for utilizing them using the collective expertise of the data community. These techniques can also be extended to provide reverse recommendations to data producers. For example, the example can suggest suitable consumption tasks / processing steps that producers of similar data have allowed, as well as the option of instantiating those consumption tasks / processing steps in a verifiable trusted environment. Examples can then facilitate sharing of resulting outputs.

[0031] Additionally, it should be understood that while in some examples, recommendations made by examples of the disclosed technology may be based on recommendation indicators or co-occurrence of certain metadata on a data set, in some other examples, recommendations may be based on other forms of analysis, including deep learning methods. For example, the extracted metadata and value indicators / credits may be input into a deep learning model, such as a deep learning recommendation model (DLRM) or Wide & Deep. In such examples, recommendations may be generated based on additional factors not captured in the indicators and co-occurrences.

[0032] Figure 1 An example data lineage representation 100 is depicted in accordance with various examples of the disclosed technology. The data lineage representation 100 represents a data lineage for downstream data-related products 122-126, which includes upstream data-related products (i.e., upstream data-related products 102-108) and processing steps (i.e., processing steps 112-116) from which the downstream data-related products are derived.

[0033] The upstream data-related products 102-108 may be various types of data-related products, such as original data sources, data sets, databases, analytical models derived from data, dashboards for displaying one or more of the aforementioned data-related products, etc. The processing steps 112-116 may be various types of processing steps (e.g., software programs / algorithms that operationalize data sets into analytical models, artificial intelligence or machine learning models, etc.), which transform the upstream data-related products into new downstream data-related products (e.g., new data sets derived from upstream data-related products, analytical models derived from upstream data-related products, dashboards for displaying one or more of the aforementioned data-related products, etc.).

[0034] As depicted, the processing step 112 receives the upstream data-related product 102 and the upstream data-related product 104 as inputs and transforms them into the downstream data-related product 122. Therefore, the upstream data-related product 102 and the upstream data-related product 104 may be referred to as “ancestor” data-related products of the downstream data-related product 122. Similarly, the processing step 112 may be referred to as a “ancestor” processing step of the downstream data-related product 122.

[0035] The processing step 114 receives the upstream data-related product 104 and the upstream data-related product 106 as inputs and transforms them into the downstream data-related product 124. Therefore, the upstream data-related product 104 and the upstream data-related product 106 can be referred to as “ancestor” data-related products of the downstream data-related product 124. Similarly, the processing step 114 can be referred to as an “ancestor” processing step of the downstream data-related product 124. Here, it can be noted that the downstream data-related product 122 and the downstream data-related product 124 share the upstream data-related product 104 as an ancestor.

[0036] The processing step 116 receives the upstream data-related product 106 and the upstream data-related product 108 as input and transforms them into the downstream data-related product 126. Therefore, the upstream data-related product 106 and the upstream data-related product 108 can be referred to as the "ancestor" data-related products of the downstream data-related product 126. Similarly, the processing step 116 can be referred to as the "ancestor" processing step of the downstream data-related product 126. Here, it can be noted that the downstream data-related product 124 and the downstream data-related product 126 share the upstream data-related product 106 as an ancestor.

[0037] exist Figure 1 In the specific example of , downstream data-related products 122 and downstream data-related products 124 are analytical models deployed in a production environment. In contrast, downstream data-related product 126 is an analytical model that has never been deployed to production.

[0038] Figure 1 A high-level conceptual overview of how examples of the disclosed technology may propagate value indicators through a data lineage representation in order to discover / identify high-value upstream data-related products and processing steps for producing new and useful (downstream) data-related products may be illustrated. As will be described in more detail in conjunction with subsequent figures, a similar process may be used to propagate value indicators through a metadata lineage representation in order to discover "high-value-impacting" metadata features.

[0039] As described above, examples of the disclosed technology observe data and its processing in terms of objective attributes (i.e., objective attributes of the data and the code used to process the data), but also observe as evidence of human judgment and human action (the examples can use the above-mentioned value indicators / valuation algorithms to quantify these human judgments and behavioral actions). This dual observation allows the examples to provide data analysis systems that examine the objective attributes of data and processing steps to predict future human judgments and actions related to the data. Therefore, these data analysis systems can examine the data lineages that currently exist (including data-related products and processing steps, which include data lineages), and can also examine whether they are likely to exist. Therefore, the examples can provide enhanced data analysis systems that predict and recommend (upstream) data-related products and processing steps that may produce useful / valuable (downstream) data-related products for data consumers' data analysis projects. In addition, these data analysis systems can make these predictions without having to understand what human judgment really means. In other words, the data analysis system does not need to know what data quality means to understand the behavior of the data community (e.g., data producers, data engineers / scientists, data consumers, etc.) as if a given data set or other data-related product is of high quality.

[0040] See again Figure 1, downstream data-related products 122 and downstream data-related products 124 represent analytical models deployed in production environments by data consumers / business users (e.g., downstream data-related products 122 and downstream data-related products 124 may be analytical models used by local governments to predict the spread of vector-borne diseases in cities). In contrast, downstream data-related products 126 represent analytical models that have never been deployed into production (e.g., downstream data-related products 126 may represent analytical models for predicting the spread of vector-borne diseases in cities that have never been operationalized). As described above, examples may use valuation algorithms to derive / calculate value indicators for downstream data-related products 122-downstream data-related products 126. These value indicators may represent quantifiable measures of the value of downstream data-related products 122-downstream data-related products 126 based on perceived human behavior associated with the use of downstream data-related products 122-downstream data-related products 126. For example, examples may utilize a simple valuation algorithm that quantifies a value measure for a given downstream data-related product based on the duration that a human / data consumer deploys the given downstream data-related product in production. Here, downstream data-related product 122 and downstream data-related product 124 may each have been deployed in production for one year. Therefore, the example may use a simple valuation algorithm to derive a value measure, such as v=1, for downstream data-related product 122 and downstream data-related product 124. In contrast, because downstream data-related product 126 has never been deployed in production, the simple valuation algorithm may calculate a lower value measure, such as v=0, for downstream data-related product 126.

[0041] In response to deriving value indicators for downstream data-related products 122-downstream data-related products 126, the example can "propagate" these value indicators back through the data lineage representation 100 to assign / attribute passing value credits to the ancestor data-related products and processing steps of the downstream data-related products 122-downstream data-related products 126. In this way, the example can discover / identify the (upstream) data-related products and processing steps that are most likely to produce new and useful / valuable (downstream) data-related products.

[0042] As described above, the examples may use various propagation algorithms to propagate value indicators through the data lineage representation 100. For example, the examples may utilize a propagation algorithm that (1) attributes a value indicator of a given downstream data-related product to each ancestor data-related product and ancestor processing step of the given downstream data-related product; and (2) aggregates one or more of the value indicators attributed to the given ancestor data-related product or ancestor processing step for a given ancestor data-related product or ancestor processing step (as depicted, in some cases, downstream data-related products 122-downstream data-related products 126 share ancestors, such as upstream data-related product 104 and upstream data-related product 106). In some instances, the examples may reduce the aggregate value indicator for a given ancestor based on the cost of implementing the aggregate value indicator for the given ancestor. Similarly, the examples may reduce the aggregate value indicator for a given ancestor based on the remoteness between the given ancestor and one or more associated downstream / descendant data-related products of the given ancestor. Here, the remoteness between a given ancestor and a given downstream / descendant data-related product may be measured by the number of ancestors between the given ancestor and the given downstream / descendant data-related product (thus, the remoteness between the upstream data-related product 102 and the downstream data-related product 122 may be greater than the remoteness between the processing step 112 and the downstream data-related product 122). Examples may utilize various propagation algorithms to perform / calculate the above-described attribution, aggregation, and reduction.

[0043] like Figure 1 As depicted in , an example may perform the above-described value propagation. Here, the relative shading of upstream data-related products, processing steps, and downstream data-related products may represent examples of value credits that have been attributed to them.

[0044] As illustrated by their relative shading, based on their respective value indicators, the example attributes a higher value to downstream data-related products 122 and downstream data-related products 124 than to downstream data-related product 126. As described above, the example may derive value indicators for downstream data-related products 122-126 using a valuation algorithm that calculates a value measure based on the duration that a given downstream data-related product has been deployed in production by a human / data consumer.

[0045] like Figure 1As depicted in the example of , through the data lineage representation 100, the example has assigned passed value credits to the processing steps and upstream data-related products used to produce the downstream data-related products 122 and the downstream data-related products 124 by back-propagating the value indicators for the downstream data-related products 122-downstream data-related products 126. In this way, the example has attributed value to the upstream data-related products and processing steps from which the downstream data-related products 122 and the downstream data-related products 124 were derived.

[0046] Due to this propagation, the example has attributed delivery value credits to processing steps 112-116. As indicated by their relative shading, the example attributes higher value credits to processing steps 112 and 114, rather than to processing step 116. This is because only processing steps 112 and 114 have produced analytical models deployed in production (i.e., downstream data-related products 122 and downstream data-related products 124). In other words, through value propagation, the example has attributed higher value indicators (e.g., v=1) of downstream data-related products 122 and downstream data-related products 124 to processing steps 112 and 114, respectively. In contrast, the example attributes a lower value indicator (i.e., v=0) of downstream data-related products 126 to processing step 116. Here, the lighter shading of processing step 112 and processing step 114 compared to downstream data-related product 122 and downstream data-related product 124 may represent a discount on the value credit for processing step 112 and processing step 114 because they are farther upstream in the data flow / data processing pipeline (in other words, they are farther away from the direct value indicator for deployment in production).

[0047] Extending the value propagation to the upstream data-related product level, the example has also attributed (and in the case of upstream data-related products 104 and 106, aggregated) the delivered value credits of upstream data-related products 102-upstream data-related products 108. As depicted by the relative shading, the example aggregated the highest value of upstream data-related product 104 because it was used to derive two in-production analytical models (i.e., downstream data-related product 122 and downstream data-related product 124). In other words, the example has aggregated the value indicators of both upstream data-related products 104 in-production descendants (i.e., downstream data-related product 122 and downstream data-related product 124) for upstream data-related product 104. Thus, the aggregated value of the input data-related information 104 may be v=2 (i.e., v=1+v=1), minus some discount level based on various factors such as the cost of achieving the aggregated value of the upstream data-related product 104 or the remoteness between the upstream data-related product 104 and its in-production descendants. As depicted, the example attributes the lowest value to the upstream data-related product 108 because no in-production analytical model was derived therefrom.

[0048] As will be described in more detail below, when implemented in a data analysis system tasked with recommending potential (upstream) data-related products and processing steps for producing new and useful (downstream) data-related products, the example may identify any of processing step 112 and processing step 114 and upstream data-related product 104 as (upstream) data-related products and processing steps that may produce new and useful / valuable (downstream) data-related products for, for example, a data consumer data analysis project.

[0049] Figure 2 An example data analysis system 200 is depicted according to various examples of the disclosed technology. As depicted, the data analysis system 200 can be used to: (1) extract, characterize, and catalog metadata (such metadata can be implicit or explicit metadata) from a data processing pipeline (such metadata extraction can include extracting data lineage information for the data processing pipeline); and (2) discover and subsequently recommend (a) (upstream) data-related products, (b) processing steps, and / or (c) metadata characteristics of (upstream) data-related products and processing steps - which may produce new and useful / valuable (downstream) data-related products for target data analysis projects of data consumers (as described above, examples can use value indicators to quantify value). The data analysis system 200 can make these discoveries by propagating value indicators via data lineage representations (and in some cases metadata lineage representations) for the data processing pipeline.

[0050] The data analysis system 200 includes a metadata enricher 210, a value indicator propagator 220, and a data pipeline catalog and recommender 230. Each subsystem and its associated functions will be described in more detail below. It should be understood that Figure 2 These are merely examples, and in other examples, the functions described below may be performed by other systems / subsystems.

[0051] Metadata Enricher 210: Metadata Enricher 210 may be a software or hardware program that extracts and characterizes metadata from a data processing pipeline, including data-related products and processing steps of the data processing pipeline.

[0052] As described above, the metadata enricher 210 can use various algorithms (e.g., artificial intelligence or machine learning algorithms) to extract metadata from data-related products and processing steps of the data processing pipeline. For data sets and other data-related products, the extracted metadata can include various types of metadata, such as field names, data sizes, data distribution, etc. For processing steps, the extracted metadata can include various types of metadata describing the processing steps, such as information about which input fields are used, information about data-related products input into the processing steps, information about data-related products produced by the processing steps, the content of the associated documents for the processing steps, the classification of the processing algorithms for the processing steps, code dependencies and whether the processing steps are in production, staging or QA status, unit testing information, the names of developers / data scientists involved in the development of the processing steps, etc. As will be described below. Examples can combine this extracted metadata with data lineage information of the data processing pipeline (which can also be part of the metadata extracted by the metadata enricher 210 from, for example, query logs, workflow records, lineage tracking tools, etc.) to create a metadata lineage representation of the data processing pipeline. In some instances, the examples can propagate the extracted metadata features forward and backward through the data lineage representations and social graphs of developers associated with the data processing pipeline. As described above, by performing such propagation, the examples can discover (upstream) data-related products, processing steps (and even people involved in processing steps), and / or metadata features of (upstream) data-related products and processing steps - potentially generating new and useful / valuable (downstream) data-related products for the data consumer's target data analysis project.

[0053] In various examples, the metadata enricher 210 can execute a metadata enricher pipeline (e.g., an artificial intelligence or machine learning algorithm) to intelligently characterize the metadata extracted from the data processing pipeline. These metadata enricher pipelines can include configurable stages that provide various forms of statistical characterizations for categories of extracted metadata. The metadata enricher 210 can also use the metadata enricher pipeline to perform basic quality checks and trustworthy data assessments, and even generate / derive initial values ​​and metadata indicators.

[0054] For example, the metadata enricher 210 can use a metadata enricher pipeline to derive / generate "implicit metadata" (e.g., content tags, metadata representations, etc.) for a given data-related product, processing step, or an entire data processing pipeline by (a) consulting a table with possible metadata values, (b) analyzing mathematical properties of metadata values, or (c) utilizing other heuristic methods.

[0055] As an example, if a given dataset has a column containing the values ​​"San Francisco", "San Jose", and "Denver", the example may determine that the labels for the column should be "CITY" and "CITY-US". Individual values ​​may have additional labels, but the column will only have labels that are common to all values ​​in the column. Thus, even though San Francisco is a city in California, Mexico, and Costa Rica (and could therefore be labeled with CITY-CA-US, CITY-MX, and CITY-COSTA-RICA), and Denver is a city in Colorado, Indiana, and Australia, the example may use the common label / metadata representation of San Francisco and Denver because they are both cities in the United States.

[0056] The example may also examine numeric values ​​when deriving / generating "implicit metadata". For example, the example may characterize / label a dataset column with values ​​of 0 or 1 as DISCRETE_BINARY. The example may also identify values ​​that appear to have a common statistical distribution such as a normal or exponential distribution and label them accordingly. Similarly, if most of the values ​​of a column are 0, but the remaining values ​​have a recognizable distribution, the example may characterize / label the column as a mixed discrete-continuous distribution. Within a specific domain, the example may identify specialized distributions based on column naming conventions combined with values. Thus, the example may characterize / label values ​​such as temperature, overdraft fees, or other types of specialized values.

[0057] Value Indicator Propagator 220: Value Indicator Propagator 220 may be a software or hardware program that propagates value and metadata indicators through a data lineage representation (or in some cases a metadata lineage representation) of a data processing pipeline using various propagation algorithms. As described above, by performing such value indicator propagation, examples may discover (upstream) data-related products, processing steps, and / or metadata characteristics of (upstream) data-related products and processing steps - potentially generating new and useful / valuable (downstream) data-related products. Although already incorporated Figure 1 is discussed, but additional examples of value indicator propagation are discussed in subsequent figures.

[0058] Data Pipeline Catalog and Recommender 230: The data pipeline catalog and recommender 230 can be a software or hardware program that catalogs extracted metadata associated with data processing pipelines (including data-related products and processing steps including them) and recommends (upstream) data-related products and processing steps for producing new (downstream) data-related products based in part on value indicator propagation.

[0059] Data steps are typically executed in the favorite environment of the data scientist or data user (which may include their personal laptop, external system, or even remote edge). To allow such decentralized platform-agnostic operation, the data pipeline catalog and recommender 230 can maintain exportable data-related packages (e.g., exportable dataset packages for individual datasets, exportable processing step packages for individual processing steps, exportable data processing pipeline packages for individual data processing pipelines, exportable data lineage representation packages, exportable metadata representation packages, etc.), which encapsulate various forms of data and metadata, including data references, information for reconstructing exportable data-related packages, information for extracting / generating data lineage representations, information for generating metadata lineage representations, etc. Any user operating within the data processing pipeline (data collector, data scientist, domain expert, etc.) can use tools that operate on exportable data-related packages to inject explicit metadata into the exportable data-related packages, or even create new exportable data-related packages. Such tools may allow users to create lineage references for exportable data-related packages (which may be used by the data analysis system 200 to extract / generate data lineage representations of data processing pipelines). Users may also utilize these tools to attribute delivery value credits to data-related products and processing steps associated with exportable data-related packages, or even inject implicit recommendation indicators into exportable data-related packages. The data pipeline catalog and recommender 230 may index the exportable data-related packages in a catalog, which may include a search engine (e.g., ElasticSearch) to provide recommendations.

[0060] As described above, examples can extract / generate data lineage representations for data processing pipelines that associate lineages between upstream and downstream data-related products and the processing steps that transform them. Examples can also extract, characterize, and catalog metadata for data-related products and processing steps of data processing pipelines. Figure 3 As described, the examples may combine a data lineage representation of a data processing pipeline and the extracted metadata to construct a metadata lineage representation for the data processing pipeline. In turn, the examples may utilize these metadata lineage representations to derive metadata indicators that represent statistically interesting / significant correspondences perceived on the metadata lineage of "high-value" data-related products. The examples may utilize the derived metadata indicators to identify "high-value-impacting metadata characteristics" (i.e., (1) metadata characteristics of (upstream) data-related products and processing steps that may increase / impact the value of (downstream) data-related products produced therefrom, and / or (2) metadata characteristics of data-related products that may increase / impact the value of the data-related products themselves). As indicated by the metadata indicators, these "high-value-impacting metadata characteristics" may be metadata characteristics that appear with unusual / statistically significant frequencies on the metadata lineage of "high-value" data-related products.

[0061] In various examples, the data pipeline catalog and recommender 230 can index data related products and processing steps based on: (a) extracted metadata, (b) value credits accumulated / propagated back through lineage representations, and (c) recommendation indicators. The index can include, for example, a search engine, a similarity index for data distribution, and a graph database.

[0062] While in some examples, recommendations may be based on recommendation indicators or co-occurrence of certain metadata on a data set, in some other examples, recommendations may be based on other forms of analysis, including deep learning methods. For example, the extracted metadata and value indicators / credits may be input into a deep learning model, such as a Deep Learning Recommendation Model (DLRM) or Wide & Deep. In such examples, recommendations may be generated based on additional factors not captured in the indicators and co-occurrences.

[0063] Figure 3 Depicted is an example metadata lineage representation 350 of a data processing pipeline 300 according to various examples of the disclosed technology. Metadata lineage representation 350 includes associations of lineages between upstream data-related products 302 and 304, processing steps 310, and downstream data-related products 320.

[0064] exist Figure 3In the simplified example of , the upstream data-related product 302 may be a data set containing data related to sales transactions, and the upstream data-related product 304 may be a data set containing locations. Although not depicted, the upstream data-related product 302 and the upstream data-related product 304 may be outputs of other processing steps of the data processing pipeline 300. The upstream data-related product 302 and the upstream data-related product 304 may also be outputs from processing steps of other data processing pipelines.

[0065] Processing step 310 may be a join type processing step that receives upstream data related product 302 and upstream data related product 304 as input and joins / transforms them into downstream data related product 320 , which may be a management sales dashboard.

[0066] As depicted, an example of the disclosed technology (e.g., data analysis system 200) may have extracted metadata describing upstream data-related products 302, upstream data-related products 304, and downstream data-related products 320. For example, the extracted metadata for upstream data-related products 302 may include table names ("sales"), column names ("location", "order", "date", and "total", etc.), and content tags ("*uuid", "*uuid", "*data", "*numeric", etc.). The extracted metadata for upstream data-related products 304 and downstream data-related products 320 may include similar descriptions.

[0067] Examples of the disclosed technology may also have extracted metadata describing the processing step 310. In particular, the extracted metadata for the processing step 310 includes a programming language description ("SQL") and a description of the query engine / algorithm used for the processing ("presto").

[0068] As depicted, metadata lineage representation 350 contains an association of the lineage of the above-described metadata extracted from data processing pipeline 300. In other words, the example may associate metadata extracted from processing step 310 with its output(s) (i.e., downstream data-related products 320). Likewise, the example may associate metadata extracted from upstream data-related products 302 and 304 with (a) the processing step that transformed it (i.e., processing step 310) and (b) the downstream data-related products into which it was transformed (i.e., downstream data-related products 320). Thus, the metadata extracted for upstream data-related products 302 and upstream data-related products 304 may be referred to as "input metadata" for processing step 310 or output downstream data-related products 320. Similarly, the metadata extracted for processing step 310 may be referred to as "producer metadata" for downstream data-related products 320. Furthermore, assuming that the data flow is acyclic, the transitive union of metadata for upstream data-related products 302-upstream data-related products 304 and processing steps 310 may be referred to as "ancestor metadata" for downstream data-related products 320. Similarly, the transitive union of metadata for processing steps 310 and downstream data-related products 320 may be referred to as "descendant metadata" for upstream data-related products 302-upstream data-related products 304. As will be described in greater detail below, examples may utilize metadata lineage representations such as metadata lineage representation 350 to discover / identify unusual common metadata characteristics shared between: (a) high-value data-related products (again, examples may use derived value indicators to attribute / quantify the value of data-related products); and (b) (upstream) data-related products and processing steps used to produce high-value data-related products. Thus, examples may provide a data analysis system that recommends these metadata properties for: (a) data-related products that may be valued by the data community; or (b) (upstream) data-related products and processing steps for producing (downstream) data-related products that may be valued by the data community.

[0069] Figure 4 An example data lineage representation 400 according to various examples of the disclosed technology is depicted. Here, Figure 4 Example use cases may be illustrated that describe how examples of the disclosed technology may recommend (1) (upstream) data sets and processing steps for producing new and useful / valuable data-related products, (2) metadata characteristics such as (upstream) data sets and processing steps; and / or (3) metadata characteristics for new / prospective data-related products that may be useful / valuable for a data consumer's targeted data analysis project.

[0070] exist Figure 4In the example of , a data consumer may want to create an analytical model for surveillance and prevention of vector-borne diseases in the United States (i.e., the data consumer's target data analysis project). As will be described below, examples of the disclosed technology (e.g., data analysis system 200) can assist a user by recommending: (1) (upstream) data sets and processing steps for generating a desired analytical model, (2) metadata features for such (upstream) data sets and processing steps; and / or (3) metadata features for a desired analytical model that may improve the value of the desired analytical model.

[0071] First, an example of the disclosed technology may utilize a value indicator to identify "valuable" data-related products utilized in a data analysis project similar to a user's target data analysis project (i.e., vector-borne disease monitoring and prevention). Thus, the example may identify data-related products 442 and 444, which may be analytical models used for disease prevention in countries outside the United States. The value indicators associated with data-related products 442 and 444 may quantify the value level of data-related products 442 and 444 based on various human behavior actions, such as (1) how long they have been deployed in production; (2) how many users / data consumers have deployed them in production; (3) the quality / accuracy scores attributed to them, etc. As described above, the example may utilize a value indicator to compare / rank similar data-related products according to perceived value. Thus, the example may have selected data-related products 442 and 444 because they have the highest perceived value (as quantified by their associated value indicators) among the identified disease prevention and monitoring models. In some instances, the example may also select data-related product 442 and data-related product 444 because they have desirable metadata characteristics (eg, weekly granularity).

[0072] As described above, the example may construct a data lineage representation (i.e., data lineage representation 400) for data-related product 442 and data-related product 444 that includes an association of lineages between data-related product 442 and data-related product 444 and their ancestor processing steps (i.e., processing steps 412-processing steps 418 and processing steps 432-processing steps 434) and ancestor datasets (i.e., datasets 402-datasets 404 and datasets 422-datasets 428). In some instances, data lineage representation 400 may be a metadata lineage representation that includes an association of lineages between metadata features extracted from data-related product 442 and data-related product 444 and their ancestors.

[0073] Using data lineage analysis, the example can sense / discover that data-related products 442 and data-related products 444 have been derived from two common ancestor datasets (i.e., dataset 402 and dataset 404). Dataset 402 may be a dataset containing quarterly disease prevalence by city data. The metadata extracted for dataset 402 may include labels representing word distributions reflecting medical terms, and the names and distributions of columns for date ranges, geographic regions, and cases by category / state, etc. Dataset 404 may be a dataset containing daily weather data. The metadata extracted for dataset 404 may include labels representing word distributions reflecting weather terms, and the names and distributions of columns for date ranges, geographic regions, temperature, humidity, precipitation, etc. Such extracted metadata may be referred to as implicit metadata as long as it is not intentionally labeled / created by the data producer / consumer.

[0074] Again using data lineage analysis, the example can sense / discover that processing step 412 and processing step 414 have been used to transform data set 402 and data set 404 into data set 422 and data set 424, respectively. In particular, processing step 412 and processing step 414 may have been used to interpolate data set 402 and data set 404 to a grid on a daily basis. Therefore, data set 422 and data set 424 may have metadata characteristics at a daily granularity.

[0075] Similarly, the example may sense / discover that processing step 416 and processing step 418 have been used to transform data set 402 and data set 404 into data set 426 and data set 428, respectively. Specifically, processing step 416 and processing step 418 may have been used to interpolate data set 402 and data set 404 to a grid on a weekly basis. Therefore, data set 426 and data set 428 may have metadata characteristics at a weekly granularity.

[0076] The example may also sense / discover that (1) processing step 432 has been used to transform data set 422 and data set 424 into data-related product 442; and (2) processing step 434 has been used to transform data set 426 and data set 428 into data-related product 444. Therefore, data-related product 442 and data-related product 444 may be analytical models that utilize daily / weekly disease prevalence and weather data to make predictions related to disease surveillance and prevention.

[0077] As will be described in more detail below, the examples may utilize metadata lineage analysis to sense / identify statistically interesting / significant correspondences (i.e., metadata indicators) on the metadata lineages of data-related products 442 and data-related products 444. The examples may utilize metadata indicators to identify "high-value-impact metadata characteristics". As described above, "high-value-impact metadata characteristics" may refer to (a) metadata characteristics of (upstream) data-related products / datasets or processing steps that may increase / impact the value of (downstream) data-related products generated from (upstream) data-related products / datasets or processing steps; and / or (b) metadata characteristics of data-related products that may increase / impact the value of the data-related products themselves. As indicated by the metadata indicators, these "high-value-impact metadata characteristics" may be metadata characteristics that appear with unusual / statistically significant frequencies on the metadata lineages of data-related products 442 and data-related products 444. The examples may recommend novel combinations of (upstream) datasets and processing steps (having these "high-value-impact metadata characteristics") for producing new and useful / valuable (downstream) data-related products. Thus, the example may recommend data sets and processing steps having these high-value-impact metadata features for use in generating valuable analytical models for disease prevention and surveillance in the U.S. Relatedly, the example may recommend certain of these high-value-impact metadata features for improved analytical models used for disease prevention and surveillance in the U.S.

[0078] In various examples, the examples can train a machine learning model that learns to discover metadata indicators representing cross-occurrences / relationships between metadata features representing data lineages 400. In other words, the machine learning model can learn to discover metadata indicators representing cross-occurrences / relationships between metadata features representing (upstream) data sets and processing steps and their corresponding descendants from these lineages, including metadata characterizing interpolation steps and resulting high-transfer value intermediate artifacts of appropriate granularity for each task.

[0079] Figure 5 An example computing system is depicted that can be used to recommend at least one of an existing data set and an existing processing step for producing a new / prospective data-related product having desired metadata characteristics in accordance with various examples of the disclosed technology.

[0080] See now Figure 5 , computing component 510 may be, for example, a server computer, a controller, or any other similar computing component capable of processing data. Figure 5 In an example implementation of , computing component 510 includes a hardware processor 512 and a machine-readable storage medium 514 .

[0081] The hardware processor 512 may be one or more central processing units (CPUs), semiconductor-based microprocessors, and / or other hardware devices suitable for retrieving and executing instructions stored in the machine-readable storage medium 514. The hardware processor 512 may fetch, decode, and execute instructions such as instructions 516-520 to control the process or operation of burst preloading for available bandwidth estimation. As an alternative or in addition to retrieving and executing instructions, the hardware processor 512 may include one or more electronic circuits including functional electronic components for executing one or more instructions, such as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other electronic circuits.

[0082] A machine-readable storage medium, such as machine-readable storage medium 514, may be any electrical, magnetic, optical, or other physical storage device that contains or stores executable instructions. Thus, machine-readable storage medium 514 may be, for example, a random access memory (RAM), a non-volatile RAM (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a storage device, an optical disk, etc. In some examples, machine-readable storage medium 514 may be a non-transitory storage medium, where the term "non-transitory" does not include a transient propagation indicator. As described in detail below, machine-readable storage medium 514 may be encoded with executable instructions, such as instructions 516-520.

[0083] As described above, computing system 500 can be used to recommend at least one of an existing data set and an existing processing step for producing a new / expected data-related product having desired metadata characteristics. The new / expected data-related product having the desired metadata characteristics can be used to perform a target data analysis project of a data consumer.

[0084] Thus, the hardware processor 512 can execute instructions 516 to receive the desired metadata characteristics for the expected data-related product. In various examples, the hardware processor 512 can receive the desired metadata characteristics from a data consumer (e.g., a business user, a data scientist, a data engineer, a data collector, a data provider, etc.) who wants to deploy the expected data-related product in a target data analysis project. In other examples, the hardware processor 512 can automatically identify the desired metadata characteristics in response to receiving other parameters related to the target data analysis project of the data consumer (e.g., the type of project, the geographic parameters of the project, the type of data-related product sought, etc.). In these examples, the hardware processor 512 can receive the desired metadata characteristics from the computing system 500.

[0085] The expected data-related products may include various types of data set-related products, such as data sets, analytical models, dashboards for displaying one or more of the aforementioned data-related products, etc. The target data analysis project may be various types of data analysis projects, such as forecasting projects (e.g., predicting weather patterns), modeling projects (e.g., modeling the spread of vector-borne diseases in a country), etc.

[0086] The expected metadata characteristics may be various types of metadata characteristics for data-related products. For example, the expected metadata characteristics may be metadata characteristics for a data set (e.g., a text document for a data set, the name of a table / column / feature of a data set, a description of the content within a data set, the size of data within a data set, the distribution of data within a data set, etc.), metadata characteristics for an analysis model (e.g., a programming language, a specific input field, a classification of algorithms, etc.), metadata characteristics for a dashboard (e.g., a text document for a dashboard, a display name, a type of display, etc.), etc. In some examples, the expected metadata characteristics may be metadata characteristics known to data consumers to be associated with high performance when performing a data analysis project similar to a target data analysis project.

[0087] As described above, in some cases, the hardware processor 512 can generate / identify the expected metadata characteristics in response to receiving parameters related to the target data analysis project from the data consumer. In these examples, the hardware processor 512 can predict that the expected metadata characteristics will be related to high performance in performing the target data analysis project. In some examples, the hardware processor 512 can use a value indicator to make this prediction.

[0088] As described above, the hardware processor 512 may derive a value indicator using a valuation algorithm that quantifies the value level of a given data-related product based on perceived human behavior associated with the use of the given data-related product (as used herein, a value indicator may refer to a quantifiable measurement that quantifies a given data-related product or processing step based on perceived human behavior associated with the use of the given data-related product / processing step). For example, the hardware processor 512 may utilize a simple valuation algorithm that quantifies the value level of an analytical model (i.e., an example of a data-related product) based on how long humans have deployed the analytical model in a production environment. Using this simple valuation algorithm, the hardware processor 512 may attribute a value of, for example, v=3 to an analytical model that has been deployed in production for three years, and a value of, for example, v=2 to an analytical model that has been deployed in production for two years. In various instances, the hardware processor 512 may utilize a more complex valuation algorithm that quantifies two or more categories of human behavioral actions associated with the use of a data-related product or processing step to a common numerical scale. For example, the hardware processor 512 may utilize a more complex valuation algorithm to quantify the level of value of an analytical model based on a common numerical scale based on any one or a combination of the following factors: (a) the duration that humans / users have deployed the analytical model in production; (b) the number of independent users who have deployed the analytical model into production; and (c) the quality score / accuracy score attributed to the analytical model (these quality scores / accuracy scores can be attributed to a given analytical model by humans, artificial intelligence, etc.). As described above, the hardware processor 512 may utilize a valuation algorithm to quantify the value level of two or more analytical models using a common numerical scale. Therefore, the hardware processor 512 may utilize these valuation algorithms to compare / rank the relative value of multiple analytical models based on multiple categories of human behavioral actions. Such valuation algorithms may provide an enhanced approximation of actual human value judgments.

[0089] As described above, the hardware processor 512 can utilize the value indicator to predict that the expected metadata characteristics will be associated with high performance in executing the target data analysis project. For example, the hardware processor 512 may discover that 10,000 data-related products have been deployed in a production environment by business users in similar data analysis projects similar to the target data analysis project of the data consumer. Here, the hardware processor 512 can utilize the derived value indicators for the 10,000 data-related products to rank the 10,000 data-related products according to perceived value. As described above (and as will be described in more detail below), the hardware processor 512 may discover that the expected metadata characteristics are unusually common among, for example, the 1000 data-related products having the highest perceived value (as quantified by their associated value indicators). Therefore, the hardware processor 512 can predict that the expected metadata characteristics will be associated with high performance in the target data analysis project of the data consumer.

[0090] In response to receiving the expected metadata characteristics for the expected data-related product, the hardware processor 512 may execute instructions 518 to determine metadata characteristics for the expected upstream data-related product for use in generating the expected data set-related output having the expected metadata characteristics.

[0091] The expected upstream data-related products may be various types of dataset-related products that may be used to produce the expected data-related products having desired metadata characteristics (e.g., datasets, analytical models, dashboards for displaying one or more of the aforementioned data-related products, combinations of the aforementioned dataset-related products, etc.).

[0092] Here, it should be understood that the expected upstream data-related product does not have to be an existing data-related product. As described below, in various examples, the hardware processor 512 can recommend, for example, an existing data set and processing steps for transforming the existing data set into an expected upstream data-related product (or a data-related product that is at least similar to the expected upstream data-related product).

[0093] In various examples, the hardware processor 512 can determine the metadata characteristics for the expected upstream data-related product by: (1) identifying one or more existing data-related products with the expected metadata characteristics; (2) generating an augmented metadata lineage representation for the one or more existing data-related products with the expected metadata characteristics, the augmented metadata lineage representation including the following sub-representations: (a) a direct metadata sub-representation; (b) an ancestor metadata sub-representation; and (c) a descendant metadata sub-representation; (3) calculating correspondences within the augmented metadata lineage representation; and (4) determining the metadata characteristics for the expected upstream data-related product based on the calculated correspondences. Each of these four steps will be described in more detail below.

[0094] Identifying one or more existing data-related products with desired metadata characteristics: The hardware processor 512 may use various techniques to identify one or more existing data-related products with desired metadata characteristics. In some examples, the hardware processor 512 may utilize a combination of Figure 2 The described data pipeline catalog 230 is used to search and identify one or more existing data-related products having desired metadata characteristics.

[0095] The identified existing data-related products may include various existing data-related products, such as existing data sets, existing analytical models, existing dashboards for displaying one or more of the aforementioned existing data-related products, and the like.

[0096] Generating an augmented metadata lineage representation for one or more identified existing data-related products having desired metadata characteristics: After identifying one or more existing data-related products having desired metadata characteristics, the hardware processor 512 can generate an augmented metadata representation for them. The augmented metadata representation can include the following sub-representations: (a) a direct metadata sub-representation; (b) an ancestor metadata sub-representation; and (c) a descendant metadata sub-representation

[0097] The direct metadata sub-representation may include a representation of metadata characteristics extracted from the identified existing data-related products having the desired metadata characteristics. The extracted metadata characteristics may include the desired metadata characteristics, as well as other metadata characteristics extracted from the identified existing data-related products having the desired metadata characteristics. In various examples, the hardware processor 512 may use a combination of Figure 2 Describes techniques to extract these metadata features.

[0098] The predecessor metadata sub-representation may include representations of metadata characteristics extracted from one or more predecessor data sets and one or more predecessor processing steps for the identified existing data-related products having the desired metadata characteristics. Here, the given predecessor data set may be an existing data set from which at least one of the identified existing data-related products having the desired metadata characteristics was derived using one or more of the predecessor processing steps.

[0099] The descendant metadata sub-representation may include metadata characteristics extracted from one or more descendant data-related products and one or more descendant processing steps for the identified existing data-related products having the desired metadata characteristics. Here, the given descendant data-related product may be an existing data-related product derived from at least one of the identified existing data-related products having the desired metadata characteristics using the one or more descendant processing steps.

[0100] As described above, the augmented metadata lineage representation may represent a "metadata lineage" for each of the identified existing data-related products having the desired metadata characteristics. By generating such an augmented metadata lineage representation, the hardware processor 512 may sense / identify statistically interesting / significant correspondences on the metadata lineage for the identified existing data-related products having the desired metadata characteristics (i.e., metadata indicators). These metadata indicators may then inform the hardware processor 512 of recommendations for processing steps and / or existing data sets for producing the desired data-related products having the desired metadata characteristics.

[0101] In various examples, the hardware processor 512 may generate an augmented metadata representation using: (1) a lineage representation (e.g., a lineage table); and (2) a metadata representation (e.g., a metadata table). For a given identified existing data-related product having desired metadata characteristics, the lineage representation may represent a representation of a lineage that relates the given identified existing data-related product to (a) its ancestor data sets and processing steps; and (b) its descendant data-related products and processing steps. The metadata representation may represent metadata extracted from (a) a given identified existing data-related product having desired metadata characteristics; (b) its ancestor data sets and processing steps; and (c) its descendant data-related products and processing steps.

[0102] As an example to illustrate this concept, a metadata representation / table may have a row for (a) each identified existing data-related product having the desired metadata characteristic; (b) each ancestor data set and processing step; and (c) each descendant data-related product and processing step. The metadata representation / table may then have a column for each atomic element of metadata.

[0103] As described above, a lineage representation / table may associate each identified existing data-related product having desired metadata characteristics with (a) its ancestor data sets and processing steps; and (b) its descendant data-related products and processing steps.

[0104] Using the lineage and metadata representation / table, the hardware processor 512 may compute an augmented metadata lineage representation that represents a "metadata lineage" for each identified existing data-related product having the desired metadata characteristics.

[0105] In various examples, the hardware processor 512 may use the following algorithm / function to compute the ancestor metadata: (i.e., metadata of the ancestor data sets and processing steps):

[0106]

[0107] Similarly, the hardware processor 512 may use the following algorithm / function to calculate the descendant metadata:

[0108]

[0109] Computing correspondences within the augmented metadata representation: In response to generating an augmented metadata pedigree representation for one or more identified existing data-related products having desired metadata characteristics, the hardware processor 512 can compute correspondences within the augmented metadata pedigree representation. In other words, the hardware processor 512 can compute correspondences between the metadata pedigrees of each identified existing data-related product having the desired metadata characteristics. In this manner, the hardware processor 512 can sense / identify statistically interesting / significant correspondences across the metadata pedigrees of the identified existing data-related products having the desired metadata characteristics. As described above, these statistically interesting / significant correspondences can be referred to as metadata indicators.

[0110] The calculated correspondences may include various types of correspondences, such as cross-occurrences within the augmented metadata representation, co-occurrences within the augmented metadata representation, etc. In some examples, the hardware processor 512 may utilize the direct metadata sub-representation to calculate co-occurrences between metadata characteristics for the identified existing data-related products with the desired metadata characteristics. In some examples, the hardware processor 512 may utilize the augmented metadata lineage representation to calculate cross-occurrences on the metadata lineage of the identified existing data-related products with the desired metadata characteristics. Such calculations may include calculating cross-occurrences between metadata characteristics of the identified existing data-related products with the desired metadata characteristics and their ancestors and / or descendants.

[0111] For example, the hardware processor 512 may calculate the cross-occurrence of metadata characteristics between the direct metadata sub-representation and the ancestor metadata sub-representation. Similarly, the hardware processor 512 may calculate the cross-occurrence of metadata characteristics between the direct metadata sub-representation and the descendant metadata sub-representation, and so on. The hardware processor 512 may then calculate the cross-occurrence of metadata characteristics between the direct metadata sub-representation and the descendant metadata sub-representation, and so on. 2 Tests such as statistical scores to identify statistically interesting / significant crossover occurrences.

[0112] As described above, while in some examples, recommendations of the disclosed technology may be based on recommendation indicators or co-occurrences of certain metadata on a data set, in some other examples, recommendations may be based on other forms of analysis, including deep learning methods. For example, the extracted metadata and value indicators / credits may be input into a deep learning model, such as a deep learning recommendation model (DLRM) or Wide & Deep. In such examples, recommendations may be generated based on additional factors not captured in the indicators and co-occurrences. For example, in some examples, a recommendation model (such as a hardware processor 512) may be trained to perform the above calculations using techniques such as matrix factorization or deep learning.

[0113] In various examples, the hardware processor 512 can generate a representation for the metadata indicators described above, which can be referred to as a metadata indicator representation. In various examples, the hardware processor 512 can utilize the metadata indicator representation to recommend metadata characteristics for data sets and processing steps that may produce valuable data-related products (similarly, the hardware processor 512 can use the metadata indicator representation to recommend metadata characteristics for valuable data-related products). In some examples, the metadata indicator representation can include a row for each direct metadata element and a column for each ancestor or descendant metadata element / value. An example metadata indicator representation / table is depicted as follows:

[0114]

[0115] The example metadata indicator representation / table described above has one row per atom direct metadata value and one column per unique value of ancestor or descendant metadata. Metadata indicator representations / tables such as the examples depicted above may typically be sparse (each row having no more than about one hundred non-zero values). In some instances, the hardware processor 512 may reduce the computations to create the metadata indicator representation / table by downsampling in the cross-occurrence computations to be linear in the total number of non-zero elements in the original augmented metadata representation.

[0116] Determine metadata characteristics for the expected upstream data-related product based on the calculated correspondences for producing the expected upstream data-related product with the desired metadata characteristics: As described above, the hardware processor 512 can derive metadata indicators (or in some examples, metadata indicator representations) from the augmented metadata pedigree representation that represent statistically interesting / significant correspondences perceived on the metadata pedigrees of the identified existing data-related products with the desired metadata characteristics. Examples can utilize these derived metadata indicators to identify metadata characteristics that appear with unusual / statistically significant frequencies on the metadata pedigrees of the identified existing data-related products with the desired metadata characteristics. As understood by examples of the disclosed technology, existing data sets and processing steps with these "key" metadata characteristics can have improved potential for producing data-related products with the desired metadata characteristics. Therefore, the hardware processor 512 can determine one or more of these "key" metadata characteristics as suitable / key metadata characteristics for the expected upstream data-related product for producing the expected data-related product with the desired metadata characteristics.

[0117] In response to determining metadata characteristics for an expected upstream data-related product for producing an expected data set-related output having the expected metadata characteristics, the hardware processor 512 may execute instructions 520 to recommend at least one of an existing data set and an existing processing step for producing the expected data-related product having the expected metadata characteristics.

[0118] As described above, the recommended existing dataset(s) and / or processing steps may include one or more of the metadata characteristics determined for the expected upstream data-related product for producing the expected data-related product having the desired metadata characteristics. In other words, individually or when combined, the recommended existing dataset(s) and / or processing steps may approximate the expected upstream data-related product for producing the expected data-related product having the desired metadata characteristics.

[0119] As described above, the expected upstream data-related product for producing the expected data-related product with the expected metadata characteristics may not be an existing data-related product. Therefore, the hardware processor 512 may recommend, for example, an existing data set and an existing processing step that have never been combined. However, the hardware processor 512 may predict that the recommended existing processing steps will transform the recommended existing data set into a data-related product that is similar to the expected upstream data-related product for producing the expected data-related product with the expected metadata characteristics. In this way, the hardware processor 512 may recommend currently existing data sets and processing steps, but may also recommend situations where they are plausible.

[0120] Figure 6 An example computing system 600 is depicted that can be used to generate metadata indicator representations according to various examples of the disclosed technology. Figure 6 , computing component 610 may be, for example, a server computer, a controller, or any other similar computing component capable of processing data. Figure 6 In an example implementation of , computing component 610 includes a hardware processor 612 and a machine-readable storage medium 614 .

[0121] Hardware processor 612 and machine-readable storage medium 614 may be the same / similar to hardware processor 512 and machine-readable storage medium 514, respectively. Thus, machine-readable storage medium 614 may be encoded with executable instructions, such as instructions 616-620.

[0122] The hardware processor 612 may execute instructions 616 to generate an augmented metadata lineage representation of the one or more primary data sets. In various examples, the hardware processor 612 may have identified the one or more primary data sets as valuable data sets based on value indicators derived from human behavior (e.g., usage of the primary data sets in numerous data processing pipelines). In other examples, the one or more primary data sets may have desirable metadata characteristics that the data consumer wants to replicate in a new / prospective data set.

[0123] As combined Figure 5 As described, an augmented metadata lineage representation may include (a) a direct metadata sub-representation; (b) an ancestor metadata sub-representation; and (c) a descendant metadata sub-representation.

[0124] The direct metadata sub-representation may include a representation of metadata features extracted from the primary dataset.

[0125] The ancestor metadata sub-representation may include representations of metadata features extracted from one or more ancestor datasets and one or more ancestor processing steps for the primary dataset. Here, a given ancestor dataset may be an existing dataset from which at least one primary dataset was derived using one or more ancestor processing steps.

[0126] The descendant metadata sub-representation may include metadata features extracted from one or more descendant datasets and one or more descendant processing steps for the primary dataset. Here, a given descendant dataset may be an existing dataset derived from at least one of the primary datasets using one or more of the descendant processing steps.

[0127] As described above, the augmented metadata lineage representation can represent the "metadata lineage" of each primary data set. By generating such an augmented metadata representation, the hardware processor 612 can identify / perceive statistically interesting / significant correspondences on and among the metadata lineages (i.e., metadata indicators) for the primary data sets, which the hardware processor 612 can use to generate the metadata indicator representation.

[0128] As combined Figure 5 As described, the hardware processor 612 may generate an augmented metadata lineage representation using (1) a lineage representation (e.g., a lineage table); and (2) a metadata representation (e.g., a metadata table). For a given primary data set, the lineage representation may represent a representation of a lineage that relates the given primary data set to (a) its ancestor data sets and processing steps; and (b) its descendant data sets and processing steps. For a given primary data set, the metadata representation may represent metadata extracted from: (a) the given primary data set; (b) its ancestor data sets and processing steps; and (c) its descendant data sets and processing steps.

[0129] As an example to illustrate this concept, a metadata representation (e.g., a metadata table) may have a row for each primary data set, ancestor data set and processing step, and descendant data set and processing step - and a column for each atomic element of metadata.

[0130] As described above, a lineage representation can associate each primary data set with: (a) its ancestor data sets and processing steps; and (b) its descendant data sets and processing steps.

[0131] Using the lineage and metadata representation, the hardware processor 612 may compute an augmented metadata lineage representation representing a “metadata lineage” for each primary data set.

[0132] In various examples, the hardware processor 612 may use the following algorithm / function to compute the ancestor metadata for a given primary dataset (i.e., metadata of the ancestor datasets and processing steps that produced the given primary dataset):

[0133]

[0134] Similarly, the hardware processor 612 may use the following algorithm / function to compute descendant metadata for a given primary data set:

[0135]

[0136]

[0137] In response to generating the augmented metadata lineage representation (including the direct metadata sub-representation, the ancestor metadata sub-representation, and the descendant metadata sub-representation), the hardware processor 612 can execute instructions 618 to calculate the correspondence between the direct metadata sub-representation and at least one of the ancestor metadata sub-representation and the descendant metadata sub-representation. In this way, the hardware processor 612 can perceive / identify statistically interesting / significant correspondences on the metadata lineage of the primary data set. As described above, these statistically interesting / significant correspondences can be referred to as metadata indicators.

[0138] The calculated correspondences described above may include various types of correspondence optics, such as cross-occurrence and co-occurrence.

[0139] For example, the hardware processor 612 may compute cross-occurrences of metadata features between the immediate metadata sub-representation and the ancestor metadata sub-representation. The hardware processor 612 may then identify the metadata features as described using, for example, G 2 Statistically interesting / significant crossovers occur as judged by statistical scores such as tests (ie, metadata indicators). As described below, hardware processor 612 can utilize these metadata indicators to generate a metadata indicator representation.

[0140] Thus, hardware processor 612 may execute instructions 620 to generate a metadata indicator representation representing the statistically interesting / significant correspondences calculated at instructions 618 .

[0141] In various examples, the metadata indicator representation may be a table including a row for each immediate metadata element and a column for each ancestor or descendant metadata element / value. An example metadata indicator representation / table is depicted as follows:

[0142]

[0143] The example metadata indicator representation / table described above has one row per atom direct metadata value and one column per unique value of ancestor or descendant metadata. Metadata indicator representations such as the examples depicted above may typically be sparse (each row having no more than about one hundred non-zero values). In some instances, the hardware processor 612 may reduce the computations to generate the metadata indicator representation / table by downsampling in the cross-occurrence computations to be linear in the total number of non-zero elements in the original augmented metadata representation.

[0144] As described above, in various examples, the hardware processor 612 may have identified one or more primary data sets as high-value data sets based on value indicators derived from human behavior (e.g., use of the data sets in numerous data processing pipelines). Therefore, the generated metadata indicator representation can represent statistically interesting / significant metadata correspondences among the metadata pedigree of the high-value data sets. Therefore, examples of the disclosed technology can utilize metadata indicator representations to identify "metadata characteristics of high value impact". As described above, "metadata characteristics of high value impact" can refer to (a) metadata characteristics of (upstream) data-related products or processing steps that may increase / influence the value of (downstream) data-related products generated from the (upstream) data-related products or processing steps; and / or (b) metadata characteristics of data-related products that may increase / influence the value of the data-related products themselves. As indicated by the metadata indicators, these "metadata characteristics of high value impact" can be metadata characteristics that appear with unusual / statistically significant frequencies on the metadata pedigree of the high-value data sets. Thus, the hardware processor 612 can identify and recommend novel combinations of existing data sets and processing steps (having these “high-value-impacting metadata characteristics”) for use in generating new / anticipated high-value data sets.

[0145] Figure 7 An example computing system 700 is depicted that can be used to recommend valuable metadata features for a prospective data set in accordance with various examples of the disclosed technology. Figure 7 , computing component 710 may be, for example, a server computer, a controller, or any other similar computing component capable of processing data. Figure 7 In an example implementation of , computing component 710 includes a hardware processor 712 and a machine-readable storage medium 714 .

[0146] Hardware processor 712 and machine-readable storage medium 714 may be the same / similar to hardware processor 512 and machine-readable storage medium 514, respectively. Thus, machine-readable storage medium 714 may be encoded with executable instructions, such as instructions 716-722.

[0147] The hardware processor 712 may execute instructions 716 to identify one or more valuable data sets using one or more value indicators. The hardware processor 712 may derive value indicators using valuation algorithms that quantify the value level of a given data-related product (e.g., a data set) based on perceived human behavior associated with the use of the given data-related product (e.g., a data set) (as used herein, a value indicator may refer to a quantifiable measurement that quantifies the value of a data-related product or a processing step based on perceived human behavior associated with the use of a given data-related product / processing step). Thus, the hardware processor 712 may utilize these valuation algorithms to compare / rank the relative value of multiple data sets based on multiple categories of human behavior actions. The hardware processor 712 may then use the ranking to identify one or more valuable data sets.

[0148] The hardware processor 712 may execute instructions 718 to generate an augmented metadata lineage representation for the identified valuable data set. The hardware processor 712 may be combined with Figure 5-Figure 6 The instructions are executed in the same / similar manner as described.

[0149] The hardware processor 712 may execute instructions 720 to calculate the corresponding relationship within the extended metadata lineage representation. The hardware processor 712 may be combined with Figure 5-Figure 6 The instructions are executed in the same / similar manner as described.

[0150] The hardware processor 712 can execute instructions 722 to recommend valuable metadata features for the expected data set based on the calculated correspondence. Figure 5-Figure 6As described, the hardware processor 712 can sense / identify statistically interesting / significant correspondences within the metadata pedigree of the identified valuable data set. As described above, these statistically interesting / significant correspondences can be referred to as metadata indicators. Using these metadata indicators, the hardware processor 712 can identify "metadata characteristics of high value impact". As described above, "metadata characteristics of high value impact" can refer to (a) metadata characteristics of (upstream) data-related products or processing steps that may increase / affect the value of (downstream) data-related products generated from (upstream) data-related products or processing steps; and / or (b) metadata characteristics of data-related products that may increase / affect the value of data-related products. As indicated by the metadata indicators, these "metadata characteristics of high value impact" can be metadata characteristics that appear at unusual / statistically significant frequencies on the metadata pedigree of the identified valuable data set. Therefore, the hardware processor 712 can recommend these metadata characteristics as valuable metadata characteristics of the expected data set. As an example of the technology of the present disclosure, the expected data set with these metadata characteristics of high value impact can be more likely to be utilized / valued by data scientists and engineers who consume the data set. Thus, data producers may utilize computing system 700 to identify / create prospective data sets (metadata characteristics having high value impact) that may be valued by members of the data community.

[0151] Figure 8 An example computing system 800 is depicted that can be used to recommend one or more existing data sets and processing steps for creating a valuable analytical model based on propagating value indicators through a lineage representation for one or more existing analytical models, in accordance with various examples of the disclosed technology. Fig. 9 Depicts can be used to illustrate the combination Figure 8 An example lineage representation 900 of various examples of value indicator propagation is depicted.

[0152] See now Figure 8 , computing component 810 may be, for example, a server computer, a controller, or any other similar computing component capable of processing data. Figure 8 In an example implementation of , computing component 810 includes a hardware processor 812 and a machine-readable storage medium 814 .

[0153] Hardware processor 812 and machine-readable storage medium 814 may be the same / similar to hardware processor 512 and machine-readable storage medium 514, respectively. Thus, machine-readable storage medium 814 may be encoded with executable instructions, such as instructions 816-822.

[0154] The hardware processor 812 can execute instructions 816 to generate a pedigree representation of one or more primary analysis models. The one or more primary analysis models can be existing analysis models of various types. Some of the primary analysis models can be deployed in a production environment, while other analysis models may not be deployed in a production environment. In some examples, the hardware processor 812 can select from, for example, a combination of Figure 2 The described data pipeline catalog retrieves / extracts the information required to generate the lineage representation for the primary analysis model.

[0155] For a given primary analysis model, the lineage representation may include a representation of the lineage of one or more predecessor processing steps and one or more predecessor data sets used to derive the given primary analysis model. The predecessor data sets and predecessor processing may be in combination with, for example, Figure 2 Describes the existing datasets and processing steps catalogued in the Data Pipeline Catalog.

[0156] See now Fig. 9 , lineage representation 900 depicts an example lineage representation that hardware processor 812 may generate.

[0157] In particular, lineage representation 900 includes a representation of a lineage of primary analysis model 952 and primary analysis model 954 and their respective ancestral processing steps and data sets.

[0158] For primary analysis model 952 moving from downstream to upstream, primary analysis model 952 has an ancestor processing step 942 that transforms ancestor dataset 912 and ancestor dataset 932 into primary analysis model 952. Ancestor dataset 932 itself has an ancestor processing step 922 that transforms ancestor dataset 914 into ancestor dataset 932. Thus, primary analysis model 952 has two ancestor processing steps (i.e., ancestor processing step 922 and ancestor processing step 942) and three ancestor datasets (i.e., ancestor dataset 912, ancestor dataset 914, and ancestor dataset 932).

[0159] For the primary analysis model 954 moving from downstream to upstream, the primary analysis model 954 has an ancestor processing step 944 that transforms the ancestor dataset 916 and the ancestor dataset 932 into the primary analysis model 954. As described above, the ancestor dataset 932 itself has an ancestor processing step 922 that transforms the ancestor dataset 914 into the ancestor dataset 932. Therefore, the primary analysis model 954 has two ancestor processing steps (i.e., ancestor processing step 922 and ancestor processing step 944) and three ancestor datasets (i.e., ancestor dataset 914, ancestor dataset 916, and ancestor dataset 932).

[0160] Here, it should be noted that primary analysis model 952 and primary analysis model 954 share certain ancestor data sets and processing steps (i.e., ancestor data set 914 and ancestor data set 932 and ancestor processing step 922). As described below, when hardware processor 812 propagates value measurements through lineage representation 900, these shared ancestor data sets and processing steps can aggregate (i.e., transitively accumulate) value measurements from primary analysis model 952 and primary analysis model 954.

[0161] See again Figure 8 , the hardware processor 812 may execute instructions 818 to assign a value indicator to each of the one or more primary analytical models. As described above, the value indicator may refer to a quantifiable measure of the value of a given primary analytical model based on perceived human behavior associated with the use of the given primary analytical model.

[0162] The hardware processor 812 may derive the value indicator using a valuation algorithm that quantifies the value level of each of the one or more primary analytical models. For example, the hardware processor 812 may utilize a simple valuation algorithm that quantifies the value level of the primary analytical models based on how long people have deployed the primary analytical models in a production environment. Using this simple valuation algorithm, the hardware processor 812 may assign a value of, for example, v=v3 to a primary analytical model that has been deployed in production for three years, and assign a value of, for example, v=v1 to a primary analytical model that has been deployed in production for one year. In various instances, the hardware processor 812 may utilize a more complex valuation algorithm that quantifies two or more categories of human behavioral actions associated with the use of the primary analytical models to a common numerical scale. For example, the hardware processor 812 may utilize a more sophisticated valuation algorithm to quantify the level of value of the primary analytic model according to a common numerical scale based on any one or a combination of the following factors: (a) the duration that humans / users have deployed the primary analytic model in production; (b) the number of independent users who have deployed the primary analytic model into production; and (c) the quality / accuracy scores attributed to the primary analytic model (these quality / accuracy scores may be attributed to the primary analytic model by humans, artificial intelligence, etc.).

[0163] See now Fig. 9 In various examples, hardware processor 812 can assign value indicators to primary analysis model 952 and primary analysis model 954. Fig. 9In a specific example of , the assigned value indicator can quantify the amount of time (e.g., hours, days, weeks, etc.) that a given primary analytical model has been deployed in production. As depicted, hardware processor 812 has assigned primary analytical model 952 with a value measurement of v=v1, and primary analytical model 954 with a value measurement of v=v3. This can indicate that primary analytical model 954 has been deployed in production three times longer than primary analytical model 952.

[0164] See again Figure 8 , the hardware processor 812 may execute instructions 820 to propagate the assigned value indicator of each primary analysis model through the pedigree representation. In other words, the hardware processor 812 may calculate (i.e., backpropagate) the value indicator in the reverse / upstream direction (e.g., Fig. 9 The value stream represented by the pipeline (depicted by the arrows in ) is then mapped to identify the predecessor datasets and processing steps that are most likely to produce valuable analytical models.

[0165] In various examples, the hardware processor 812 may perform value propagation by: (1) attributing a value indicator of a given primary analysis model to each ancestor data set and ancestor processing step of the given primary analysis model; and (2) for a given ancestor data set or ancestor processing step, aggregating one or more of the value indicators attributed to the given ancestor data set or ancestor processing step. In some of these examples, the hardware processor 812 may reduce the aggregate value indicator based on the cost of implementing the aggregate value indicator for the given ancestor. Similarly, the hardware processor 812 may reduce the aggregate value indicator for the given ancestor based on the remoteness between the given ancestor and one or more of its associated primary analysis models. Here, the remoteness between a given ancestor and a given offspring may be measured by the number of ancestors between the given ancestor and the given offspring along a representation of the pedigree of the given ancestor and the given offspring (e.g., mother to grandmother to great-grandmother, etc.).

[0166] Now see again Fig. 9 As depicted, hardware processor 812 can propagate (backward) the value indicators assigned to primary analytic models 952 and primary analytic models 954 through lineage representation 900 to assign pass-through value credits to their ancestors. As depicted, hardware processor 812 aggregates the value indicators of common ancestors shared by primary analytic models 952 and primary analytic models 954. In particular, ancestor data set 932 and ancestor data set 914 and ancestor processing step 922 all have the same aggregate value indicator: v=v1 U v3.

[0167] like Fig. 9As depicted in the examples of , the hardware processor 812 may also reduce (i.e., discount) the aggregate value indicator for a given ancestor based on various factors. In these examples, the hardware processor 812 may propagate the value indicator as a single dimension / parameter of a multi-dimensional vector that includes other dimensions / parameters for reducing the attributed / aggregate value indicator.

[0168] For example, the hardware processor 812 may propagate the value indicator (and Fig. 9 (shown in the figure):

[0169] v (i.e., the downstream social value indicator of the predecessor) = the union / aggregation of the values ​​assigned to the predecessors;

[0170] c (i.e., downstream cost to achieve the social value indicator of the ancestor) = sum of step costs on all steps in the path(s) leading to the downstream social value indicator v (e.g., when computational cost or effort is not available, this can be approximated as the upstream distance from the initial social value indicator). Here, the cost of downstream steps is shared between overlapping paths that feed directly from other ancestors to any intermediate ancestors in the path.

[0171] p (ie, the fraction of input paths to the same downstream social value indicator) = the ratio of direct peer paths (partition / dataset inputs) that contribute to and share the downstream social value indicator (distributed by the propagation logic).

[0172] like Fig. 9 As depicted in , by propagating these vectors through the lineage representation 900, the hardware processor 812 can (1) attribute to each ancestor the value indicator (v) of its descendant primary analytical model; and (2) for a given ancestor, aggregate one or more of the value indicators (v) attributed to the given ancestor. In various examples, the hardware processor 812 can then reduce the attributed / aggregated value indicator of each ancestor by its downstream cost to achieve (c) and / or the portion of the input path to the same downstream value (p).

[0173] Examples of the disclosed technology can use propagated value indicators such as those described above in various ways. For example, they can be used when training a data analysis system to infer possible value tuples associated with a recommended data set based on its metadata characteristics. As another example, when the data analysis system generates candidate data sets and processing steps that match the recommended metadata characteristics, the example can rank the candidates based on the tuples calculated by this value back-propagation mechanism. For example, a simple ranking method can sort the candidates in the following manner: descending by (v), ascending by (c), descending by (p), etc.

[0174] See again Figure 8 , after propagating the value indicator of each of the one or more primary analytical models through the lineage representation, the hardware processor 812 can execute instructions 822 to recommend one or more of an ancestor data set and an ancestor processing step for use in creating the intended analytical model. Here, the hardware processor 812 can recommend an ancestor having the highest aggregate value indicator after propagation of the instructions 820. As described above, in various examples, the hardware processor 812 can recommend an ancestor having the highest aggregate value indicator after considering one or more reduction / discount factors.

[0175] For conceptual illustration, see again Fig. 9 , both ancestor dataset 932 and ancestor dataset 914 have aggregate value indicators of v=v1U v3. However, if these aggregate value indicators are reduced / discounted by their respective implementation costs (i.e., c=2 for ancestor dataset 932 and c=4 for ancestor dataset 914), then hardware processor 812 may recommend ancestor dataset 932 over ancestor dataset 914 because it has the same aggregate value indicator and a lower implementation cost. Similarly, hardware processor 812 may recommend ancestor processing step 944 over ancestor processing step 942 because it has a higher aggregate value indicator (i.e., v3 vs. v1) at the same implementation cost (i.e., c=1).

[0176] Fig.10 An example computing system 1000 is depicted that can be used to recommend data-related products for a target data analytics project in accordance with various examples of the disclosed technology.

[0177] See now Fig.10 , computing component 1010 may be, for example, a server computer, a controller, or any other similar computing component capable of processing data. Fig.10 In an example implementation of , computing component 1010 includes a hardware processor 1012 and a machine-readable storage medium 1014 .

[0178] Hardware processor 1012 and machine-readable storage medium 1014 may be the same / similar to hardware processor 512 and machine-readable storage medium 514, respectively. Thus, machine-readable storage medium 1014 may be encoded with executable instructions, such as instructions 1016-1024.

[0179] The hardware processor 1012 may execute instructions 1016 to generate a lineage representation of one or more downstream data-related products (e.g., data sets derived from upstream / ancestor data-related products and processing steps, analytical models derived from upstream / ancestor data-related products and processing steps, dashboards for displaying one or more of the aforementioned data-related products, etc.). As described above, a given lineage representation of a given downstream data-related product may include a representation of the lineage between the given downstream data-related product and the ancestor data sets and ancestor processing steps of the given downstream data-related product (i.e., the data sets and processing steps from which the given downstream data-related product was derived).

[0180] The hardware processor 1012 may execute instructions 1018 to generate implicit metadata for one or more downstream data-related products and their associated predecessor processing steps and predecessor data sets. As used herein, implicit metadata may refer to data that is not intentionally created by a human (as used herein, "explicit metadata" may refer to metadata that is intentionally created by a human, for example, by manually tagging a data-related product with supplemental information that characterizes the data-related product). The hardware processor 1012 may use instructions such as in conjunction with Figure 2 The above-described metadata indicators (ie, statistically interesting / significant correspondences perceived across the metadata lineage of downstream data-related products) may be considered examples of implicit metadata.

[0181] The hardware processor 1012 may execute instructions 1020 to propagate the value indicator for each downstream data-related product in the one or more downstream data-related products through the lineage representation. The hardware processor 1012 may be configured in conjunction with Figure 8 and Fig. 9 The instructions may be executed in the same / similar manner as described above. As described above, the given value indicator may be a quantifiable measure of the value of the given downstream data-related product derived from human behavioral actions associated with the use of the given downstream data-related product.

[0182] The hardware processor 1012 may execute instructions 1022 to receive parameters associated with a target data analysis project. The received parameters may generally describe the target data analysis project. For example, the received parameters may specify a type of data analysis project (e.g., a project to predict the spread of vector-borne diseases in New York City), a desired type of data-related product to be used in the target data analysis project (e.g., a dataset or an analytical / ML model), and / or desired metadata characteristics of the data-related product to be utilized in the target data analysis project (e.g., a weekly granularity of a dataset).

[0183] The hardware processor 1012 may execute instructions 1024 to recommend at least one of the predecessor datasets for the target data analysis project with reference to the propagated value indicator and the generated implicit metadata.

[0184] Here, the hardware processor 1012 may also recommend at least one of the predecessor processing steps. For example, the hardware processor may recommend a first predecessor data set (e.g., a data set containing weekly disease prevalence data by city), a second predecessor data set (e.g., a data set containing daily weather data by city), a first predecessor processing step (e.g., a processing step for joining / combining the data sets), and a second predecessor processing step (e.g., a processing step for operationalizing the data sets into an analysis model). Here, the recommended first and second predecessor data sets and the first and second predecessor processing steps may have never been combined before. However, the hardware processor 1012 may (via the propagation of value indicators represented by the pedigree) identify these predecessors as predecessors of high-value data-related products that have been used in data analysis projects similar to the target data analysis project. Therefore, the hardware processor 1012 may determine that the novel combination of these predecessor data sets and processing steps can produce useful / valuable new data-related products for the target data analysis project. For example, the hardware processor 1012 may recommend using a first predecessor processing step to combine / transform the first predecessor data set and the second predecessor data set into a new data set (e.g., a data set that combines daily weather data by city with weekly disease prevalence data by city). The hardware processor 1012 may also recommend using a second processing step to transform / operationalize this new data set into a new analysis model (e.g., an analysis model that predicts the spread of vector-borne diseases in a city based on weather data). This new analysis model may be consistent with the parameters received by the target data analysis project.

[0185] Fig.11 A block diagram of an example computer system 1100 is depicted in which various embodiments described herein may be implemented. The computer system 1100 includes a bus 1112 or other communication mechanism for communicating information, one or more hardware processors 1104 coupled to the bus 1112 for processing information. The hardware processor 1104 may be, for example, one or more general purpose microprocessors.

[0186] The computer system 1100 also includes a main memory 1106, such as a random access memory (RAM), a cache, and / or other dynamic storage device, coupled to the bus 1112 for storing instructions and information to be executed by the processor 1104. The main memory 1106 may also be used to store temporary variables or other intermediate information during execution of instructions to be executed by the processor 1104. Such instructions, when stored in a storage medium accessible to the processor 1104, render the computer system 1100 into a special-purpose machine customized to perform the operations specified in the instructions.

[0187] The computer system 1100 also includes a read only memory (ROM) 1108 or other static storage device coupled to the bus 1112 for storing static information and instructions for the processor 1104. A storage device 1110, such as a magnetic disk, optical disk, or USB thumb drive (flash drive), is provided and coupled to the bus 1112 for storing information and instructions.

[0188] The computer system 1100 may be coupled to a display 1112, such as a liquid crystal display (LCD) (or touch screen), via bus 1112 for displaying information to a computer user. An input device 1114, including alphanumeric and other keys, is coupled to bus 1112 for communicating information and command selections to processor 1104. Another type of user input device is a cursor control 1116, such as a mouse, trackball, or cursor direction keys, for communicating direction information and command selections to processor 1104 and for controlling cursor movement on display 1112. In some embodiments, the same direction information and command selections as a cursor control may be achieved via receiving touches on a touch screen without a cursor.

[0189] The computing system 1100 may include a user interface module for implementing a GUI, which may be stored in a mass storage device as executable software code executed by the computing device(s). This and other modules may include, by way of example, components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0190] In general, as used herein, the terms "component", "engine", "system", "database", "data repository", etc. may refer to logic embodied in hardware or firmware, or to a set of software instructions written in a programming language that may have entry points and exit points, such as, for example, Java, C, or C++. Software components may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpreted programming language such as BASIC, Perl, or Python. It should be understood that software components may be called from other components or from themselves, and / or may be called in response to detected events or interrupts. Software components configured for execution on a computing device may be provided on a computer-readable medium, such as a compressed disk, a digital video disk, a flash drive, a magnetic disk, or any other tangible medium, or as a digital download (and may be initially stored in a compressed or installable format that requires installation, decompression, or decryption prior to execution). Such software code may be stored in part or in whole on a memory device of an executing computing device for execution by the computing device. Software instructions may be embedded in firmware such as an EPROM. It should also be appreciated that hardware components may be comprised of connected logic units, such as gates and flip-flops, and / or may be comprised of programmable units, such as programmable gate arrays or processors.

[0191] The computer system 1100 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, in combination with the computer system, makes the computer system 1100 a special purpose machine or programs it as a special purpose machine. According to one embodiment, the techniques herein are performed by the computer system 1100 in response to the processor 1104 executing one or more sequences of one or more instructions contained in the main memory 1106. Such instructions may be read into the main memory 1106 from another storage medium such as the storage device 1110. Execution of the sequences of instructions contained in the main memory 1106 causes the processor(s) 1104 to perform the processing steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0192] As used herein, the term "non-transitory media" and similar terms refer to any media that stores data and / or instructions that cause a machine to operate in a specific manner. Such non-transitory media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 1110. Volatile media include dynamic memory, such as main memory 1106. Common forms of non-transitory media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a pattern of holes, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips or cassettes, and networked versions thereof.

[0193] Non-transitory media are distinct from transmission media, but can be used in conjunction with transmission media. Transmission media participate in the transfer of information between non-transitory media. For example, transmission media include coaxial cables, copper wire, and optical fiber, including the wires that make up bus 1112. Transmission media can also take the form of sound or light waves, such as those generated during radio wave and infrared data communications.

[0194] The computer system 1100 also includes a communication interface 1118 coupled to the bus 1112. The network interface 1118 provides a two-way data communication coupled to one or more network links connected to one or more local area networks. For example, the communication interface 1118 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem for providing a data communication connection to a corresponding type of telephone line. As another example, the network interface 1118 can be a local area network (LAN) card to provide a data communication connection to a compatible LAN (or a WAN component that communicates with a WAN). A wireless link can also be implemented. In any such implementation, the network interface 1118 sends and receives electrical, electromagnetic, or optical indicators that carry digital data streams representing various types of information.

[0195] A network link typically provides data communications to other data devices through one or more networks. For example, a network link may provide a connection to a host computer or to data devices operated by an Internet Service Provider (ISP) through a local area network. The ISP, in turn, provides data communications services through a global packet data communications network now commonly referred to as the "Internet." Both local area networks and the Internet use electrical, electromagnetic, or optical indicators that carry digital data streams. Indicators through various networks and indicators on network links and through communication interface 1118 are example forms of transmission media that carry digital data to and from computer system 1100.

[0196] Computer system 1100 can send messages and receive data, including program code, through the network(s), network links and communication interface 1118. In the Internet example, a server can transmit the requested code for an application program through the Internet, an ISP, a local area network and communication interface 1118.

[0197] The received code may be executed by processor 1104 as it is received, and / or stored in storage device 1110 or other non-volatile storage for later execution.

[0198] Each of the processes, methods, and algorithms described in the foregoing sections may be embodied in a code component executed by one or more computer systems or computer processors including computer hardware, and may be fully or partially automated by it. One or more computer systems or computer processors may also be operable to support the execution of related operations in a "cloud computing" environment or operate as "software as a service" (SAAS). The processes and algorithms may be implemented in part or in whole in a dedicated circuit. The various features and processes described above may be used independently of each other, or may be combined in various ways. Different combinations and sub-combinations are intended to fall within the scope of the present disclosure, and certain methods or process blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular order, and the blocks or states associated therewith may be executed in other appropriate orders, or may be executed in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The execution of certain operations or processes may be distributed between computer systems or computer processors, which may reside in a single machine or be deployed on several machines.

[0199] As used herein, any form of hardware, software or its combination can be utilized to realize circuit.For example, one or more processors, controllers, ASIC, PLA, PAL, CPLD, FPGA, logic components, software routines or other mechanisms can be realized to constitute circuit.In implementation, various circuits described herein can be implemented as discrete circuits, or described functions and features can be shared between one or more circuits in part or in whole.Even if various features or functional elements can be described or claimed individually as separated circuits, these features and functionality can also be shared between one or more common circuits, and such description should not require or imply that separated circuits are needed to realize such features or functionality.In the case where the circuit is implemented in whole or in part using software, such software can be implemented as operating together with a computing or processing system (such as computer system 1100) that can perform the functionality described therein.

[0200] As used herein, the term "or" may be interpreted in an inclusive or exclusive sense. In addition, descriptions of resources, operations, or structures in the singular should not be interpreted to exclude the plural. Unless otherwise expressly stated or otherwise understood in the context of use, conditional language such as "may," "can," "can," or "might" is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not include certain features, elements, and / or steps.

[0201] Unless expressly stated otherwise, the terms and phrases used in this document and variations thereof should be interpreted as open ended and not restrictive. Adjectives such as "conventional," "traditional," "normal," "standard," "known," and terms of similar meaning should not be interpreted as limiting the items described to a given time period or items available at a given time, but should be interpreted as covering conventional, traditional, normal, or standard technologies that may be available or known at any time now or in the future. In some cases, the presence of broad words and phrases such as "one or more," "at least," "but not limited to," or other similar phrases should not be interpreted as meaning that a narrower situation is intended or required where such broad phrases may not be present.

[0202] As used herein, a data set may refer to an ordered collection of data.As described above, a data set may be a data-related product of a data processing pipeline.

[0203] As used herein, data analysis (which may be implemented using machine learning and / or artificial intelligence) may refer to techniques for making predictive determinations about, for example, modeled scenarios and systems. As described above, analytical models derived from data are common data-related products of a data processing pipeline.

[0204] As used herein, machine learning may refer to methods that can automatically transform data sets into analytical models through the use of algorithms. These analytical models can, in turn, make predictions based on patterns or inferences gleaned from other data / data sets. There has been a push to implement machine learning in enterprise settings (e.g., businesses) so that these entities can leverage machine learning to provide better services and products to their customers, become more efficient in their operations, and so on. Implementing machine learning in an enterprise setting (also referred to as operationalization) may involve the deployment (and management) of analytical models, i.e., putting analytical models into production.

[0205] As used herein, metadata may refer to data that provides information about other data. Within a data processing pipeline, each data-related product and processing step may have metadata associated with it. For example, metadata for a data set may include a text document for the data set, the name of the table / column / feature of the data set, a description of the content within the data set, the size of the data within the data set, the distribution of data within the data set, etc. Examples of metadata for a processing step may include the language in which the processing step / program is written, information about the input fields used for the processing step, the content of the associated document for the processing step, the classification of the algorithm used for the processing step, the classification or quality assurance (QA) status for the processing step, the name of the developer / data scientist related to the processing step, etc.

Claims

1. A system comprising: Graphical User Interface GUI; Data pipeline database; one or more processing resources; a non-transitory computer-readable medium coupled to the one or more processing resources, having stored thereon instructions that, when executed by the one or more processing resources, cause the system to: generating a lineage representation for one or more downstream data-related products, wherein a given lineage representation for a given downstream data-related product comprises a directed graph including associations of the lineage between the given downstream data-related product and ancestor data sets and ancestor processing steps of the given downstream data-related product; generating implicit metadata for the one or more downstream data-related products and predecessor processing steps and predecessor data sets associated with the one or more downstream data-related products, wherein the generated implicit metadata includes statistically significant correspondences between metadata characteristics of the one or more downstream data-related products and predecessor processing steps and predecessor data sets associated with the one or more downstream data-related products; propagating a value indicator for each of the one or more downstream data-related products through the lineage representation, a given value indicator being a quantifiable measure of the value of the given downstream data-related product derived from human behavioral actions associated with use of the given downstream data-related product; updating the lineage representation to include the propagated value indicator and the generated implicit metadata; storing the updated representation of the lineage as one or more exportable data-related packages in a non-transitory computer-readable memory of the data pipeline database; receiving, via the GUI, parameters associated with a target data analysis project; referencing the one or more exportable data related packages to predict a subset of an ancestor data set most likely to be used for the target data analysis project; as well as At least one of the predicted predecessor datasets is recommended via the GUI for the target data analysis project.

2. The system of claim 1 , wherein propagating the value indicator through the lineage representation comprises: attributing the value indicator for a given downstream data-related product to each ancestor data set of the given downstream data-related product, and For a given ancestor data set, one or more of the value indicators attributed to the given ancestor data set are aggregated.

3. The system of claim 2, wherein propagating the value indicator through the lineage representation further comprises: A given aggregate value indicator for a given ancestor data set is reduced based on a cost of achieving the given aggregate value indicator for the given ancestor data set. 4 . The system of claim 1 , wherein the received parameters associated with the target data analytics project include desired metadata characteristics for data-related products to be utilized in the target data analytics project.

5. The system of claim 4, wherein the instructions, when executed by the one or more resources, further cause the system to: referencing the one or more exportable data related packages to predict a subset of predecessor processing steps most likely to be used for the target data analysis project; and Predicted predecessor processing steps are recommended for transforming the recommended predecessor data set into a new data-related product for the target data analytics project. 6 . The system of claim 5 , wherein the new data-related product for the target data analytics project has desired metadata characteristics specified in the received parameters associated with the target data analytics project.

7. The system of claim 5, wherein: The recommended predecessor data set is a predecessor data set of a first downstream data related product among the one or more downstream data related products; and The recommended predecessor processing step is a predecessor processing step of a second data-dependent product of the one or more downstream data-dependent products.

8. A non-transitory computer readable medium storing instructions that, when executed by one or more processing resources, cause the one or more processing resources to: generating a lineage representation for one or more downstream data-related products, wherein a given lineage representation for a given downstream data-related product comprises a directed graph including associations of the lineage between the given downstream data-related product and ancestor data sets and ancestor processing steps of the given downstream data-related product; generating implicit metadata for the one or more downstream data-related products and predecessor processing steps and predecessor data sets associated with the one or more downstream data-related products, wherein the generated implicit metadata includes statistically significant correspondences between metadata characteristics of the one or more downstream data-related products and predecessor processing steps and predecessor data sets associated with the one or more downstream data-related products; propagating a value indicator for each of the one or more downstream data-related products through the lineage representation, a given value indicator being a quantifiable measure of the value of the given downstream data-related product derived from human behavioral actions associated with use of the given downstream data-related product; updating the lineage representation to include the propagated value indicator and the generated implicit metadata; storing the updated representation of the lineage as one or more exportable data-related packages in a non-transitory computer-readable memory of a data pipeline database; Receiving parameters related to a target data analysis project via a graphical user interface GUI; referencing the one or more exportable data related packages to predict a subset of an ancestor data set most likely to be used for the target data analysis project; as well as At least one of the predicted predecessor datasets is recommended via the GUI for the target data analysis project.

9. The non-transitory computer-readable medium storing instructions of claim 8, wherein propagating the value indicator through the lineage representation comprises: attributing the value indicator for a given downstream data-related product to each ancestor data set of the given downstream data-related product, and For a given ancestor data set, one or more of the value indicators attributed to the given ancestor data set are aggregated.

10. The non-transitory computer readable medium storing instructions of claim 9, wherein propagating the value indicator through the lineage representation further comprises: A given aggregate value indicator for a given ancestor data set is reduced based on a cost of achieving the given aggregate value indicator for the given ancestor data set.

11. The non-transitory computer-readable medium storing instructions of claim 8, wherein the received parameters associated with the target data analytics project include desired metadata characteristics for a data-related product to be utilized in the target data analytics project.

12. The non-transitory computer readable medium storing instructions of claim 8, wherein the non-transitory computer readable medium storing instructions further causes the one or more processing resources to: referencing the one or more exportable data related packages to predict a subset of predecessor processing steps most likely to be used for the target data analysis project; and Predicted predecessor processing steps are recommended for transforming the recommended predecessor data set into a new data-related product for the target data analytics project.

13. The non-transitory computer-readable medium storing instructions of claim 12, wherein the new data-related product for the target data analytics project has desired metadata characteristics specified in the received parameters associated with the target data analytics project.

14. The non-transitory computer readable medium storing instructions of claim 12, wherein: The recommended predecessor data set is a predecessor data set of a first downstream data related product among the one or more downstream data related products; and The recommended predecessor processing step is a predecessor processing step of a second data-dependent product of the one or more downstream data-dependent products.

15. A method comprising: generating a lineage representation for one or more downstream data-related products, wherein a given lineage representation for a given downstream data-related product comprises a directed graph including associations of the lineage between the given downstream data-related product and ancestor data sets and ancestor processing steps of the given downstream data-related product; generating implicit metadata for the one or more downstream data-related products and predecessor processing steps and predecessor data sets associated with the one or more downstream data-related products, wherein the generated implicit metadata includes statistically significant correspondences between metadata characteristics of the one or more downstream data-related products and predecessor processing steps and predecessor data sets associated with the one or more downstream data-related products; propagating a value indicator for each of the one or more downstream data-related products through the lineage representation, a given value indicator being a quantifiable measure of the value of the given downstream data-related product derived from human behavioral actions associated with use of the given downstream data-related product; updating the lineage representation to include the propagated value indicator and the generated implicit metadata; storing the updated representation of the lineage as one or more exportable data-related packages in a non-transitory computer-readable memory of a data pipeline database; Receiving parameters related to a target data analysis project via a graphical user interface GUI; referencing the one or more exportable data related packages to predict a subset of an ancestor data set most likely to be used for the target data analysis project; as well as At least one of the predicted predecessor datasets is recommended via the GUI for the target data analysis project.

16. The method of claim 15, wherein propagating the value indicator through the lineage representation comprises: attributing the value indicator for a given downstream data-related product to each ancestor data set of the given downstream data-related product, and For a given ancestor data set, one or more of the value indicators attributed to the given ancestor data set are aggregated.

17. The method of claim 16, wherein propagating the value indicator through the lineage representation further comprises: A given aggregate value indicator for a given ancestor data set is reduced based on a cost of achieving the given aggregate value indicator for the given ancestor data set.