Methods of tracking and cloning artifacts associated with distributed machine learning pipelines

Abstract

This disclosure relates to methods for tracking and cloning artifacts associated with a distributed machine learning pipeline. Systems and methods are provided for automatically constructing data hierarchy representations for distributed data processing pipelines. These data hierarchy representations (which are constructed and stored in a central repository shared by multiple data processing sites) can be used to clone the distributed data processing pipeline for quality assurance or debugging purposes, etc. Examples of the currently disclosed techniques are capable of constructing data hierarchy representations for distributed data processing pipelines.

Description

Background Technology

[0001] A data processing pipeline can refer to a series of executions (i.e., processing steps) that transform one or more data artifacts (e.g., raw data, datasets, analyses or machine learning (ML) models derived from the data, etc.) into one or more new data artifacts (e.g., new datasets, analyses or ML models derived from the data, dashboards displaying one or more data from the aforementioned data artifacts, etc.). In many cases, a data processing pipeline consists of processing stages (e.g., data discovery / preparation stages, ML model preparation stages, ML model training stages, etc.). A given processing stage can consist of one or more executions. An execution can be a data processing step that transforms one or more data artifacts. In other words, an execution can receive one or more input data artifacts (i.e., data artifacts received as input to the execution) and transform them into one or more output data artifacts (i.e., data artifacts produced by the execution). A data artifact can be the output of one execution and / or the input to one or more different executions. For example, in addition to being the output of the first execution of the first processing stage, the first data artifact can also be the input to the first execution of the second processing stage.

[0002] In many cases, multiple processing stages of a data processing pipeline can be executed by different data processing sites (e.g., different data centers, edge computing sites, etc.). A data processing pipeline that executes across multiple data processing sites can be referred to as a "distributed data processing pipeline." In many cases, the data processing sites in a distributed data processing pipeline will be disconnected from each other, but may at least intermittently connect to a central repository (e.g., a central data center) for storing data artifacts consumed / generated by the distributed data processing pipeline. Attached Figure Description

[0003] This disclosure is described in detail with reference to the following figures, based on one or more different examples. These figures are provided for illustrative purposes only and describe only typical or exemplary cases.

[0004] Figure 1 It describes various examples of distributed data processing pipelines based on currently disclosed technologies.

[0005] Figure 2 This paper describes an example data processing pipeline abstraction layer structure based on various examples of currently disclosed technologies, representing distributed data processing pipelines.

[0006] Figure 3 Example workflows for example distributed data processing pipelines are described, based on various examples of currently disclosed technologies.

[0007] Figure 4 Examples of computational systems, based on currently disclosed technologies, are described that can be used to construct data hierarchy representations for distributed data processing pipelines.

[0008] Figure 5 An example computing system, based on various examples of currently disclosed technologies, is described, which can be used to export information related to data artifacts processed by the computing system to a central repository shared by multiple data processing sites in a distributed data processing pipeline.

[0009] Figure 6 It is an example computing system based on various currently disclosed technologies that can be used to clone distributed data processing pipelines.

[0010] Figure 7 Example data hierarchy representations for distributed data processing pipelines are depicted based on various examples of currently disclosed technologies.

[0011] Figure 8 Another example data hierarchy representation for a distributed data processing pipeline is depicted, based on various examples of currently disclosed technologies.

[0012] Figure 9 A block diagram depicts an example computer system in which the various examples described herein can be implemented.

[0013] The accompanying drawings are not exhaustive and do not limit this disclosure to the precise form disclosed. Detailed Implementation

[0014] Interactions within a distributed data processing pipeline can be highly complex. As mentioned above, a distributed data processing pipeline can consist of multiple processing stages distributed across various data processing sites. Each processing stage may include multiple executions that transform data artifacts. In a complex distributed data processing pipeline, a given data artifact can be the input and / or output of multiple executions performed across multiple data processing sites.

[0015] In (distributed or otherwise) data processing pipelines, dependencies often exist between processing stages and execution, which can affect performance metrics of the final / downstream output data artifacts (e.g., datasets, ML models, etc.). In other words, data artifacts consumed / produced (along with execution) in earlier processing stages of a data processing pipeline can affect data quality metrics (e.g., ML model accuracy) of data artifacts produced in later processing stages. Accordingly, it may be important to track metadata across the data processing pipeline to provide functionality such as data lineage tracking, provenance, and reproducibility. Relatedly, government regulations often require inspection of data artifacts (and / or execution) of data processing pipelines when certain events occur (e.g., poor model performance, model bias, etc.). To perform regulatory (or non-regulatory) inspections, it may be advantageous to clone the entire data processing pipeline, including its code, data artifacts, metadata, and dependencies, to locally reproduce its execution for testing and debugging purposes.

[0016] However, tracking the metadata and data lineage of a distributed data processing pipeline presents significant challenges—problems that are difficult to address with existing technologies. Specifically, the multiple data processing sites in a distributed data processing pipeline are often not connected and may only have intermittent connections to a central repository used to store, for example, data artifacts consumed / produced by the data processing pipeline. This lack of direct connection / collaboration between distributed data processing sites / teams often hinders comprehensive and consistent tracking of the data lineage and metadata of a distributed data processing pipeline. Consequently, existing technologies struggle to clone distributed data processing pipelines for testing and debugging purposes. Relevant factors that may make comprehensive and consistent tracking of the data lineage and metadata of a distributed data processing pipeline difficult include: (1) different data processing sites often assign the same names to different data artifacts; and (2) different data processing sites often assign different names to the same data artifacts. As mentioned above, this inconsistency in data artifact naming can pose significant challenges when it comes to tracking the metadata and data lineage of a distributed data processing pipeline and (2) cloning a distributed data processing pipeline.

[0017] In this context, examples of currently disclosed technologies provide computerized systems and methods for automatically constructing data hierarchy representations for distributed data processing pipelines. These data hierarchy representations (which are constructed and stored in a central repository shared by multiple data processing sites) can be used to clone distributed data processing pipelines for quality assurance or debugging purposes, etc.

[0018] Examples of currently disclosed technologies enable the construction of a data hierarchy representation for a distributed data processing pipeline by: (1) generating hash content values ​​to generically identify each data artifact of the distributed data processing pipeline across multiple processing stages / processing stations; and (2) creating an abstract hierarchical structure for the data processing pipeline to associate each data artifact with input and output events of a given execution (performed by multiple data processing stations) of a given data processing stage.

[0019] In various instances, examples may provide a computerized central repository (e.g., a centralized data center) shared by multiple data processing sites in a distributed data processing pipeline, which: (1) receives specially tailored information from each data processing site in the distributed data processing pipeline, the specially tailored information including, for a given information: (a) a unique hash content value identifying a given data artifact; and (b) an indication of the input or output of a given execution of a given processing stage performed by a given data processing site; (2) constructs a data hierarchy representation for the distributed data processing pipeline based on the received information; and (3) outputs the constructed data hierarchy representation to one or more of the data processing sites. Accordingly, one or more data processing sites may utilize the constructed data hierarchy representation to locally reproduce / clone the distributed data processing pipeline for, for example, testing and debugging purposes.

[0020] As described above, examples of currently disclosed techniques are capable of constructing a data hierarchy representation for a distributed data processing pipeline, where the prior art has attempted / failed to: (1) generate hash content values ​​to generically identify each data artifact of the distributed data processing pipeline across multiple processing stages / stations; and (2) create an abstract hierarchical structure for the data processing pipeline to associate each data artifact with the input and output events of a given execution (performed by multiple data processing stations) of a given data processing stage. Examples can then leverage these constructed data hierarchy representations to clone the distributed data processing pipeline for purposes such as testing and debugging.

[0021] Here, the generated hash content value can be used as an identifier key / link, allowing examples of the currently disclosed techniques to construct a data lineage representation for a distributed data processing pipeline. In other words, examples of the currently disclosed techniques can utilize the generated hash content value to determine, for example, that the same data artifact (e.g., the first data artifact) is both the output of an execution of a first processing stage and the input of an execution of a second processing stage—even when these two processing stages are executed by different individuals at different data processing sites. In the case of multiple data processing sites operating on data artifacts in a distributed data processing pipeline, the unique (and consistent with the description below) identifier of the data artifact provided by the hash content value plays a crucial role. As mentioned above, this is because in many cases: (1) different data processing sites often assign the same name to different data artifacts; and (2) different data processing sites often assign different names to the same data artifacts. As mentioned above, such naming inconsistencies of data artifacts can pose significant challenges when it comes to tracking metadata and data lineage in a distributed data processing pipeline. As described below, by enabling multiple data processing sites in a distributed data processing pipeline to utilize the same hash algorithm / function to generate hash content values ​​for data artifacts, the example ensures that data artifacts are consistently named / identified across multiple data processing sites. This common naming / identification scheme for data artifacts allows the currently disclosed example of the technology to construct a more accurate genealogical representation of data for a distributed data processing pipeline than existing technologies.

[0022] As mentioned above, the specially tailored information received by the central repository also includes an indication that a given data artifact (identified by the hash content value of the given data artifact) is an input or output of a given execution in a given data processing stage. Here, examples of the currently disclosed techniques can leverage a data processing pipeline abstraction hierarchy (combined with...) Figure 2(To be described in more detail) to make this instruction. The data processing pipeline abstraction hierarchy of the currently disclosed technology may include: (1) data processing pipeline level; (2) processing stage level under the data processing pipeline level; (3) execution level under the processing stage level; and (4) input / output event relationship for the execution level. Using the data processing pipeline abstraction hierarchy, examples may associate each data artifact of the data processing pipeline with: (1) at least one of the input or output events; (2) multiple given executions associated with (multiple) input / output events; (3) multiple given processing stages associated with (multiple) given executions; and (4) a distributed data processing pipeline associated with (multiple) given processing stages. As described above, based on these associations, examples of the currently disclosed technology (e.g., a central repository) can construct a data hierarchy representation for a distributed data processing pipeline more accurately and consistently than prior art. Examples of the currently disclosed technology then use these constructed data hierarchy representations to, for example, test or debug cloned distributed data processing pipelines.

[0023] Figure 1 An example distributed data processing pipeline 100 is described, which is based on various examples of currently disclosed technologies.

[0024] As shown in the figure, the distributed data processing pipeline 100 includes five processing stages (i.e., processing stages 101a, 102a, 103a, 104a, and 105a) executed by five data processing stations (i.e., data processing stations 101, 102, 103, 104, and 105), respectively. Each data processing station can be, for example, a data center, an edge computing station, etc. The multiple data processing stations of the distributed data processing pipeline 100 can be disconnected from each other, but can be connected at least intermittently to a data artifact repository 110. As will be described below, the data artifact repository 110 can be (logically) a central location, storing data artifacts (i.e., data artifacts A through G) consumed and / or generated during the execution of the data processing pipeline 100.

[0025] Data processing station 101 can execute processing stage 101a of data processing pipeline 100 (e.g., data discovery / preparation stage). In various examples, this can be the first processing stage of distributed data processing pipeline 100. Data processing station 102 can execute processing stage 102a of data processing pipeline 100 (e.g., ML model preparation stage). In various examples, this can be the second processing stage of distributed data processing pipeline 100. Data processing station 103 can execute processing stage 103a of distributed data processing pipeline 100 (e.g., ML model training stage). In various examples, this can be the third processing stage of distributed data processing pipeline 100. Similarly, data processing stations 104 and 105 can execute processing stages 104a and 105a, respectively. In various examples, processing stages 104a and 105a can be the fourth and fifth processing stages of distributed data processing pipeline 100.

[0026] At a given processing stage, a given data processing station can execute one or more executions of that processing stage. An execution can be a data processing step that transforms one or more data artifacts. In other words, an execution can receive one or more input data artifacts (i.e., data artifacts received as input to the execution) and transform them into one or more output data artifacts (i.e., data artifacts produced by the execution). Data artifacts (including data artifacts A through F) can be data-related inputs or outputs of the distributed data processing pipeline 100 (e.g., raw data, processed datasets, analyses or ML models derived from the data, etc.). A data artifact can be the output of one execution and / or the input to one or more different executions. For example, as shown, data artifact B is the output of a first execution of processing stage 102a and the input to a first execution of processing stage 103a. Similarly, data artifact C is the output of a second execution of processing stage 102a and the input to both the first execution of processing stage 103a and the first execution of processing stage 104a.

[0027] Even as Figure 1As illustrated in the simplified example, a distributed data processing pipeline can include a complex network of data transformations / executions, where a given data artifact can be the input and / or output of multiple executions of the distributed data processing pipeline. Since these executions are performed across multiple data processing sites, such a network of transformations / executions can become increasingly complex and difficult to track. As will be described below, by enabling multiple data processing sites of the distributed data processing pipeline to utilize a common hash algorithm / function to generate hash content values ​​for identifying data artifacts, the examples of the currently disclosed techniques can ensure that data artifacts are consistently named / identified across multiple data processing sites. This common naming / identification scheme for data artifacts allows the examples of the currently disclosed techniques to construct a more accurate and consistent data lineage representation for the distributed data processing pipeline than existing techniques. Using these data lineage representations, the examples can clone the entire data processing pipeline, including its code, data, metadata, and all associated dependencies. The examples can then use the clone to locally reproduce the execution of the data processing pipeline for testing and debugging purposes. The example can also clone a distributed data processing pipeline (or a portion thereof) to extend the distributed data processing pipeline or to perform additional experiments / executions at a specific stage of the distributed data processing pipeline.

[0028] Figure 2 An example data processing pipeline abstract hierarchical structure 210 is described, which is based on various examples of distributed data processing pipelines 200 according to currently disclosed technologies.

[0029] As described above, examples of currently disclosed technologies can define abstraction / abstraction levels that can be used to consistently construct a data hierarchy representation for data processing pipelines (and their constituent data artifacts) across distributed data processing sites. Accordingly, the data processing pipeline abstraction hierarchy 210 can represent the entire distributed data processing pipeline 200 comprising multiple processing stages, each executed by a different data processing site. As shown, the data processing pipeline abstraction hierarchy 210 includes: (1) a data processing pipeline level 210a; (2) a processing stage level 210b below the data processing pipeline level 210a; (3) an execution level 210c below the processing stage level 210b; and (4) an input / output event relationship 210d for the execution level 210c. Using the data processing pipeline abstraction hierarchy 210, examples can associate each data artifact of the distributed data processing pipeline 200 with: (1) at least one of the input or output events; (2) a given execution associated with the input / output event; (3) a given processing stage associated with the given execution; and (4) a given data processing pipeline (i.e., data processing pipeline 200) associated with the given processing stage. As described above, based on these associations, examples of the currently disclosed techniques can construct a data hierarchy representation for the distributed data processing pipeline 200 (and / or each of its constituent data artifacts). Each level of the data processing pipeline abstraction hierarchy 210 will be described in more detail in the following paragraphs.

[0030] Data Processing Pipeline Level 210a: As shown in the figure, data processing pipeline level 210a is a pipeline level of the data processing pipeline abstraction hierarchy 210. Using this abstraction level, examples of the currently disclosed techniques can assign unique identifiers (e.g., pipeline names) to each unique data processing pipeline they operate on. These unique identifiers can be used to compile / document the processing stages, execution, input / output events, and data artifacts associated with a given data processing pipeline.

[0031] Processing Stage Level 210b: As shown in the figure, the next level below data processing pipeline level 210a is processing stage level 210b. A given data processing pipeline can consist of multiple processing stages (e.g., data discovery / preparation stage, ML model preparation stage, ML model training stage, ML model inference stage, etc.). As mentioned above, even when multiple processing stages are performed across different data processing sites, the examples of currently disclosed techniques can construct a data hierarchy representation for the data processing pipeline.

[0032] Using this level of abstraction, examples of currently disclosed techniques can assign unique identifiers (e.g., processing stage names) to each unique processing stage of a given data processing pipeline. These unique identifiers can be used to compile / log the execution, input / output events, and data artifacts associated with a given processing stage.

[0033] Execution Level 210c: As shown in the figure, the next level below Processing Stage Level 210b is Execution Level 210c. A given processing stage can consist of one or more executions. An execution can be a data processing step that transforms one or more data artifacts. In other words, an execution can receive one or more input data artifacts (i.e., data artifacts received as input to the execution) and transform them into one or more output data artifacts (i.e., data artifacts produced by the execution).

[0034] Using this level of abstraction, examples of currently disclosed techniques can assign unique identifiers (e.g., execution names) to each unique execution in the processing phase. These unique identifiers can be used to compile / log input / output events and data artifacts associated with a common execution.

[0035] Input / Output Event Relationship 210d: As shown in the figure, a given execution can consist of input events and output events. Input events can involve the process of inputting data artifacts into a given execution. As described above, using the data processing pipeline abstraction hierarchy 210, the example can associate one or more data artifacts input (and then transformed) by a given execution with input events for that given execution. Output events can involve the process of outputting data artifacts from a given execution. As described above, using the data processing pipeline abstraction hierarchy 210, the example can associate one or more data artifacts output from / generated by a given execution with output events of that given execution.

[0036] In this way, the example can associate each data artifact of the distributed data processing pipeline 200 with: (1) at least one of the input or output events; (2) a given execution associated with the input / output event; (3) a given processing stage associated with the given execution; and (4) a given data processing pipeline (i.e., the distributed data processing pipeline 200) associated with the given processing stage. As described above, based on these associations, the examples of the currently disclosed technology can construct a data hierarchy representation for the distributed data processing pipeline 200 (and / or each of its constituent data artifacts).

[0037] Figure 3 An example workflow for an example distributed data processing pipeline 300, based on various examples of currently disclosed technologies, is described.

[0038] The distributed data processing pipeline 300 comprises three processing stages performed by three data processing sites (e.g., data centers, edge computing sites, etc.). Specifically, data processing site 310 performs the raw data acquisition stage, data processing site 320 performs the data processing stage, and data processing site 330 performs the inspection stage. As shown, each data processing site is connected to a central repository 350 (e.g., a central data center, GitHub, etc.) at least intermittently. However (and importantly), the data processing sites do not need to be connected to each other (i.e., they can be distributed). Accordingly, the data processing sites can work independently of each other (and in many cases offline) while performing their respective processing stages (as will be described below, when each data processing site completes its work, it can export its results to the central repository 350). This independent operating mode can be beneficial for data security and privacy purposes. This operation can also facilitate collaboration between multiple remote individuals / teams in a way that enhances data security for those involved.

[0039] A central repository 350 (e.g., a central data center, GitHub, etc.) may be a repository logically located centrally relative to data processing sites 310 to 330 (however, it should be understood that the central repository 350 may be physically distributed, for example, between data processing sites 310 and 330). As shown, the central repository 350 may include multiple sub-repositories, such as: a central artifact repository 352 (i.e., a central sub-repository for storing data artifacts of the distributed data processing pipeline 300); a central code repository 354 (i.e., a central sub-repository for storing code used in the execution of the distributed data processing pipeline 300); and a central metadata repository 356 (i.e., a central sub-repository for storing metadata associated with the data artifacts and execution of the distributed data processing pipeline 300). In various examples, the central repository 350 may utilize Git as an index for the stored data artifacts, code, and metadata.

[0040] As described above, data processing station 310 can perform the first processing stage (i.e., raw data acquisition stage) of the distributed data processing pipeline 300, data processing station 320 can perform the second processing stage (i.e., data processing stage) of the distributed data processing pipeline 300, and data processing station 330 can perform the third processing stage (i.e., inspection stage) of the distributed data processing pipeline 300. In various examples, other data processing stations of the distributed data processing pipeline 300 can perform additional processing stages of the distributed data processing pipeline 300.

[0041] At a given processing stage, a given data processing station can execute one or more executions for that processing stage. An execution can be a data processing step that transforms one or more data artifacts. In other words, an execution can receive one or more input data artifacts (e.g., one or more raw data artifacts) and transform them into one or more output data artifacts (e.g., one or more distilled data artifacts, such as datasets). Data artifacts can be data-related inputs or outputs of a data processing pipeline (e.g., raw data, processed datasets, analyses or ML models derived from the data, etc.). Data artifacts can be the output of one execution and / or the input to one or more different executions. For example, a given raw data artifact can also be the input to an execution of a data processing stage, in addition to being the output of an execution in the raw data acquisition stage. Each stage of the distributed data processing pipeline 300 will be described in more detail in the following paragraphs.

[0042] Raw data acquisition phase (performed by data processing station 310): At the raw data acquisition phase, data processing station 310 collects raw data artifacts for future processing by distributed data processing pipeline 300. Data processing station 310 also collects code (or at least one version of that code) used to collect the raw data artifacts. This code may be associated with the execution used to collect the raw data artifacts.

[0043] In addition to collecting raw data artifacts and the code (or at least one version of the code) used to collect them, data processing station 310 also creates metadata associated with the collected raw data artifacts. The metadata created for a given collected raw data artifact includes: (1) a hash content value identifying the given collected raw data artifact; and (2) an indication that the given collected raw data artifact is the output of a given execution of the raw data acquisition phase.

[0044] As described above, the created hash content value can be used as an identification key / link, allowing examples of the currently disclosed technology (including central repository 350) to construct a data lineage representation for distributed data processing pipeline 300 and / or its constituent data artifacts. In other words, examples of the currently disclosed technology (including central repository 350) can utilize the created hash content value to determine, for example, that the same data artifact (e.g., the raw data artifact of the first collection) is both the output of the raw data acquisition phase and the input of the data processing phase—even if these two processing phases are performed by different individuals at different data processing sites (i.e., data processing sites 310 and 320, respectively). In cases where multiple data processing sites in the data processing pipeline operate on data artifacts, the unique identifier of the data artifact provided by the hash content value of the currently disclosed technology plays a crucial role. This is because in many cases, distributed data processing stations will, for example: (1) use different names / labels for the same data artifact (i.e., a first data processing station may use a first name / label for a first data artifact, and a second data processing station may use a second name / label for the (same) first data artifact); or (2) use the same name / label for different data artifacts (i.e., a first data processing station may use a first name / label for a first data artifact, and a second data processing station may use the (same) first name / label for a second (different) data artifact). As will be described below, by having multiple data processing stations in a distributed data processing pipeline utilize the same hash algorithm / function to generate hash content values ​​for data artifacts, examples can ensure that data artifacts are consistently named / identified across multiple data processing stations. Such a common naming / identification scheme for data artifacts allows examples of currently disclosed techniques to construct a more accurate and consistent data lineage representation for data processing pipelines (and / or their constituent data artifacts) than existing techniques.

[0045] A hash content value can be considered a unique fingerprint of a data artifact. Typically, a hash content value will be a numeric and / or alphabetic string value that uniquely identifies the data artifact. Hash content values ​​can be generated using hash functions (e.g., cryptographic algorithms) that generate hash content values ​​based on the content of the data artifact. Two common hash functions / algorithms used to generate hash content values ​​for data artifacts are MD5 and SHA1.

[0046] Accordingly, data processing station 310 can use hash functions / algorithms to create / generate unique hash content values ​​for each collected raw data artifact.

[0047] In various examples, all data processing stations of the distributed data processing pipeline 300 can utilize a common hash function / algorithm to create hash content values. For example, data processing stations 310 to 330 can utilize the MD5 algorithm to create hash content values ​​for data artifacts. Therefore, a first hash content value created by data processing station 310 (identifying, for example, the original data artifact of a first collection) can include the same numeric / alphanumeric string as the corresponding hash content value created by data processing station 320 (identifying the original data artifact of the first collection). In contrast, a second hash content value created by data processing station 310 (identifying, for example, the original data artifact of a second collection) can include a different numeric / alphanumeric string than the first hash content value. As described above, by having multiple data processing stations in the data processing pipeline utilize the same hash algorithm / function to generate hash content values ​​for data artifacts, the examples ensure that data artifacts are consistently named / identified across multiple data processing stations. This common naming / identification scheme for data artifacts allows examples of currently disclosed techniques to construct accurate and complete data lineage representations for distributed data processing pipelines and / or their constituent data artifacts.

[0048] As described above, the metadata of the given collected raw data artifact (created by data processing station 310) also includes an indication that the given collected raw data artifact (identified by the hash content value of the given collected raw data artifact) is the output of a given execution from the raw data acquisition phase. Here, data processing station 310 can utilize a combination of Figure 2 The described data processing pipeline uses an abstract, hierarchical structure to make this instruction. (For example, combining...) Figure 2 The data processing pipeline abstraction hierarchy described herein may include: (1) a data processing pipeline level; (2) a processing stage level below the data processing pipeline level; (3) an execution level below the processing stage level; and (4) input / output event relationships at the execution level. Using this data processing pipeline abstraction hierarchy, examples may associate each data artifact of the data processing pipeline with: (1) at least one of an input event or an output event; (2) a given execution associated with the input / output event(s); (3) a given processing stage(s) associated with the given execution(s); and (4) a given data processing pipeline associated with the given processing stage(s). As described above, based on these associations, examples of the currently disclosed technology may construct a data hierarchy representation for a given data processing pipeline and / or its constituent data artifacts.

[0049] In various examples, the metadata created by data processing station 310 may also include a description of the physical location of a given collected raw data artifact within a data artifact repository (e.g., a central artifact repository 352). In some examples, data processing station 310 may create / collect additional metadata associated with the collected raw data artifacts and the code / execution used to collect them (e.g., a text document of the data source for the collected raw data artifacts, a description of the contents within the collected raw data artifacts, the data size and distribution of the collected raw data artifacts, the programming language used in the execution, information about the input fields used in the execution, the type of algorithm used in the execution, the stage or quality assurance (QA) status of the execution, the names of the developers / data scientists involved in the execution, etc.).

[0050] Once collected / created, data processing station 310 can export to central repository 350: (1) raw data artifacts, (2) code used in execution to collect raw data artifacts, and (3) metadata associated with the collected raw data artifacts (and in some cases, metadata associated with the execution used to collect raw data artifacts). For example, data processing station 310 can export the collected code associated with the raw data acquisition phase to central code repository 354 (i.e., a central repository for storing code used in the execution of distributed data processing pipeline 300). Similarly, data processing station 310 can use data versioning software such as data version control (DVC) to export the collected raw data artifacts to central artifact repository 352 (i.e., a central repository for storing data artifacts of distributed data processing pipeline 300). Data processing station 310 can also export the collected metadata to central metadata repository 356 (i.e., a central repository for storing metadata associated with data artifacts and execution of distributed data processing pipeline 300).

[0051] As described above, the derived metadata for a given collected raw data artifact may include: (1) a hash content value identifying the given collected raw data artifact; (2) an indication that the given collected raw data artifact is the output of a given execution of the raw data acquisition phase; and (3) other metadata associated with the given collected raw data artifact. When the central metadata repository 356 receives a hash content value for the first time, the central metadata repository 356 may create a new metadata entry for the data artifact identified by the (newly encountered) hash content value. Conversely, when the central metadata repository 356 has previously encountered a hash content value, the central metadata repository 356 may merge the newly received metadata entry with one or more existing metadata entries associated with the data artifact identified by the previously encountered hash content value. In this way, the central metadata repository 356 can stitch together both the data lineage representation and metadata representation for the data artifact, as well as the distributed data processing pipeline 300, as a whole.

[0052] Data processing phase (performed by data processing station 320): In the data processing phase, data processing station 320 processes the raw data artifacts collected (and then exported) by data processing station 310. Here, data processing station 320 can process the collected / exported raw data artifacts into more refined data artifacts (e.g., processed datasets, machine learning models, etc.). Data processing station 320 can retrieve the raw data artifacts from the central artifact repository 352.

[0053] Similar to data processing station 310, data processing station 320 can create metadata for each data artifact consumed (i.e., raw data artifacts extracted from the central artifact repository 352) and generated (i.e., refined data artifacts) during the data processing phase. As described above, the metadata created for a given data artifact may include: (1) a hash content value identifying the given data artifact; and (2) an indication that the given data artifact is at least one of the inputs and outputs of a given execution of the data processing phase.

[0054] Here, data processing station 320 can utilize the same hash function / algorithm as data processing station 310 to create hash content values. Accordingly, data processing station 320 and data processing station 310 should create the same hash content values ​​(i.e., the same string of numbers and / or letters) for public data artifacts (e.g., the first original data artifact collected / exported by data processing station 310 and extracted / processed by data processing station 320). As described above (and will be described in more detail below), examples of the currently disclosed technology can utilize these hash content values ​​to stitch together the data lineage representations for the distributed data processing pipeline 300 (and / or its constituent data artifacts).

[0055] As described above, the metadata created by data processing site 320 may also include an identifier of the physical location of a given data artifact within a data artifact repository (e.g., data artifact repository 352). In some examples, data processing site 320 may create / collect additional metadata associated with the consumed / generated data artifacts and the code / execution used to process them (e.g., the text document of the data artifact, a description of the contents within the data artifact, the data size and data distribution of the data artifact, the programming language used for the execution of the data artifact, information about the input fields used for the execution, the type of algorithm used for the execution, the stage or quality assurance (QA) status of the execution, the names of the developers / data scientists involved in the execution, etc.).

[0056] Data processing station 320 may export to central repository 350: (1) newly generated refined data artifacts (and, in some examples, the extracted original data artifacts used to generate the refined data artifacts), (2) code used in execution to generate the refined data artifacts, and (3) metadata associated with the refined data artifacts (and, in some cases, metadata associated with the execution used to generate the refined artifacts). For example, data processing station 320 may export code associated with a data processing stage to central code repository 354 (i.e., a central repository for storing code used in the execution of distributed data processing pipeline 300). Similarly, data processing station 320 may use data version control software, such as data version control (DVC), to export refined data artifacts to central artifact repository 352 (i.e., a central repository for storing data artifacts of distributed data processing pipeline 300). Data processing station 310 can also export metadata from the data processing stage to a central metadata repository 356 (i.e., a central repository for storing metadata associated with data artifacts and execution of the distributed data processing pipeline 300).

[0057] As described above, the metadata of a given data artifact derived may include: (1) a hash content value identifying the given data artifact; (2) an indication that the given data artifact is at least one of the inputs or outputs of a given execution of a data processing phase; and (3) other metadata associated with the given data artifact. When the central metadata repository 356 receives a hash content value for the first time (e.g., the hash content value of a newly generated refined data artifact), the central metadata repository 356 may create a new metadata entry for the data artifact identified by the (newly encountered) hash content value. When the central metadata repository 356 has previously encountered a hash content value (e.g., for a raw data artifact collected by data processing site 310 and extracted / processed by data processing site 320), the central metadata repository 356 may merge the newly derived metadata entry with existing metadata entries associated with the given data artifact identified by the previously encountered hash content value. In this way, the central metadata repository 356 can stitch together both the data lineage representation and metadata representation for a given data artifact, as well as the distributed data processing pipeline 300, as a whole. For example, the central metadata repository 356 may have received metadata associated with the first raw data artifact from the data processing site 310 and recorded a metadata entry for the associated first raw data artifact. This recorded metadata entry may include an indication that the first raw data artifact is the output of a given execution of the raw data acquisition phase. The central metadata repository 356 may then receive metadata associated with the first raw data artifact from the data processing site 320, indicating that the first raw data artifact is the input of a given execution of the data processing phase. Here, the central metadata repository 356 may use the hash content value of the received first raw data artifact to determine that the two executions involve the same (i.e., the first raw data artifact). Accordingly, the central metadata repository 356 may merge these two metadata entries to construct at least a partial data lineage representation for the first raw data artifact.

[0058] As described above, the merged metadata stored in the central metadata repository 356 may include data lineage information for each data artifact of the distributed data processing pipeline 300 (i.e., a linear association between a given data artifact and (1) its ancestor data artifacts and executions; and / or (2) its descendant data artifacts and executions), as well as data lineage information for the distributed data processing pipeline 300 as a whole (i.e., a linear association between all data artifacts of the distributed data processing pipeline 300 and the executions that are transformed and / or produced). In various examples, the central repository 350 may construct: (1) a data lineage representation for each data artifact in the data processing pipeline 300 (e.g., a lineage directed acyclic graph (DAG)); and / or (2) a data lineage representation for the distributed data processing pipeline 300 as a whole (e.g., a data lineage representation for all data artifacts of the distributed data processing pipeline 300 as a whole) (e.g., a lineage directed acyclic graph (DAG)). In some examples, the central repository 350 can construct these data lineage representations using recursive queries on the stored metadata and artifacts (e.g., all child artifacts of a given artifact; all ancestor artifacts of a given artifact; all artifacts associated with the same data processing pipeline; and so on).

[0059] Inspection Phase (performed by Data Processing Station 330): At the inspection phase, Data Processing Station 330 inspects (and / or performs) the data artifacts of the distributed data processing pipeline 300. As described above, Data Processing Station 330 may perform this inspection for regulatory purposes and / or troubleshoot / debug the distributed data processing pipeline 300. In various examples, Data Processing Station 330 may “clone” the data processing pipeline 300 using content stored in Central Repository 350 for inspection purposes. Such cloning may include extracting: (1) all metadata of the distributed data pipeline 300, including data lineage information for the distributed data processing pipeline 300 and its data artifacts; (2) all data artifacts of the distributed data processing pipeline 300; and (3) all code used in the distributed data processing pipeline (or at least one version of that code). By extracting the above content, data processing station 330 can locally reproduce distributed data processing pipeline 300 for troubleshooting / debugging purposes (data processing station 330 can also clone distributed data processing pipeline 300 or parts thereof in order to extend distributed data processing pipeline 300 or perform additional experiments / executions at specific stages of distributed data processing pipeline 300).

[0060] For example, data processing station 330 can (e.g., using a code pull request) extract and then clone code stored in central code repository 354. Similarly, data processing station 330 can (e.g., using a metadata pull command) extract merged metadata stored in central metadata repository 356. As described above, the merged metadata stored in central metadata repository 356 may include: (1) a data hierarchy representation (e.g., a directed acyclic graph (DAG)) for each data artifact in the distributed data processing pipeline 300, individually; and / or (2) a data hierarchy representation (e.g., a directed acyclic graph (DAG)) for the distributed data processing pipeline 300 as a whole (i.e., a hierarchy representation for all data artifacts of the distributed data processing pipeline 300 as a whole). In some examples, data processing station 330 may extract these data hierarchy representations from central metadata repository 356 as SQLite files. Upon receipt, data processing station 330 can traverse each node of the extracted data lineage representation (here, each node may represent a data artifact of data processing pipeline 300) to identify the data artifacts of the extracted data lineage. Data processing station 330 can then retrieve these data artifacts from central artifact repository 352. In some examples, data processing station 330 may utilize a description of the physical location of a given data artifact in the central artifact repository (as described above, this may be an element of stored metadata associated with the given data artifact) for retrieval purposes. Here, by traversing the extracted data lineage representation, data processing station 330 can ensure that all relevant data artifacts and execution code are retrieved for inspection and analysis.

[0061] Figure 4 Example computing systems that can be used to construct data hierarchy representations for distributed data processing pipelines are described, based on various currently disclosed techniques.

[0062] Now for reference Figure 4 The computing component 410 can be, for example, a server computer, a controller, or any other similar computing component capable of processing data. Figure 4 In the example implementation, computing component 410 includes hardware processor 412 and machine-readable storage medium 414.

[0063] Hardware processor 412 may be one or more central processing units (CPUs), semiconductor-based microprocessors, and / or other hardware devices adapted to retrieve and execute instructions stored in machine-readable storage medium 414. Hardware processor 412 may fetch, decode, and execute instructions (such as instructions 416 to 420) to control a burst preloading process or operation for estimating available bandwidth. Alternatively or supplementary to retrieving and executing instructions, hardware processor 412 may include one or more electronic circuits comprising electronic components for the function of executing one or more instructions, such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or other electronic circuits.

[0064] Machine-readable storage media (such as machine-readable storage media 414) can be any electronic, magnetic, optical, or other physical storage device that contains or stores executable instructions. Therefore, machine-readable storage media 414 can be, for example, random access memory (RAM), non-volatile RAM (NVRAM), electrically erasable programmable read-only memory (EEPROM), storage devices, optical discs, etc. In some examples, machine-readable storage media 414 can be a non-transitory storage medium, where the term "non-transitory" does not cover transient propagation indicators. As described in detail below, machine-readable storage media 414 can be encoded with executable instructions (e.g., instructions 416 to 420).

[0065] As described above, the computing system 400 can be used to construct a data hierarchy representation for a distributed data processing pipeline executed across multiple data processing sites.

[0066] Accordingly, the hardware processor 412 may execute instruction 416a to receive first information relating to the first data artifact from the first data processing station. The first information relating to the first data artifact may include: (1) a first hash content value identifying the first data artifact; and (2) an indication that the first data artifact is the output of a first execution of a first processing stage in a distributed data processing pipeline, wherein the first processing stage is executed at the first data processing station.

[0067] Accordingly, the hardware processor 412 may execute instruction 416b to receive second information relating to the first data artifact from the second data processing station. The second information relating to the first data artifact may include: (1) a first hash content value identifying the first data artifact; and (2) the first data artifact being an indication of input to a first execution of a second processing stage of a distributed data processing pipeline, wherein the second processing stage is executed at the second data processing station.

[0068] The hardware processor 412 may also execute instruction 416c to receive first information relating to the second data artifact from the second data processing station. The first information relating to the second data artifact may include: (1) a second hash content value identifying the second data artifact, and (2) an indication that the second data artifact is the output of the first execution of the second processing stage.

[0069] The first and second data processing stations can be two of multiple data processing stations in a distributed data processing pipeline. Each data processing station can be, for example, a data center, an edge computing station, etc. The multiple data processing stations in the distributed data processing pipeline can be disconnected from each other, but can be connected to the computing system 400 at least intermittently.

[0070] As described above, the first data processing station can perform the first processing stage of the distributed data processing pipeline (e.g., the data discovery / preparation stage), and the second data processing station can perform the second processing stage of the distributed data processing pipeline (e.g., the ML model preparation stage). Other data processing stations in the distributed data processing pipeline can perform additional processing stages. For example, the third data processing station can perform the third processing stage (e.g., the ML model training stage), the fourth data processing station can perform the fourth processing stage (e.g., the ML model inference stage), and so on.

[0071] At a given processing stage, a given data processing station can perform one or more executions for that given processing stage. An execution (including a first execution of a first data processing stage and a first execution of a second data processing stage) can be a data processing step that transforms one or more data artifacts. In other words, an execution can receive one or more input data artifacts (i.e., data artifacts received as input to the execution) and transform them into one or more output data artifacts (i.e., data artifacts produced by the execution). Data artifacts (including a first data artifact and a second data artifact) can be data-related inputs or outputs of a data processing pipeline (e.g., raw data, processed datasets, analyses or ML models derived from data, etc.). A data artifact can be the output of one execution and / or the input to one or more different executions. For example, in addition to being the output of a first execution of a first processing stage, a first data artifact can also be the input of a first execution of a second processing stage.

[0072] As described above, the hardware processor 412 can execute instructions 416a to 416c to receive specially tailored information (i.e., first information relating to a first data artifact, second information relating to a first data artifact, first information relating to a second data artifact, etc.) from the first and second data processing stations, allowing it to construct a data hierarchy representation for the distributed data processing pipeline (even if the distributed data processing pipeline is executed across multiple distributed stations). The constructed data hierarchy representation may include linear relationships between: (1) a first execution of a first processing stage; (2) a first artifact; (3) a first execution of a second processing stage; (4) a second data artifact; etc. Examples of the currently disclosed technology can utilize such a data hierarchy representation to, for example, clone a distributed data processing pipeline for testing and debugging purposes.

[0073] As described above, each specially customized information (unit) includes: (1) a hash content value that identifies a given data artifact; and (2) an indication that the given data artifact is an input or output of a given execution of a given data processing stage.

[0074] The received hash content value can be used as an identification key / link, allowing the hardware processor 412 to stitch together data lineage representations. In other words, the hardware processor 412 can use the received hash content value to determine, for example, that the same data artifact (i.e., the first data artifact) is both the output of a first execution from a first processing stage and the input to a first execution of a second processing stage—even when the first and second processing stages are executed at different data processing sites (i.e., the first data processing site and the second data processing site, respectively). In the case of multiple data processing sites operating on data artifacts in a distributed data processing pipeline, the unique identification of the data artifact provided by the hash content value of currently disclosed techniques can be crucial. This is because, in many cases, distributed data processing stations will, for example, use different names / labels for the same data artifact (e.g., a first data processing station may use a first name / label for a first data artifact, and a second data processing station may use a second name / label for the (same) first data artifact), or use the same name / label for different data artifacts (e.g., a first data processing station may use a first name / label for a first data artifact, and a second data processing station may use the (same) first name / label for a second data artifact). As will be described below, by having multiple data processing stations in a distributed data processing pipeline utilize the same hash algorithm / function to generate hash content values ​​for data artifacts, the example can ensure that data artifacts are consistently named / identified across multiple data processing stations. This common naming / identification scheme for data artifacts allows the examples of currently disclosed techniques to construct a more accurate and consistent data lineage representation for distributed data processing pipelines (and / or their constituent data artifacts) than existing techniques.

[0075] As mentioned above, a hash content value can be considered a unique fingerprint of a data artifact. Typically, a hash content value will be a numeric and / or alphabetic string value that uniquely identifies the data artifact. Hash content values ​​can be generated using hash functions (e.g., cryptographic algorithms) that generate hash content values ​​based on the content of the data artifact. Two common hash functions / algorithms used to generate hash content values ​​for data artifacts are MD4 and SHA1.

[0076] Accordingly, the first hash content value may be a first numeric and / or alphanumeric string value that uniquely identifies the first data artifact across multiple distributed data processing stations / stages in the data processing pipeline. Similarly, the second hash content value may be a second numeric and / or alphanumeric string value that uniquely identifies the second data artifact across multiple distributed data processing stations / stages in the data processing pipeline.

[0077] In various examples, the first and second data processing stations can utilize a common hash function / algorithm to generate hash content values. For example, the first and second data processing stations can utilize the MD5 algorithm to generate a first hash content value for the first data artifact. Accordingly, the first hash content value (identifying the first data artifact) received by the hardware processor 412 from the first data processing station may include the same string of numbers / letters as the first hash content value (identifying the first data artifact) received by the hardware processor 412 from the second data processing station. In contrast, the second hash content value (identifying the second data artifact) received by the hardware processor 412 from the second data processing station may include a different string of numbers / letters than the first hash content value.

[0078] As described above, the specially tailored information received by the hardware processor 412 also includes an indication that a given data artifact (identified by a hash content value of the given data artifact) is an input or output of a given execution of a given data processing stage. Here, examples of the currently disclosed technology can utilize a combination of Figure 2 This instruction is given by describing an abstract, hierarchical structure of the data processing pipeline. For example, combining... Figure 2 The data processing pipeline abstraction hierarchy described herein may include: (1) a data processing pipeline level; (2) a processing stage level below the data processing pipeline level; (3) an execution level below the processing stage level; and (4) input / output event relationships for the execution level. Using this data processing pipeline abstraction hierarchy, examples may associate each data artifact of the data processing pipeline with: (1) at least one of an input event or an output event; (2) a given execution associated with the input / output event(s); (3) a given processing stage(s) associated with the given execution(s); and (4) a given data processing pipeline associated with the given processing stage(s). As described above, based on these associations, examples of the currently disclosed technology may construct a data hierarchy representation for a distributed data processing pipeline.

[0079] In various examples, the information received by the hardware processor 412 may also include metadata associated with the data artifact and execution. For example, first information relating to the first data artifact may also include metadata associated with: (1) the first data artifact (e.g., a text document of the first data artifact, a description of the contents within the first data artifact, the data size and data distribution of the first data artifact, etc.); and (2) the first execution of the first processing stage (e.g., the programming language of the execution, information about the input fields used for the execution, the type of algorithm used for the execution, the stage or quality assurance (QA) status of the execution, the name of the developer / data scientist involved in the execution, etc.). Similarly, second information relating to the first data artifact may also include metadata associated with: (1) the first data artifact; and (2) the first execution of the second processing stage. First information relating to the second data artifact may also include metadata associated with: (1) the second data artifact; and (2) the first execution of the second processing stage. Figure 3 The hash content value of a given data artifact and an indication that a given data artifact is an input or output of a given execution can also be considered metadata associated with a given data artifact.

[0080] As described in conjunction with instruction 418, based on the received metadata, hardware processor 412 can compile a merged metadata representation that provides a “global view” of the metadata for the data processing pipeline. As will be described below, examples of the currently disclosed technology can utilize this merged metadata representation, along with a constructed data hierarchy representation for the distributed data processing pipeline, to reproduce / clone the distributed data processing pipeline for, for example, testing and debugging purposes.

[0081] Hardware processor 412 can execute instructions 418 to construct a data hierarchy representation for a distributed data processing pipeline based on information received after executing instructions 416a to 416c.

[0082] As described above, the constructed data hierarchy representation may include linear relationships between the following items: (1) a first execution of the first processing stage; (2) a first artifact; (3) a first execution of the second processing stage; (4) a second data artifact; and so on. In other words, the constructed data hierarchy representation may relate to the following items: (1) a first data artifact as the output of the first execution of the first processing stage; (2) a first data artifact as the input of the first execution of the second processing stage; (3) a second data artifact as the output of the first execution of the second processing stage; and so on.

[0083] In an example where the information received by the hardware processor 412 also includes (other) metadata associated with data artifacts and execution, the execution of instruction 418 by the hardware processor 412 may further include compiling a merged metadata representation for the data processing pipeline. As described above, the merged metadata representation can provide a “global view” of the metadata for the data processing pipeline. Examples of the currently disclosed technology can utilize this merged metadata representation, along with a constructed data lineage representation for the data processing pipeline, to reproduce / clone the data processing pipeline for, for example, testing and debugging purposes.

[0084] Accordingly, hardware processor 412 may execute instructions 420 to export the constructed data hierarchy representation of the distributed data processing pipeline to a third data processing station of the distributed data processing pipeline. In some examples, execution of instructions 420 by hardware processor 412 may also include exporting a merged metadata representation of the data pipeline to the third data processing station.

[0085] As described above, a third data processing station can use the constructed data hierarchy representation and / or merged metadata representation for the distributed data processing pipeline to clone the distributed data processing pipeline for, for example, testing and debugging purposes.

[0086] In various examples, hardware processor 412 can clone the distributed data processing pipeline itself using the constructed data hierarchy representation and the merged metadata representation, instead of outputting the constructed data hierarchy representation and the merged metadata representation to a third data processing station. Accordingly, hardware processor 412 can use the cloned distributed data processing pipeline for, for example, testing and debugging the distributed data processing pipeline.

[0087] Figure 5 An example computing system 500, depicting various examples of currently disclosed techniques, is associated with a second data processing station in a distributed data processing pipeline. This example computing system 500 can be used to export information related to data artifacts processed by the computing system 500 to a central repository shared by multiple data processing stations (including the second data processing station) of the distributed data processing pipeline. As described above, the central repository can utilize information received from the multiple data processing stations of the distributed data processing pipeline to construct a data hierarchy representation for the distributed data processing pipeline. Reference is now made to... Figure 5 The computing component 510 can be, for example, a server computer, a controller, or any other similar computing component capable of processing data. Figure 5 In the example implementation, computing component 510 includes hardware processor 512 and machine-readable storage medium 514. In various examples, computing system 500 may be associated with a second data processing station in a data processing pipeline.

[0088] The hardware processor 512 and the machine-readable storage medium 514 may be the same as or similar to the hardware processor 412 and the machine-readable storage medium 414, respectively. Accordingly, the machine-readable storage medium 414 may be encoded with executable instructions (e.g., instructions 516 to 522).

[0089] The hardware processor 512 can execute instructions 516 to process the first data artifact in a first execution of the second data processing stage of the distributed data processing pipeline, wherein the processing produces a second data artifact.

[0090] The second data processing station can be one of multiple data processing stations in a distributed data processing pipeline. Each data processing station can be, for example, a data center, an edge computing station, etc. The multiple data processing stations in the distributed data processing pipeline can be disconnected from each other, but can be connected to the central repository at least intermittently.

[0091] As described above, a second data processing station can execute a second processing stage of the data processing pipeline (e.g., an ML model preparation stage). This second processing stage can follow a first processing stage (e.g., a data discovery / preparation stage) executed by the first data processing station. In various examples, the first data processing station / stage can produce a first data artifact. Other data processing stations in the distributed data processing pipeline can execute additional processing stages. For example, a third data processing station can execute a third processing stage (e.g., an ML model training stage), a fourth data processing station can execute a fourth processing stage (e.g., an ML model inference stage), and so on.

[0092] At a given processing stage, a given data processing station may perform one or more executions of that processing stage. An execution (including a first execution of a second data processing stage) may be a data processing step that transforms one or more data artifacts. In other words, an execution may receive one or more input data artifacts (i.e., data artifacts received as input to the execution) and transform them into one or more output data artifacts (i.e., data artifacts produced by the execution). Accordingly, processing a first data artifact in a first execution of a second data processing stage may include transforming the first data artifact into a second data artifact.

[0093] Data artifacts (including first and second data artifacts) can be data-related inputs or outputs (e.g., raw data, processed datasets, ML models, etc.) of a distributed data processing pipeline. A data artifact can be the output of an execution and / or the input to one or more different executions. For example, in addition to being the input to the first execution of a second processing stage, a first data artifact can also be, for example, the output of the first execution of a first processing stage.

[0094] The hardware processor 512 can execute instructions 518 to generate: (1) a first hash content value for the first data artifact that identifies the first data artifact; and (2) a second hash content value for the second data artifact that identifies the second data artifact.

[0095] As described above, the generated hash content value can be used as an identifier key, allowing the central repository to stitch together the data lineage representations for the distributed data processing pipeline. In other words, the central repository can use the hash content value it receives to determine, for example, that the same data artifact (i.e., the first data artifact) is both the output of the first execution of the first processing stage and the input to the first execution of the second processing stage—even when the first and second processing stages are executed at different data processing sites (i.e., the first data processing site and the second data processing site, respectively).

[0096] As mentioned above, a hash content value can be considered a unique fingerprint of a data artifact. Typically, a hash content value will be a numeric and / or alphabetic string value that uniquely identifies the data artifact. Hash content values ​​can be generated using hash functions (e.g., cryptographic algorithms) that generate hash content values ​​based on the content of the data artifact. Two common hash functions / algorithms for generating hash content values ​​for data artifacts are MD5 and SHA1.

[0097] Accordingly, the hardware processor 512 can utilize a hash function / algorithm (e.g., MD5 or SHA1 algorithm) to generate a first hash content value for the first data artifact and a second hash content value for the second data artifact. The first hash content value may be a first numeric and / or alphanumeric string value that uniquely identifies the first data artifact across multiple distributed data processing stations / stages in the distributed data processing pipeline. Similarly, the second hash content value may be a second numeric and / or alphanumeric string value that uniquely identifies the second data artifact across multiple distributed data processing stations / stages in the distributed data processing pipeline.

[0098] In various examples, the second data processing station can utilize the same hash function / algorithm as the other data processing stations in the distributed data processing pipeline to generate hash content values. For example, all data processing stations in the distributed data processing pipeline (including the second data processing station) can utilize the MD5 algorithm to generate hash content values ​​for the data artifact. Accordingly, the first hash content value generated by the hardware processor 512 for the first data artifact may include the same string of numbers / letters as the first hash content value generated by another data processing station for the first data artifact. The central repository can leverage this commonality to stitch together received information related to public data artifacts (e.g., first information related to the first data artifact and second information related to the first data artifact).

[0099] Accordingly, the hardware processor can execute instruction 520 to export to a central repository shared by at least the second data processing station and the first data processing station: (1) second information relating to the first data artifact; and (2) first information relating to the second data artifact (here, it should be understood that the terms first and second do not necessarily imply order).

[0100] The second information relating to the first data artifact may include: (a) a first hash content value identifying the first data artifact; and (b) an indication that the first data artifact is an input to the first execution of the second processing stage.

[0101] The first information relating to the second data artifact may include: (a) a second hash content value identifying the second data artifact; and (b) an indication that the second data artifact is the output of the first execution of the second processing stage.

[0102] As described above, the information derived by the hardware processor 512 may include specially tailored information that allows the central repository to construct a data hierarchy representation for the distributed data processing pipeline. The constructed data hierarchy representation may include, for example, linear relationships between: (1) a first execution of a first processing stage that produces a first data artifact; (2) the first data artifact; (3) a first execution of a second processing stage; (4) a second data artifact; and so on. Examples of the currently disclosed technology can utilize such a data hierarchy representation to, for example, clone a distributed data processing pipeline for testing and debugging purposes.

[0103] As described above, the specially customized information derived by the hardware processor 512 includes an indication that a given data artifact (identified by the hash content value of the given data artifact) is an input or output of a given execution of a given data processing stage. Here, the hardware processor 512 can utilize a combination of Figure 2 The described data processing pipeline uses an abstract, hierarchical structure to make / generate this instruction. (For example, combining...) Figure 2The data processing pipeline abstraction hierarchy described in the presently disclosed technology may include: (1) a data processing pipeline level; (2) a processing stage level below the data processing pipeline level; (3) an execution level below the processing stage level; and (4) input / output event relationships at the execution level. Using this data processing pipeline abstraction hierarchy, examples of the presently disclosed technology (including hardware processor 512) can associate a given data artifact of a data processing pipeline with: (1) at least one of an input event or an output event; (2) a given execution associated with the input / output event(s); (3) a given processing stage associated with the given execution(s); and (4) a given data processing pipeline associated with the given processing stage(s). As described above, based on the associations indicated in the information derived by hardware processor 512 (along with similar information derived by other data processing sites of the distributed data processing pipeline), a central repository can construct a data hierarchy representation for the distributed data processing pipeline.

[0104] In various examples, the information derived by the hardware processor 512 may also include information relating to metadata associated with the data artifact and the execution. For example, the second information relating to the first data artifact may also include metadata associated with: (1) the first data artifact (e.g., a text document of the first data artifact, a description of the contents within the first data artifact, the data size and data distribution of the first data artifact, etc.); and (2) the first execution of the second processing stage (e.g., the programming language of the execution, information about the input fields used for the execution, the type of algorithm used for the execution, the stage or quality assurance (QA) status of the execution, the name of the developer / data scientist of the execution, etc.). Similarly, the first information relating to the second data artifact may also include metadata associated with: (1) the second data artifact; and (2) the first execution of the second processing stage.

[0105] Based on the metadata it receives, the hardware processor 512 can compile a merged metadata representation that provides a “global view” of the metadata for the data processing pipeline. As will be described below, examples of the currently disclosed technology can utilize this merged metadata representation, along with a constructed data hierarchy representation for the distributed data processing pipeline, to reproduce / clone the distributed data processing pipeline for, for example, testing and debugging purposes.

[0106] Accordingly, the hardware processor 512 may execute instruction 522 to extract a data hierarchy representation for the distributed data processing pipeline from a central repository. In some examples, the execution of instruction 522 by the hardware processor 512 may also include extracting a merged metadata representation for the distributed data processing pipeline from the central repository. The extracted data hierarchy representation and the merged metadata representation may be based in part on information derived by the hardware processor 512 to the central repository. Accordingly, the extracted data hierarchy representation may at least represent a hierarchy representation for the first data artifact and the second data artifact. In other words, the extracted data hierarchy representation may at least include a linear relationship between: (1) the first data artifact; (2) the first execution of the second processing stage; and (3) the second data artifact.

[0107] As described above, in various examples, hardware processor 512 can use the extracted data hierarchy representation and merged metadata representation to clone / reproduce a distributed data processing pipeline for purposes such as debugging and testing.

[0108] Figure 6 An example computing system 600, which can be used to clone a distributed data processing pipeline, is depicted according to various examples of currently disclosed technologies. Reference is now made to... Figure 6 The computing component 610 can be, for example, a server computer, a controller, or any other similar computing component capable of processing data. Figure 6 In the example implementation, computing component 610 includes hardware processor 612 and machine-readable storage medium 614.

[0109] The hardware processor 612 and the machine-readable storage medium 614 may be the same as or similar to the hardware processor 412 and the machine-readable storage medium 414, respectively. Accordingly, the machine-readable storage medium 614 may be encoded with executable instructions (e.g., instructions 616 to 618).

[0110] Hardware processor 612 can execute instructions 616a to extract a data hierarchy representation for the distributed data processing pipeline from a central repository shared by multiple data processing sites of the distributed data processing pipeline. The distributed data processing pipeline may include multiple executions and multiple data artifacts, each execution having been performed in one of multiple processing stages, each of the multiple processing stages having been performed by one of the multiple data processing sites. Here, each of the multiple data artifacts may be at least one of: (1) an input to one or more of the multiple executions; and (2) an output from one or more of the multiple executions.

[0111] In various examples, hardware processor 612 may execute instruction 616b to extract from a central repository a merged metadata representation of metadata associated with multiple data artifacts and multiple executions (e.g., text documents of multiple data artifacts, descriptions of the contents within multiple data artifacts, data size and data distribution of multiple data artifacts, programming languages ​​of multiple executions, information about input fields used for multiple executions, algorithm categories used for multiple executions, stage or quality assurance (QA) status of multiple executions, names of developers / data scientists involved in multiple executions, etc.).

[0112] A data hierarchy representation may include multiple data artifacts and linear associations of multiple executions, wherein the linear association indicates that each data artifact is at least one of the following: (a) an input to a given execution; and (b) an output from a given execution.

[0113] Multiple data processing sites can be multiple data processing sites in a distributed data processing pipeline. Each data processing site can be, for example, a data center, an edge computing site, etc. The multiple data processing sites in the distributed data processing pipeline can be disconnected from each other, but can be connected to the central repository at least intermittently.

[0114] As described above, each of the multiple data processing stations can perform a processing stage of the distributed data processing pipeline. For example, the first data processing station can perform a first processing stage (e.g., data discovery / preparation stage), the second data processing station can perform a second processing stage (e.g., ML model preparation stage), the third data processing station can perform a third processing stage (e.g., ML model training stage), and so on.

[0115] At a given processing stage, a given data processing station can execute one or more executions of that processing stage. An execution (including multiple executions) can be a data processing step that transforms one or more data artifacts. In other words, an execution can receive one or more input data artifacts (i.e., data artifacts received as input to the execution) and transform them into one or more output data artifacts (i.e., data artifacts produced by the execution). Data artifacts (including multiple data artifacts) can be data-related inputs or outputs of a data processing pipeline (e.g., raw data, processed datasets, ML models, etc.). A data artifact can be the output of one execution and / or the input to one or more different executions. For example, in addition to being the output of the first execution of the first processing stage, the first data artifact can also be the input of the first execution of the second processing stage.

[0116] As mentioned above, the central repository can construct a data lineage representation based on specially tailored information received from multiple data processing sites in a distributed data processing pipeline—even when the distributed data processing pipeline is executed across multiple data processing sites.

[0117] As described above, each specially customized piece of information may include: (1) a hash content value that identifies a given data artifact; and (2) an indication that the given data artifact is an input or output of a given execution of a given data processing stage.

[0118] Here, the hash content value can be used as an identifier key, which allows the central repository to stitch together the extracted data lineage representations. In other words, the central repository can use the received hash content value to determine, for example, that the same data artifact (e.g., the first data artifact) is both the output of the first execution of the first processing stage and the input of the first execution of the second processing stage—even when the first and second processing stages are executed at different data processing sites (e.g., the first data processing site and the second data processing site, respectively).

[0119] As mentioned above, a hash content value can be considered a unique fingerprint of a data artifact. Typically, a hash content value will be a numeric and / or alphabetic string value that uniquely identifies the data artifact. Hash content values ​​can be generated using hash functions (e.g., cryptographic algorithms) that generate hash content values ​​based on the content of the data artifact. Two common hash functions / algorithms for generating hash content values ​​for data artifacts are MD5 and SHA1.

[0120] Accordingly, the first hash content value may be a first numeric and / or alphabetic string value that uniquely identifies the first data artifact across multiple data processing stations / stages in the data processing pipeline. Similarly, the second hash content value may be a second numeric and / or alphabetic string value that uniquely identifies the second data artifact across multiple data processing stations / stages in the distributed data processing pipeline.

[0121] In various examples, multiple data processing sites may utilize a common hash function / algorithm to generate hash content values. For instance, multiple data processing sites may use the MD5 algorithm to generate a first hash content value for a first data artifact. Accordingly, the first hash content value (identifying the first data artifact) received by the central repository from the first data processing site may include the same string of numbers / letters as the first hash content value (identifying the first data artifact) received by the central repository from the second data processing site. In contrast, the second hash content value (identifying the second data artifact) received by the central repository from the second data processing site may include a different string of numbers / letters than the first hash content value.

[0122] As described above, the specially tailored information received by the central repository may also include an indication that a given data artifact (identified by the hash content value of the given data artifact) is an input or output of a given execution of a given data processing stage. Here, examples of the currently disclosed techniques can utilize a combination of Figure 2 This instruction is given by describing an abstract, hierarchical structure of the data processing pipeline. For example, combining... Figure 2 The data processing pipeline abstraction hierarchy described in the presently disclosed technology may include: (1) a data processing pipeline level; (2) a processing stage level below the data processing pipeline level; (3) an execution level below the processing stage level; and (4) input / output event relationships at the execution level. Using this data processing pipeline abstraction hierarchy, examples can associate each data artifact of the data processing pipeline with: (1) at least one of an input event or an output event; (2) a given execution associated with the input / output event; (3) a given processing stage associated with the given execution; and (4) a given data processing pipeline associated with the given processing stage. As described above, based on these associations, examples of the presently disclosed technology can construct a data hierarchy representation for a distributed data processing pipeline.

[0123] As described above, in various examples, hardware processor 612 can execute instruction 616b to extract a merged metadata representation from a central repository, representing metadata associated with multiple data artifacts and multiple executions.

[0124] The extracted merged metadata representation may include a “global” view of the metadata of the distributed data processing pipeline. As will be described below, the hardware processor 612 may utilize the extracted merged metadata representation (along with the extracted data lineage representation) to clone / reproduce the distributed data processing pipeline for, for example, testing and debugging purposes.

[0125] Accordingly, the hardware processor 612 can execute instructions 620 to clone the distributed data processing pipeline based on the extracted data hierarchy representation and the extracted metadata representation. The cloned distributed data processing pipeline may include all the code, data, metadata, and dependencies of the distributed data processing pipeline.

[0126] As described above, examples of currently disclosed techniques can clone a distributed data processing pipeline for inspection / troubleshooting. In some cases, this inspection / troubleshooting may be a regulatory requirement to check all data artifacts of the data processing pipeline when certain events occur (poor model performance, model bias, etc.). Accordingly, it is highly advantageous to be able to clone the entire distributed data processing pipeline (including its code, data, metadata, and all associated dependencies) to locally reproduce the execution of the data processing pipeline for testing and debugging purposes. Using the cloned distributed data processing pipeline, examples can include performing one or more quality assurance checks on the distributed data processing pipeline to analyze the data trustworthiness of one or more data artifacts of the distributed data processing pipeline, etc.

[0127] Such as combination Figure 3 As described, in addition to extracting the data lineage representation and merging the metadata representation, the hardware processor 612 can also extract from a central repository: (1) multiple executions of code for the distributed data processing pipeline (using, for example, code pull requests); and (2) multiple data artifacts of the distributed data processing pipeline. Accordingly, the hardware processor 612 can essentially use the extracted data lineage representation and merged metadata representation as a roadmap for cloning / reproducing the distributed data processing pipeline from the extracted data artifacts and code.

[0128] Figure 7 An example data hierarchy representation 710 for an example data processing pipeline 700 is depicted according to an example of the currently disclosed technology. Specifically, the data hierarchy representation 710 may represent a simple directed acyclic graph (i.e., a directed graph without cycles) that represents the data hierarchy of the data processing pipeline 700.

[0129] exist Figure 7 In a specific example, the data processing pipeline 700 operates (i.e., consumes and / or produces) seven data artifacts (i.e., data artifacts 701 to 707). As described above, the data lineage representation 710 may include linear relationships between these seven data artifacts. For example, the data lineage representation 710 includes a linear relationship between data artifact 707 and its two direct ancestor data artifacts (data artifacts 705 and 706 (i.e., the two data artifacts used to produce data artifact 707)). Similarly, the data lineage representation 710 includes a linear relationship between data artifact 706 and its two direct ancestor data artifacts (i.e., data artifacts 703 and 704), and so on. Here, data artifacts 703 and 704 may also be considered (indirect) ancestors of data artifact 707 (correspondingly, data artifact 707 may be considered a descendant of data artifacts 701 to 706).

[0130] exist Figure 7 In a specific example, data lineage representation 710 does not contain a specific node for the execution that produces data artifacts 701 to 707 (however, in various examples, the arrows in data lineage representation 710 may represent such executions). (As will be combined...) Figure 8 The examples described can also construct a data hierarchy representation that includes specific nodes for execution.

[0131] Figure 8 An example data hierarchy representation 810 is depicted for an example data processing pipeline 800 according to an example of currently disclosed technology.

[0132] As described above, the data lineage representation 810 can represent the linear relationship between data artifacts (i.e., data artifacts 801 to 806) and executions (i.e., executions 821 to 824) of the data processing pipeline 800. For example, the data lineage representation 810 shows execution 821 for transforming data artifact 801 (e.g., a first raw data artifact) into data artifact 803 (e.g., a first processed dataset). Similarly, the data lineage representation 810 shows execution 822 for transforming data artifact 802 (e.g., a second raw data artifact) into data artifact 804 (e.g., a second processed dataset). The data lineage representation 810 also shows execution 824 for combining data artifacts 803 and 804 and then transforming them into data artifact 806 (e.g., a machine learning model). Accordingly, data artifacts 801 to 804 can be considered as ancestor data artifacts of data artifact 806. Similarly, executions 821, 822, and 824 can be considered as ancestor executions of data artifact 806.

[0133] Figure 9 A block diagram of an example computer system 900 in which various embodiments described herein may be implemented is depicted. The computer system 900 includes a bus 902 or other communication mechanism for conveying information, and one or more hardware processors 904 coupled to the bus 902 for processing information. For example, the hardware processors 904 may be one or more general-purpose microprocessors.

[0134] Computer system 900 also includes main memory 906, such as random access memory (RAM), cache, and / or other dynamic storage devices, coupled to bus 902 for storing information and instructions to be executed by processor 904. Main memory 906 can also be used to store temporary variables or other intermediate information during instruction execution by processor 904. When stored in storage media accessible to processor 904, these instructions make computer system 900 a dedicated machine customized to perform the operations specified in the instructions.

[0135] The computer system 900 also includes a read-only memory (ROM) 908 or other static storage device coupled to the bus 902 for storing static information and instructions for the processor 904. Storage devices 910, such as disks, optical discs, or USB thumb drives (flash drives), are provided and coupled to the bus 902 for storing information and instructions.

[0136] Computer system 900 may be coupled to display 912 (such as a liquid crystal display (LCD) (or touchscreen)) via bus 902 for displaying information to a computer user. Input device 914, including alphanumeric keys and other keys, is coupled to bus 902 for conveying information and command selections to processor 904. Another type of user input device is cursor control 916, such as a mouse, trackball, or cursor arrow keys, for conveying directional information and command selections to processor 904 and for controlling cursor movement on display 912. In some embodiments, the same directional information and command selections as cursor control may be implemented via receiving touches on the touchscreen without a cursor.

[0137] The computing system 900 may include a user interface module for implementing a GUI, which may be stored as executable software code executed by the computing device(s)(s). As an example, this module and other modules may include components (such as software components, object-oriented software components, class components, and task components), processes, functions, properties, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0138] Generally, the terms “component,” “engine,” “system,” “database,” and “data storage” used herein can refer to logic contained in hardware or firmware, or to a collection of software instructions (possibly with entry and exit points written in a programming language such as Java, C, or C++). Software components can be compiled and linked into executable programs, installed in dynamic link libraries, or written in interpreted programming languages ​​such as BASIC, Perl, or Python. It should be understood that software components can be invoked from other components or themselves, and / or can be invoked in response to detected events or interrupts. Software components configured to execute on a computing device may be provided on computer-readable media (such as optical discs, digital video discs, flash drives, disks, or any other tangible media) or as digital downloads (and may be initially stored in a compressed or installable format that requires installation, decompression, or decryption before execution). Such software code may be stored, in part or in whole, on a memory device executing the computing device for execution by the computing device. Software instructions may be embedded in firmware (such as EPROM). It should also be understood that hardware components may consist of connected logic units (such as gates and flip-flops) and / or may consist of programmable units (such as programmable gate arrays or processors).

[0139] Computer system 900 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic combined with the computer system to turn or program the computer system 900 into a special-purpose machine. According to one embodiment, computer system 900 performs the techniques described herein in response to processor(s) 904 executing one or more sequences of one or more instructions contained in main memory 906. These instructions may be read into main memory 906 from another storage medium (such as storage device 910). Execution of the instruction sequence contained in main memory 906 causes processor(s) 904 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0140] As used herein, the term "non-transitory media" and similar terms refer to any medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such non-transitory media can include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs or magnetic disks, such as storage device 910. Volatile media include dynamic memory, such as main memory 906. Common forms of non-transitory media include, for example, floppy disks, foldable disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips or cartridges, and their networking versions.

[0141] Non-transient media differ from transmission media, but can be used in conjunction with them. Transmission media participate in the transmission of information between non-transient media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including conductors containing bus 902. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0142] Computer system 900 also includes a communication interface 918 coupled to bus 902. Network interface 918 provides bidirectional data communication coupled to one or more network links connected to one or more local networks. For example, communication interface 918 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem providing data communication connectivity to a corresponding type of telephone line. As another example, network interface 918 may be a Local Area Network (LAN) card to provide data communication connectivity to a LAN-compatible network (or a WAN component communicating with a WAN). Wireless links may also be implemented. In any such implementation, network interface 918 transmits and receives electrical, electromagnetic, or optical indicators carrying streams of digital data representing various types of information.

[0143] Network links typically provide data communication to other data devices over one or more networks. For example, a network link can provide a connection to a host or a data device operated by an Internet Service Provider (ISP) via a local network. ISPs, in turn, provide data communication services through a global packet data communication network now commonly referred to as the "Internet." Both local area networks (LANs) and the Internet use electrical, electromagnetic, or optical indicators that carry streams of digital data. Indicators over various networks, as well as those on network links and via communication interface 918, are example forms of transmission media that carry digital data to and from computer system 900.

[0144] Computer system 900 can send messages and receive data (including program code) through (multiple) networks, network links, and communication interface 918. In the example of the Internet, the server can send requested code for the application through the Internet, ISP, local network, and communication interface 918.

[0145] The received code can be executed by processor 904 when it is received, and / or stored in storage device 910 or other non-volatile memory for later execution.

[0146] Each of the processes, methods, and algorithms described in the preceding sections may be embodied in code components executed by one or more computer systems or computer processors including computer hardware, and may be fully or partially automated. One or more computer systems or computer processors may also operate to support the performance of related operations in a “cloud computing” environment or as “Software as a Service” (SaaS). These processes and algorithms may be implemented, partially or wholly, in dedicated circuitry. The various features and processes described above may be used independently of each other or may be combined in various ways. Different combinations and sub-combinations are intended to fall within the scope of this disclosure, and certain method or process boxes may be omitted in some implementations. The methods and processes described herein are not limited to any particular order, and the boxes or states associated with them may be executed in other suitable orders, or may be executed in parallel, or may be executed in some other manner. Boxes or states may be added to or removed from the disclosed example embodiments. The performance of certain operations or processes may be distributed across computer systems or computer processors, residing not only in a single machine but also deployed across multiple machines.

[0147] As used herein, the circuit can be implemented using any form of hardware, software, or a combination thereof. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logic components, software routines, or other mechanisms can be used to compose the circuit. In implementation, the various circuits described herein can be implemented as discrete circuits, or the described functions and features can be shared partially or entirely among one or more circuits. Even though various features or functional elements can be described individually or required to be protected as discrete circuits, these features and functions can be shared among one or more common circuits, and such description should not require or imply the need for discrete circuits to implement such features or functions. When the circuit is implemented entirely or partially using software, such software can be implemented to operate using a computing or processing system (such as computer system 900) capable of performing the functions described therein.

[0148] As used herein, the term “or” can be interpreted as including or excluding. Furthermore, descriptions of resources, operations, or structures in the singular form should not be construed as excluding the plural form. Unless otherwise specifically stated or understood in the context in which they are used, conditional language (such as “can,” “may,” “perhaps,” or “may”) is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not.

[0149] Unless otherwise expressly stated, the terms and phrases used herein, and their variations thereof, should be interpreted as open-ended rather than restrictive. Adjectives such as “regular,” “traditional,” “normal,” “standard,” “known,” and similar terms should not be interpreted as limiting the described items to a given time period or to items available at a given time, but should be understood to include regular, traditional, normal, or standard techniques available or known now or at any time in the future. In some cases, the presence of expanded words and phrases (such as “one or more,” “at least,” “but not limited to,” or other similar phrases) should not be interpreted as implying a desire or need for a narrower scope where such expanded phrases might not be present.

Claims

1. A method for data processing, comprising: Receive a first hash content value identifying a data artifact from a first data processing station, and a first indication that the data artifact is the output of a first processing stage from a data processing pipeline, wherein the first processing stage is performed at the first data processing station; Receive from the second data processing station a second hash content value identifying the data artifact, and a second indication that the data artifact is an input to a second processing stage of the data processing pipeline, wherein the second processing stage is performed at the second data processing station; A first node generates the first hash content value from the first data processing station and a second node generates the second hash content value from the second data processing station, wherein the output of the second processing stage generates descendant nodes that depend on the first node. In response to determining that the first hash content value and the second hash content value are the same hash, the first node and the second node are determined to represent the same data artifact based on the same hash; Merge the first node and the second node to generate a merged node with the same hash; Construct a data lineage representation that includes the merged node as the predecessor of the descendant node, the data lineage representation including a linear relationship of the data artifacts at the first processing stage and the second processing stage, the linear relationship associating the merged node with the descendant node; The data hierarchy representation is exported to the first data processing station or the second data processing station; as well as This enables the first data processing station or the second data processing station to use the data hierarchy representation to locally reproduce the data processing pipeline for the data artifact.

2. The method according to claim 1, wherein, The data artifact is a first data artifact, and the method further includes: Receive from the second data processing station a third hash value identifying the second data artifact and a third indication that the second data artifact is the output of the second processing stage of the data processing pipeline; and A second data hierarchy representation is generated for the data processing pipeline, the second data hierarchy representation representing a linear relationship between the first data artifact and the second data artifact.

3. The method of claim 2, wherein the second data spectrum represents a linear relationship between the first data artifact and the second data artifact.

4. The method of claim 2, wherein the second data hierarchy representation for the data processing pipeline includes a linear relationship between the following: The first execution of the first processing stage; The first data artifact; The first execution of the second processing phase; and The second data artifact.

5. The method of claim 1, wherein the data artifact is a first data artifact, and the method further comprises: Receive metadata associated with the first data artifact and the first processing stage from the first data processing station; Receive metadata associated with the second data artifact and the second processing stage from the second data processing site, wherein the first data artifact is different from the second data artifact; as well as Based on the metadata associated with the first data artifact and the metadata associated with the second data artifact, a metadata association is created for the first data artifact and the second data artifact for the data processing pipeline.

6. The method of claim 1, wherein the data artifact comprises a first dataset.

7. The method of claim 6, wherein the data artifact is a first data artifact, and wherein the second data artifact comprises at least one of a second dataset and a machine learning model different from the first data artifact.

8. The method of claim 1, wherein the first processing stage of the data processing pipeline is a training stage for a machine learning model, and the second processing stage of the data processing pipeline is an inference stage for a machine learning model.

9. The method according to claim 1, further comprising: Use Git to index the data genealogy representation.

10. The method of claim 1, wherein the data processing pipeline for local reproduction of the data artifact comprises a complete data processing pipeline, the complete data processing pipeline including code, data, metadata, and associated dependencies.

11. The method according to claim 1, further comprising: The locally reproduced data processing pipeline is locally merged with the metadata for the data processing pipeline.

12. The method according to claim 1, further comprising: Receive metadata including a description of the physical location of the data artifact in the data artifact repository; as well as Create additional metadata associated with the data artifact and the code used to collect the data artifact.

13. The method of claim 1, further comprising: An abstract hierarchical structure of the data processing pipeline is generated based on the first instruction from the first data processing station and the second instruction from the second data processing station. as well as The data hierarchy representation is updated based on the abstract hierarchical structure.

14. A non-transitory computer-readable medium storing instructions, said instructions, when executed by one or more processing resources of a data processing pipeline, causing said one or more processing resources to: Receive a first hash content value identifying a data artifact from a first processing stage of the data processing pipeline, and a first indication that the data artifact is the output of the first processing stage of the data processing pipeline, wherein the first processing stage is executed at a first data processing station; Receive a second hash content value that identifies the data artifact, and a second indication that the data artifact is an input to a second processing stage of the data processing pipeline, wherein the second processing stage is performed at a second data processing station; A first node generates the first hash content value from the first data processing station and a second node generates the second hash content value from the second data processing station, wherein the output of the second processing stage generates descendant nodes that depend on the first node. In response to determining that the first hash content value and the second hash content value are the same hash, the first node and the second node are determined to represent the same data artifact based on the same hash; Merge the first node and the second node to generate a merged node with the same hash; Construct a data lineage representation that includes the merged node as the predecessor of the descendant node, the data lineage representation including a linear relationship of the data artifacts at the first processing stage and the second processing stage, the linear relationship associating the merged node with the descendant node; The data hierarchy representation is exported to the first data processing station or the second data processing station; as well as This enables the first data processing station or the second data processing station to use the data hierarchy representation to locally reproduce the data processing pipeline for the data artifact.

15. The non-transitory computer-readable medium for storing instructions according to claim 14, further comprising instructions that cause the one or more processing resources to: Receive metadata associated with the data artifact, and the first execution of the data processing phase.

16. The non-transitory computer-readable medium for storing instructions according to claim 14, wherein the data hierarchy representation for the data processing pipeline includes a linear correlation among the following: The data artifact; and Second data product.

17. The non-transitory computer-readable medium for storing instructions according to claim 14, wherein the data artifact comprises a first dataset.

18. The non-transitory computer-readable medium for storing instructions according to claim 17, wherein the data artifact is a first data artifact, and wherein the second data artifact includes at least one of a second dataset and a machine learning model different from the first data artifact.

19. The non-transitory computer-readable medium for storing instructions according to claim 14, wherein the first node is input to a first execution of the second processing stage associated with the second node.

20. The non-transitory computer-readable medium for storing instructions according to claim 14, wherein the second node is the output of a first execution of the first processing phase associated with the first node.

Images