Machine Learning Pipeline Skeleton Instantiation

Abstract

To instantiate a machine learning pipeline skeleton.SOLUTION: A method includes acquiring a ML pipeline skeleton that indicates a set of first functional blocks used for processing a new dataset of a new machine learning (ML) project. Additionally, for each first functional block, the method includes acquiring existing code snippets from existing ML pipelines. Each of the existing code snippets performs instantiation on a second functional block of the existing ML pipelines and is a potential instantiation of the respective first functional block. The method determines respective adaptability of each of the existing code snippets with respect to the new dataset and selects a particular existing code snippet for implementing the respective first functional block on the basis of the determined adaptability. Further, the method includes instantiation of the pipeline skeleton on the basis of the particular existing code snippet.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The embodiments discussed in this disclosure relate to instantiating a machine learning pipeline skeleton. [Background technology]

[0002] Machine learning (ML) typically uses ML models trained on training data to make predictions that automatically become more accurate with ongoing training. ML may be used in a wide range of applications, including, but not limited to, traffic forecasting, web search, online fraud detection, medical diagnosis, speech recognition, email filtering, image recognition, virtual personal assistants, and automatic translation.

[0003] The claimed subject matter in this disclosure is not limited to embodiments that solve any drawbacks or that operate only in environments such as those described above. Rather, this background is provided only to illustrate one example technology where some embodiments described in this disclosure may be practiced. Summary of the Invention

[0004] According to an aspect of an embodiment, operations may include obtaining an ML pipeline skeleton indicating a set of first function blocks to be used to process a new dataset for a new machine learning (ML) project, wherein each first function block of the set of first function blocks has a respective functionality. Additionally, for each respective first function block of the set of first function blocks, the operations may include obtaining a plurality of existing code snippets from one or more existing ML pipelines of one or more existing ML projects, wherein each existing code snippet instantiates a second function block of the existing ML pipeline and is a potential instantiation of the respective first function block. The operations also include determining a respective suitability of each of the existing code snippets with respect to the new dataset and selecting a particular existing code snippet to implement the respective first function block based on the determined suitability. Furthermore, the operations include instantiating a pipeline skeleton based on the particular existing code snippet selected for each of the first function blocks.

[0005] The object and advantages of the embodiments will be realized and attained at least by the elements, features, and combinations particularly pointed out in the claims.

[0006] Both the foregoing general description and the following detailed description are offered by way of example and are explanatory only and are not intended to be limitations on the invention as claimed. [Brief explanation of the drawings]

[0007] The exemplary embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0008] [Figure 1] FIG. 1 illustrates an exemplary environment involved in automatically generating a new machine learning project based on an existing machine learning project. [Figure 2] 1 illustrates an example set of operations that may be performed to modify a pipeline skeleton for a new machine learning project to generate a refined pipeline skeleton. [Figure 3A] 1 is a flowchart of an example method for determining function block dependencies. [Figure 3B] 1 shows an example table that may illustrate the use of three different function blocks for different columns of a data set. [Figure 4] 1 is a flowchart of an example method for determining relationship mappings between function blocks and dataset features. [Figure 5] 1 is a flowchart of an example method for determining block instantiations for a pipeline skeleton. [Figure 6] 1 is a flowchart of an exemplary method for refining a pipeline skeleton into a refined skeleton. [Figure 7] 1 illustrates an exemplary set of operations that may be performed to instantiate a pipeline skeleton into a concrete pipeline skeleton. [Figure 8] 1 is a flowchart of an example method for obtaining code snippets for instantiation of a pipeline skeleton. [Figure 9] 10 is a flowchart of another example method for obtaining code snippets for instantiation of a pipeline skeleton. [Figure 10] 1 is a flowchart of an example method for determining the suitability of a code snippet for implementation on a pipeline skeleton. [Figure 11] 1 is a flowchart of an example method for generating a set of candidate pipelines. [Figure 12] 1 shows a block diagram of an exemplary computing system. DETAILED DESCRIPTION OF THE INVENTION

[0009] Some embodiments described in this disclosure relate to methods and systems for automatically adapting existing machine learning (ML) projects to new ML projects.

[0010] As ML becomes increasingly common, there is often a shortage of available ML experts (e.g., skilled data scientists) to implement new ML projects. To address the ever-increasing challenge of implementing new ML projects in the face of a shortage of ML experts, various AutoML solutions (e.g., Auto-Sklrean, AutoPandas) have been proposed. However, current AutoML solutions only provide overly simplified, partial solutions that are insufficient for non-experts to fully implement new ML projects. Furthermore, open-source software (OSS) databases of existing ML projects (e.g., Kaggle, GitHub, etc.) have also been proposed as another solution to the challenge of implementing new ML projects by non-experts. However, it can be difficult or impossible for non-experts to find potentially useful existing ML projects in these databases. Even if non-experts succeed in finding potentially useful existing ML projects in these databases, it can be difficult or impossible for non-experts to modify potentially useful existing ML projects to meet the new requirements of the new ML project.

[0011] In this disclosure, the term "ML project" may refer to a project that includes a dataset, an ML task defined for the dataset, and an ML pipeline (e.g., script or program code) that is configured to implement a set of operations for training an ML model on the dataset for the ML task and using the ML model for new predictions. In this disclosure, the term "computational notebook" may refer to a computational structure (e.g., a Jupyter notebook) used to develop and / or express an ML pipeline, particularly during the development phase. While embodiments disclosed herein are illustrated with ML pipelines in the Python programming language and in computational notebooks structured as Jupyter notebooks, it is understood that other embodiments may include ML pipelines written in computational notebooks structured in different languages ​​and on other platforms.

[0012] According to one or more embodiments of the present disclosure, operations may be performed to automatically adapt an existing ML project to a new ML project. For example, in some embodiments, a computer system may organically support a data scientist's natural workflow by building in a "search and adapt" style workflow, where the data scientist first searches for existing ML projects that may serve as suitable starting points for building the new ML project, and then suitably adapts the existing ML projects to build ML pipelines for the new datasets and new ML tasks of the new ML project.

[0013] For example, in some embodiments, a computer system may automatically mine raw ML projects from an OSS database of existing ML projects and automatically curate the raw ML projects before storing them in a corpus of existing ML projects. In some embodiments, mining and curating existing ML projects from this large repository may result in a corpus of diverse, high-quality existing ML projects that can be used in search and adaptation workflows. This curation may also involve cleaning the ML pipelines of existing ML projects (e.g., using dynamic program slicing) and computing a set of features to capture the quality and diversity of each ML project and select an optimal number of existing ML projects that meet these goals.

[0014] Also, in some embodiments, this curation may require operations performed to automatically identify and index function blocks in the ML pipeline of an existing ML project. Unlike traditional software programs, the ML pipeline of an ML project generally follows a well-defined workflow based on the properties of the dataset and can be viewed as a series of function blocks. Therefore, some embodiments may involve techniques to automatically extract and label the function blocks in the ML pipeline and properly index them in a corpus so that they can be efficiently searched to synthesize new ML pipelines for new ML tasks. More specifically, this technique may extract the ML pipeline at an appropriate level and may use graph-based sequence mining algorithms to extract both custom and idiomatic function blocks. Finally, each function block may be semantically labeled.

[0015] In this disclosure, references to "function blocks" may refer to operations that may be performed by an ML pipeline, where a particular function block may correspond to a particular type of functionality. Semantic labeling may indicate the functionality of the corresponding function block. Furthermore, each function block may be instantiated in its corresponding ML pipeline with a particular code snippet that is configured to cause execution of the functionality of the corresponding function block. In many cases, the same function block across different ML pipelines may have different instantiations in each of the different ML pipelines.

[0016] In some embodiments, upon receiving a new dataset and a new ML task for a new ML project, such as from a non-expert data scientist, the computer system may automatically use a hierarchical approach to first synthesize a function block-level pipeline skeleton for the new ML project using the ML model. Additionally or alternatively, the computer system may obtain the pipeline skeleton through another mechanism (e.g., from user input). The pipeline skeleton may indicate which function blocks may be used for the new ML project.

[0017] In some instances, the obtained pipeline skeleton may include function blocks that may be technically different from one another but may be similar enough that they are considered redundant. Additionally or alternatively, as described above, the pipeline skeleton may indicate which function blocks may be used for a new ML project, but in some instances may not indicate the order of use of the function blocks. As discussed in detail below, in some embodiments, the computer system may be configured to refine the obtained pipeline skeleton by removing function blocks according to a redundancy analysis. Additionally or alternatively, the computer system may be configured to identify the order of the function blocks in the pipeline skeleton and refine the pipeline skeleton accordingly.

[0018] The pipeline skeleton may indicate which function blocks to use for a new ML project, but may not indicate the instantiation of the function blocks. As discussed in detail below, in some embodiments, the computer system may be configured to determine to which portions of a new dataset each of the function blocks of the pipeline skeleton applies. Additionally or alternatively, the computer system may be configured to identify existing code snippets of an existing ML project that can be used to instantiate the pipeline skeleton into a concrete pipeline skeleton for the new ML project.

[0019] Thus, in some embodiments, a non-expert data scientist may simply formulate a new dataset and a new ML task for a new ML project, and then a computer system may implement a tool-assisted interactive search and adaptation workflow to automatically generate a new ML pipeline for the ML project that can be immediately executed to perform the new ML task on the new dataset, without modification by the non-expert data scientist. Thus, some embodiments can empower novice data scientists to efficiently create new, high-quality end-to-end ML pipelines for new ML projects.

[0020] According to one or more embodiments of the present disclosure, the technical field of ML project development may be improved by configuring a computing system to automatically generate new ML projects based on existing ML projects, compared to tasking data scientists (e.g., often non-experts) with manually finding potentially useful existing ML projects and modifying them to the requirements of a new ML project. Such a configuration may enable the computing system to better search for relevant existing ML projects and use them to generate new ML projects by identifying and extracting function blocks and their corresponding instantiations from existing ML pipelines and automatically using and modifying them for use in the new ML pipeline.

[0021] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0022] 1 is a diagram depicting an example environment 100 related to automatically generating a new ML project based on an existing ML project, arranged in accordance with at least one embodiment described in this disclosure. The environment 100 may include a modification module 120 configured to modify a pipeline skeleton 102 to generate a concrete pipeline 122 that may be used for implementing the new ML project 115. In some embodiments, the modification module 120 may be configured to modify an existing ML project 110 that is based on the pipeline skeleton 102, which may be included in an ML project corpus 105.

[0023] The ML project corpus 105 may include any suitable repository of existing ML projects 110. Each existing ML project 110 may include electronic data including at least a dataset 109, an ML task defined for the dataset, and an ML pipeline 111 (e.g., script or program code) configured to implement a sequence of operations for training an ML model for the ML task and using the ML model for new predictions. In some embodiments, each existing ML project 110 may include a computational notebook, which may be a computational structure used to develop and / or express the corresponding ML pipeline, particularly during the development phase. One example of a computational notebook is a Jupyter notebook.

[0024] In some embodiments, the ML project corpus 105 may include one or more OSS ML project databases, which may be large repositories of existing ML projects. Examples of large repositories of existing ML projects 110 include, but are not limited to, Kaggle and GitHub.

[0025] Additionally or alternatively, in some embodiments, the existing ML projects 110 in the ML project corpus 105 may be curated and selected from one or more of the OSS ML project databases. The curation may enable the ML project corpus 105 to be a large-scale corpus of cleaned, high-quality indexed existing ML projects that can be used for automated "search and adapt" style workflows. The curation may be performed according to any suitable technique.

[0026] The pipeline skeleton 102 may include a set of function blocks that may represent functionality that may be used to accomplish a new ML task 108 on a new dataset 106 for a new ML project 115. In some embodiments, the function blocks may be unordered in the pipeline skeleton 102. Additionally or alternatively, the pipeline skeleton 102 may include one or more function blocks that may be relatively redundant compared to one or more other function blocks of the pipeline skeleton 102.

[0027] In some embodiments, the pipeline skeleton 102 may be generated using a pipeline skeleton model 104. The pipeline skeleton model 104 may include one or more ML models trained to learn a mapping between dataset metafeatures and function block semantic labels (e.g., based on existing ML project information included with existing ML projects 110 in an ML project corpus 105). For example, given the metafeatures of a new dataset 106, the pipeline skeleton model 104 may use the mapping to identify function blocks that correspond to the metafeatures of the new dataset 106 and synthesize the pipeline skeleton 102 accordingly. Additionally or alternatively, the pipeline skeleton 102 may be generated manually or by any other suitable technique.

[0028] In some embodiments, the pipeline skeleton model 104 may include a multivariate multi-class classifier that is trained before generating the pipeline skeleton. The multivariate multi-class classifier may be configured to map metafeatures of a new dataset to an unordered set of function blocks (indicated by corresponding semantic labels) that the pipeline skeleton should include. This training may include performing relational mapping as described below with respect to FIG. 2. For example, training may include extracting dataset features from an existing dataset of an existing ML project that correlate to specific semantic labels, identifying a set of all labels from the function blocks of the existing ML project, preparing training data including input vectors with the dataset features and binary output tuples indicating the presence or absence of each of the set of all labels, and training the pipeline skeleton model 104 to learn the mapping between the dataset features and corresponding labels in the set of all labels. In some embodiments, training the pipeline skeleton model 104 may enable the pipeline skeleton model 104 to use salient properties of the new dataset 106 and the new ML task 108 (metafeatures) to predict a set of function blocks as skeleton blocks of the pipeline skeleton.

[0029] Modification module 120 may include code and routines configured to enable a computing device to perform one or more operations. Additionally or alternatively, modification module 120 may be implemented using hardware, including a processor, a microprocessor (e.g., for performing or controlling the execution of one or more operations), a field programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In some other examples, modification module 120 may be implemented using a combination of hardware and software. In this disclosure, operations described as being performed by modification module 120 may include operations that modification module 120 may instruct a corresponding system to perform.

[0030] The modification module 120 may be configured to obtain the pipeline skeleton 102 and modify the pipeline skeleton 102 to generate the concrete pipeline 122. For example, in some embodiments, the modification module 120 may be configured to modify the pipeline skeleton 102 to refine the pipeline skeleton 102. For example, the modification module 120 may refine the pipeline skeleton 102 by determining which portions of the new dataset 106 apply different function blocks of the pipeline skeleton 102. Additionally or alternatively, the modification module 120 may be configured to identify an order of function blocks included in the pipeline skeleton 102 as part of the refinement. In these or other embodiments, the modification module 120 may be configured to refine the pipeline skeleton 102 by performing a redundancy analysis on the pipeline skeleton 102. Additionally or alternatively, the modification module 120 may remove one or more function blocks from the pipeline skeleton 102 based on the redundancy analysis. In some embodiments, the modification module 120 may be configured to modify the pipeline skeleton 102 to generate a refined pipeline skeleton, as described below with respect to FIGS.

[0031] In these or other embodiments, the modification module 120 may be configured to identify code snippets from the existing ML pipeline 111 that can be used to instantiate functional blocks of the pipeline skeleton 102 and, accordingly, concrete the pipeline skeleton 102 into a concrete pipeline 122. Additionally or alternatively, the modification module 120 may be configured to determine the suitability of the identified code snippets with respect to adapting the identified code snippets for use as part of the concrete pipeline 122. In these or other embodiments, the modification module 120 may be configured to generate one or more candidate pipelines, which may be different concrete pipelines of the pipeline skeleton 102. The candidate pipelines may each include different instantiations of the same functional blocks of the pipeline skeleton 102 using different identified code snippets. Additionally or alternatively, the modification module 120 may be configured to analyze the candidate pipelines to determine their performance. In these or other embodiments, the modification module 120 may select one of the candidate pipelines as the concrete pipeline 122 based on the performance determination. In some embodiments, the modification module 120 may be configured to identify, select, and implement code snippets for the generation and selection of concrete pipelines 122, as described below with respect to FIGS.

[0032] Thus, the modification module 120 may be configured to modify the pipeline skeleton 102 to generate a concrete pipeline 122 for use as part of a new ML project 115. This operation may improve the automation of the generation and implementation of new ML projects by the computer system and improve the ability to apply machine learning to a larger number of projects.

[0033] Modifications, additions, or omissions may be made to Figure 1 without departing from the scope of this disclosure. For example, environment 100 may include more or fewer elements than shown and described in this disclosure.

[0034] 2 illustrates an example set of operations 200 (“operation set 200”) that may be performed to modify a pipeline skeleton 202 of a new ML project 215 to generate an improved pipeline skeleton 212. Operation set 200 may be performed by any suitable system or device. For example, one or more operations of operation set 200 may be performed by or directed for execution by modification module 120 of FIG. 1. Additionally or alternatively, operation set 200 may be performed by a computing system such as computing system 1202 of FIG. 12 (e.g., as directed by modification module 120).

[0035] In general, operation set 200 may be configured to perform one or more operations on pipeline skeleton 202, new dataset 206, and one or more existing ML projects 210 to generate refined pipeline skeleton 212. In some embodiments, operation set 200 may include dependency analysis 250, existing ML mapping 252, instantiation determination 254, and pipeline refinement 256 to generate refined pipeline skeleton 212.

[0036] Pipeline skeleton 202 may be similar to pipeline skeleton 102 of FIG. 1 and may include a set of functional blocks (referred to as "skeleton blocks") associated with new ML project 215. New dataset 206 may also be part of new ML project 215 and may be similar to new dataset 106 of FIG. 1. Existing ML project 210 may be similar to existing ML project 110 of FIG. 1 and includes existing ML pipeline 211 and corresponding existing dataset 209, which may be similar to existing ML pipeline 111 and corresponding existing dataset 109 of FIG. 1, respectively.

[0037] Dependency analysis 250 may include operations that may be used to determine one or more function block dependencies 258. Function block dependencies 258 may indicate whether pairs of function blocks depend on each other based on whether each of the function blocks of each pair applies to the same portion of the same dataset. In some embodiments, dependency analysis 250 may determine function block dependencies 258 based on the use of one or more function blocks in existing ML pipelines 211. Function blocks in existing ML pipelines 211 may be referred to as "existing function blocks."

[0038] In some embodiments, the dependency analysis 250 usage determination may include determining to which portions of the existing data set 209 the existing function block applies. For example, the dependency analysis 250 may include determining to which columns of the existing data set 209 the existing function block applies.

[0039] In these or other embodiments, dependency analysis 250 may include determining which existing function blocks apply to the same portion of existing data set 209. In these or other embodiments, existing function blocks that apply to the same portion may be considered dependent with respect to one another in function block dependencies 258. Conversely, different existing function blocks not previously identified as applying to the same portion may be considered independent with respect to one another.

[0040] For example, a first function block and a second function block of the existing function blocks may both apply to a particular column of a particular existing data set 209. In some embodiments, function block dependencies 258 may therefore indicate that the first function block and the second function block are a dependent pair. As another example, a first function block and a third function block of the existing function blocks may never be identified as applying to the same column of any of the existing data sets 209. In some embodiments, function block dependencies 258 may therefore indicate the first function block and the third function block as an independent pair of function blocks.

[0041] In some embodiments, dependency analysis 250 may be performed with respect to multiple function block pairs of existing function blocks. In these or other embodiments, dependency analysis 250 may be performed for each possible pair of existing function blocks. Additionally or alternatively, function block dependencies 258 may include a representation of all different dependencies. It is understood that existing ML pipeline 211 may include multiple instances of the same existing function block, such that reference to “all pairs” of existing function blocks does not necessarily include all pairs of all instances of existing function blocks, but instead may refer to all pairs of all different existing function block types. In some embodiments, dependency analysis 250 may include one or more of the operations described below with respect to FIGS. 3A and 3B. As discussed further below, function block dependencies 258 may be used in pipeline refinement 256 in some embodiments.

[0042] Existing ML mapping 252 ("ML mapping 252") may include operations to generate relationship mapping 260 ("relationship mapping 260"). Relationship mapping 260 may indicate relationships between certain dataset features and function block usage for portions of the dataset having those features. In some embodiments, ML mapping 252 may determine mapping 260 based on existing function block usage of one or more existing ML pipelines 211.

[0043] In some embodiments, the usage determination of ML mapping 252 may include determining usage information indicating to which portions of existing dataset 209 an existing function block applies. For example, ML mapping 252 may include determining to which columns of existing dataset 209 an existing function block applies. In some embodiments, this information may be obtained from the same determination made with respect to dependency analysis 250.

[0044] In these or other embodiments, ML mapping 252 may include identifying one or more meta-features of different portions of the existing dataset 209 "dataset features." Dataset features may include, but are not limited to, number of rows, number of features, presence of numbers, presence of missing values, presence of numbers, presence of number categories, presence of string categories, presence of text, and type of target.

[0045] In some embodiments, ML mapping 252 may include determining a relationship between an existing function block and a dataset feature of a portion to which the existing function block is applied. The relationship may be determined based on usage information and may indicate to what extent a particular function block may be used with respect to a portion having certain dataset features. ML mapping 252 may generate relationship mapping 260 based on the determined relationship. For example, relationship mapping 260 may provide a mapping indicating which dataset features different function blocks correspond to, as determined from the relationship.

[0046] In some embodiments, ML mapping 252 may include one or more of the operations described below with respect to Figure 4. As discussed further below, relationship mapping 260 may be used in instantiation determination 254 in some embodiments.

[0047] As discussed above, in some instances, pipeline skeleton 202 may include a set of skeleton blocks that indicate operations to be performed on new ML project 215, but may not indicate which portions of new dataset 206 different skeleton blocks to apply to. Instantiation decision 254 may include an operation that determines which portions (e.g., columns) of new dataset 206 different skeleton blocks to apply to. In some embodiments, instantiation decision 254 may be determined by applying relational mapping 260 to skeleton blocks and new dataset 206 based on dataset characteristics of different portions of new dataset 206. In some embodiments, instantiation decision 254 may generate block instantiations 262, which may indicate which portions of new dataset 206 different skeleton blocks to apply based on the determination. In some embodiments, instantiation decision 254 may include one or more operations described below with respect to FIG. 5. As discussed further below, instantiation decision 254 may, in some embodiments, be used in pipeline refinement 256.

[0048] Pipeline refinement 256 may include operations related to refining pipeline skeleton 202. For example, pipeline refinement 256 may include removing one or more skeleton blocks from pipeline skeleton 202. In these or other embodiments, the removal of one or more of the skeleton blocks may be based on redundancy analysis, which may use function block dependencies 258. Additionally or alternatively, the removal of one or more of the skeleton blocks may be based on block instantiation 262. In some embodiments, the removal of one or more of the skeleton blocks may include one or more of the operations described below with respect to FIG. 6.

[0049] In these or other embodiments, pipeline refinement 256 may include determining an order of execution of skeleton blocks. This order may be determined by first inferring a partial order of block order from the ML pipelines 211 in the existing ML project 210. For example, in some embodiments, this partial order may be represented as a graph, where there is a node for each function block that appears in any of the ML pipelines 211. In these or other embodiments, the graph may include edges between nodes that indicate the order of execution of the corresponding function blocks. For example, an edge directed from a first node to a second node may be included in instances where a first function block corresponding to the first node occurs before a second function block corresponding to the second node in all ML pipelines 211 where the two blocks co-occur. This partial order may then be used to determine an order of execution of the skeleton blocks (e.g., to determine an overall order for the skeleton function blocks as any order consistent with the inferred partial order).

[0050] Additionally or alternatively, pipeline refinement 256 may include annotating pipeline skeleton 202 with one or more instantiations selected from block instantiations 262. For example, block instantiations 262 related to skeleton blocks remaining in refined pipeline skeleton 212 after pipeline refinement 256 may be indicated in refined pipeline skeleton 212.

[0051] Thus, operation set 200 may be configured to modify pipeline skeleton 102 to generate refined pipeline skeleton 212. Refined pipeline skeleton 212 may be more suitable for instantiation than pipeline skeleton 202 by indicating the execution order of skeleton blocks and / or removing skeleton blocks that are redundant or not required.

[0052] Modifications, additions, or omissions may be made to FIG. 2 without departing from the scope of the present disclosure. For example, operation set 200 may include more or fewer operations than those shown and described in this disclosure. Furthermore, the order of description of the operations in operation set 200 does not imply that the operations must be performed in the order described. Additionally, in some instances, the same operation may be described with respect to different portions of operation set 200 (e.g., dependency analysis 250 and usage determination for ML mapping 252), but in some instances may be performed only once and used for different portions of operation set 200.

[0053] 3A is a flowchart of an example method 300 for determining functional block dependencies according to at least one embodiment described in this disclosure. Method 300 may be performed by any suitable system, apparatus, or device. For example, modification module 120 of FIG. 1 or computing system 1202 of FIG. 12 (e.g., as directed by the modification module) may perform one or more of the operations associated with method 300. Further, as mentioned above, in some embodiments, one or more of the operations of method 300 may be performed in conjunction with dependency analysis 250 of FIG. 2. Although shown with discrete blocks, the steps and operations associated with one or more of the blocks of method 300 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

[0054] At block 302, the method 300 may include performing a dataset and abstract syntax tree (AST) analysis of one or more pipelines and one or more corresponding datasets of one or more ML projects. For example, the AST analysis may be performed on one or more existing pipelines and their respective existing datasets of one or more existing ML projects stored in a corpus. The AST analysis may include generating an AST for each of the existing pipelines based on the code of the existing pipelines. The dataset analysis may include identifying names of portions of the existing dataset (e.g., names of columns in the existing dataset). The AST may indicate which code elements may be related to certain function blocks. For example, the AST may indicate calls to application program interfaces (APIs) corresponding to the function blocks. Furthermore, the AST may indicate which portions of the dataset may be targets of certain operations and API calls.

[0055] Based on these indications, the dataset and AST analysis may include identifying the use of existing function blocks for different portions of the existing dataset. For example, a determination may be made as to which portions (e.g., which columns) of the existing dataset different existing function blocks may be applied. For example, FIG. 3B shows an exemplary table 350 that may indicate the use of three different function blocks for different columns of the dataset, as may be determined based on the dataset and AST analysis. In the example of FIG. 3B, table 350 indicates that the function block "drop" is applied to the column "Year" of the dataset. In the example of FIG. 3B, table 350 also indicates that the function block "LabelEncoder" is applied to the columns "Publisher" and "Genre" of the dataset. In the example of FIG. 3B, table 350 further indicates that the function block "fillna" is applied to the column "Publisher" of the dataset.

[0056] Returning to Figure 3A, function blocks that apply to the same portion of the data set may be identified at block 304. For example, with respect to the example of Figure 3B, the function blocks "LabelEncoder" and "fillna" may be identified as applying to the same column of "Publisher."

[0057] At block 306, dependent function blocks may be identified based on the identification performed at block 304. For example, function blocks that apply to the same part may be identified as dependent with respect to each other. For example, the function blocks "fillna" and "LabelEncoder" in FIG. 3B may be identified as a dependent pair based on both applying to the column "Publisher."

[0058] At block 308, independent function blocks may be identified based on the identification performed at block 304. For example, function blocks that are not identified as applying to the same portion may be identified as being independent of one another. For example, the function blocks "drop" and "LabelEncoder" in FIG. 3B may be identified as an independent pair based on the fact that they do not apply to any of the same columns.

[0059] In some embodiments, the dependency analysis of blocks 306 and 308 may be performed for multiple pairs of existing function blocks. In these or other embodiments, the dependency analysis of blocks 306 and 308 may be performed for each possible pair of existing function blocks. Furthermore, such an analysis may be performed for each ML pipeline 211 and dataset 209 in FIG. 2, as well as for results aggregated across all pipelines, where pairs of blocks are considered dependent if they are considered dependent in one or more pipelines in which they co-occur, and otherwise independent.

[0060] Modifications, additions, or omissions may be made to method 300 without departing from the scope of the present disclosure. For example, some of the operations of method 300 may be implemented in a different order. Additionally, or alternatively, two or more operations may be performed simultaneously. Furthermore, the outlined operations and actions are provided only as examples, and some of the operations and actions may be optional, combined into fewer operations and actions, or expanded into additional operations and actions without detracting from the disclosed embodiments.

[0061] 4 is a flowchart of an example method 400 for determining relationship mappings between function blocks and dataset features, according to at least one embodiment described in this disclosure. Method 400 may be performed by any suitable system, apparatus, or device. For example, correction module 120 of FIG. 1 or computing system 1202 of FIG. 12 (e.g., as directed by the correction module) may perform one or more of the operations associated with method 400. Although shown with discrete blocks, steps and operations associated with one or more of the blocks of method 400 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

[0062] In general, method 400 may be configured to determine the usage of existing function blocks of an existing ML pipeline with respect to features of a portion of an existing dataset to which the existing function blocks are applied. The usage may be used to determine a mapping between the function blocks and dataset features, which may indicate which dataset features the function blocks may correspond to. Additionally, as mentioned above, in some embodiments, one or more of the operations of method 400 may be performed with respect to ML mapping 252 of FIG. 2.

[0063] Method 400 may include block 402, where dataset features are obtained. For example, in some embodiments, one or more existing datasets of an existing ML project may be obtained. In these or other embodiments, one or more features of the existing dataset may be obtained. For example, one or more meta-features of the existing dataset may be obtained. In these or other embodiments, different portions (e.g., different columns) of the existing dataset may have different features. In these or other embodiments, identifying the dataset features may also include identifying which portions have which features. In these or other embodiments, identifying the different dataset features may be based on semantic labels that may be applied to different portions having different dataset features. The semantic labels may indicate the corresponding dataset features of the corresponding portions.

[0064] In some embodiments, method 400 may include block 404, where function block usage may be determined with respect to the dataset feature. For example, in some embodiments, determining usage may include determining the number of times each existing function block is used with respect to different portions having each dataset feature (also referred to as "function block occurrences"). For example, the number of times a particular function block is used with respect to portions having a given dataset feature may be determined.

[0065] As another example, determining usage may include determining a frequency of use of the function block with respect to portions having respective dataset characteristics. For example, the number of times a particular function block is used with respect to portions having a given dataset characteristic may be compared to the total number of dataset portions having the given dataset characteristic to determine a frequency of use (e.g., a usage rate) of the particular function block with respect to dataset portions having the given dataset characteristic.

[0066] In some embodiments, determining the use may include determining one or more conditional probabilities based on the use. The conditional probabilities may indicate the likelihood that the corresponding function block may be applied to dataset portions having certain dataset characteristics. For example, a first conditional probability may be determined for a particular function block with respect to the first dataset characteristic. The first conditional probability may be determined based on the use of the particular function block with dataset portions having the first dataset characteristic and may indicate the degree to which the particular function block may be used in dataset portions having the first dataset characteristic. A second conditional probability may similarly be determined for the particular function block with respect to the second dataset characteristic. In some embodiments, a different conditional probability may be determined for each function block and each dataset characteristic.

[0067] In some embodiments, determining the usage information may be based on AST and dataset analysis, such as described above with respect to Figure 3A. Additionally or alternatively, identified dataset features may be used to determine features of the portion to which the function block is applied.

[0068] At block 406, a mapping between dataset features and function blocks may be determined based on the determined usage. For example, in response to one or more usage factors for a certain function block and a given dataset feature satisfying a threshold, the certain function block and the given dataset feature may be mapped to one another so as to correspond to one another.

[0069] For example, a first function block may be determined to be used a first number of times with respect to a dataset portion having a first dataset feature. For example, a first function block may be determined to be used a second number of times with respect to a dataset portion having a second dataset feature. In these or other embodiments, the first number may satisfy an occurrence threshold, but the second number may not satisfy the occurrence threshold. In these or other embodiments, the first function block may be mapped to the first dataset feature, but not to the second dataset feature.

[0070] As another example, a second function block may be determined as having a first conditional probability with respect to a first dataset feature and a second conditional probability with respect to a second dataset feature. In some embodiments, the second conditional probability may satisfy a probability threshold, but the first conditional probability may not. In these or other embodiments, the second function block may therefore be mapped to a second dataset feature, but not to a first dataset feature.

[0071] In these or other embodiments, the mapping may indicate the determined correspondence without regard to a threshold. For example, in some embodiments, the mapping may indicate the state probability, occurrence, and / or frequency of use of each of the function blocks with respect to each of the different features.

[0072] Modifications, additions, or omissions may be made to method 400 without departing from the scope of the present disclosure. For example, some of the operations of method 400 may be implemented in a different order. Additionally, or alternatively, two or more operations may be performed simultaneously. Furthermore, the outlined operations and actions are provided only as examples, and some of the operations and actions may be optional, combined into fewer operations and actions, or expanded into additional operations and actions without detracting from the disclosed embodiments.

[0073] FIG. 5 is a flowchart of an example method 500 for determining block instantiations for a pipeline skeleton, according to at least one embodiment described in this disclosure. Method 500 may be performed by any suitable system, apparatus, or device. For example, modification module 120 of FIG. 1 or computing system 1200 of FIG. 12 (e.g., as directed by the modification module) may perform one or more of the operations associated with method 500. Although shown in discrete blocks, steps and operations associated with one or more of the blocks of method 500 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation. Additionally, as mentioned above, in some embodiments, one or more of the operations of method 500 may be performed with respect to instantiation determination 254 of FIG. 2.

[0074] The method 500 may include block 502, where a dataset portion of the new dataset may be selected. For example, a column of the new dataset may be selected. The new dataset may be the dataset from which the pipeline skeleton was generated.

[0075] At block 504, one or more dataset features of the selected portion may be obtained. For example, one or more meta-features of the selected portion may be determined.

[0076] At block 506, functional blocks of the pipeline skeleton (“skeleton blocks”) that correspond to the dataset features of the selected portion may be identified. In some embodiments, the corresponding skeleton blocks may be identified based on a relationship mapping, such as relationship mapping 260 of FIG. 2 or the relationship mapping described with respect to FIG. 4. For example, the selected portion may have a first dataset feature that is indicated as corresponding to a first function block in the relationship mapping. Thus, a skeleton block of the pipeline skeleton that is the same as (e.g., has the same functionality as) the first function block may be identified as corresponding to the selected portion of the new dataset.

[0077] As another example, the relationship mapping may indicate the frequency of use, conditional probability, and / or occurrence of different function blocks with respect to different dataset features. In these or other embodiments, the correspondence may be based on dataset features of the selected portion that correspond to the function blocks according to a certain threshold. For example, the relationship mapping may indicate that a second function block may have a conditional probability with respect to a second dataset feature of the selected portion. In these or other embodiments, a skeleton block corresponding to the second function block may be mapped to the selected portion accordingly. In contrast, a skeleton block corresponding to a third function block that has a conditional probability with respect to a second dataset feature that does not meet the probability threshold may not be mapped to the selected portion.

[0078] At block 508, one or more block instantiations may be determined for the selected portion. As described above, the block instantiations may indicate which skeleton blocks to apply to the selected portion. In some embodiments, the block instantiations may be determined based on the correspondence determined at block 506. For example, the correspondence determined at block 506 may indicate that a first skeleton block and a second skeleton block correspond to the selected portion. Accordingly, the first block instantiation may be determined to indicate that the first skeleton block should be applied to the selected portion. Additionally, the second block instantiation may also be determined to indicate that the second skeleton block should be applied to the selected portion.

[0079] In some embodiments, method 500 may be performed for multiple portions of the new dataset. In these or other embodiments, method 500 may be performed for every different portion (e.g., each column) of the new dataset. As such, in some embodiments, every different portion of the new dataset may be mapped to one or more skeleton blocks through the generation of block instantiations.

[0080] Modifications, additions, or omissions may be made to method 500 without departing from the scope of the present disclosure. For example, some of the operations of method 500 may be implemented in a different order. Additionally, or alternatively, two or more operations may be performed simultaneously. Furthermore, the outlined operations and actions are provided only as examples, and some of the operations and actions may be optional, combined into fewer operations and actions, or expanded into additional operations and actions without detracting from the disclosed embodiments.

[0081] 6 is a flowchart of an example method 600 for refining a pipeline skeleton into a refined skeleton, according to at least one embodiment described in this disclosure. Method 600 may be performed by any suitable system, apparatus, or device. For example, modification module 120 of FIG. 1 or computing system 1202 of FIG. 12 (e.g., as directed by the modification module) may perform one or more of the operations associated with method 600. Although shown in discrete blocks, steps and operations associated with one or more of the blocks of method 600 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

[0082] In general, method 600 may be configured to remove one or more skeleton blocks from the pipeline skeleton. Additionally or alternatively, method 600 may be configured to determine an execution order of the skeleton blocks. In these or other embodiments, method 600 may include annotating the pipeline skeleton with one or more instantiations selected from the block instantiations determined in method 500. Further, as mentioned above, in some embodiments, one or more of the operations of method 600 may be performed with respect to pipeline refinement 256 of FIG. 2.

[0083] Method 600 may include block 602, in which functional blocks of a pipeline skeleton ("skeleton blocks") are identified. At block 604, block instantiations, such as those determined using method 500, may be obtained.

[0084] At block 606, one or more skeleton blocks may be removed from the pipeline skeleton according to the block instantiations. For example, in some embodiments, skeleton blocks that are not included in any of the block instantiations may be removed. In these or other embodiments, all skeleton blocks that are not included in any block instantiations may be removed.

[0085] At block 608, function block dependencies may be obtained. For example, function block dependencies determined based on method 300 may be obtained. Additionally or alternatively, at block 608, usage information related to existing function blocks of one or more existing ML pipelines of one or more existing ML projects may be obtained. For example, the usage information may be the same or similar to that determined with respect to block 404 of method 400 of FIG. 4. For example, the usage information may include occurrences of function blocks with respect to each dataset feature, usage frequencies with respect to each dataset feature, and / or conditional probabilities with respect to each dataset feature.

[0086] One or more skeleton blocks may be removed at block 610. Removal may occur such that one or more function blocks representing duplicate functions applied to the same portion of the new data set may be removed.

[0087] In some embodiments, removal may be based on block instantiation, dependency, and usage information. For example, using function block dependency information, the dependencies of skeleton blocks may be determined by matching the skeleton blocks to function blocks indicated in the dependency information (e.g., based on the same functionality, the same name, etc.). After matching, the dependencies indicated in the dependency information may be applied to the skeleton blocks according to the dependencies of the function blocks identified as matching the skeleton blocks.

[0088] Additionally or alternatively, skeleton blocks that map to the same portion of the new dataset may be identified from the block instantiation. In these or other embodiments, pairs of skeleton blocks that map to the same portion ("mapped pairs") may be identified as being independent or dependent with respect to one another using dependency information determined for the skeleton blocks. In response to a mapped pair of skeleton blocks being independent with respect to one another, one of the skeleton blocks of the mapped pair may be removed.

[0089] In some embodiments, removal may be based on usage information. For example, skeleton blocks of mapped pairs with lower conditional probabilities may be removed. As another example, skeleton blocks of mapped pairs with lower occurrences or lower usage frequencies may be removed.

[0090] Modifications, additions, or omissions may be made to method 600 without departing from the scope of the present disclosure. For example, some of the operations of method 600 may be implemented in a different order. Additionally, or alternatively, two or more operations may be performed simultaneously. Furthermore, the outlined operations and actions are provided only as examples, and some of the operations and actions may be optional, combined into fewer operations and actions, or expanded into additional operations and actions without detracting from the disclosed embodiments.

[0091] 7 illustrates an example set of operations 700 (“operation set 700”) that may be performed to instantiate a pipeline skeleton 702 in accordance with one or more embodiments of the present disclosure. Operation set 700 may be performed by any suitable system or device. For example, one or more operations of operation set 700 may be performed by or directed for execution by modification module 120 of FIG. 1. Additionally or alternatively, operation set 700 may be performed by a computing system such as computing system 1202 of FIG. 12 (e.g., as directed by modification module 120).

[0092] The set of operations 700 may include one or more operations performed on a pipeline skeleton 702, a pipeline skeleton model 704, one or more existing ML projects 710, and / or a new dataset 706 to instantiate the pipeline skeleton 702 into a concrete pipeline 732. In some embodiments, the set of operations 700 may also include code snippet identification 720, suitability analysis 722, candidate pipeline generation 724, and pipeline analysis 726 to instantiate the pipeline skeleton 702 into a concrete pipeline 732.

[0093] The pipeline skeleton 702 may include a set of functional blocks (referred to as "skeleton blocks") related to the new ML project. In some embodiments, the pipeline skeleton 702 may be similar to the pipeline skeleton 102 of FIG. 1 or the pipeline skeleton 202 of FIG. 2. Additionally or alternatively, the pipeline skeleton 702 may be similar to the refined pipeline skeleton 212 of FIG. 2. In these or other embodiments, the pipeline skeleton 702 may include one or more block instantiations 718. The block instantiations 718 may be similar to the block instantiations 262 of FIG. 2 in that the block instantiations 718 may indicate which skeleton blocks of the pipeline skeleton 702 to apply to which portions (e.g., which columns) of the new dataset 706.

[0094] New dataset 706 may also be part of a new ML project and may be similar to new dataset 106 of FIG. 1. Existing ML project 710 may be similar to existing ML project 110 of FIG. 1 and includes an existing ML pipeline 711 and a corresponding existing dataset 709, which may be similar to existing ML pipeline 111 and corresponding existing dataset 109 of FIG. 1, respectively. Pipeline skeleton model 704 may, in some embodiments, be configured to generate pipeline skeleton 702. Pipeline skeleton model 704 may be similar to pipeline skeleton model 104 of FIG. 1. Concrete pipeline 732 may be similar to concrete pipeline 122 of FIG. 1.

[0095] Code snippet identification may include operations that may be used to identify one or more code snippets 728. The code snippets 728 may include one or more existing code snippets from an existing ML pipeline 711. Existing code snippets that may be identified as code snippets 728 may be identified as potentially being used to instantiate respective skeleton blocks of the pipeline skeleton 702.

[0096] In some embodiments, the code snippet 728 may be identified based on a similarity between the new dataset 706 and the existing dataset 709 to the code snippet 728. The similarity may be determined based on a similarity between one meta-feature of the existing dataset 709 and one meta-feature of the new dataset 706. In some embodiments, the identification of the code snippet 728 based on the determined similarity may include one or more of the operations described below with respect to FIG.

[0097] In these or other embodiments, the code snippets 728 may be identified based on an analysis of the generation of the pipeline skeleton 702 via the pipeline skeleton model 704. For example, which training data of the pipeline skeleton model 704 may be used to determine which function blocks to include in the pipeline skeleton 702. In these or other embodiments, the identified training data may be obtained from an existing ML project 710. For example, the identified training data may illustrate a correlation between certain features of existing datasets and the presence of certain existing function blocks in the pipeline, which may cause the pipeline skeleton model 704 to include certain function blocks in the pipeline skeleton 702 predicted for the new dataset 706. Thus, in these or other embodiments, the identified training data may in some instances represent the most appropriate instantiation of a function block in the context of the new dataset 706. Additionally or alternatively, the identified training data may include or be used to identify code snippets that instantiate existing function blocks in the identified training data. Code snippets associated with the identified training data may be useful for instantiating the pipeline skeleton 702. In some embodiments, identifying code snippets 728 based on training data used to generate pipeline skeleton 702 may include one or more of the acts described below with respect to FIG.

[0098] Suitability analysis 722 may include operations related to determining how suitable code snippets 728 are for implementation with respect to pipeline skeleton 702. Suitability analysis 722 may include determining element suitability of code snippets 728 based on program elements of code snippets 728. Additionally or alternatively, suitability analysis 722 may include determining dataflow suitability of code snippets 728 based on dataflow of code snippets 728. In these or other embodiments, suitability analysis 722 may include determining cardinality suitability of code snippets 728 based on cardinality suitability of each code snippet 728. In some embodiments, suitability analysis 722 may include determining an overall suitability of each of each code snippet 728 based on a combination of two or more of element suitability, dataflow suitability, or cardinality suitability of each code snippet 728.

[0099] In some embodiments, the suitability analysis may output expanded code snippet information 730 ("expanded information 730") for the code snippets 728. The expanded information 730 may include a respective suitability determination for each code snippet 728. In these or other embodiments, the expanded information 730 may include the code snippets 728. In some embodiments, the suitability analysis 722 may include one or more of the operations described below with respect to FIG. 10.

[0100] Additionally or alternatively, the extended information 730 may include a ranking of the code snippets 728 with respect to one another. For example, different code snippets may be potential candidates for the same skeleton block instantiation. In some embodiments, the different code snippets may be ranked with respect to one another with respect to the same skeleton block instantiation. In some embodiments, the different code snippets may be ranked as described below with respect to FIGS. 8, 9, and / or 10.

[0101] Candidate pipeline generation 724 may include operations that may generate one or more candidate pipelines 734 based on augmentation information 730. Candidate pipelines 734 may each be a materialized instantiation of pipeline skeleton 702 using a set of selected code snippets 728. Code snippets 728 may, in some embodiments, be selected based on adaptability information included in augmentation information 730. In these or other embodiments, code snippets 728 may be selected based on rankings that may be included in augmentation information 730. In some embodiments, candidate pipeline generation 724 may include one or more operations described below with respect to FIG. 11 .

[0102] Pipeline analysis 726 may analyze candidate pipelines 734 and select one of the candidate pipelines 734 for use as the concrete pipeline 732. For example, in some embodiments, each of the candidate pipelines 734 may be applied to new dataset 706 to determine a performance level for each candidate pipeline. In these or other embodiments, a particular candidate pipeline 734 may be selected as the concrete pipeline 732 based on the determined performance level. In some embodiments, pipeline analysis 726 may be performed using any suitable technique. Additionally or alternatively, in some embodiments, new dataset 706 may be relatively large, and data sampling (e.g., layered data sampling) may be used to prune new dataset 706 to reduce the amount of data to analyze candidate pipelines 734.

[0103] Modifications, additions, or omissions may be made to FIG. 7 without departing from the scope of the present disclosure. For example, set of operations 700 may include more or fewer operations than those shown and described in this disclosure. Furthermore, the order of description of the operations in set of operations 700 does not imply that the operations must be performed in the order described. Additionally, in some instances, the same operation may be described with respect to different portions of set of operations 700, but in some instances may be performed only once and used for different portions of set of operations 200.

[0104] FIG. 8 is a flowchart of an example method 800 for obtaining code snippets for pipeline skeleton instantiation, according to at least one embodiment described in this disclosure. Method 800 may be performed by any suitable system, apparatus, or device. For example, modification module 120 of FIG. 1 or computing system 1202 of FIG. 12 (e.g., as directed by the modification module) may perform one or more of the operations associated with method 800. Although shown in discrete blocks, steps and operations associated with one or more of the blocks of method 800 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation. Additionally, as noted above, in some embodiments, one or more of the operations of method 800 may be performed with respect to code snippet identification 720 of FIG. 7.

[0105] The method 800 may include block 802, where information related to a new ML project may be obtained. The new ML information may include a new dataset for the new ML project and / or a pipeline skeleton for the new ML project. For example, the new ML information may include the pipeline skeleton 702 of FIG. 7 and / or the new dataset 706 of FIG. 7.

[0106] In these or other embodiments, information related to one or more existing ML projects may be obtained at block 802. The existing ML information may include one or more existing ML projects, such as existing ML project 710, and corresponding information. For example, the existing ML information may include an existing pipeline and / or an existing dataset for the existing ML project. Additionally or alternatively, the existing ML information may include code for an existing ML pipeline. In these or other embodiments, the existing ML information may include usage information that indicates to which portions of an existing dataset different existing function blocks and corresponding existing code snippets may be applied. In these or other embodiments, the usage information may be obtained based on the existing ML information as described above with respect to block 302 of method 300 of FIG. 3A.

[0107] At block 804, one or more existing function blocks of an existing ML pipeline may be identified. In some embodiments, the existing function blocks may be identified based on the function blocks of the pipeline skeleton of the new ML project (“skeleton blocks”). For example, in some embodiments, the existing function blocks may be identified based on being identical to the skeleton blocks (e.g., based on having the same name and / or functionality).

[0108] At block 806, one or more existing datasets for the existing ML project may be identified based on the identified existing function blocks. For example, existing datasets to which the existing function blocks apply may be identified. In some embodiments, the existing datasets may be identified using usage information included in the existing ML project information.

[0109] At block 808, a respective similarity may be determined between the new dataset and each of the identified existing datasets. In some embodiments, the similarity may be determined based on a comparison between one or more obtained metafeatures of the existing datasets and the new dataset. For example, the metafeatures may include the number of rows of the dataset, the number of columns of the dataset, and / or the column types of the columns of the dataset. In these or other embodiments, each similarity score may be determined based on how similar different metafeatures of the new dataset are to corresponding metafeatures of each existing dataset. For example, the similarity scores may be determined using any suitable distance metric. In these or other embodiments, each of the existing datasets may be given a similarity ranking with respect to other existing datasets with respect to their similarity to the new dataset. For example, the existing datasets may be ranked such that the most similar existing datasets, as indicated by the similarity analysis, are ranked highest. In some embodiments, the code snippets themselves may be ranked according to the rankings of their corresponding existing datasets.

[0110] At block 810, one or more existing code snippets may be identified and selected based on a determination of the similarity of the existing datasets. For example, existing code snippets derived from the pipeline for the highest-ranked existing datasets may be identified. In these or other embodiments, existing code snippets applied to existing datasets that meet a similarity threshold with respect to the new dataset may be identified. In these or other embodiments, the existing code snippets may instantiate existing function blocks identified at block 804, which function blocks are also associated with the highest-ranked existing datasets. In these or other embodiments, the identified code snippets may be identified based on code snippet rankings, which may correspond to existing dataset rankings. In some embodiments, the code snippet rankings may be included in expanded code snippet information, such as expanded code snippet information 730 of FIG. 7.

[0111] The identified existing code snippet may, in some embodiments, be identified as a potential instantiation of a skeleton block of a pipeline skeleton. For example, the identified existing code snippet may, in some embodiments, be code snippet 728 of FIG. 7.

[0112] Modifications, additions, or omissions may be made to method 800 without departing from the scope of the present disclosure. For example, some of the operations of method 800 may be implemented in a different order. Additionally, or alternatively, two or more operations may be performed simultaneously. Furthermore, the outlined operations and actions are provided only as examples, and some of the operations and actions may be optional, combined into fewer operations and actions, or expanded into additional operations and actions without detracting from the disclosed embodiments.

[0113] FIG. 9 is a flowchart of an example method 900 for obtaining code snippets for pipeline skeleton instantiation, according to at least one embodiment described in this disclosure. Method 900 may be performed by any suitable system, apparatus, or device. For example, modification module 120 of FIG. 1 or computing system 1202 of FIG. 12 (e.g., as directed by the modification module) may perform one or more of the operations associated with method 900. Although shown in discrete blocks, steps and operations associated with one or more of the blocks of method 900 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation. Additionally, as mentioned above, in some embodiments, one or more of the operations of method 900 may be performed with respect to code snippet identification 720 of FIG. 7.

[0114] Method 900 may include block 902, where a pipeline skeleton model is obtained. As described above, the pipeline skeleton model may be an ML model used to generate a pipeline skeleton from which code snippets can be identified. In some embodiments, the pipeline skeleton model may have been previously generated. In these or other embodiments, the pipeline skeleton model may be generated at block 902 as part of obtaining the pipeline skeleton model. For example, the pipeline skeleton model may be generated by training a multivariate multiclass classifier as described above. In these or other embodiments, the pipeline skeleton may be generated at block 902 using the pipeline skeleton model.

[0115] At block 904, training data to be used by the pipeline skeleton model to generate the pipeline skeleton may be identified. In some embodiments, identifying the training data may include identifying metafeatures of the new ML project (e.g., metafeatures of the new dataset and / or metafeatures of the corresponding new task) to be used by the pipeline skeleton model to predict function blocks to include in the pipeline skeleton. In these or other embodiments, metafeatures may be identified on a function block by function block basis of skeleton block, such that metafeatures to be used to determine each individual skeleton block of the pipeline skeleton may be identified.

[0116] In some embodiments, identifying metafeatures to be used in making predictions may be based on one or more “white-box” techniques where the structure of the pipeline skeleton model is known. For example, in instances where the pipeline skeleton model is based on a decision tree family, the paths used to reach a particular decision for a particular function block for inclusion in the pipeline skeleton may be identified. In these or other embodiments, for each of the respective function blocks of the pipeline skeleton, each path used to reach each of the respective decisions may be identified. As another example, a “white-box” technique may include discovering the dominant terms used in a linear regression model of the pipeline skeleton model. The above are merely exemplary “white-box” techniques; any other suitable white-box technique may be used.

[0117] Additionally or alternatively, the identification of metafeatures to be used in making predictions may be based on one or more "black box" techniques in which the specific structure of the pipeline skeleton model is not known or required. Such "model-agnostic" techniques may include any suitable techniques, including the Local Interpretable Model-agnostic Explanation (LIME) technique or the Shapely Additive exPlanations (SHAP) technique.

[0118] In some embodiments, the identified metafeatures may be used to generate one or more dataset vectors for a new dataset ("new dataset vectors") for the pipeline skeleton model. These new dataset vectors may be vectors of values ​​of the identified metafeatures for the new dataset. In some embodiments, a new dataset vector may be identified for each skeleton block.

[0119] In these or other embodiments, the new dataset vector may be used to identify existing ML projects to be used as training data that can influence the generation of the pipeline skeleton. For example, a dataset vector for an existing ML project ("existing dataset vector") may be constructed from an existing ML dataset, similar to the construction of a new dataset vector from a new dataset, by computing previously identified metafeatures in the context of the existing dataset. In these or other embodiments, the closest existing dataset vector to the new dataset vector may be identified. For example, existing dataset vectors within a threshold distance of the new dataset vector may be identified.

[0120] In some embodiments, the determination regarding the closest existing dataset vector may be made with respect to different new dataset vectors corresponding to different skeleton blocks of the pipeline skeleton. In some embodiments, the closest existing dataset vector may be identified by performing any suitable closest point analysis between each respective new dataset vector and each respective existing dataset vector. In some embodiments, training data associated with an existing ML project that corresponds to the closest existing dataset vector may be identified as training data that influenced the generation of the pipeline skeleton.

[0121] At block 906, one or more code snippets may be identified from the training data identified at block 904. For example, in some embodiments, existing ML pipelines of existing ML projects associated with (e.g., included in or indicated by) the training data may be identified. Additionally, code snippets that instantiate existing functional blocks of the existing ML pipelines may also be identified.

[0122] In some embodiments, different code snippets from different existing ML pipelines associated with different ML projects that may contribute to the selection of the same functional block may be ranked relative to each other. In some embodiments, the ranking may be based on the distance of the new dataset vector relative to the existing dataset vector of the existing ML project corresponding to the code snippet. For example, a first code snippet may correspond to a first ML project having a first existing dataset vector that is a first distance from the new dataset vector. Additionally, a second code snippet may correspond to a second ML project having a second existing dataset vector that is a second distance from the new dataset vector, where the second distance is greater than the first distance. The first code snippet in this example may be ranked higher than the second code snippet.

[0123] At block 908, one or more of the identified code snippets may be selected. In some embodiments, the selected code snippets may be selected based on those code snippets that instantiate existing functional blocks indicated by the training data that correspond to skeleton blocks of the pipeline skeleton. Thus, the selected code snippets may be selected based on the training data used to determine the skeleton blocks of the pipeline skeleton. In some embodiments, the selected code snippets may have a certain ranking. In some embodiments, the ranking of the code snippets may be included in expanded code snippet information, such as expanded code snippet information 730 of FIG. 7.

[0124] Modifications, additions, or omissions may be made to method 900 without departing from the scope of the present disclosure. For example, some of the operations of method 900 may be implemented in a different order. Additionally, or alternatively, two or more operations may be performed simultaneously. Furthermore, the outlined operations and actions are provided only as examples, and some of the operations and actions may be optional, combined into fewer operations and actions, or expanded into additional operations and actions without detracting from the disclosed embodiments.

[0125] FIG. 10 is a flowchart of an example method 1000 for determining the suitability of a code snippet for implementation with respect to a pipeline skeleton, according to at least one embodiment described in this disclosure. Method 1000 may be performed by any suitable system, apparatus, or device. For example, modification module 120 of FIG. 1 or computing system 1202 of FIG. 12 (e.g., as directed by the modification module) may perform one or more of the operations associated with method 1000. Although shown in discrete blocks, steps and operations associated with one or more of the blocks of method 1000 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation. As mentioned above, in some embodiments, one or more of the operations of method 1000 may be performed with respect to suitability analysis 722 of FIG. 7. Additionally or alternatively, method 1000 may be performed with respect to code snippets identified using methods 800 and / or 900 of FIGS. 8 and 9, respectively.

[0126] Method 1000 may include block 1002, where code snippets identified for potential instantiation of a pipeline skeleton may be obtained. For example, in some embodiments, code snippet 728 of FIG. 7 may be identified using methods 800 or 900.

[0127] At block 1004, element applicability may be determined for each respective code snippet. In some embodiments, element applicability may be based on the program elements of the corresponding code snippet. For example, the program elements may be identified, and a determination may be made as to whether the program elements are general code elements or domain-specific code elements. General code elements may include code elements that may be generally applicable across multiple data sets. In contrast, domain-specific code elements may include code elements that are specific to the data set to which the corresponding code snippet applies.

[0128] In some embodiments, identifying the program elements may include extracting all constants from the corresponding code snippet and identifying the constants as program elements. Additionally or alternatively, the identified constants may be compared to names or values ​​of a dataset to which the code snippet applies. For example, the constants may be compared to column names of the dataset. Additionally or alternatively, the constants may be compared to values ​​contained in fields of the dataset. In response to a particular constant matching a name or value of the dataset, the corresponding program element may be determined to be a domain-specific code element. In contrast, in response to a particular constant not matching a name or value of the dataset, the corresponding program element may be determined to be a general code element.

[0129] In these or other embodiments, a determination may be made as to whether a domain-specific code element can be mapped to a new dataset of a new ML project to which the pipeline skeleton corresponds. For example, it may be determined whether the new dataset includes names or values ​​that can be mapped to names or values ​​of an existing dataset to which the particular domain-specific code element corresponds. In response to the new dataset including names or values ​​that can be mapped to names or values ​​of the existing dataset, the particular domain-specific code element may be deemed mappable to the new dataset.

[0130] The element adaptability of each code snippet may be based on whether the respective code snippet includes any domain-specific code elements. For example, in some embodiments, a particular code snippet may be deemed to potentially not have element adaptability in response to the particular code snippet including one or more domain-specific code elements. In these or other embodiments, it may be determined whether the domain-specific code elements are mappable to the new dataset. A particular code snippet may be deemed to have element adaptability in response to the domain-specific code elements being mappable to the new dataset. In contrast, a particular code snippet may be deemed to not have element adaptability in response to one or more of the domain-specific code elements not being mappable to the new dataset. Additionally or alternatively, a particular code snippet may be deemed to have element adaptability in response to the particular code snippet only having general code elements.

[0131] At block 1006, dataflow adaptability may be determined for each respective code snippet. In some embodiments, dataflow adaptability may be based on the flow of inputs that may be input to the corresponding code snippet and the flow of outputs that may be output by the corresponding code snippet. In particular, it may be determined whether the inputs are derived from data frames of the corresponding dataset and whether the outputs are sent to data frames of the corresponding dataset. A corresponding code snippet may be deemed to have dataflow adaptability in response to its inputs and outputs all corresponding to data frames (e.g., derived from or sent to data frames). In contrast, a corresponding code snippet may be deemed not to have dataflow adaptability in response to one or more inputs and / or one or more outputs not corresponding to data frames.

[0132] In some embodiments, static analysis may be performed to determine the inputs and outputs for each code snippet. Further, the static analysis may indicate from which portion of a corresponding existing dataset the inputs may be obtained and / or to which portion of a corresponding existing dataset the outputs may be sent. Thus, the static analysis may indicate whether the inputs or outputs correspond to data frames of the corresponding existing dataset.

[0133] At block 1008, cardinality adaptability may be determined for each code snippet. The cardinality adaptability may be based on the cardinality adaptability of the corresponding code snippet with respect to the new data set. For example, the cardinality adaptability may be based on the number of portions (e.g., number of columns) that the corresponding code snippet provides with respect to its corresponding existing data set compared to the number of portions to which the corresponding code snippet applies with respect to the new data set.

[0134] For example, to determine cardinality suitability, a determination may be made as to how many portions (e.g., columns) of a new dataset a corresponding code snippet can be applied to. In some embodiments, this determination may be made based on block instantiations (e.g., block instantiations 718 in FIG. 7 ) that may be included in the pipeline skeleton. For example, the corresponding code snippet may be selected as a potential instantiation of a particular function block of the pipeline skeleton. Furthermore, the block instantiation may indicate to which portions of the new dataset a particular function block should be applied. Thus, a determination may be made as to how many portions of a new dataset a corresponding code snippet can be applied as an instantiation of a particular function block. The determined number of portions of a new dataset to which a corresponding code snippet can be applied may be referred to as the “number of new dataset portions.” Additionally, it may be determined how many portions of a corresponding existing dataset a corresponding code snippet can be determined to. This determination may be based on any suitable analysis of the corresponding code snippet. The determined number of portions of an existing dataset to which a corresponding code snippet can be applied may be referred to as the “number of existing dataset portions.”

[0135] The cardinality adaptability may be based on a comparison of the number of new dataset portions to the number of existing dataset portions. For example, a first code snippet may have a first new dataset portion number equal to the first existing dataset portion number. Furthermore, a second code snippet may have a second new dataset portion number unequal to the second existing dataset portion number. In this example, the first code snippet may have a higher cardinality adaptability than the second code snippet.

[0136] In some embodiments, in response to the new dataset portion number not matching the existing dataset portion number, it may be determined whether one or more transformations can be applied to the corresponding code snippet to improve cardinality adaptability. For example, in response to the existing dataset portion number being 1 for a particular code snippet and the new dataset portion number being greater than 1, the particular code snippet may be placed in a loop that iterates a number of times matching the new dataset portion number. In these or other embodiments, a code snippet that can be transformed to improve cardinality may be considered to have lower cardinality adaptability than a code snippet that still has a matching cardinality. Additionally or alternatively, a code snippet that can be transformed to improve cardinality may be considered to have higher cardinality adaptability than a code snippet that cannot be transformed to improve cardinality. In these or other embodiments, a code snippet may generally be considered to have cardinality adaptability in response to the code snippet having matching new dataset portion number and existing dataset portion number, or in response to the code snippet being transformable to match a different dataset portion number.

[0137] In some embodiments, method 1000 may include block 1010, where an overall adaptability may be determined for each respective code snippet. In some embodiments, the overall adaptability may be based on a combination of two or more of element adaptability, dataflow adaptability, or cardinality adaptability. In these or other embodiments, the overall adaptability may be based on all combinations of element adaptability, dataflow adaptability, and cardinality adaptability.

[0138] For example, in some embodiments, the corresponding code snippet may be deemed to have an overall adaptability, where the corresponding code snippet is deemed to be either adaptable or non-adaptable. In some embodiments, the corresponding code snippet may be deemed to have an overall adaptability in response to the corresponding code snippet being determined to have element adaptability, dataflow adaptability, and cardinality adaptability.

[0139] In these or other embodiments, code snippets that are potential instantiations of the same skeleton block of a pipeline skeleton may be ranked with respect to each other and with respect to their respective adaptability. For example, a first code snippet may have domain-specific code elements that can be mapped to a new data set, such that the first code snippet has program element adaptability. Additionally, a second code snippet that instantiates the same skeleton block may have program element adaptability because it may have only general code elements. Thus, the second code snippet may be ranked higher than the first code snippet with respect to program element adaptability. Additionally or alternatively, all other things being equal, the second code snippet may be ranked higher than the first code snippet with respect to overall adaptability. Similarly, a code snippet that has cardinality adaptability through transformation may be ranked lower than a code snippet that has cardinality adaptability without requiring transformation.

[0140] In some embodiments, the suitability determinations for different code snippets may be included in expanded code snippet information, such as expanded code snippet information 730 of Figure 7. Additionally or alternatively, a ranking of the code snippets based on the suitability determinations may be included in the expanded code snippet information.

[0141] Modifications, additions, or omissions may be made to method 1000 without departing from the scope of the present disclosure. For example, some of the operations of method 1000 may be implemented in a different order. Additionally, or alternatively, two or more operations may be performed simultaneously. Furthermore, the outlined operations and actions are provided only as examples, and some of the operations and actions may be optional, combined into fewer operations and actions, or expanded into additional operations and actions without detracting from the disclosed embodiments.

[0142] 11 is a flowchart of an example method 1100 for generating a set of candidate pipelines according to at least one embodiment described in this disclosure. Method 1100 may be performed by any suitable system, apparatus, or device. For example, modification module 120 of FIG. 1 or computing system 1202 of FIG. 12 (e.g., as directed by the modification module) may perform one or more of the operations associated with method 1100. Although shown in discrete blocks, steps and operations associated with one or more of the blocks of method 1100 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

[0143] Generally, method 1100 may include generating multiple concrete pipelines as potential instantiations of a pipeline skeleton for a new ML project. Accordingly, the multiple concrete pipelines may be candidate pipelines for a new ML project to be applied to a new dataset for the new ML project. As described above, in some embodiments, one or more of the operations of method 1100 may be performed with respect to candidate pipeline generation 724 of FIG. 7. Additionally or alternatively, method 1100 may be performed with respect to code snippet 728 of FIG. 7 using expanded code snippet information 730, which may be identified using methods 800, 900, and / or 1000 of FIGS. 8, 9, and 10, respectively.

[0144] Method 1100 may include block 1102, where code snippet rankings may be obtained for different code snippets that may be identified for instantiating a pipeline skeleton. The code snippet rankings may be based on a skeleton block by skeleton block basis of the pipeline skeleton. For example, code snippets may be grouped according to the skeleton blocks that each code snippet may instantiate. In these or other embodiments, a code snippet ranking may be provided for each different group of code snippets. For example, a first group of code snippets corresponding to a first skeleton block may be ranked relative to each other, and a second group of code snippets corresponding to a second skeleton block may be ranked relative to each other. The code snippet rankings may, in some embodiments, include any suitable combination of adaptive rankings, such as those described above with respect to method 1000. Additionally or alternatively, the code snippet rankings may include rankings used to select code snippets, such as those described above with respect to method 800 or method 900.

[0145] At block 1104, a respective code snippet may be selected for each skeleton block of the pipeline skeleton. In some embodiments, the respective code snippets may be selected based on their respective rankings in their respective sets. For example, the highest ranked code snippet of the first set of code snippets may be selected for the first skeleton block, and the highest ranked code snippet of the second set of code snippets may be selected for the second skeleton block. In these or other embodiments, multiple different code snippets may be selected for each of the skeleton blocks such that multiple instantiations of each skeleton block are evaluated.

[0146] In some embodiments, at block 1104, the operations may also include transforming each selected code snippet in the context of the new data set. This transformation may include resolving inconsistencies in variable or object names of the code snippet (e.g., adapting names based on program analysis).

[0147] At block 1106, a set of candidate pipelines may be generated using the selected code snippets. For example, each candidate pipeline may be a concrete pipeline that includes an instantiation of each of the skeleton blocks of the pipeline skeleton. As such, in some embodiments, the set of selected code snippets may be selected as a pipeline group that can each be used to generate a candidate pipeline.

[0148] For example, a pipeline skeleton may include skeleton blocks A through D. Further, multiple pipeline groups may be selected to generate multiple candidate pipelines for the pipeline skeleton. For example, a first pipeline group may be used to generate a first candidate pipeline for the pipeline skeleton that includes a first code snippet that instantiates skeleton block A, a second code snippet that instantiates skeleton block B, a third code snippet that instantiates skeleton block C, and a fourth code snippet that instantiates skeleton block D. Additionally, a second pipeline group may be used to generate a second candidate pipeline for the pipeline skeleton that includes a fifth code snippet that instantiates skeleton block A, a sixth code snippet that instantiates skeleton block B, and an eighth code snippet that instantiates skeleton block D. In this example, the first and fifth code snippets may be part of the same set of code snippets corresponding to skeleton block A, the second and sixth code snippets may be part of the same set of code snippets corresponding to skeleton block B, the third and seventh code snippets may be part of the same set of code snippets corresponding to skeleton block C, and the fourth and eighth code snippets may be part of the same set of code snippets corresponding to skeleton block D.

[0149] In some embodiments, a different skeleton group may be generated for each different permutation of different combinations of code snippets. As such, in some embodiments, the set of candidate pipelines generated at block 1106 may include a different candidate pipeline for each different permutation.

[0150] In some embodiments, template code may be added to each of the candidate pipelines in the set of candidate pipelines. The template code may provide standard instantiations of common operations that appear in all pipelines, including reading in a dataset, splitting it into training and test datasets, fitting a model from the instantiated pipeline skeleton to the training data, and evaluating the trained model on the test data. Because such operations may typically not change in functionality or syntax from one pipeline or dataset to another, in some embodiments, standard boilerplate code instantiated with appropriate parameters may be used to complete these portions of the instantiated pipeline.

[0151] A set of candidate pipelines may be output at block 1108. Each concrete pipeline in the set of candidate pipelines may be a candidate instantiation of a pipeline skeleton.

[0152] Modifications, additions, or omissions may be made to method 1100 without departing from the scope of the present disclosure. For example, some of the operations of method 1100 may be implemented in a different order. Additionally, or alternatively, two or more operations may be performed simultaneously. Furthermore, the outlined operations and actions are provided only as examples, and some of the operations and actions may be optional, combined into fewer operations and actions, or expanded into additional operations and actions without detracting from the disclosed embodiments.

[0153] 12 shows a block diagram of an exemplary computing system 1202 in accordance with at least one embodiment of the present disclosure. The computing system 1202 may be configured to implement or direct one or more operations associated with a modification module (e.g., modification module 120 of FIG. 1). The computing system 1202 may include a processor 1250, a memory 1252, and data storage 1254. The processor 1250, the memory 1252, and the data storage 1254 may be communicatively coupled.

[0154] Generally, processor 1250 may include any suitable special-purpose or general-purpose computer, computing entity, or processing device, including various computer hardware or software modules, and may be configured to execute instructions stored on any applicable computer-readable storage medium. For example, processor 1250 may include a microprocessor, microcontroller, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or any other digital or analog circuit configured to interpret and / or execute program instructions and / or process data. Although shown as a single processor in FIG. 12, processor 1250 may include any number of processors configured, individually or collectively, to perform or direct the execution of any number of operations described in this disclosure. Additionally, one or more processors may reside in one or more different electronic devices, e.g., different servers.

[0155] In some embodiments, processor 1250 may be configured to interpret and / or execute program instructions and / or process data stored in memory 1252, data storage 1254, or memory 1252 and data storage 1254. In some embodiments, processor 1250 may fetch program instructions from data storage 1254 and load the program instructions into memory 1252. After the program instructions are loaded into memory 1252, processor 1250 may execute the program instructions.

[0156] For example, in some embodiments, the modification modules may be included as program instructions in data storage 1254. Processor 1250 may fetch the program instructions for the corresponding modules from data storage 1254 and load the program instructions for the corresponding modules in memory 1252. After the program instructions for the corresponding modules are loaded into memory 1252, processor 1250 may execute the program instructions such that the computing system may implement the operations associated with the corresponding modules as directed by the instructions.

[0157] Memory 1252 and data storage 1254 may include computer-readable storage media for carrying or having computer-executable instructions or data structures stored thereon. Such computer-readable storage media may include any available medium that can be accessed by a general-purpose or special-purpose computer, such as processor 1250. By way of example, and not limitation, such computer-readable storage media may include tangible or non-transitory computer-readable storage media, including random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory devices (e.g., solid-state memory devices), or any other storage medium that can be used to carry or store specific program code in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer. Combinations of the above may also be included within the scope of computer-readable storage media. Computer-executable instructions may include, for example, instructions and data configured to cause processor 1250 to perform a certain operation or group of operations.

[0158] Modifications, additions, or omissions may be made to computing system 1202 without departing from the scope of the present disclosure. For example, in some embodiments, computing system 1202 may include any number of other components that may not be explicitly shown or described.

[0159] As mentioned above, the embodiments described in this disclosure may involve the use of special purpose or general purpose computers including various computer hardware or software modules, as discussed in more detail below. Further, as noted above, the embodiments described in this disclosure may be implemented using computer-readable media for carrying or having computer-executable instructions or data structures stored thereon.

[0160] As used in this disclosure, the term "module" or "component" may refer to a specific hardware implementation configured to perform the actions of the module or component and / or a software object or software routine that may be stored and / or executed on general-purpose hardware of a computer system (e.g., a computer-readable medium, a processing device, etc.). In some embodiments, different components, modules, engines, and services described in this disclosure may be implemented as objects or processes running on a computing system (e.g., as separate threads). Although some of the systems and methods described in this disclosure are generally described as being implemented in software (stored on and / or executed by general-purpose hardware), specific hardware implementations or combinations of software and specific hardware implementations are also possible and contemplated. In this description, a "computing entity" may be any computing system, or any combination of modules or modulators operating on a computing system, as defined previously in this disclosure.

[0161] Terms used in this disclosure, particularly in the appended claims (e.g., the body of the appended claims), are generally intended as "open" terms (e.g., the term "including" should be interpreted as "including, but not limited to," the term "having" should be interpreted as "having at least," the term "including" should be interpreted as "including, but not limited to," etc.).

[0162] Furthermore, where a specific number of introduced claim provisions is intended, such intention will be expressly set forth in the claim; where such a provision is absent, such intention does not exist. For example, to aid in understanding, the following appended claims may contain the use of the introductory phrases "at least one" and "one or more" to introduce claim provisions. However, the use of such phrases should not be construed to suggest that the introduction of a claim provision with the indefinite article "a" or "an" limits any particular claim containing such introduced claim provision to embodiments containing only one such provision, even when the same claim includes the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" or "an" (e.g., "a" and / or "an" should be interpreted to mean "at least one" or "one or more"), and the same is true for any use of an indefinite article used to introduce a claim provision.

[0163] Additionally, even when a specific number of provisions in an introduced claim is explicitly recited, those skilled in the art will recognize that such provision should be interpreted to mean at least the recited number (e.g., the mere provision of "two provisions," without other modifiers, means at least two provisions, or more than two provisions). Furthermore, when a convention similar to "at least one of A, B, and C" or "one or more of A, B, and C, etc." is used, it is generally intended that such a structure include A alone, B alone, C alone, A and B, A and C, B and C, or A, B, and C, etc. This interpretation of the phrase "A" or "B" remains applicable even though the term "A and / or B" is sometimes used to include the possibilities of "A" or "B," or "A" and "B."

[0164] Furthermore, any word or phrase, whether in the specification, claims, or drawings, that presents two or more alternative terms should be understood to contemplate one of the terms, either of the terms, or both of the terms. For example, the phrase "A or B" should be understood to include the possibilities of "A" or "B" or "A and B."

[0165] All examples and conditional language set forth in this disclosure are intended for educational purposes to aid the reader in understanding the disclosure and the concepts the inventors have contributed to furthering the art, and should not be construed as being limited to the examples and conditions so specifically set forth. While embodiments of the present disclosure have been described in detail, various changes, substitutions, and alterations may be made thereto without departing from the spirit and scope of the present disclosure.

[0166] The following additional notes are provided regarding the above-described embodiments. (Appendix 1) Obtaining a machine learning (ML) pipeline skeleton indicating a set of first function blocks to be used to process a new dataset for a new ML project, each first function block of the set of first function blocks having a respective functionality; for each respective first function block of the set of first function blocks: Retrieving a plurality of existing code snippets from one or more existing ML pipelines of one or more existing ML projects, each of the existing code snippets instantiating a second function block of the existing ML pipeline and being a potential instantiation of the respective first function block; determining the suitability of each of the existing code snippets with respect to the new data set; selecting a particular existing code snippet for implementing the respective first functional block based on the determined suitability; and and instantiating the pipeline skeleton based on the particular existing code snippets selected for each of the first functional blocks, respectively. (Appendix 2) Obtaining a particular existing code snippet of the plurality of existing code snippets for each first function block includes: identifying the second function block corresponding to the particular existing code snippet based on the second function block having functionality corresponding to the functionality of the respective first function block; identifying an existing dataset of the existing ML project based on the application of the second function block to the existing dataset via an implementation of the second function block through the particular existing code snippet; determining the similarity between the new dataset and the identified existing dataset; and selecting the particular existing code snippet as a potential instance of the respective first functional block based on the similarity and in response to the particular existing code snippet corresponding to the identified existing dataset. (Appendix 3) 3. The method of claim 2, wherein determining the similarity is based on similarity between one or more features of the new dataset with respect to the one or more features of the identified dataset. (Appendix 4) 3. The method of claim 2, wherein selecting the particular existing code snippet based on the determined similarity is based on a similarity ranking of a particular identified dataset corresponding to the particular existing code snippet. (Appendix 5) 2. The method of claim 1, further comprising ranking the particular existing code snippets based on the determined similarity between the new dataset and the identified existing dataset. (Appendix 6) 10. The method of claim 1, wherein instantiating the pipeline skeleton includes generating a set of candidate concrete pipelines based on the selected particular existing code snippet, and the method further includes selecting a particular concrete pipeline for the new ML project from the set of candidate concrete pipelines based on performance of the particular concrete pipeline applied to the new dataset. (Appendix 7) 2. The method of claim 1, wherein determining the suitability of each existing code snippet is based on one or more of whether the each existing code snippet includes code elements specific to the corresponding existing dataset, whether the each existing code snippet includes data types of input data and output data, or whether the each existing code snippet includes compatibility with respect to cardinality of the new dataset. (Appendix 8) One or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed, cause the system to perform operations, including: obtaining a machine learning (ML) pipeline skeleton indicating a set of first function blocks to be used to process a new dataset for a new ML project, each first function block of the set of first function blocks having a respective functionality; for each respective first function block of the set of first function blocks: Retrieving a plurality of existing code snippets from one or more existing ML pipelines of one or more existing ML projects, each of the existing code snippets instantiating a second function block of the existing ML pipeline and being a potential instantiation of the respective first function block; determining the suitability of each of the existing code snippets with respect to the new data set; selecting a particular existing code snippet for implementing the respective first functional block based on the determined suitability; and and instantiating the pipeline skeleton based on the particular existing code snippets selected for each of the first function blocks. (Appendix 9) Obtaining a particular existing code snippet of the plurality of existing code snippets for each first function block includes: identifying the second function block corresponding to the particular existing code snippet based on the second function block having functionality corresponding to the functionality of the respective first function block; identifying an existing dataset of the existing ML project based on the application of the second function block to the existing dataset via an implementation of the second function block through the particular existing code snippet; determining the similarity between the new dataset and the identified existing dataset; and selecting the particular existing code snippet as a potential instance of the respective first functional block based on the similarity and in response to the particular existing code snippet corresponding to the identified existing data set. (Appendix 10) 10. The one or more non-transitory computer-readable storage media of claim 9, wherein determining the similarity is based on similarity between one or more features of the new dataset relative to the one or more features of the identified dataset. (Appendix 11) 10. The one or more non-transitory computer-readable storage media of claim 9, wherein selecting the particular existing code snippet based on the determined similarity is based on a similarity ranking of a particular identified dataset corresponding to the particular existing code snippet. (Appendix 12) 9. The one or more non-transitory computer-readable storage media of claim 8, wherein the operations further include ranking the particular existing code snippets based on the determined similarity between the new dataset and the identified existing dataset. (Appendix 13) 9. The one or more non-transitory computer-readable storage media of claim 8, wherein instantiating the pipeline skeleton includes generating a set of candidate concrete pipelines based on the selected particular existing code snippet, and the operations further include selecting a particular concrete pipeline for the new ML project from the set of candidate concrete pipelines based on performance of the particular concrete pipeline applied to the new dataset. (Appendix 14) 9. The one or more non-transitory computer-readable storage media of claim 8, wherein determining the suitability of each existing code snippet is based on one or more of whether the each existing code snippet includes code elements specific to the corresponding existing dataset, whether the each existing code snippet includes data types of input data and output data, or whether the each existing code snippet includes compatibility with respect to cardinality of the new dataset. (Appendix 15) one or more processors; one or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed, cause the system to perform operations, including: obtaining a machine learning (ML) pipeline skeleton indicating a set of first function blocks to be used to process a new dataset for a new ML project, each first function block of the set of first function blocks having a respective functionality; for each respective first function block of the set of first function blocks: Retrieving a plurality of existing code snippets from one or more existing ML pipelines of one or more existing ML projects, each of the existing code snippets instantiating a second function block of the existing ML pipeline and being a potential instantiation of the respective first function block; determining the suitability of each of the existing code snippets with respect to the new data set; selecting a particular existing code snippet for implementing the respective first functional block based on the determined suitability; and and instantiating the pipeline skeleton based on the particular existing code snippets selected for each of the first function blocks. (Appendix 16) Obtaining a particular existing code snippet of the plurality of existing code snippets for each first function block includes: identifying the second function block corresponding to the particular existing code snippet based on the second function block having functionality corresponding to the functionality of the respective first function block; identifying an existing dataset of the existing ML project based on the application of the second function block to the existing dataset via an implementation of the second function block through the particular existing code snippet; determining the similarity between the new dataset and the identified existing dataset; and selecting the particular existing code snippet as a potential instance of the respective first functional block based on the similarity and in response to the particular existing code snippet corresponding to the identified existing dataset. (Appendix 17) 17. The system of claim 16, wherein determining the similarity is based on similarity between one or more features of the new dataset with respect to the one or more features of the identified dataset. (Appendix 18) 16. The system of claim 15, further comprising ranking the particular existing code snippets based on the determined similarity between the new dataset and the identified existing dataset. (Appendix 19) 16. The system of claim 15, wherein instantiating the pipeline skeleton includes generating a set of candidate concrete pipelines based on the selected particular existing code snippet, and the operations further include selecting a particular concrete pipeline for the new ML project from the set of candidate concrete pipelines based on performance of the particular concrete pipeline applied to the new dataset. (Appendix 20) 16. The system of claim 15, wherein determining the suitability of each existing code snippet is based on one or more of whether the each existing code snippet includes code elements specific to the corresponding existing dataset, whether the each existing code snippet includes data types of input data and output data, or whether the each existing code snippet includes compatibility with respect to cardinality of the new dataset. [Explanation of symbols]

[0167] 104 Pipeline Skeleton Model 102 Pipeline Skeleton 120 Correction Module 122 Concrete Pipeline 105 ML Project Corpus 110 Existing ML Projects 111 ML Pipeline 109 datasets 115 New ML Projects 108 New ML Tasks 106 new datasets

Claims

1. A method performed by a system, comprising: Obtaining a pipeline skeleton indicating a set of first function blocks to be used to process a new dataset for a new machine learning (ML) project, each first function block of the set of first function blocks having a respective functionality; for each respective first function block of the set of first function blocks: Retrieving a plurality of existing code snippets from one or more existing ML pipelines of one or more existing ML projects, each of the existing code snippets instantiating a second function block of the existing ML pipeline and being a potential instantiation of the respective first function block; determining the suitability of each of the existing code snippets with respect to the new data set; selecting a particular existing code snippet for implementing the respective first functional block based on the determined suitability; and and instantiating the pipeline skeleton based on the particular existing code snippets selected respectively for each of the first functional blocks.

2. Obtaining a particular existing code snippet of the plurality of existing code snippets for each first function block includes: identifying the second function block corresponding to the particular existing code snippet based on the second function block having functionality corresponding to the functionality of the respective first function block; identifying an existing dataset of the existing ML project based on the second function block being applied to the existing dataset via an implementation of the second function block through the particular existing code snippet; determining the similarity between the new dataset and the identified existing dataset; and selecting the particular existing code snippet as a potential instance of the respective first functional block based on the similarity and in response to the particular existing code snippet corresponding to the identified existing data set.

3. The method of claim 2 , wherein determining the similarity is based on a similarity between one or more features of the new dataset with respect to the one or more features of the identified dataset.

4. The method of claim 2 , wherein selecting the particular existing code snippet based on the determined similarity is based on a similarity ranking of a particular identified data set that corresponds to the particular existing code snippet.

5. The method of claim 2 , further comprising: ranking the particular existing code snippets based on the determined similarity between the new dataset and the identified existing dataset.

6. 2. The method of claim 1, wherein instantiating the pipeline skeleton includes generating a set of candidate concrete pipelines based on the selected particular existing code snippet, the method further including selecting a particular concrete pipeline for the new ML project from the set of candidate concrete pipelines based on performance of the particular concrete pipeline applied to the new dataset.

7. 2. The method of claim 1, wherein determining the suitability of each existing code snippet is based on one or more of whether the each existing code snippet includes code elements specific to the corresponding existing dataset, whether the each existing code snippet includes data types of input data and output data, or whether the each existing code snippet includes compatibility with respect to cardinality of the new dataset.

8. One or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed, cause the system to perform operations, including: Obtaining a pipeline skeleton indicating a set of first function blocks to be used to process a new dataset for a new machine learning (ML) project, each first function block of the set of first function blocks having a respective functionality; for each respective first function block of the set of first function blocks: Retrieving a plurality of existing code snippets from one or more existing ML pipelines of one or more existing ML projects, each of the existing code snippets instantiating a second function block of the existing ML pipeline and being a potential instantiation of the respective first function block; determining the suitability of each of the existing code snippets with respect to the new data set; selecting a particular existing code snippet for implementing the respective first functional block based on the determined suitability; and and instantiating the pipeline skeleton based on the particular existing code snippets selected for each of the first function blocks.

9. one or more processors; one or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed by the one or more processors, cause the system to perform operations, including: Obtaining a pipeline skeleton indicating a set of first function blocks to be used to process a new dataset for a new machine learning (ML) project, each first function block of the set of first function blocks having a respective functionality; for each respective first function block of the set of first function blocks: Retrieving a plurality of existing code snippets from one or more existing ML pipelines of one or more existing ML projects, each of the existing code snippets instantiating a second function block of the existing ML pipeline and being a potential instantiation of the respective first function block; determining the suitability of each of the existing code snippets with respect to the new data set; selecting a particular existing code snippet for implementing the respective first functional block based on the determined suitability; and and instantiating the pipeline skeleton based on the particular existing code snippets selected for each of the first functional blocks, respectively.

Images