Method, computer-readable medium, and system for augmenting a machine learning pipeline with annotations

Abstract

Methods, computer-readable media, and systems for augmenting a machine learning pipeline with annotations are provided. A method can include obtaining a trained machine learning (ML) pipeline skeleton model configured to predict functional blocks within a new ML pipeline based on meta-features of a dataset associated with the new ML pipeline; obtaining parameterized templates, each of the parameterized templates including fillable portions and static text portions that, in combination, describe a given functional block; receiving a request to generate the new ML pipeline; determining functional blocks to populate the new ML pipeline based on the pipeline skeleton model; extracting decision conditions that lead to the functional blocks; generating annotations for the functional blocks using the parameterized templates, wherein at least one of the fillable portions is populated based on the decision conditions that lead to the functional blocks; and instantiating the new ML pipeline including the functional blocks with the generated annotations.

Description

Technical Field

[0001] The implementations discussed in this disclosure relate to using annotations to enhance machine learning pipelines. Background Technology

[0002] Machine learning (ML) typically uses ML models trained on training data to make predictions that automatically become more accurate with continued training. ML can be used in a variety of applications, including but not limited to traffic prediction, web search, online fraud detection, medical diagnosis, speech recognition, email filtering, image recognition, virtual personal assistants, and automatic translation.

[0003] The subject matter claimed in this disclosure is not limited to implementations that address any shortcomings or operate only in environments such as those described above. Rather, this background art is provided merely to illustrate an example technical field in which some of the implementations described in this disclosure can be practiced. Summary of the Invention

[0004] According to one aspect of the implementation, the operation may include: obtaining a trained machine learning (ML) pipeline skeleton model configured to predict one or more functional blocks within a new ML pipeline based on meta-features of a dataset associated with the new ML pipeline; and obtaining parameterized templates, wherein each parameterized template may include one or more fillable portions and one or more static text portions that combine to describe a given functional block. The method may further include: receiving a request to generate a new ML pipeline based on a dataset; and determining functional blocks for filling the new ML pipeline based on the trained ML pipeline skeleton model. The method may additionally include: extracting decision conditions leading to at least one functional block; and generating annotations for at least one functional block using the parameterized templates, wherein at least one fillable portion may be filled based on the decision conditions leading to the functional block. The method may further include instantiating the new ML pipeline including the functional blocks using the generated annotations.

[0005] The objectives and advantages of the implementation will be realized and achieved, at least by means of the elements, features and combinations specifically pointed out in the claims.

[0006] Both the foregoing general description and the following detailed description are given by way of example and are illustrative, not limiting of the claimed invention. Attached Figure Description

[0007] Example embodiments will be described and illustrated with additional features and details using the accompanying drawings, in which:

[0008] Figure 1 This is a diagram representing an example environment related to automatically generating new machine learning projects based on existing machine learning projects;

[0009] Figure 2 An example set of operations can be performed to modify the pipeline skeleton of a new machine learning project to generate a refined pipeline skeleton;

[0010] Figure 3A This is a flowchart of an example method for determining the relevance of functional blocks;

[0011] Figure 3B An example table is shown that can indicate the use of different columns in a dataset for three different function blocks;

[0012] Figure 4 This is a flowchart of an example method for determining the mapping between functional blocks and dataset features;

[0013] Figure 5 This is a flowchart of an example method for determining the block instantiation of the pipeline skeleton;

[0014] Figure 6 This is a flowchart illustrating an example method for refining a pipeline skeleton into a refined skeleton;

[0015] Figure 7 A set of example operations that can be performed to instantiate a pipe skeleton into a concrete pipe skeleton is shown.

[0016] Figure 8 This is a flowchart of an example method for obtaining a code snippet used to instantiate the pipeline skeleton;

[0017] Figure 9 This is a flowchart of another example method for obtaining a code snippet used to instantiate the pipeline skeleton;

[0018] Figure 10 This is a flowchart of an example method for determining the adaptability of the code snippet used for implementation to the pipeline skeleton;

[0019] Figure 11 This is a flowchart of an example method for generating candidate pipeline groups;

[0020] Figure 12 A block diagram of an example computing system is shown;

[0021] Figure 13 A flowchart is shown for an example method of generating annotated ML pipelines;

[0022] Figure 14 A flowchart illustrating an example method for gathering information in the preparation for generating an annotated ML pipeline is shown.

[0023] Figure 15 A flowchart illustrating an example method for training a skeleton model is shown;

[0024] Figure 16 A flowchart is shown for another example method of generating annotated ML pipelines;

[0025] Figure 17 A flowchart is shown illustrating an example method for generating annotations related to preprocessing function blocks in an ML pipeline;

[0026] Figure 18 A flowchart is shown for another example method for generating annotations related to ML models in an ML pipeline; and

[0027] Figure 19 A flowchart is shown for another example method of generating annotated ML pipelines. Detailed Implementation

[0028] Some implementations described in this disclosure relate to methods and systems for automatically adapting existing machine learning (ML) projects to new ML projects.

[0029] As ML becomes increasingly prevalent, there is a common lack of ML experts (e.g., skilled data scientists) available to implement new ML projects. While various AutoML solutions (e.g., Auto-Sklearn, AutoPandas, etc.) have been proposed to address the growing challenge of implementing new ML projects in the absence of ML experts, current AutoML solutions offer only simple and partial solutions, insufficient for non-experts to fully implement new ML projects. Furthermore, while open-source software (OSS) databases of existing ML projects (e.g., Kaggle, GitHub, etc.) have been proposed as another solution to address the challenge of implementing new ML projects by non-experts, finding potentially useful existing ML projects in these databases can be difficult or impossible for non-experts. Moreover, even if non-experts successfully find potentially useful existing ML projects in these databases, modifying these potentially useful existing ML projects to meet the new requirements of the new ML project is also difficult or impossible for them.

[0030] In this disclosure, the term "ML project" can refer to a project that includes a dataset, ML tasks defined on the dataset, and an ML pipeline (e.g., scripts or program code) configured to implement a series of operations for training an ML model on the dataset for an ML task and using the ML model to make new predictions. In this disclosure, the term "computation notebook" can refer to a computational structure (e.g., a Jupyter notebook) used to develop and / or represent an ML pipeline, particularly during the development phase. Although the embodiments disclosed herein utilize ML pipelines in the Python programming language and computation notebooks constructed as Jupyter notebooks for illustration, it should be understood that other embodiments may include ML pipelines written in different languages ​​and computation notebooks constructed in other platforms.

[0031] According to one or more embodiments of this disclosure, operations can be performed to automatically adapt existing ML projects to new ML projects. For example, in some embodiments, a computer system can systematically support the natural workflow of data scientists by constructing a "search and adapt" style workflow, in which data scientists would first search for existing ML projects that can serve as a good starting point for building new ML projects, and then appropriately adapt the existing ML projects to build ML pipelines for the new datasets and new ML tasks of the new ML projects.

[0032] For example, in some implementations, a computer system can automatically mine raw ML projects from an OSS database of existing ML projects and can automatically manage these raw ML projects before storing them in a corpus of existing ML projects. In some implementations, such mining and management of existing ML projects from a large repository may produce a corpus of high-quality existing ML projects that can be used to search and adapt to workflows. Furthermore, this management may involve cleaning the ML pipeline of existing ML projects (e.g., using dynamic program slicing) and may involve computing a set of features to capture the quality and diversity of each ML project and selecting an optimal number of existing ML projects that align with these objectives.

[0033] Furthermore, in some implementations, this management may require operations to automatically identify and index functional blocks within the ML pipeline of an existing ML project. Unlike traditional software programs, ML pipelines in ML projects typically follow well-defined workflows based on dataset attributes and can be viewed as a series of functional blocks. Therefore, some implementations may involve a technique that automatically extracts and labels functional blocks within the ML pipeline to correctly index them in a corpus, thereby enabling efficient searching of functional blocks to synthesize new ML pipelines for new ML tasks. More specifically, this technique can abstract the ML pipeline at an appropriate level and can employ graph-based sequence mining algorithms to extract both custom and idiomatic functional blocks. Finally, semantic tagging can be performed on each functional block.

[0034] Additionally, in some implementations, annotations may be provided so that human operators can observe and understand the decision-making process performed by the automated system to generate the new ML pipeline. In some implementations, each functional block may include a corresponding annotation when the new ML pipeline is instantiated. For example, for a preprocessing functional block, the annotation may include a description of the decision in the decision tree that led to the inclusion of the preprocessing functional block in the new ML pipeline. This information can be used to populate the populateable portion of the parameterized template, which, together with the natural language static text portion of the parameterized template, can provide the text for the annotation. As another example, the annotation may include a description of which meta-features of the dataset are most influential in selecting the model used in the ML pipeline. In some implementations, the annotation may include suggestions for alternative functional blocks to be included in the new ML pipeline.

[0035] In this disclosure, the reference to "functional block" can refer to an operation that can be performed by an ML pipeline, where a specific functional block can correspond to a specific type of functionality. Semantic tags can indicate the functionality of a corresponding functional block. Furthermore, each functional block can be instantiated in its corresponding ML pipeline using a specific code snippet configured to execute the functionality of the corresponding functional block. In many cases, the same functional block across different ML pipelines can be instantiated differently in each different ML pipeline.

[0036] In some implementations, upon receiving a new dataset and new ML tasks for a new ML project—for example, from a non-expert data scientist—the computer system can automatically use a hierarchical approach to first synthesize a function block-level pipeline skeleton for the new ML project using the ML model. Alternatively, the computer system can obtain the pipeline skeleton via another mechanism (e.g., from user input). The pipeline skeleton can indicate which function blocks can be used for the new ML project.

[0037] In some cases, the obtained pipeline skeleton may include functional blocks that may be technically different from each other, but may also be similar enough that they can be considered redundant. Alternatively, as described above, the pipeline skeleton may indicate which functional blocks can be used for a new ML project, but in some cases may not indicate the order in which the functional blocks can be used. As discussed in detail below, in some implementations, the computer system may be configured to refine the obtained pipeline skeleton by removing functional blocks based on redundancy analysis. Alternatively, the computer system may be configured to identify the order of the functional blocks in the pipeline skeleton and can refine the pipeline skeleton accordingly.

[0038] A pipeline skeleton can indicate which functional blocks are used in a new ML project, but may not indicate the instantiation of those functional blocks. As discussed in detail below, in some implementations, the computer system can also be configured to determine which parts of a new dataset each functional block of the pipeline skeleton should be applied to. Alternatively or additionally, the computer system can be configured to identify existing code snippets from existing ML projects that can be used to instantiate the pipeline skeleton into a specific pipeline skeleton for the new ML project.

[0039] Therefore, in some implementations, non-expert data scientists can simply define a new dataset and new ML tasks for a new ML project, and then the computer system can implement a tool-assisted interactive search and adaptation workflow to automatically generate a new ML pipeline for the ML project that can be immediately executed to perform the new ML tasks on the new dataset, without requiring any modifications from the non-expert data scientist. Thus, some implementations enable novice data scientists to effectively create new, high-quality end-to-end ML pipelines for new ML projects.

[0040] According to one or more embodiments of this disclosure, the technical field of ML project development can be improved by configuring a computing system to automatically generate new ML projects based on existing ML projects, compared to requiring data scientists (e.g., who are typically non-experts) to manually find potentially useful existing ML projects and modify them to meet the new requirements of new ML projects. Such a configuration enables the computing system to better search for relevant existing ML projects and to better utilize relevant existing ML projects to generate new ML projects by identifying and extracting functional blocks and their corresponding instantiations from existing ML pipelines and automatically using and modifying them for use in new ML pipelines.

[0041] Implementation methods of this disclosure are illustrated with reference to the accompanying drawings.

[0042] Figure 1This diagram illustrates an example environment 100 arranged according to at least one embodiment described in this disclosure, relating to the automatic generation of new ML projects based on existing ML projects. Environment 100 may include a modification module 120 configured to modify a pipeline skeleton 102 to generate a concrete pipeline 122 that can be used for the implementation of a new ML project 115. In some embodiments, modification module 120 may be configured to modify a pipeline skeleton 102 based on an existing ML project 110, which may be included in an ML project corpus 105.

[0043] The ML project corpus 105 may include any suitable repository of existing ML projects 110. Each existing ML project 110 may include electronic data comprising at least a dataset 109, ML tasks defined on the dataset, and ML pipelines 111 (e.g., scripts or program code) configured to implement a series of operations for training an ML model for the ML task and using the ML model to make new predictions. In some implementations, each existing ML project 110 may include a computation notebook, which may be a computational structure used, particularly during the development phase, to develop and / or represent the corresponding ML pipeline. An example of a computation notebook is a Jupyter notebook.

[0044] In some implementations, the ML project corpus 105 may include one or more OSS ML project databases, which may be large repositories of existing ML projects. Some examples of large repositories of existing ML projects 110 include, but are not limited to, Kaggle and GitHub.

[0045] Alternatively or concurrently, in some implementations, the existing ML projects 110 of the ML project corpus 105 can be managed and selected from one or more OSS ML project databases. Management allows the ML project corpus 105 to be a large corpus of clean, high-quality indexed existing ML projects that can be adopted in an automated "search and adapt" style workflow. Management can be carried out using any suitable technology.

[0046] Pipeline skeleton 102 may include a set of functional blocks that indicate functions that can be used to complete a new ML task 108 for a new dataset 106 of a new ML project 115. In some embodiments, functional blocks may not be ranked in pipeline skeleton 102. Alternatively or additionally, pipeline skeleton 102 may include one or more functional blocks that may be relatively redundant compared to one or more other functional blocks in pipeline skeleton 102.

[0047] In some implementations, pipeline skeleton 102 can be generated using pipeline skeleton model 104. Pipeline skeleton model 104 may include one or more ML models trained to learn a mapping between dataset meta-features and function block semantic labels (e.g., based on existing ML project information included in existing ML projects 110 of ML project corpus 105). For example, given meta-features of a new dataset 106, pipeline skeleton model 104 can use the mapping to identify function blocks corresponding to the meta-features of the new dataset 106, and pipeline skeleton 102 can be synthesized accordingly. Alternatively or concurrently, pipeline skeleton 102 can be generated manually or by any other suitable technique.

[0048] In some implementations, the pipeline skeleton model 104 may include a multivariate multivalued classifier trained prior to generating the pipeline skeleton. The multivariate multivalued classifier may be configured to map meta-features of a new dataset to a set of unordered functional blocks (represented by corresponding semantic labels) that the pipeline skeleton should contain. This training may include performing relational mapping, as shown below regarding... Figure 2 As described. For example, training may include: extracting dataset features from an existing dataset of an existing ML project associated with a specific semantic label; identifying groups of all labels from the function blocks of the existing ML project; preparing training data containing input vectors with dataset features and binary output tuples representing the presence or absence of each label in the groups of all labels; and training the pipeline skeleton model 104 to learn the mapping between the dataset features and the corresponding labels in the groups of all labels. In some implementations, training the pipeline skeleton model 104 may enable the pipeline skeleton model 104 to predict a set of function blocks as the skeleton blocks of the pipeline skeleton using salient features (meta-features) of the new dataset 106 and the new ML task 108.

[0049] Modification module 120 may include code and routines configured to enable a computing device to perform one or more operations. Alternatively or additionally, modification module 120 may be implemented using hardware including a processor, a microprocessor (e.g., to perform one or more operations or control the execution of one or more operations), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In some other cases, modification module 120 may be implemented using a combination of hardware and software. In this disclosure, operations described as being performed by modification module 120 may include operations that modification module 120 may instruct a corresponding system to perform.

[0050] Modification module 120 can be configured to obtain pipeline skeleton 102 and modify pipeline skeleton 102 to generate specific pipelines 122. For example, in some embodiments, modification module 120 can be configured to modify pipeline skeleton 102 to refine pipeline skeleton 102. For example, modification module 120 can refine pipeline skeleton 102 by determining which parts of a new dataset 106 to apply different functional blocks of pipeline skeleton 102 to. Additionally or alternatively, modification module 120 can be configured to identify the order of functional blocks included in pipeline skeleton 102 as part of refinement. In these or other embodiments, modification module 120 can be configured to refine pipeline skeleton by performing redundancy analysis on pipeline skeleton 102. Additionally or alternatively, modification module 120 can remove one or more functional blocks from pipeline skeleton 102 based on redundancy analysis. In some embodiments, modification module 120 can be configured to modify pipeline skeleton 102 to generate a refined pipeline skeleton, as described below. Figures 2 to 6 As described.

[0051] In these or other embodiments, modification module 120 may be configured to identify code snippets from an existing ML pipeline 111 that can be used to instantiate functional blocks of pipeline skeleton 102 and thus materialize pipeline skeleton 102 into a concrete pipeline 122. Alternatively, modification module 120 may be configured to determine the suitability of the identified code snippets for adapting them to be used as part of concrete pipeline 122. In these or other embodiments, modification module 120 may be configured to generate one or more candidate pipelines, which may be different concrete pipelines of pipeline skeleton 102. Each candidate pipeline may include different instantiations of the same functional blocks of pipeline skeleton 102 using different identified code snippets. Alternatively, modification module 120 may be configured to analyze candidate pipelines to determine their performance. In these or other embodiments, modification module 120 may select one of the candidate pipelines as concrete pipeline 122 based on the performance determination. In some embodiments, modification module 120 may be configured to identify, select, and implement code snippets for the generation and selection of concrete pipeline 122, as described below. Figures 7 to 12 As described.

[0052] Modification module 120 can be configured accordingly to modify pipeline skeleton 102 to generate specific pipelines 122 for use as part of a new ML project 115. These operations can improve the automation of the generation and implementation of new ML projects through computer systems, which can enhance the ability to apply machine learning to an increasing number of projects.

[0053] Without departing from the scope of this disclosure, the following may be made: Figure 1Modifications, additions, or omissions may be made. For example, environment 100 may include more or fewer elements than those shown and described in this disclosure.

[0054] Figure 2 An example set of operations 200 (“Operation Set 200”) is shown that can be executed to modify the pipe skeleton 202 of a new ML project 215 to generate a refined pipe skeleton 212. Operation Set 200 can be executed by any suitable system or device. For example, one or more operations of Operation Set 200 can be performed by… Figure 1 The modification module 120 is executed or by Figure 1 The modification module 120 guides its execution. Alternatively, the operation group 200 can be configured by, for example, Figure 12 The computing system of computing system 1202 (e.g., as pointed to by the modification module 120) is executed.

[0055] Typically, operation group 200 can be configured to perform one or more operations on pipeline skeleton 202, new dataset 206, and one or more existing ML projects 210 to generate a refined pipeline skeleton 212. In some implementations, operation group 200 may include correlation analysis 250, existing ML mapping 252, instantiation determination 254, and pipeline refinement 256 to generate a refined pipeline skeleton 212.

[0056] Pipe skeleton 202 can be similar to Figure 1 The pipeline skeleton 102 may include a set of functional blocks (referred to as "skeleton blocks") associated with the new ML project 215. The new dataset 206 may also be part of the new ML project 215 and may be analogous to... Figure 1 The new dataset 106. Existing ML project 210 can be analogous to... Figure 1 The existing ML project 110 may include an existing ML pipeline 211 and a corresponding existing dataset 209, wherein the existing ML pipeline 211 and the corresponding existing dataset 209 may be respectively similar to Figure 1 The existing ML pipeline 111 and the corresponding existing dataset 109.

[0057] Correlation analysis 250 may include operations that can be used to determine the correlation 258 of one or more function blocks. Function block correlation 258 may indicate whether a pair of function blocks is related to each other based on whether each function block in a corresponding pair applies to the same portion of the same dataset. In some implementations, correlation analysis 250 may determine function block correlation 258 based on the use of function blocks in one or more existing ML pipelines 211. Function blocks in existing ML pipelines 211 may be referred to as "existing function blocks".

[0058] In some implementations, the use of correlation analysis 250 may include determining which parts of the existing function block apply to the existing dataset 209. For example, correlation analysis 250 may include determining which columns of the existing function block apply to the existing dataset 209.

[0059] In these or other implementations, correlation analysis 250 may include determining which existing function blocks are applied to the same portion of the existing dataset 209. In these or other implementations, existing function blocks applied to the same portion may be considered relevant to each other in function block correlation 258. Conversely, different existing function blocks not identified as having been applied to the same portion may be considered unrelated to each other.

[0060] For example, both the first and second function blocks of an existing function block can be applied to a specific column of a particular existing dataset 209. In some implementations, function block relevance 258 can accordingly indicate that the first and second function blocks are a related pair. As another example, the first and third function blocks of an existing function block may never be identified as being applied to the same column of any existing dataset 209. In some implementations, function block relevance 258 can accordingly indicate that the first and third function blocks are an unrelated pair of function blocks.

[0061] In some implementations, correlation analysis 250 may be performed on multiple pairs of existing functional blocks. In these or other implementations, correlation analysis 250 may be performed on each possible pair of existing functional blocks. Alternatively or concurrently, functional block correlation 258 may include indications of all different correlations. It should be understood that an existing ML pipeline 211 may include multiple instances of the same existing functional block, such that a reference to “each pair” of existing functional blocks may not necessarily include every instance of each pair of existing functional blocks, but may refer to each pair of each different existing functional block type. In some implementations, correlation analysis 250 may include the following regarding… Figure 3A and Figure 3B One or more operations are described. As discussed further below, in some implementations, function block dependency 258 can be used in pipeline refinement 256.

[0062] An existing ML mapping 252 (“ML mapping 252”) may include operations for generating a relational mapping 260 (“relational mapping 260”). Relational mapping 260 may indicate the relationship between specific features of a dataset and the use of function blocks relative to the portions of the dataset having those features. In some implementations, ML mapping 252 may determine mapping 260 based on the use of existing function blocks of one or more existing ML pipelines 211.

[0063] In some implementations, the use determination of ML mapping 252 may include determining use information that indicates which parts of the existing function block are applied to the existing dataset 209. For example, ML mapping 252 may include determining which columns of the existing dataset 209 are applied to the existing function block. In some implementations, this information may be obtained from the same determination made regarding correlation analysis 250.

[0064] In these or other implementations, ML mapping 252 may include one or more meta-features (“dataset features”) that identify different portions of an existing dataset 209. Dataset features may include, but are not limited to, number of rows, number of features, presence of numbers, presence of missing values, presence of numbers, presence of number categories, presence of string categories, presence of text, and target type.

[0065] In some implementations, ML mapping 252 may include determining relationships between existing functional blocks and dataset features of portions of the datasets to which the existing functional blocks are applied. These relationships may be determined based on usage information and may indicate how likely a particular functional block is to be used relative to portions having specific dataset features. ML mapping 252 may generate relationship mapping 260 based on the determined relationships. For example, relationship mapping 260 may provide mappings indicating which dataset features different functional blocks correspond to, as determined from the relationships.

[0066] In some implementations, ML mapping 252 may include the following about Figure 4 One or more operations are described. As discussed further below, in some implementations, relation mapping 260 may be used in instantiation determination 254.

[0067] As discussed above, in some cases, the pipeline skeleton 202 may include a set of skeleton blocks that indicate operations to be performed for a new ML project 215, but may not indicate which parts of the new dataset 206 to apply different skeleton blocks to. Instantiation determination 254 may include operations to determine which parts (e.g., columns) of the new dataset 206 to apply skeleton blocks to. In some implementations, instantiation determination 254 may be determined by applying relation mapping 260 to skeleton blocks and the new dataset 206 based on dataset features of different parts of the new dataset 206. In some implementations, instantiation determination 254 may generate block instantiation 262 that may indicate which parts of the new dataset 206 to apply different skeleton blocks to based on the determination. In some implementations, instantiation determination 254 may include the following regarding... Figure 5 One or more operations are described. As discussed further below, in some implementations, instantiation determination 254 can be used in pipeline refinement 256.

[0068] Pipe refinement 256 may include operations related to refining the pipe skeleton 202. For example, pipe refinement 256 may include removing one or more skeleton blocks from the pipe skeleton 202. In these or other embodiments, the removal of one or more skeleton blocks may be based on redundancy analysis that may utilize functional block dependency 258. Alternatively or alternatively, the removal of one or more skeleton blocks may be based on block instantiation 262. In some embodiments, the removal of one or more skeleton blocks may include the following regarding... Figure 6 Describe one or more operations.

[0069] In these or other embodiments, pipeline refinement 256 may include determining the execution order of skeleton blocks. This order can be determined by first inferring a partial order of block sorting from the ML pipelines 211 in the existing ML project 210. For example, in some embodiments, this partial order may be represented as a graph, where there is a node for each functional block appearing in any ML pipeline 211. In these or other embodiments, the graph may include edges between nodes that indicate the execution order of corresponding functional blocks. For example, a directed edge from a first node to a second node may be included in instances where, in each ML pipeline 211 where two blocks co-occur, a first functional block corresponding to the first node appears before a second functional block corresponding to the second node. This partial order can then be used to determine the execution order of skeleton blocks (e.g., to determine the overall order on the skeleton functional blocks, such as any order consistent with the inferred partial order).

[0070] Alternatively or concurrently, pipeline refinement 256 may include annotating the pipeline skeleton 202 with one or more instantiations selected from block instantiations 262. For example, block instantiation 262 associated with the skeleton block retained in the refined pipeline skeleton 212 after pipeline refinement 256 may be indicated in the refined pipeline skeleton 212.

[0071] Operation group 200 can be configured accordingly to modify pipeline skeleton 102 to generate a refined pipeline skeleton 212. By instructing the execution order of skeleton blocks and / or removing potentially redundant or unnecessary skeleton blocks, the refined pipeline skeleton 212 can be more suitable for instantiation than pipeline skeleton 202.

[0072] Without departing from the scope of this disclosure, the following may be made: Figure 2Modifications, additions, or omissions may be made. For example, operation group 200 may include more or fewer operations than those shown and described in this disclosure. Furthermore, the order in which the operations of operation group 200 are described does not imply that they must be performed in the described order. Additionally, in some cases, the same operation may be described for different parts of operation group 200 (e.g., for the use of correlation analysis 250 and ML mapping 252), but in other cases, the same operation may be performed only once and used for different parts of operation group 200.

[0073] Figure 3A This is a flowchart of an example method 300 for determining the relevance of functional blocks according to at least one embodiment described in this disclosure. Method 300 can be performed by any suitable system, apparatus, or device. For example, Figure 1 Modify module 120 or Figure 12 The computing system 1202 (e.g., as pointed to by the modification module) can perform one or more operations associated with method 300. Furthermore, as described above, in some embodiments, one or more operations of method 300 can be targeted at... Figure 2 The correlation analysis 250 is performed. Although illustrated using discrete blocks, the steps and operations associated with one or more blocks of method 300 can be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the specific implementation.

[0074] Method 300 may include, at block 302, performing dataset and abstract syntax tree (AST) analysis on one or more pipelines and one or more corresponding datasets of one or more ML projects. For example, AST analysis may be performed on one or more existing pipelines and corresponding existing datasets of one or more existing ML projects stored in a corpus. AST analysis may include generating the corresponding AST of the existing pipeline based on its code. Dataset analysis may include identifying the names of parts of the existing dataset (e.g., the names of columns in the existing dataset). The AST may indicate which code elements are likely associated with a specific function block. For example, the AST may indicate calls to the application programming interface (API) corresponding to the function block. Furthermore, the AST may indicate which parts of the dataset are likely targets of specific operations and API calls.

[0075] Based on these instructions, dataset and AST analysis can include identifying the use of existing function blocks relative to different parts of the existing dataset. For example, it can be determined which parts of the existing dataset (e.g., which columns) can be applied to different existing function blocks. Figure 3B Example Table 350 shows a table that can indicate the use of different columns relative to the dataset for three different functional blocks, which can be determined based on the dataset and AST analysis. Figure 3B In the example, Table 350 indicates that the function block "drop" is applied to the "Year" column of the dataset. Figure 3B In the example, Table 350 also indicates that the function block "LabelEncoder" is applied to the "Publisher" and "Genre" columns of the dataset. Figure 3B In the example, Table 350 also indicates that the function block “fillna” is applied to the “Publisher” column of the dataset.

[0076] Return to Figure 3A At block 304, functional blocks that apply to the same part of the dataset can be identified. For example, regarding... Figure 3B For example, the function blocks “LabelEncoder” and “fillna” can be identified as “Publisher” applied to the same column. In some implementations, this may include features or meta-features applied to the same column.

[0077] At block 306, related functional blocks can be identified based on the identification performed at block 304. For example, functional blocks applied to the same part can be identified as related to each other. Figure 3B The function blocks “fillna” and “LabelEncoder” can be identified as a related pair based on the fact that both are applied to the “Publisher” column.

[0078] At block 308, unrelated functional blocks can also be identified based on the identification performed at block 304. For example, functional blocks that were not identified as applying to the same part can be identified as unrelated to each other. Figure 3B The function blocks “drop” and “LabelEncoder” can be identified as unrelated pairs based on the fact that they are not applied to any of the same columns.

[0079] In some implementations, the correlation analysis of blocks 306 and 308 can be performed on multiple pairs of existing functional blocks. In these or other implementations, the correlation analysis of blocks 306 and 308 can be performed on each possible pair of existing functional blocks. Furthermore, it can be performed on... Figure 2 Each ML pipeline 211 and dataset 209 performs such an analysis and aggregates the results across all pipelines such that if a pair of blocks is considered relevant in one or more pipelines where the blocks co-occur, then the pair of blocks is considered relevant; otherwise, if a pair of blocks is considered irrelevant in one or more pipelines where the blocks co-occur, then the pair of blocks is considered irrelevant.

[0080] Method 300 may be modified, added to, or omitted without departing from the scope of this disclosure. For example, some operations of method 300 may be implemented in a different order. Alternatively, two or more operations may be performed simultaneously. Furthermore, the operations and actions outlined are provided as examples only, and some operations and actions may be optional, combined into fewer operations and actions, or extended into additional operations and actions without departing from the disclosed embodiments.

[0081] Figure 4 This is a flowchart of an example method 400 for determining the mapping of relationships between functional blocks and dataset features according to at least one embodiment described in this disclosure. Method 400 can be performed by any suitable system, apparatus, or device. For example, Figure 1 Modify module 120 or Figure 12 The computational system 1202 (e.g., as pointed to by the modification module) can perform one or more operations associated with method 400. Although illustrated using discrete blocks, the steps and operations associated with one or more blocks of method 400 can be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the specific implementation.

[0082] Typically, method 400 can be configured to determine the use of features of an existing function block of an existing ML pipeline relative to a portion of an existing dataset. This use can be used to determine a mapping between function blocks and dataset features, indicating which dataset features a function block may correspond to. Furthermore, as described above, in some embodiments, one or more operations of method 400 may be targeted at… Figure 2 The ML mapping 252 is used to execute.

[0083] Method 400 may include block 402, where dataset features may be obtained. For example, in some embodiments, one or more existing datasets from existing ML projects may be obtained. In these or other embodiments, one or more features of the existing datasets may be obtained. For example, one or more meta-features of the existing datasets may be obtained. In these or other embodiments, different parts of the existing datasets (e.g., different columns) may have different features. In these or other embodiments, the identification of dataset features may further include identifying which parts have which features. In these or other embodiments, the identification of different dataset features may be based on semantic labels that can be applied to different parts with different dataset features. Semantic labels may indicate corresponding dataset features for corresponding parts.

[0084] In some implementations, method 400 may include block 404, where the use of a function block relative to a dataset feature may be determined. For example, in some implementations, determining the use may include determining the number of times a corresponding existing function block is used relative to different portions having the corresponding dataset features (also referred to as the "feature block occurrence rate"). For example, the number of times a particular function block is used relative to a portion having a given dataset feature may be determined.

[0085] As another example, determining usage can include determining the frequency of use of a function block relative to a portion of the dataset having the characteristics of that dataset. For example, the number of times a particular function block is used relative to a portion of the dataset having the characteristics of a given dataset can be compared to the total number of dataset portions having the characteristics of the given dataset to determine the frequency of use (e.g., percentage of use) of a particular function block relative to a dataset portion having the characteristics of the given dataset.

[0086] In some implementations, determining usage may include determining one or more conditional probabilities based on usage. A conditional probability may indicate the likelihood that a corresponding function block can be applied to a portion of a dataset having specific dataset characteristics. For example, a first conditional probability may be determined for a specific function block relative to a first dataset characteristic. The first conditional probability may be determined based on the usage of the specific function block relative to a portion of a dataset having the first dataset characteristics, and may indicate how likely the specific function block is to be used in a portion of a dataset having the first dataset characteristics. A second conditional probability may also be determined for a specific function block relative to a second dataset characteristic. In some implementations, different conditional probabilities may be determined for each function block and each dataset characteristic.

[0087] In some implementations, determining usage information can be based on AST and dataset analysis, such as the above regarding... Figure 3A As described. Alternatively, the identified dataset features can be used to determine the features of parts of the application functional blocks.

[0088] At block 406, the mapping between dataset features and function blocks can be determined based on the determined usage. For example, in response to one or more usage factors for a particular function block and a given dataset feature satisfying a threshold, the particular function block and the given dataset feature can be mapped to correspondences with each other.

[0089] For example, a first functional block can be determined as having used a first number relative to a portion of the dataset having features of a first dataset. Alternatively, a first functional block can be determined as having used a second number relative to a portion of the dataset having features of a second dataset. In these or other embodiments, the first number may satisfy an occurrence rate threshold, but the second number may not. In these or other embodiments, the first functional block may be mapped to features of the first dataset rather than features of the second dataset.

[0090] As another example, the second functional block can be determined to have a first conditional probability relative to features of the first dataset and can be determined to have a second conditional probability relative to features of the second dataset. In some implementations, the second conditional probability may satisfy a probability threshold, but the first conditional probability may not. In these or other implementations, the second functional block may be mapped to features of the second dataset instead of features of the first dataset.

[0091] In these or other implementations, the mapping can indicate the determined correspondence, independent of the threshold. For example, in some implementations, the mapping can indicate the conditional probability, occurrence rate, and / or frequency of use of each functional block relative to each distinct feature.

[0092] Method 400 may be modified, added to, or omitted without departing from the scope of this disclosure. For example, some operations of method 400 may be implemented in a different order. Alternatively, two or more operations may be performed simultaneously. Furthermore, the operations and actions outlined are provided as examples only, and some operations and actions may be optional, combined into fewer operations and actions, or extended into additional operations and actions without departing from the disclosed embodiments.

[0093] Figure 5 This is a flowchart of an example method 500 for determining the instantiation of a pipe skeleton block according to at least one embodiment described in this disclosure. Method 500 can be performed by any suitable system, apparatus, or device. For example, Figure 1 Modify module 120 or Figure 12 The computing system 1200 (e.g., as pointed to by the modification module) can perform one or more operations associated with method 500. Although illustrated using discrete blocks, the steps and operations associated with one or more blocks of method 500 can be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the specific implementation. Furthermore, as described above, in some embodiments, one or more operations of method 500 can be targeted at… Figure 2 The instantiation is determined to be 254 for execution.

[0094] Method 500 may include block 502, where a portion of the dataset of the new dataset can be selected. For example, columns of the new dataset can be selected. The new dataset may be a dataset for which a pipeline skeleton has already been generated.

[0095] At block 504, one or more dataset features of the selected portion can be obtained. For example, one or more meta-features of the selected portion can be determined.

[0096] At block 506, functional blocks (“skeleton blocks”) corresponding to the dataset features of the selected portion of the pipeline skeleton can be identified. In some implementations, this can be based on relational mapping, for example... Figure 2 Relationship mapping 260 or about Figure 4 A relational mapping is described to identify the corresponding skeleton blocks. For example, the selected portion may have features of a first dataset that are indicated in the relational mapping to correspond to a first functional block. Skeleton blocks of the pipeline skeleton that are identical to the first functional block (e.g., have the same function) can therefore be identified as corresponding to the selected portion of the new dataset.

[0097] As another example, a relational mapping can indicate the frequency of use, conditional probability, and / or occurrence rate of different functional blocks relative to different dataset features. In these or other implementations, the correspondence can be based on dataset features of the selected portion corresponding to the functional block according to a specific threshold. For example, a relational mapping can indicate that a second functional block can have a conditional probability relative to a second dataset feature of the selected portion. In these or other implementations, the skeleton block corresponding to the second functional block can be mapped to the selected portion accordingly. In contrast, the skeleton block corresponding to a third functional block having a conditional probability relative to the second dataset feature that does not meet a probability threshold may not be mapped to the selected portion.

[0098] At block 508, one or more block instantiations can be determined for the selected portion. As described above, block instantiation can indicate which skeleton blocks are applied to the selected portion. In some embodiments, block instantiation can be determined based on the correspondence determined at block 506. For example, the correspondence determined at block 506 can indicate that a first skeleton block and a second skeleton block correspond to the selected portion. Therefore, a first block instantiation indicating that the first skeleton block will be applied to the selected portion can be determined. Additionally, a second block instantiation indicating that the second skeleton block will be applied to the selected portion can also be determined.

[0099] In some implementations, method 500 can be executed for multiple parts of the new dataset. In these or other implementations, method 500 can be executed for each distinct part of the new dataset (e.g., each column). Thus, in some implementations, all distinct parts of the new dataset can be mapped to one or more skeleton blocks through block instantiation.

[0100] Method 500 may be modified, added to, or omitted without departing from the scope of this disclosure. For example, some operations of method 500 may be implemented in a different order. Alternatively, two or more operations may be performed simultaneously. Furthermore, the operations and actions outlined are provided as examples only, and some operations and actions may be optional, combined into fewer operations and actions, or extended into additional operations and actions without departing from the disclosed embodiments.

[0101] Figure 6 This is a flowchart of an example method 600 for refining a pipe skeleton into a refined skeleton according to at least one embodiment described in this disclosure. Method 600 can be performed by any suitable system, apparatus, or device. For example, Figure 1 Modify module 120 or Figure 12 The computational system 1202 (e.g., as pointed to by the modification module) can perform one or more operations associated with method 600. Although illustrated using discrete blocks, the steps and operations associated with one or more blocks of method 600 can be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the specific implementation.

[0102] Typically, method 600 can be configured to remove one or more skeleton blocks from the pipeline skeleton. Alternatively or additionally, method 600 can be configured to determine the execution order of the skeleton blocks. In these or other embodiments, method 600 may include annotating the pipeline skeleton with one or more instantiations selected from the block instantiations determined at method 500. Furthermore, as described above, in some embodiments, one or more operations of method 600 may be targeted at… Figure 2 The pipeline refinement is carried out in 256 steps.

[0103] Method 600 may include block 602, where a functional block (“skeleton block”) of the pipeline skeleton can be identified. At block 604, a block instantiation, such as that determined using block 500, can be obtained.

[0104] At block 606, one or more skeleton blocks can be removed from the pipeline skeleton based on the block instantiation. For example, in some implementations, skeleton blocks not included in any block instantiation can be removed. In these or other implementations, all skeleton blocks not included in the block instantiation can be removed.

[0105] At block 608, function block dependencies can be obtained. For example, function block dependencies determined based on method 300 can be obtained. Alternatively or additionally, at block 608, usage information associated with existing function blocks of one or more existing ML pipelines in one or more existing ML projects can be obtained. For example, usage information may be similar to or analogous to information about... Figure 4 The usage information is determined by block 404 of method 400. For example, the usage information may include the occurrence rate of a function block relative to a feature of the corresponding dataset, the frequency of use relative to a feature of the corresponding dataset, and / or the conditional probability relative to a feature of the corresponding dataset.

[0106] At block 610, one or more skeleton blocks can be removed. Removal allows one or more feature blocks representing duplicate functions applied to the same part of a new dataset to be removed.

[0107] In some implementations, removal can be based on block instantiation and relevance, as well as usage information. For example, using function block relevance information, the relevance of a skeleton block can be determined by matching it with function blocks indicated in the relevance information (e.g., based on the same function, the same name, etc.). After matching, the relevance indicated in the relevance information can be applied to the skeleton block based on the relevance of the function blocks identified as matching the skeleton block.

[0108] Alternatively or concurrently, skeleton blocks mapped to the same portion of the new dataset can be identified from block instantiation. In these or other implementations, pairs of skeleton blocks mapped to the same portion (“mapping pairs”) can be identified as either unrelated or related to each other using the determined relevance information of the skeleton blocks. In response to a mapped pair of skeleton blocks being unrelated to each other, one of the skeleton blocks in the mapping pair can be removed.

[0109] In some implementations, removal can be based on usage information. For example, skeleton blocks with mapping pairs having a low conditional probability can be removed. As another example, skeleton blocks with mapping pairs having a low occurrence count or low usage frequency can be removed.

[0110] Method 600 may be modified, added to, or omitted without departing from the scope of this disclosure. For example, some operations of method 600 may be implemented in a different order. Alternatively, two or more operations may be performed simultaneously. Furthermore, the operations and actions outlined are provided as examples only, and some operations and actions may be optional, combined into fewer operations and actions, or extended into additional operations and actions without departing from the disclosed embodiments. For example, blocks 604 and / or 606 may be omitted entirely.

[0111] Figure 7 Example operation groups 700 (“Operation Group 700”) that can be executed to instantiate pipeline skeleton 702 according to one or more embodiments of this disclosure are shown. Operation group 700 can be executed by any suitable system or device. For example, one or more operations of operation group 700 can be performed by… Figure 1 The modification module 120 is executed or by Figure 1 The modified module 120 guides its execution. Alternatively, the operation group 700 can be controlled by, for example... Figure 12 The computing system of computing system 1202 (e.g., as pointed to by the modification module 120) is executed.

[0112] Operation group 700 may include one or more operations performed on pipeline skeleton 702, pipeline skeleton model 704, one or more existing ML projects 710, and / or new datasets 706 to instantiate pipeline skeleton 702 into concrete pipeline 732. In some embodiments, operation group 700 may also include code snippet identification 720, adaptive analysis 722, candidate pipeline generation 724, and pipeline analysis 726 to instantiate pipeline skeleton 702 into concrete pipeline 732.

[0113] The pipeline skeleton 702 may include a set of functional blocks (referred to as "skeleton blocks") associated with a new ML project. In some implementations, the pipeline skeleton 702 may be analogous to... Figure 1 Pipe skeleton 102 or Figure 2 The pipe skeleton 202. Alternatively or concurrently, the pipe skeleton 702 may be similar to... Figure 2 A refined pipe skeleton 212. In these or other embodiments, the pipe skeleton 702 may include one or more block instantiations 718. Block instantiations 718 may be similar to... Figure 2 Block instantiation 262, because block instantiation 718 can indicate which skeleton block of pipeline skeleton 702 should be applied to which parts of the new dataset 706 (e.g., to which column).

[0114] The new dataset 706 can also be part of a new ML project and can be similar to... Figure 1 The new dataset is 106. Existing ML project 710 can be considered similar. Figure 1 The existing ML project 110 may include an existing ML pipeline 711 and a corresponding existing dataset 709, wherein the existing ML pipeline 711 and the corresponding existing dataset 709 may be respectively similar to Figure 1 The existing ML pipeline 111 and the corresponding existing dataset 109. In some implementations, the pipeline skeleton model 704 can be configured to generate a pipeline skeleton 702. The pipeline skeleton model 704 can be analogous to... Figure 1The pipe skeleton model 104. The specific pipe 732 can be analogous to... Figure 1 The specific pipeline is 122.

[0115] Code snippet identification may include operations that can be used to identify one or more code snippets 728. Code snippets 728 may include one or more existing code snippets from an existing ML pipeline 711. Existing code snippets that can be identified as code snippets 728 may be identified as corresponding skeleton blocks that can potentially be used to instantiate pipeline skeleton 702.

[0116] In some implementations, code snippet 728 can be identified based on the similarity between the existing dataset 709 and the new dataset 706 to which code snippet 728 corresponds. The similarity can be determined based on the similarity between a meta-feature of the existing dataset 709 and a meta-feature of the new dataset 706. In some implementations, the identification of code snippet 728 based on the determined similarity may include the following... Figure 8 Describe one or more operations.

[0117] In these or other embodiments, code snippet 728 may be identified based on analysis of the generation of pipeline skeleton 702 via pipeline skeleton model 704. For example, it may be determined which training data of pipeline skeleton model 704 is used to determine which functional blocks to include in pipeline skeleton 702. In these or other embodiments, the identified training data may have been obtained from an existing ML project 710. For example, the identified training data may exemplify the correlation between specific features of an existing dataset and the presence of specific existing functional blocks in the pipeline, which may cause pipeline skeleton model 704 to include specific functional blocks in pipeline skeleton 702 predicted for a new dataset 706. In these or other embodiments, in some cases, the identified training data may therefore represent the most appropriate instantiation of functional blocks in the context of the new dataset 706. Additionally or alternatively, the identified training data may include code snippets that instantiate existing functional blocks of the identified training data, or may be used to identify code snippets that instantiate existing functional blocks of the identified training data. Code snippets associated with the identified training data may be used to instantiate pipeline skeleton 702. In some implementations, the identification of code snippet 728 based on the training data used to generate pipeline skeleton 702 may include the following: Figure 9 Describe one or more operations.

[0118] Adaptability analysis 722 may include operations relating to determining how suitable code segment 728 is for the implementation of pipeline skeleton 702. Adaptability analysis 722 may include determining element adaptability of code segment 728 based on program elements of code segment 728. Alternatively or additionally, adaptation analysis 722 may include determining data flow adaptability of code segment 728 based on data flow of code segment 728. In these or other embodiments, adaptation analysis 722 may include determining cardinality adaptability of code segment 728 based on cardinality compatibility of the corresponding code segment 728. In some embodiments, adaptation analysis 722 may include determining the corresponding overall adaptability of the corresponding code segment 728 based on a combination of two or more of element adaptability, data flow adaptability, or cardinality adaptability of the corresponding code segment 728.

[0119] In some embodiments, adaptive analysis may output enhanced code snippet information 730 (“enhanced information 730”) regarding code snippet 728. Enhanced information 730 may include a corresponding adaptive determination for the corresponding code snippet 728. In these or other embodiments, enhanced information 730 may include code snippet 728. In some embodiments, adaptive analysis 722 may include the following regarding… Figure 10 Describe one or more operations.

[0120] Alternatively or concurrently, the enhanced information 730 may include the ranking of code snippets 728 relative to each other. For example, different code snippets may be potential candidates for instantiation of the same skeleton block. In some implementations, different code snippets may be ranked relative to each other with respect to instantiation of the same skeleton block. In some implementations, different code snippets may be ranked, for example, with respect to the following... Figure 8 , Figure 9 and / or Figure 10 As described.

[0121] Candidate pipeline generation 724 may include operations that can generate one or more candidate pipelines 734 based on the enhanced information 730. Each candidate pipeline 734 may be a specific instantiation of a pipeline skeleton 702 using a selected set of code snippets 728. In some embodiments, code snippets 728 may be selected based on adaptive information included in the enhanced information 730. In these or other embodiments, code snippets 728 may be selected based on a ranking that may be included in the enhanced information 730. In some embodiments, candidate pipeline generation 724 may include the following regarding... Figure 11 Describe one or more operations.

[0122] Pipeline analysis 726 may include operations that analyze candidate pipelines 734 to select one of them as a specific pipeline 732. For example, in some embodiments, each candidate pipeline 734 may be applied to a new dataset 706 to determine the performance level of the corresponding candidate pipeline. In these or other embodiments, a particular candidate pipeline 734 may be selected as a specific pipeline 732 based on the determined performance level. In some embodiments, any suitable technique may be used to perform pipeline analysis 726. Additionally or alternatively, in some embodiments, the new dataset 706 may be relatively large and data sampling (e.g., stratified data sampling) may be used to prune the new dataset 706 to reduce the amount of data used to analyze the candidate pipelines 734.

[0123] Without departing from the scope of this disclosure, the following may be made: Figure 7 Modifications, additions, or omissions may be made. For example, operation group 700 may include more or fewer operations than those shown and described in this disclosure. Furthermore, the order in which the operations of operation group 700 are described does not imply that the operations must be performed in the described order. Additionally, in some cases, the same operation may be described for different parts of operation group 700, but in other cases, the same operation may be performed only once and used for different parts of operation group 700.

[0124] Although Figure 7 One method for instantiating the pipeline skeleton 702 has been shown, but it will be appreciated that many other possible methods exist for instantiating the pipeline skeleton 702. For example, instead of using code snippets and determining the suitability of the code snippets, a database query can be used where the functionality of each discrete functional block can be implemented via a template associated with the functionality of the functional block. In some implementations, such a template can be prepared manually by a user such as a data scientist or programmer.

[0125] Figure 8 This is a flowchart of an example method 800 for obtaining a code snippet for instantiating a pipeline skeleton according to at least one embodiment described in this disclosure. Method 800 can be performed by any suitable system, apparatus, or device. For example, Figure 1 Modify module 120 or Figure 12 The computing system 1202 (e.g., as pointed to by the modification module) can perform one or more operations associated with method 800. Although illustrated using discrete blocks, the steps and operations associated with one or more blocks of method 800 can be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the specific implementation. Furthermore, as described above, in some embodiments, one or more operations of method 800 can be targeted at… Figure 7 The code snippet is identified as 720 and executed.

[0126] Method 800 may include block 802, where information related to the new ML project can be obtained. The new ML information may include the new dataset for the new ML project and / or the pipeline skeleton for the new ML project. For example, the new ML information may include... Figure 7 Pipe skeleton 702 and / or Figure 7 The new dataset 706.

[0127] In these or other implementations, information relating to one or more existing ML projects can be obtained at block 802. Existing ML information may include one or more existing ML projects and corresponding information, such as existing ML project 710. For example, existing ML information may include existing pipelines and / or existing datasets of existing ML projects. Alternatively or concurrently, existing ML information may include code from existing ML pipelines. In these or other implementations, existing ML information may include usage information indicating which parts of the existing dataset can be applied to different existing functional blocks and corresponding existing code snippets. In these or other implementations, usage information may be obtained based on existing ML information, such as the information above regarding... Figure 3A Method 300 is described in block 302.

[0128] At block 804, one or more existing functional blocks of an existing ML pipeline can be identified. In some implementations, existing functional blocks can be identified based on functional blocks (“skeleton blocks”) of the pipeline skeleton of a new ML project. For example, in some implementations, existing functional blocks can be identified based on being identical to skeleton blocks (e.g., based on having the same name and / or function).

[0129] At block 806, one or more existing datasets of an existing ML project can be identified based on the identified existing functional blocks. For example, existing datasets that apply existing functional blocks can be identified. In some implementations, usage information included in the existing ML project information can be used to identify existing datasets.

[0130] At block 808, the corresponding similarity between the new dataset and each identified existing dataset can be determined. In some implementations, similarity can be determined based on a comparison between one or more obtained meta-features of the existing dataset and one or more obtained meta-features of the new dataset. For example, meta-features may include the number of rows in the dataset, the number of columns in the dataset, and / or the column types of the columns in the dataset. In these or other implementations, a corresponding similarity score can be determined based on how similar the different meta-features of the new dataset are to their corresponding meta-features in the respective existing datasets. For example, any suitable distance metric can be used to determine the similarity score. In these or other implementations, each existing dataset can be given a similarity ranking relative to other existing datasets regarding its similarity to the new dataset. For example, existing datasets can be ranked such that the most similar existing dataset (as indicated by similarity analysis) is ranked highest. In some implementations, code snippets themselves can be ranked based on their ranking in their corresponding existing datasets.

[0131] At block 810, one or more existing code snippets can be identified and selected based on existing dataset similarity determination. For example, existing code snippets obtained from a pipeline used for the highest-ranking existing dataset can be identified. In these or other embodiments, existing code snippets applied to existing datasets that meet a similarity threshold for a new dataset can be identified. In these or other embodiments, existing code snippets can be those instantiated from existing function blocks identified at block 804, which are also associated with the highest-ranking existing dataset. In these or other embodiments, the identified code snippets can be identified based on a code snippet ranking that may correspond to an existing dataset ranking. In some embodiments, the ranking of a code snippet can include enhanced code snippet information, such as... Figure 7 The enhanced code snippet information is in 730.

[0132] In some implementations, the identified existing code snippets can be identified as potential instantiations of skeleton blocks of the pipeline skeleton. For example, in some implementations, the identified existing code snippets may be... Figure 7 Code snippet 728.

[0133] Method 800 may be modified, added to, or omitted without departing from the scope of this disclosure. For example, some operations of method 800 may be implemented in a different order. Alternatively, two or more operations may be performed simultaneously. Furthermore, the operations and actions outlined are provided as examples only, and some operations and actions may be optional, combined into fewer operations and actions, or extended into additional operations and actions without departing from the disclosed embodiments.

[0134] Figure 9 This is a flowchart of an example method 900 for obtaining a code snippet for instantiating a pipeline skeleton according to at least one embodiment described in this disclosure. Method 900 can be performed by any suitable system, apparatus, or device. For example, Figure 1 Modify module 120 or Figure 12 The computing system 1202 (e.g., as pointed to by the modification module) can perform one or more operations associated with method 900. Although illustrated using discrete blocks, the steps and operations associated with one or more blocks of method 900 can be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the specific implementation. Furthermore, as described above, in some embodiments, one or more operations of method 900 can be targeted at… Figure 7 The code snippet is identified as 720 and executed.

[0135] Method 900 may include block 902, where a pipeline skeleton model can be obtained. As described above, the pipeline skeleton model can be an ML model used to generate a pipeline skeleton that can identify code snippets targeting it. In some embodiments, the pipeline skeleton model may have been previously generated. In these or other embodiments, the pipeline skeleton model can be generated at block 902 as part of obtaining the pipeline skeleton model. For example, the pipeline skeleton model can be generated by training a multivariate multivalued classifier as described above. In these or other embodiments, the pipeline skeleton model can be used at block 902 to generate the pipeline skeleton.

[0136] At block 904, training data used by the pipeline skeleton model to generate the pipeline skeleton can be identified. In some embodiments, identifying the training data may include identifying meta-features of new ML projects (e.g., meta-features of new datasets and / or corresponding new tasks), which are used by the pipeline skeleton model to predict functional blocks included in the pipeline skeleton. In these or other embodiments, meta-features may be identified on a functional block-by-block basis, such that meta-features used to determine each individual skeleton block of the pipeline skeleton can be identified.

[0137] In some implementations, the identification of meta-features used in making predictions can be based on one or more "white-box" techniques, where the structure of the pipeline skeleton model is known. For example, for an instance where the pipeline skeleton model is based on a family of decision trees, paths used to reach a specific decision for a specific functional block included in the pipeline skeleton can be identified. In these or other implementations, each path used to reach each corresponding decision for each corresponding functional block of the pipeline skeleton can be identified. Another example of a "white-box" technique may include finding the main terms used in the linear regression model of the pipeline skeleton model. The above are merely examples of "white-box" techniques, and any other suitable white-box techniques may be used.

[0138] Alternatively, the identification of meta-features used in making predictions can be based on one or more "black box" techniques, where a particular structure of the pipeline skeleton model may be unknown or unnecessary. Such "model-agnostic" techniques can include any suitable techniques, including LIME (Locally Interpretable Model-Agnostic Interpretation) or SHAP (Shape Additive Interpretation).

[0139] In some implementations, the identified meta-features can be used to generate one or more dataset vectors (“new dataset vectors”) for a new dataset based on the pipeline skeleton model. These new dataset vectors can be vectors of the values ​​of the identified meta-features for the new dataset. In some implementations, a new dataset vector can be identified for each skeleton block.

[0140] In these or other implementations, the new dataset vector can be used to identify existing ML projects that may be used as training data and influence the generation of the pipeline skeleton. For example, a dataset vector (“existing dataset vector”) for existing ML projects can be constructed based on an existing ML dataset by computing previously identified meta-features in the context of an existing dataset, which is equivalent to constructing a new dataset vector based on a new dataset. In these or other implementations, the existing dataset vector that is closest to the new dataset vector can be identified. For example, an existing dataset vector within a threshold distance of the new dataset vector can be identified.

[0141] In some implementations, the determination of the closest existing dataset vector can be performed on different new dataset vectors corresponding to different skeleton blocks of the pipeline skeleton. In some implementations, the closest existing dataset vector can be identified by performing any suitable closest point analysis between each corresponding new dataset vector and each corresponding existing dataset vector. In some implementations, training data associated with existing ML projects corresponding to the closest existing dataset vector can be identified as training data that influences the generation of the pipeline skeleton.

[0142] At block 906, one or more code snippets can be identified based on the training data identified at block 904. For example, in some implementations, existing ML pipelines of existing ML projects associated with the training data (e.g., included in or indicated by the training data) can be identified. Furthermore, code snippets that instantiate existing functional blocks of existing ML pipelines can also be identified.

[0143] In some implementations, different code snippets from different existing ML pipelines associated with different ML projects (potentially facilitating the selection of the same functional blocks) can be ranked relative to each other. In some implementations, the ranking can be based on the distance of the new dataset vector relative to the existing dataset vector of the existing ML project corresponding to the code snippet. For example, a first code snippet might correspond to a first ML project with a first existing dataset vector, which is at a first distance from the new dataset vector. Conversely, a second code snippet might correspond to a second ML project with a second existing dataset vector, which is at a second distance from the new dataset vector, where the second distance is greater than the first distance. In this example, the first code snippet could be ranked higher than the second code snippet.

[0144] At block 908, one or more of the identified code segments can be selected. In some implementations, the selected code segments can be selected based on those code segments instantiated from existing functional blocks corresponding to the skeleton blocks of the pipeline skeleton as indicated by training data. Therefore, the selected code segments can be selected based on the training data used to determine the skeleton blocks of the pipeline skeleton. In some implementations, the selected code segments may be those code segments with a specific ranking. In some implementations, the ranking of the code segments can be included in enhanced code segment information, such as... Figure 7 Enhanced code snippet information 730.

[0145] Method 900 may be modified, added to, or omitted without departing from the scope of this disclosure. For example, some operations of method 900 may be performed in a different order. Alternatively, two or more operations may be performed simultaneously. Furthermore, the operations and actions outlined are provided as examples only, and some operations and actions may be optional, combined into fewer operations and actions, or extended into additional operations and actions without departing from the disclosed embodiments.

[0146] Figure 10 This is a flowchart of an example method 1000 for determining the adaptability of a code segment for implementation relative to a pipeline skeleton, according to at least one embodiment described in this disclosure. Method 1000 can be performed by any suitable system, apparatus, or device. For example, Figure 1Modify module 120 or Figure 12 The computing system 1202 (e.g., as pointed to by the modification module) can perform one or more operations associated with method 1000. Although illustrated using discrete blocks, the steps and operations associated with one or more blocks of method 1000 can be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the specific implementation. As described above, in some embodiments, one or more operations of method 1000 can be targeted at... Figure 7 The adaptive analysis 722 is performed. Alternatively, method 1000 can be used for different purposes. Figure 8 Method 800 and / or Figure 9 The method 900 identifies and executes code snippets.

[0147] Method 1000 may include block 1002, where a code snippet for potential instantiation identification of the pipeline skeleton can be obtained. For example, Figure 7 The code snippet 728 can be identified using 800 or method 900 in some implementations.

[0148] At block 1004, element adaptability can be determined for each corresponding code snippet. In some implementations, element adaptability can be based on program elements of the corresponding code snippet. For example, program elements can be identified and it can be determined whether a program element is a general code element or a domain-specific code element. General code elements can include those code elements that can generally be applied to multiple datasets. In contrast, domain-specific code elements can include those code elements specific to the dataset to which the corresponding code snippet is applied.

[0149] In some implementations, identifying program elements may include extracting all constants from the corresponding code snippet and identifying the constants as program elements. Alternatively, the identified constants may be compared to the names or values ​​of a dataset to which the code snippet is applied. For example, constants may be compared to column names of the dataset. Alternatively, constants may be compared to values ​​included in fields of the dataset. In response to a specific constant that matches the name or value of the dataset, its corresponding program element may be identified as a domain-specific code element. In contrast, in response to a specific constant that does not match the name or value of the dataset, its corresponding program element may be identified as a generic code element.

[0150] In these or other implementations, it can be determined whether a domain-specific code element can be mapped to a new dataset in a new ML project corresponding to the pipeline skeleton. For example, it can be determined whether the new dataset includes names or values ​​of existing datasets that can be mapped to a particular domain-specific code element. In response to the new dataset including names or values ​​of existing datasets that can be mapped to a particular domain-specific code element, the particular domain-specific code element can be considered mappable to the new dataset.

[0151] The element adaptability of a code snippet can be based on whether the snippet includes any domain-specific code elements. For example, in some implementations, a particular code snippet may be considered potentially unadaptable in response to including one or more domain-specific code elements. In these or other implementations, it can be determined whether the domain-specific code elements can be mapped to a new dataset. A particular code snippet may be considered element-adaptable in response to the domain-specific code elements being mappable to a new dataset. Conversely, a particular code snippet may be considered unadaptable in response to one or more domain-specific code elements not being mappable to a new dataset. Alternatively, a particular code snippet may be considered element-adaptable in response to having only general code elements.

[0152] At block 1006, data stream adaptability can be determined for each corresponding code segment. In some implementations, data stream adaptability can be based on input streams that can be input to the corresponding code segment and output streams that can be output by the corresponding code segment. Specifically, it can be determined whether the input originates from a data frame of the corresponding dataset and whether the output is sent to a data frame of the corresponding dataset. In response to both inputs and outputs corresponding to data frames (e.g., originating from or sent to data frames), the corresponding code segment can be considered to have data stream adaptability. In contrast, in response to one or more inputs and / or one or more outputs not corresponding to data frames, the corresponding code segment can be considered not to have data stream adaptability.

[0153] In some implementations, static analysis can be performed to determine inputs and outputs for each code snippet. Furthermore, static analysis can indicate which parts of the corresponding existing dataset can be used for input and / or to which parts of the corresponding existing dataset output can be sent. Therefore, static analysis can indicate whether the input or output corresponds to a data frame of the corresponding existing dataset.

[0154] At block 1008, cardinality fitness can be determined for each corresponding code snippet. Cardinality fitness can be based on the cardinality compatibility of the corresponding code snippet relative to the new dataset. For example, cardinality compatibility can be based on the number of parts of the corresponding code snippet that can be applied relative to the new dataset (e.g., the number of columns) relative to its corresponding existing dataset, rather than the number of parts of the corresponding code snippet that can be applied relative to the new dataset.

[0155] For example, to determine cardinality adaptability, it can be determined how much of the new dataset (e.g., columns) can be used to apply the corresponding code snippets. In some implementations, instantiation can be based on blocks that can be included in the pipeline skeleton (e.g., Figure 7 This determination is made by instantiating (718) a block. For example, a corresponding code snippet can be selected for potential instantiation of a specific functional block in the pipeline skeleton. Furthermore, block instantiation can indicate which parts of a new dataset a specific functional block should be applied to. Therefore, it can be determined how many parts of the new dataset can apply the corresponding code snippet as an instantiation of the specific functional block. The number of parts of the new dataset to which the corresponding code snippet can be applied can be referred to as the "number of new dataset parts". Additionally, it can be determined how many parts of a corresponding existing dataset should apply the corresponding code snippet. This determination can be based on any suitable analysis of the corresponding code snippet. The number of parts of the existing dataset to which the corresponding code snippet is applied can be referred to as the "number of existing dataset parts".

[0156] Cardinality fitness can be based on a comparison between the number of new dataset parts and the number of existing dataset parts. For example, a first code snippet can have a first number of new dataset parts equal to the first number of existing dataset parts. Furthermore, a second code snippet can have a second number of new dataset parts that is not equal to the second number of existing dataset parts. In this example, the first code snippet can have a higher cardinality fitness than the second code snippet.

[0157] In some implementations, in response to a mismatch between the number of new dataset portions and the number of existing dataset portions, it can be determined whether one or more transformations can be applied to the corresponding code snippet to improve cardinality fitness. For example, for a particular code snippet, in response to an existing dataset portion number of one and a new dataset portion number greater than one, the particular code snippet can be placed in a loop that is iterated multiple times to match the number of new dataset portions. In these or other implementations, code snippets that can be transformed to improve cardinality can still be considered to have lower cardinality fitness than code snippets with a matching cardinality. Alternatively or additionally, code snippets that can be transformed to improve cardinality can be considered to have higher cardinality fitness than code snippets that cannot be transformed to improve cardinality. In these or other implementations, code snippets can generally be considered to have cardinality fitness in response to a code snippet having a matching number of new dataset portions and an existing dataset portion number, or in response to a code snippet being transformable to match different dataset portion numbers.

[0158] In some implementations, method 1000 may include block 1010, where an overall fitness can be determined for each corresponding code segment. In some implementations, the overall fitness may be based on a combination of two or more of element fitness, data flow fitness, or cardinality fitness. In these or other implementations, the overall fitness may be based on a combination of all fitnesss of element fitness, data flow fitness, and cardinality fitness.

[0159] For example, in some implementations, a corresponding code segment may be considered to have overall adaptability, wherein the corresponding code segment is considered adaptable or not adaptable. In some implementations, a corresponding code segment may be considered to have overall adaptability in response to being determined to have element adaptability, data flow adaptability, and cardinality adaptability.

[0160] In these or other implementations, code snippets that are potentially instantiated from the same skeleton block, serving as the pipeline skeleton, can be ranked relative to each other and their respective fitness. For example, a first code snippet may have domain-specific code elements that can be mapped to a new dataset, making it program element fitness. Additionally, a second code snippet instantiating from the same skeleton block may have program element fitness because it may only have general code elements. Therefore, the second code snippet may rank higher than the first code snippet in terms of program element fitness. Alternatively, all else being equal, the second code snippet may rank higher than the first code snippet in terms of overall fitness. Similarly, a code snippet with cardinality fitness due to transformation may rank lower than a code snippet with cardinality fitness without transformation.

[0161] In some implementations, the determination of adaptability for different code snippets may include enhanced code snippet information, such as... Figure 7 The enhanced code snippet information 730. Alternatively, the ranking of code snippets determined based on adaptability may be included in the enhanced code snippet information.

[0162] Method 1000 may be modified, added to, or omitted without departing from the scope of this disclosure. For example, some operations of method 1000 may be implemented in a different order. Alternatively, two or more operations may be performed simultaneously. Furthermore, the operations and actions outlined are provided as examples only, and some operations and actions may be optional, combined into fewer operations and actions, or extended into additional operations and actions without departing from the disclosed embodiments.

[0163] Figure 11 This is a flowchart of an example method 1100 for generating candidate pipeline groups according to at least one embodiment described in this disclosure. Method 1100 can be performed by any suitable system, apparatus, or device. For example, Figure 1 Modify module 120 or Figure 12 The computational system 1202 (e.g., as pointed to by the modification module) can perform one or more operations associated with method 1100. Although illustrated using discrete blocks, the steps and operations associated with one or more blocks of method 1100 can be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the specific implementation.

[0164] Typically, method 1100 may include generating multiple concrete pipelines as potential instantiations of the pipeline skeleton for a new ML project. These multiple concrete pipelines can therefore be candidate pipelines for the new ML project, which can be applied to a new dataset for the new ML project. As described above, in some embodiments, one or more operations of method 1100 may be targeted at... Figure 7 724 candidate pipelines are generated for execution. Alternatively, or alternatively, information about... Figure 7 The enhanced code snippet information 730 of code snippet 728 is used to execute method 1100. Figure 7 Code snippet 728 can be used separately Figure 8 Method 800 Figure 9 Method 900 and / or Figure 10 Method 1000 is used to identify it.

[0165] Method 1100 may include block 1102, where a ranking of different code segments that can be identified for instantiation of the pipeline skeleton can be obtained. The code segment ranking may be based on individual skeleton blocks of the pipeline skeleton. For example, code segments may be grouped according to the skeleton blocks in which each code segment can be instantiated. In these or other embodiments, the code segment ranking may be for each distinct group of code segments. For example, a first group of code segments corresponding to a first skeleton block may be ranked relative to each other, and a second group of code segments corresponding to a second skeleton block may be ranked relative to each other. In some embodiments, the code segment ranking may include any suitable combination of adaptive rankings such as those described above with respect to method 1000. Additionally or alternatively, the code segment ranking may include rankings for selecting code segments, such as those described above with respect to method 800 or method 900.

[0166] At block 1104, a corresponding code snippet can be selected for each skeleton block of the pipeline skeleton. In some implementations, the code snippets can be selected based on their respective rankings within their respective groups. For example, the highest-ranking code snippet in a first code snippet group can be selected for a first skeleton block, and the highest-ranking code snippet in a second code snippet group can be selected for a second skeleton block. In these or other implementations, multiple different code snippets can be selected for each skeleton block, thereby allowing for the evaluation of more than one instantiation of each skeleton block.

[0167] In some implementations, at block 1104, the operation may also include transforming each selected code snippet in the context of the new dataset. Such transformation may include resolving any differences in variable names or object names of the code snippets (e.g., adjusting names based on program analysis).

[0168] At block 1106, the selected code snippets can be used to generate a candidate pipeline group. For example, each candidate pipeline can be a concrete pipeline instantiated from each skeleton block that includes the pipeline skeleton. Therefore, in some implementations, the selected group of code snippets can be selected as a pipeline group that can be used to generate candidate pipelines.

[0169] For example, a pipeline skeleton may include skeleton blocks A through D. Multiple additional pipeline groups can be selected to generate multiple candidate pipelines for the pipeline skeleton. For instance, a first pipeline group can be used to generate a first candidate pipeline for the pipeline skeleton, wherein the first pipeline group includes a first code snippet instantiated from skeleton block A, a second code snippet instantiated from skeleton block B, a third code snippet instantiated from skeleton block C, and a fourth code snippet instantiated from skeleton block D. Additionally, a second pipeline group can be used to generate a second candidate pipeline for the pipeline skeleton, wherein the second pipeline group includes a fifth code snippet instantiated from skeleton block A, a sixth code snippet instantiated from skeleton block B, a seventh code snippet instantiated from skeleton block C, and an eighth code snippet instantiated from skeleton block D. In this example, the first and fifth code snippets may be part of the same code snippet group corresponding to skeleton block A, the second and sixth code snippets may be part of the same code snippet group corresponding to skeleton block B, the third and seventh code snippets may be part of the same code snippet group corresponding to skeleton block C, and the fourth and eighth code snippets may be part of the same code snippet group corresponding to skeleton block D.

[0170] In some implementations, different skeleton groups can be generated for each of the different permutations of different combinations of code snippets. Therefore, in some implementations, the candidate pipeline group generated at block 1106 may include different candidate pipelines for each of the different permutations.

[0171] In some implementations, template code can be added to each candidate pipeline in a candidate pipeline group. The template code provides standard instantiations of common operations that appear in all pipelines, including reading the dataset, splitting the dataset into training and testing datasets, fitting the model from the instantiated pipeline skeleton to the training data, and evaluating the trained model on the testing data. Since such operations typically do not change functionally or syntactically from one pipeline or dataset to another, in some implementations, these parts of the pipeline instantiation can be accomplished using standard boilerplate code instantiated with appropriate parameters.

[0172] At block 1108, candidate pipe groups can be output. Each specific pipe in the candidate pipe group can be a candidate instantiation of the pipe skeleton.

[0173] Method 1100 may be modified, added to, or omitted without departing from the scope of this disclosure. For example, some operations of method 1100 may be implemented in a different order. Alternatively, two or more operations may be performed simultaneously. Furthermore, the operations and actions outlined are provided as examples only, and some operations and actions may be optional, combined into fewer operations and actions, or extended into additional operations and actions without departing from the disclosed embodiments.

[0174] Figure 12 A block diagram of an example computing system 1202 according to at least one embodiment of the present disclosure is shown. The computing system 1202 can be configured to implement or guide and modify modules (e.g., Figure 1 The modification module 120) is associated with one or more operations. The computing system 1202 may include a processor 1250, a memory 1252, and a data storage device 1254. The processor 1250, memory 1252, and data storage device 1254 may be communicatively coupled.

[0175] Typically, processor 1250 may include any suitable special-purpose or general-purpose computer, computing entity, or processing device comprising various computer hardware or software modules, and may be configured to execute instructions stored on any suitable computer-readable storage medium. For example, processor 1250 may include a microprocessor, microcontroller, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or any other digital or analog circuit system configured to interpret and / or execute program instructions and / or process data. Although in Figure 12 While shown as a single processor, processor 1250 may include any number of processors configured to perform, individually or collectively, any number of operations described in this disclosure, or to direct the execution of any number of operations described in this disclosure. Additionally, one or more processors may reside in one or more different electronic devices, such as different servers.

[0176] In some embodiments, processor 1250 may be configured to interpret and / or execute program instructions and / or process data stored in memory 1252, data storage device 1254, or both. In some embodiments, processor 1250 may retrieve program instructions from data storage device 1254 and load the program instructions into memory 1252. After the program instructions are loaded into memory 1252, processor 1250 may execute the program instructions.

[0177] For example, in some embodiments, the modification module may be included as program instructions in the data storage device 1254. The processor 1250 may retrieve the program instructions for the corresponding module from the data storage device 1254 and load the program instructions for the corresponding module into the memory 1252. After the program instructions for the corresponding module are loaded into the memory 1252, the processor 1250 may execute the program instructions, causing the computing system to perform the operations associated with the corresponding module as indicated by the instructions.

[0178] Memory 1252 and data storage device 1254 may include computer-readable storage media for carrying or having computer-executable instructions or data structures stored thereon. Such computer-readable storage media may include any available medium accessible by a general-purpose or special-purpose computer, such as processor 1250. By way of example and not limitation, such computer-readable storage media may include tangible or non-transitory computer-readable storage media, including random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), optical disc read-only memory (CD-ROM) or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory devices (e.g., solid-state memory devices), or any other storage medium that can be used to carry or store specific program code in the form of computer-executable instructions or data structures and is accessible by a general-purpose or special-purpose computer. Combinations of the above may also be included within the scope of computer-readable storage media. Computer-executable instructions may include, for example, instructions and data configured to cause processor 1250 to perform an operation or a set of operations.

[0179] Modifications, additions, or omissions may be made to the computing system 1202 without departing from the scope of this disclosure. For example, in some embodiments, the computing system 1202 may include any number of other components that may not be explicitly shown or described.

[0180] Figures 13 to 19 This involves the preparation and generation of new ML pipelines, including annotations for functional blocks included in the new ML pipelines. These methods can be performed by any suitable system, apparatus, or device. For example, Figure 1 System 100 or one or more of its components or Figure 12 The computational system 1202 can perform one or more operations associated with these methods. Although illustrated using discrete blocks, the steps and operations associated with one or more blocks of these methods can be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the specific implementation.

[0181] Without departing from the scope of this disclosure, the following may be made: Figures 13 to 19 The methods in the text can be modified, added to, or omitted. For example, Figures 13 to 19 Some operations of the method described herein may be implemented in a different order. Alternatively, two or more operations may be performed simultaneously. Furthermore, the operations and actions outlined are provided as examples only, and some operations and actions may be optional, combined into fewer operations and actions, or extended into additional operations and actions without departing from the disclosed implementation.

[0182] Figure 13 A flowchart of an example method 1300 for generating annotated ML pipelines according to one or more embodiments of the present disclosure is shown.

[0183] At block 1310, a trained ML pipeline skeleton model can be obtained. For example, the pipeline skeleton model can be similar to or equivalent to pipeline skeleton models 104 and / or 704. For example, the trained ML pipeline skeleton model can include one or more ML models trained to learn the mapping between dataset meta-features and function block semantic labels. For example, given meta-features of a new dataset, the pipeline skeleton model can use the mapping to identify function blocks corresponding to the meta-features of the new dataset, and can synthesize the pipeline skeleton accordingly.

[0184] At block 1320, one or more parameterized templates may be obtained. A parameterized template may include a text frame within which annotations can be generated. For example, a given potential functional block may include a corresponding parameterized template. In some implementations, a parameterized template may include infillable portions and static text portions that can be combined to describe the aspects and / or reasons why a given functional block is included in the generated ML pipeline.

[0185] Below are examples of parameterized templates:

[0186] Since the dataset has a {relevant_meta_feature_list}, the pipeline requires a {target_functional_block_name}. The relevant feature is: {relevant_column_list}. Terms enclosed in curly braces {} can include populateable data, which may be collected regarding the dataset, functional blocks, and / or other data specific to the generated pipeline, and this populateable data can be used to generate annotations. Italicized text can include static text that can be used to generate annotations. While examples are provided, other fields, tables, and / or text are also considered within the scope of this disclosure. For example, parameterized templates can provide annotations about the order of functional blocks, alternative functional blocks, meta-features related to functional blocks, and / or annotations or other information about functional blocks, their purpose, their function, or other information.

[0187] At block 1330, a request for generating a new ML pipeline based on a dataset can be received. For example, a user can submit a dataset and request a new ML pipeline to facilitate some prediction or analysis process to be performed on the dataset. In these and other implementations, the user can be a non-technical user, and the generation of the new ML pipeline can utilize simplified commands so that non-technical users who may not be familiar with programming ML projects can still utilize the ML pipeline and understand how the new ML pipeline is instantiated and how it can be modified.

[0188] At block 1340, a trained ML pipeline skeleton model can be used to determine the functional blocks used to populate the new ML pipeline. An example of this process can be found in [reference needed]. Figure 1 This can be described and / or performed in accordance with any other implementation of this disclosure. For example, a trained ML pipeline skeleton model can take the dataset on which the new ML pipeline is based as input. Using the dataset, the trained ML pipeline skeleton model can identify functional blocks associated with the ML pipeline.

[0189] At block 1350, the decision conditions leading to the functional block can be extracted. For example, the path along the decision tree model that leads to the preprocessing functional block can be extracted. As another example, the most relevant meta-features that help select a given model for the ML pipeline can be extracted.

[0190] In some implementations, block 1350 may include decision conditions for functional blocks not included in the ML pipeline. For example, if it is determined at block 1340 that two functional blocks include those performing redundant functions, one of the redundant functions can be removed. See also... Figure 1 and / or Figure 2Examples of such deduplication are described. In these and other implementations, decision conditions that lead to the removal of redundant functional blocks and / or the retention of other blocks in the ML pipeline can be extracted. As another example, decision conditions relating to models other than those determined to be included as functional blocks can be extracted (e.g., decision conditions relating to why these models are not included).

[0191] At block 1360, the dependencies between functional blocks can be determined. For example, one functional block may utilize data or data processing that occurs at another functional block, and those dependencies between functional blocks can be determined. See also... Figure 16 Describe such a specific example.

[0192] At block 1370, parameterized templates can be used to generate annotations for each function block. For example, for a given function block, the associated parameterized template can be identified, and the corresponding data can be inserted into the fillable portion to generate annotations. Following the example provided above for a function block that includes a function named SimpleImputer and whose missing values ​​are based on the "age" column of the dataset, the parameterized template is:

[0193] Since the dataset contains {relevant_meta_feature_list}, this pipeline requires {target_functional_block_name}. The relevant feature is: {relevant_column_list}. This can be translated into annotations for reading as follows:

[0194] Because the dataset has missing data values, SimpleImputer is needed in this pipeline. The relevant feature is: age.

[0195] At block 1380, the new ML pipeline can be instantiated using the function block and the comments generated at block 1370. For example, for each function block, the corresponding comments can be provided exactly before or exactly after the computer programming code that implements the function block. See [reference needed]. Figure 19 Examples that describe such instances in more detail.

[0196] Figure 14 A flowchart of an example method 1400 for collecting information in preparation for generating an annotated ML pipeline, according to one or more embodiments of this disclosure, is shown. For example, the operation of method 1400 can be performed before receiving a request for generating a new ML pipeline.

[0197] At block 1410, one or more parameterized templates can be created. The creation of parameterized templates may include a manual process in which the user generates a combination of static and / or populated text to provide information about a given functional block within the ML pipeline. In some implementations, the parameterized template may be drafted in a manner that provides information about: the function block (e.g., the title of the function invoked in the function block), the valid reason for the function block (e.g., the rationale for the decision or the reason why the function block is included in a given ML pipeline), the purpose of the function block (e.g., a description of what the function block accomplishes or performs), the form of the function block (e.g., what the inputs and outputs of the function block are and / or what data format is used in the inputs and / or outputs), alternative function blocks (e.g., what other options are available to replace the identified function block if the user expects to change the ML pipeline), and any ordering factors associated with the function block (e.g., if the function block utilizes data generated in an earlier function block, the relevance to the earlier function block may be included as part of the parameterized template, e.g.: {target_functional_block_name} should be executed before / after {dependent_functional_block_name}).

[0198] At block 1420, application programming interface (API) data may be collected. In some implementations, block 1420 may include fetching the API documentation for a given functional block. For example, project pages, readme.txt files, help databases, or other information sources for a given functional block may be analyzed and / or data collected regarding that functional block. In some implementations, Hypertext Markup Language (HTML) tag paths for various types of information may be identified, such as API name, summary, hyperparameter descriptions, and / or attributes. Such methods may leverage the uniform HTML structural attributes of the document. While block 1420 has been described with reference to collecting information about the API, it will be understood that block 1420 may include collecting data about functional blocks that may be implemented in a manner different from the API.

[0199] In some implementations, block 1420 may include storage of the collected information (e.g., API name, summary, hyperparameters, attribute descriptions, and / or other data). In these and other implementations, the collected information may be stored as key-value pairs in a structured data format file (e.g., a JSON file).

[0200] At block 1430, a data flow model can be constructed. For example, the flow of a dataset through function blocks in a new ML pipeline can be traced and / or tracked. In some implementations, the data flow model may include the use of a Directed Acyclic Graph (DAG), where each node represents a function block and each edge represents a data flow correlation from a source node to a destination node. After constructing the DAG, each pair of function blocks applied at least column-by-column in the sample data is collected. In these and other implementations, such a pair of function blocks can be used to generate edges within the DAG. For example, if for a given column of data in all pipelines of the sample data, the first function block in the pair is always applied before the second function block in the pair, an edge from the first function block to the second function block can be created.

[0201] At block 1440, the ML skeleton model can be trained. For example, a set of meta-features describing the dataset can be defined, such as the number of rows, the number of columns, the presence of missing values, and whether columns have numeric or string values. For function blocks that preprocess the data, a decision tree classifier can be trained on sample data to predict the probability score of the function block's presence in the ML pipeline based on the dataset's meta-features. For ML model function blocks, a logistic regression model can be trained to calculate the probability of each model being used for a given dataset. See [reference needed]. Figure 15 Examples of training ML skeleton models are described in more detail in and / or in other embodiments of this disclosure.

[0202] Figure 15 A flowchart of an example method 1500 for training a skeleton model according to one or more embodiments of the present disclosure is shown.

[0203] At block 1510, items can be selected from the training corpus. For example, a training corpus of sample ML pipelines with their associated datasets can be stored as a training corpus, and a given item can be selected from the training corpus.

[0204] At block 1520, meta-features can be extracted from the dataset and used to create the input feature vector. For example, the input feature vector can include a vector of values ​​associated with the meta-features, such as {8, 36, 0} if there are eight columns and thirty-six rows, where 0 represents no missing data. While a simple example of three meta-features is provided here, it will be understood that any number of meta-features can be included.

[0205] At block 1530, for each functional block in the ML pipeline of the item selected at block 1510, an output tuple indicating the presence of each functional block in the ML pipeline can be generated. For example, the output tuple could be a string of values, where each value corresponds to each potential functional block. As another example, each functional block could include its own output as key-value pairs that serve as the functional block name or other identifier and an indication of whether the functional block is included in the ML pipeline (e.g., as 0 or 1).

[0206] At block 1540, the input feature vectors and output tuples of blocks 1520 and 1530 can be added to the training data. In this way, the training data can include a data-based representation of the relationship between meta-features and which feature blocks were previously included in the ML pipeline to serve the respective datasets.

[0207] At block 1550, it can be determined whether there are any additional items that have not yet been analyzed and added to the training data. If there are additional items, method 1500 can return to block 1510 to select additional items for analysis. If all items have been considered (e.g., no additional items exist), method 1500 can proceed to block 1560.

[0208] At block 1560, a skeleton model can be trained using the training data. For example, the skeleton model can be trained such that it can then provide one or more meta-features with probabilities or numerical correlation scores between them and the feature blocks. In this way, the trained skeleton model can be able to receive a dataset as input and propose possible or potential feature blocks to be included in a new ML pipeline.

[0209] Figure 16 A flowchart is shown of another example method 1600 for generating annotated ML pipelines according to one or more embodiments of the present disclosure.

[0210] At block 1610, a list of functional blocks can be obtained. For example, when a dataset is provided as input, a trained ML pipeline skeleton model can generate a list of functional blocks to be included in a new ML pipeline based on the dataset. In some implementations, block 1610 may include deduplication of redundant or functionally similar functional blocks to remove blocks that perform duplicate functions. In some implementations, block 1610 may be related to... Figure 5 , Figure 6 , Figure 11 The methods are similar to or equivalent to those used in other diagrams.

[0211] At block 1620, decision conditions for the preprocessing function blocks can be extracted. For example, a respective decision tree model associated with each function block included in the list obtained at block 1610 can be obtained. Paths through the decision tree, starting from the root node and leading to the leaf nodes corresponding to the decisions that include the preprocessing function blocks in the list, can be traversed. Each decision along the decision tree, along with the associated meta-features, can be collected. The decision tree can be analyzed for each preprocessing function block.

[0212] In some implementations, block 1620 may include using the collected decision and correlation meta-features to populate at least a portion of the populateable portion of the parameterized template to generate annotations about the basis for including the preprocessing function block.

[0213] At block 1630, meta-features that influence the decision of which model feature blocks to include can be extracted. For example, an ML process (e.g., a logistic regression model) can be used to select an ML model for a new pipeline. In doing so, the ML process can assign certain weights and / or values ​​to various meta-features when selecting a model. Block 1630 can include weights for different meta-features for the selected ML model. The weights of meta-features can be normalized relative to their original meta-feature values ​​to calculate proportional meta-feature weights (e.g., how important the value of that particular meta-feature is in determining the ML model). The proportional meta-feature weights can be sorted in descending order, and a number of top meta-features can be selected to be included in the annotation explaining why the ML model was selected. In some implementations, in addition to the top K meta-features, a set number of meta-features, the number of meta-features with proportional meta-feature weights above a threshold, or any other equivalent or similar metric can be used to select the number of meta-features that influence the decision of selecting an ML model for a new ML pipeline.

[0214] In some implementations, block 1630 may include using the collected meta-features to populate at least a portion of the populateable portion of the parameterized template to generate annotations about the basis for selecting the model.

[0215] At block 1640, alternative functional blocks can be identified. For example, for a preprocessing functional block, functional blocks removed based on redundancy can be listed as alternative functional blocks. In these and other implementations, a given preprocessing functional block can be selected from multiple functional blocks performing the same or similar functions. Other preprocessing functional blocks that are not selected can be identified as alternative functional blocks. As another example, for an ML model functional block, a given ML model can be selected based on a combination of specific meta-feature values, weighted by those specific meta-feature values, and values ​​that can reflect the expected probability of success in performing the target task in the ML pipeline. A given ML model can include the highest combination of these values; for example, when selecting a given ML model, a ranked list of ML models can be generated at block 1610, and the next K ML models after the given ML model can be identified as alternative ML models to the given ML model. In some implementations, the identified alternative functional blocks can be included in the generated annotation. For example, if a user indicates a desire to adjust a given functional block (whether a preprocessing functional block or an ML model functional block) in a new ML pipeline, the annotation can provide suggestions for alternative functional blocks as identified herein.

[0216] At block 1650, the relevant functional blocks and their order can be determined. For example, a data flow model can be generated that includes functional blocks from the functional blocks included in block 1610. In some implementations, a directed acyclic graph can be constructed based on the functional blocks involved in data processing / transmission. In these and other implementations, annotations can be generated based on data correlations observed using the acyclic graph. For example, block 1650 can be similar to or equivalent to block 1430, where correlations between functional blocks can be identified.

[0217] In some implementations, block 1650 may include annotations generated based on identified relevance. For example, at least some of the fillable portions of the parameterized template may be supplemented with sequential relevance and related functional blocks. An example of such a parameterized template may include: {target_functional_block_name} should be executed before / after {dependent_functional_block_name}.

[0218] Figure 17 A flowchart is shown of an example method 1700 for generating annotations related to preprocessing function blocks in an ML pipeline, according to one or more embodiments of this disclosure.

[0219] At block 1710, the decision tree model associated with the preprocessing function block can be accessed. For example, for a given preprocessing function block included in the ML pipeline skeleton, the skeleton model may have used a model such as a decision tree model to determine that the given preprocessing function block will be included in the ML pipeline skeleton.

[0220] At block 1720, the path between the root node of the decision tree model and the leaf node corresponding to the decision including the preprocessing function block can be traversed.

[0221] At block 1730, decisions traversed through the decision tree model can be collected in terms of dataset attributes (meta-features) and the value conditions for presenting decisions along the path of block 1720. For example, for each decision along the path, the meta-feature and its associated value can be collected together with the decision considering the meta-feature and its associated value.

[0222] At block 1740, a parameterized template can be populated with decisions and attributes (e.g., meta-features) to generate annotations. For example, the populated portion of the parameterized template can be supplemented with relevant meta-features and / or their values, such that, combined with the static portion of the text, the annotation details why a given functional block is included in the ML pipeline skeleton.

[0223] Figure 18 A flowchart is shown of another example method 1800 for generating annotations relating to ML models in an ML pipeline, according to one or more embodiments of this disclosure.

[0224] At block 1810, the meta-model that predicts the probability scores for each model function block can be accessed. For example, the ML skeleton model can assign scores indicating the accuracy of each candidate model in performing the task to each candidate model based on the meta-features of the dataset associated with the ML pipeline being generated.

[0225] At block 1820, for the selected model function block, the meta-feature weights can be obtained. For example, when determining the model function block with the highest accuracy score, various meta-features can be assigned certain weights when determining the accuracy scores of different model function blocks.

[0226] At block 1830, each meta-feature weight can be normalized relative to the original meta-feature value to compute the proportional meta-feature weight. For example, the dataset may include the original value of the meta-feature (e.g., the number of columns in the dataset, the number of features, binary values ​​indicating whether the dataset has missing values, etc.), and this value can be divided by the weight to produce a normalized value.

[0227] At block 1840, meta-features can be sorted in descending order based on their proportional meta-feature weights (e.g., how much a given meta-feature contributes to determining the factors that include a given model feature block). After sorting, the top K meta-features and their values ​​can be selected to provide annotations for the selection of model feature blocks. For example, the top three or top five most influential or relevant meta-features can be presented along with their respective values, enabling the user to understand why a given model was selected instead of another. In these and other implementations, block 1840 may include supplementing the fillable portion of the parameterized template with the selected meta-features and / or their values.

[0228] Figure 19 A flowchart of another example method 1900 for generating annotated ML pipelines according to one or more embodiments of this disclosure is shown. In some embodiments, method 1900 may be performed after a new ML pipeline has already been generated based on a dataset, according to any of the methods described herein.

[0229] At block 1910, a function block can be selected from the ML pipeline. For example, a function block can be selected according to the time order or linear order in the ML pipeline, or according to any other method or order.

[0230] At block 1920, for the selected function block, the parameterized template can be instantiated by applying the collected data to the populated portion of the parameterized template. For example, the collected data can be used to instantiate the function block's name, purpose, relevance, reason for including the function block, or any other part or aspect of the annotation.

[0231] At block 1930, comments can be formatted according to the format of the ML pipeline. For example, comments can be presented as text comments in computer programming code (e.g., Python program code), markdown cells in a Jupyter notebook, or any other comment or text feature accompanying a computer program or code. In some implementations, comments can be formatted based on the format of the ML pipeline. For example, if the user requires and / or the system is designed to output the ML pipeline as a Jupyter notebook, comments can be formatted as markdown cells in a Jupyter notebook. As another example, if the ML pipeline outputs Python programming code, comments can be included as text comments in the computer programming code.

[0232] At block 1940, comments can be placed before the code associated with the function block. For example, a comment can be placed as a text comment immediately before the code snippet associated with the function block in the ML pipeline.

[0233] At block 1950, it can be determined whether there are any additional functional blocks in the ML pipeline that have not generated annotations. If there are additional functional blocks, method 1900 can return to block 1910 to select another functional block in the ML pipeline. If there are no additional functional blocks (e.g., all functional blocks have annotations), method 1900 can continue to block 1960.

[0234] At block 1960, the resulting ML pipeline with function blocks and associated annotations can be stored. For example, an ML pipeline including associated annotations can be stored as a structured data structure, such as a JSON or XML file. As another example, an ML pipeline including associated annotations can be stored as a Jupyter notebook. As yet another example, an ML pipeline including associated annotations can output as a regular computer programming script, such as a Python script.

[0235] As described above, the embodiments described in this disclosure may include the use of a dedicated or general-purpose computer comprising various computer hardware or software modules, as discussed in more detail below. Furthermore, as described above, the embodiments described in this disclosure may be implemented using a computer-readable medium for carrying or having computer-executable instructions or data structures stored thereon.

[0236] As used herein, the terms "module" or "component" may refer to a specific hardware implementation configured to perform the actions of a module or component and / or a software object or software routine that may be stored on and / or executed by the general-purpose hardware of a computing system (e.g., a computer-readable medium, processing device, etc.). In some embodiments, the different components, modules, engines, and services described herein may be implemented as objects or processes (e.g., as separate threads) that execute on a computing system. While some systems and methods described herein are generally described as being implemented in software (stored on and / or executed by general-purpose hardware), specific hardware implementations or combinations of software with specific hardware implementations are possible and contemplated. In this specification, a "computing entity" may be any computing system as previously defined in this disclosure or any module or combination of modules running on a computing system.

[0237] The terms used in this disclosure and particularly in the appended claims (e.g., the body of the appended claims) are generally intended to be “open-ended” terms (e.g., the term “comprising” should be interpreted as “including but not limited to”, the term “having” should be interpreted as “having at least”, the term “including” should be interpreted as “including but not limited to”, etc.).

[0238] Furthermore, if a specific number of claims is intended to be introduced, such an intention will be explicitly stated in the claims, and such an intention will not exist without such a statement. For example, to aid understanding, the appended claims may include the use of the introductory phrases “at least one” and “one or more” to introduce the claim statements. However, even when the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” or “one” (e.g., “a” and / or “one” should be interpreted as meaning “at least one” or “one or more”), the use of such phrases should not be interpreted as implying that the claim statements introduced by the indefinite article “a” or “one” limit any particular claim containing such introduced claim statements to an implementation containing only one such statement; the same applies to the use of definite articles to introduce claim statements.

[0239] Furthermore, even if the specific number of claims referred to in the introduced claims is explicitly stated, those skilled in the art will recognize that such a statement should be interpreted as meaning at least the number stated (e.g., an unmodified statement of "two statements" without other modifiers means at least two statements, or two or more statements). Moreover, in cases where idiomatic expressions such as "at least one of A, B, and C" or "one or more of A, B, and C" are used, such a construction is generally intended to include only A, only B, only C, A and B together, A and C together, B and C together, or A, B, and C together, etc. Even though the term "A and / or B" may sometimes be used to include the possibility of "A" or "B" or "A and B," this interpretation of the phrase "A or B" still applies.

[0240] Furthermore, any separator or phrase presenting two or more alternative terms, whether in the specification, claims, or drawings, should be understood to include the possibility of including one, any, or all of the terms. For example, the phrase "A or B" should be understood to include the possibility of including "A" or "B" or "A and B".

[0241] All examples and conditional language described in this disclosure are intended for educational purposes to help the reader understand this disclosure and the concepts contributed by the inventors to promote development in the art, and should be construed as not being limited to such specific examples and conditions described. Although embodiments of this disclosure have been described in detail, various changes, substitutions, and modifications may be made to embodiments of this disclosure without departing from the spirit and scope of this disclosure.

Claims

1. A method comprising: obtaining a trained machine learning (ML) pipeline skeleton model configured to predict one or more functional blocks within a new ML pipeline based on meta-features of a dataset associated with the new ML pipeline; obtaining a plurality of parameterized templates, each of the parameterized templates including one or more fillable portions and one or more static text portions that collectively describe a given functional block in natural language; receiving a request to generate the new ML pipeline based on the dataset; determining, based on the trained ML pipeline skeleton model, a plurality of functional blocks to populate the new ML pipeline; extracting decision conditions that lead to at least one of the functional blocks; generating, using the parameterized templates, annotations for at least one of the functional blocks, at least one of the fillable portions populated based on the decision conditions that lead to at least one of the functional blocks; and instantiating the new ML pipeline including the plurality of functional blocks with the generated annotations.

2. The method of claim 1, further comprising: determining correlations between the plurality of functional blocks; and generating annotations regarding an order of the functional blocks within the new ML pipeline based on the correlations. Determining correlations includes constructing a loop-free graph of the functional blocks using a dataflow model of the dataset within the new ML pipeline. Determining the functional blocks includes determining, based on meta-features of the dataset, an ML model to use from a plurality of models.

3. The method of claim 2, wherein, Generating the annotations related to the ML model includes identifying at least one of the meta-features of the dataset that most influenced the determination of the ML model from the plurality of models.

4. The method of claim 1, wherein, Generating annotations for a given functional block includes:

5. The method of claim 4, wherein, traversing a path in a decision tree model from a root to a leaf corresponding to a decision that includes the given functional block; 6. The method of claim 1, wherein, collecting decisions made along the decision tree model based on meta-features of the dataset; and populating the one or more fillable portions of a given parameterized template corresponding to the given functional block based on the collected decisions, the meta-features, or both the collected decisions and the meta-features. Populating the fillable portions includes applying data obtained from a third-party source that hosts the given functional block. Generating annotations for a given functional block includes providing a suggestion of an alternative functional block for the given functional block.

7. The method of claim 6, wherein, 9. The method of claim 8, wherein:

8. The method of claim 1, wherein, Determining the plurality of functional blocks includes removing a second functional block that performs a function duplicative of the given functional block; and The alternative functional block is the removed second functional block.

10. One or more non-transitory computer-readable media containing instructions that, when executed by one or more processors, cause a system to perform operations comprising: obtaining a trained machine learning (ML) pipeline skeleton model configured to predict one or more functional blocks within a new ML pipeline based on meta-features of a dataset associated with the new ML pipeline; ​ ​ obtaining a plurality of parameterized templates, each of the parameterized templates including one or more fillable portions and one or more static text portions that collectively describe a given functional block in natural language structure; receiving a request to generate the new ML pipeline based on the dataset; determining a plurality of functional blocks to populate the new ML pipeline based on the trained ML pipeline skeleton model; extracting decision conditions that lead to at least one of the functional blocks; generating annotations for at least one of the functional blocks using the parameterized templates, at least one of the fillable portions populated based on the decision conditions that lead to at least one of the functional blocks; and instantiating the new ML pipeline including the plurality of functional blocks with the generated annotations.

11. The non-transitory computer-readable medium of claim 10, the operations further comprising: determining a correlation between the plurality of functional blocks; and generating annotations regarding an order of the functional blocks within the new ML pipeline based on the correlation. Determining a correlation includes constructing a loop-free graph of the functional blocks using a dataflow model of the dataset within the new ML pipeline.

12. The non-transitory computer readable medium of claim 11, wherein, Determining the functional blocks includes determining an ML model to use from a plurality of models based on meta-features of the dataset.

13. The non-transitory computer-readable medium of claim 10, wherein, Generating the annotations related to the ML model includes identifying at least one of the meta-features of the dataset that most influences the determination of the ML model from the plurality of models.

14. The non-transitory computer-readable medium of claim 13, wherein, Generating annotations for a given functional block includes:

15. The non-transitory computer-readable medium of claim 10, wherein, traversing a path in a decision tree model from a root to a leaf corresponding to a decision that includes the given functional block; collecting decisions made along the decision tree model based on meta-features of the dataset; and populating the one or more fillable portions of a given parameterized template corresponding to the given functional block based on the collected decisions, the meta-features, or both the collected decisions and the meta-features. Populating the fillable portions includes applying data obtained from a third-party source that hosts the given functional block.

16. The non-transitory computer readable medium of claim 15, wherein, Generating annotations for a given functional block includes providing a suggestion of an alternative functional block for the given functional block.

17. The non-transitory computer-readable medium of claim 10, wherein, 18. The non-transitory computer-readable medium of claim 17, wherein, determining the plurality of functional blocks includes removing a second functional block that performs a function duplicative of the given functional block; and the alternative functional block is the removed second functional block.

19. A system comprising: one or more processors; and one or more non-transitory computer-readable media containing instructions that, when executed by the one or more processors, cause the system to perform operations comprising: obtaining a trained machine learning (ML) pipeline skeleton model configured to predict one or more functional blocks within a new ML pipeline based on meta-features of a dataset associated with the new ML pipeline; ​ ​ obtaining a plurality of parameterized templates, each of the parameterized templates including one or more fillable portions and one or more static text portions that collectively describe a given functional block in natural language; receiving a request to generate the new ML pipeline based on the dataset; determining, based on the trained ML pipeline skeleton model, a plurality of functional blocks to populate the new ML pipeline; extracting decision conditions that lead to at least one of the functional blocks; generating, using the parameterized templates, annotations for at least one of the functional blocks, at least one of the fillable portions populated based on the decision conditions that lead to at least one of the functional blocks; and instantiating the new ML pipeline including the plurality of functional blocks with the generated annotations.

20. The system of claim 19, the operations further comprising: determining a correlation between the plurality of functional blocks; and generating annotations regarding an order of the functional blocks within the new ML pipeline based on the correlation.

Images