Automatic labeling of functional blocks within the pipeline of an existing machine learning project into an adaptable corpus for use in new machine learning projects

Abstract

To provide automatic labeling of function blocks in a pipeline of an existing machine learning project in corpuses applicable for use in a new machine learning project.SOLUTION: According to one or more embodiments, an operation may include a step for normalizing a machine learning (ML) pipeline of an existing ML project stored in the corpus of the existing ML project. The operation may further include a step for extracting function blocks from the normalized ML pipeline. Furthermore, the operation may include a step for assigning a label to each of the function blocks in the normalized ML project. Furthermore, the operation may include a step for indexing each of ML pipelines in the corpus on the basis of the labels assigned to the function blocks. Furthermore, the operation may include a step for utilizing the labels assigned to the function blocks in the corpus and generating a new ML pipeline for executing new ML tasks with a new dataset of the new ML project.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The embodiments discussed in this disclosure relate to the automatic labeling of function blocks in the pipeline of an existing machine learning project within a corpus that can be adapted for use in a new machine learning project. [Background technology]

[0002] Machine learning (ML) typically utilizes ML models trained with training data to make predictions that automatically become more accurate with ongoing training. ML can be used in a wide range of applications, including, but not limited to, traffic prediction, web search, online fraud detection, medical diagnosis, speech recognition, email filtering, image recognition, virtual personal assistance, and automatic translation.

[0003] As ML becomes increasingly common, there is often a shortage of ML experts (e.g., skilled data scientists) available to implement new ML projects. For example, by some estimates, the majority of data scientists currently engaged in developing new ML projects are non-experts (e.g., relatively inexperienced or novice), and only about two in five people with master's or doctoral degrees are qualified to develop increasingly complex ML projects.

[0004] Automated ML (AutoML) is the process of automating the application of ML to real-world problems. AutoML can enable non-experts to utilize ML models and techniques without first requiring them to become ML experts. AutoML has been proposed as a solution to the growing challenge of implementing new ML projects despite a shortage of ML experts. However, current AutoML solutions provide only simple and partial solutions that are insufficient to enable non-experts to fully implement new ML projects.

[0005] The subject matter claimed in this disclosure is not limited to embodiments that solve any disadvantages or that operate only in environments such as those described above. Rather, this background is only provided to illustrate one example technology area where some embodiments described in this disclosure may be practiced. Summary of the Invention

[0006] According to aspects of an embodiment, operations may include normalizing machine learning (ML) pipelines of an existing ML project stored in a corpus of the existing ML project. The operations may further include extracting function blocks from the normalized ML pipelines. Further, the operations may include assigning labels to each of the function blocks in the normalized ML pipelines. Further, the operations may include indexing each of the ML pipelines in the corpus based on the labels assigned to the function blocks. Further, the operations may include generating a new ML pipeline for performing a new ML task on a new dataset for a new ML project using the labels assigned to the function blocks in the corpus.

[0007] The object and advantages of the embodiments will be realized and achieved at least by the elements, features, and combinations particularly pointed out in the claims.

[0008] Both the foregoing general description and the following detailed description are provided by way of example and explanation, and are not limitations of the invention, as claimed. [Brief explanation of the drawings]

[0009] Exemplary embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0010] [Figure 1] FIG. 1 illustrates an exemplary environment related to automatically searching for existing ML projects and adapting to new ML projects.

[0011] [Figure 2] FIG. 1 depicts an exemplary environment related to automatically curating existing ML projects into a corpus.

[0012] [Figure 3] FIG. 1 depicts an example environment related to automatically generating a pipeline for a new ML project from a pipeline for an existing ML project stored in a corpus.

[0013] [Figure 4] FIG. 1 illustrates a block diagram of an exemplary computing system.

[0014] [Figure 5] 1 is a flowchart of an exemplary method for automatically curating existing ML projects into an adaptable corpus for use in new ML projects.

[0015] [Figure 6] 1 is a flowchart of an example method for collecting a set of ML projects from one or more repositories of ML projects based on filtering criteria.

[0016] [Figure 7] 1 is a flowchart of an example method for ensuring the viability of ML pipelines in a set of ML projects.

[0017] [Figure 8A] 1 shows example ML pipeline code for an existing ML project.

[0018] [Figure 8B] 8B illustrates exemplary cleaned ML pipeline code resulting from the cleaned exemplary ML pipeline code of FIG. 8A.

[0019] [Figure 9] 1 is a table of exemplary quality features.

[0020] [Figure 10] 1 is a table of exemplary diversity features.

[0021] [Figure 11] 1 is a flowchart of an example method for selecting a subset of ML projects from a set of ML projects based on quality features and diversity features.

[0022] [Figure 12] 1 is a flowchart of an exemplary method of auto-labeling function blocks in a pipeline of an existing ML project into a corpus that can be adapted for use in a new ML project.

[0023] [Figure 13A] 1 shows an example original ML pipeline code for an existing ML project before normalization.

[0024] [Figure 13B] 13B illustrates an example normalized ML pipeline code after normalization of the example original ML pipeline code of FIG. 13A.

[0025] [Figure 14] 1 is a flowchart of an example method for identifying idiomatic function blocks and custom function blocks.

[0026] [Figure 15] We demonstrate the use of directed graphs to extract function blocks from a normalized ML pipeline.

[0027] [Figure 16] 1 is a flowchart of an example method for assigning a label to each of the function blocks in a normalized ML pipeline.

[0028] [Figure 17]Demonstrates automatic labeling of functional blocks in an ML pipeline.

[0029] [Figure 18] 1 is a flowchart of an example method for automatically generating a pipeline for a new ML project from a pipeline for an existing ML project stored in a corpus.

[0030] [Figure 19] Shows the sequence graph and pipeline skeleton for a new ML project.

[0031] [Figure 20] 19 shows the pipeline skeleton of FIG. 19 and a table of ML pipelines that can be searched for function blocks that match the pipeline skeleton. DETAILED DESCRIPTION OF THE INVENTION

[0032] Some embodiments described in this disclosure relate to methods and systems for automatically searching for existing ML projects and adapting to new ML projects.

[0033] As ML becomes increasingly common, a shortage of available ML experts (e.g., skilled data scientists) to implement new ML projects often occurs. Various AutoML solutions (e.g., Auto-Sklearn, AutoPandas, etc.) have been proposed to solve the ever-growing challenge of implementing new ML projects despite the shortage of ML experts. However, current AutoML solutions only provide simplified and partial solutions that allow non-experts to fully implement new ML projects. Furthermore, open-source software (OSS) databases of existing ML projects (e.g., Kaggle, GitHub, etc.) have also been proposed as another solution to the challenge of implementing new ML projects by non-experts. However, it may be difficult or impossible for non-experts to find potentially useful existing ML projects in these databases. Furthermore, even if non-experts succeed in finding potentially useful existing ML projects in these databases, it may be difficult or impossible for non-experts to modify potentially useful existing ML projects to meet the new requirements of the new ML project.

[0034] In this disclosure, the term "ML project" may refer to a project that includes a dataset, an ML task defined on the dataset, and an ML pipeline (e.g., script or program code) configured to perform a sequence of operations to train an ML model for the ML task on the dataset and use the ML model for new predictions. In this disclosure, the term "computational notebook" may refer to a computational structure (e.g., Jupyter notebook) used to develop and / or represent an ML pipeline, particularly during the development phase. While embodiments disclosed herein are illustrated with an ML pipeline in the Python programming language and a computational notebook structured with Jupyter notebook, it should be understood that other embodiments may include ML pipelines written in different languages ​​and computational notebooks structured on other platforms.

[0035] According to one or more embodiments of the present disclosure, operations may be performed to automatically search for existing ML projects and adapt for new ML projects. For example, in some embodiments, a computer system may systematically support a data scientist's natural workflow by building a workflow in a "search-and-adapt" style in which the data scientist first searches for existing ML projects that provide a good starting point for building a new ML project, and then appropriately adapts the existing ML projects to build an ML pipeline for the new database and new ML tasks for the new ML project.

[0036] For example, in some embodiments, a computer system may automatically mine raw ML projects from an OSS database of existing ML projects and automatically curate new ML projects before storing them in a corpus of existing ML projects. In some embodiments, this mining and curation of existing ML projects from a large repository may result in a diverse corpus of high-quality existing ML projects that can be used in search and adaptation workflows. This curation may also include cleaning the ML pipelines of the existing ML projects (e.g., using dynamic program slicing), computing feature sets to capture the quality and diversity of each ML project, and selecting an optimal number of existing ML projects that meet these objectives.

[0037] Additionally, in some embodiments, this curation may involve operations performed to automatically identify and index function blocks in the ML pipeline of an existing ML project. Unlike traditional software programs, the ML pipeline of an ML project typically follows a well-defined workflow based on dataset properties and can be thought of as a sequence of function blocks. Accordingly, some embodiments include techniques to automatically extract and label function blocks in an ML pipeline and properly index them in a corpus so that they can be efficiently searched for synthesizing new ML pipelines for new ML tasks. More specifically, this technique may abstract the ML pipeline at an appropriate level and utilize graph-based sequence mining algorithms to extract both custom and idiomatic function blocks. Finally, each function block may be semantically labeled.

[0038] In some embodiments, upon receiving a new database and a new task for a new ML project, such as from a non-expert data scientist, the computer system may first automatically use a hierarchical approach to synthesize a function block-level pipeline skeleton for the new ML project using an ML model, and then explicitly search through a curated and labeled corpus to identify relevant existing ML projects for instantiating this pipeline skeleton. Next, the computer system may automatically select function blocks from ML pipelines of a set of relevant existing ML projects and concretize the pipeline skeleton into a new ML pipeline for the new ML project. Finally, the computer system may adapt the function blocks of the new ML pipeline so that the new ML pipeline is executable to perform the new ML task on the new dataset.

[0039] Thus, in some embodiments, a non-expert data scientist may simply formulate a new dataset and a new ML task for a new ML project. The computer system may then implement a tool-assisted interactive search and adaptation workflow to automatically generate a new ML pipeline for the new ML project, which can be immediately executed to run the new ML task on the new dataset without any modifications by the non-expert data scientist. Thus, some embodiments may empower novice data scientists to efficiently generate new, high-quality end-to-end ML pipelines for new ML projects.

[0040] According to one or more embodiments of the present disclosure, the technical field of ML project development may be improved by configuring a computer system to automatically search for and adapt existing ML projects to a new ML project, as opposed to tasking a data scientist (e.g., often a non-expert) with manually discovering and modifying potentially useful existing ML projects for the new requirements of the new ML project. Such a configuration may enable the computing system to better search for relevant existing ML projects and adapt them to the new ML project by identifying and extracting function blocks from the existing ML pipeline and automatically adapting them for use in the new ML pipeline.

[0041] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0042] 1 illustrates an example environment 100 related to automatically searching and adapting existing ML projects to a new ML project, configured in accordance with at least one embodiment described herein. The environment 100 may include an OSS ML project database 102a-102n, a curation module 114 configured to curate existing ML projects into an ML project corpus 104, a search module configured to search the ML project corpus 104 for relevant existing ML projects for the new ML project based on a new dataset 106 and new ML tasks 108 for the new ML project (e.g., provided by a data scientist 118), and an adaptation module 120 configured to synthesize and adapt function blocks from ML pipelines 111 of relevant existing ML projects 110 into a new ML pipeline 112 for the new ML project.

[0043] The OSS ML project databases 102a-102n may be large repositories of existing ML projects, where each ML project includes at least a dataset, an ML task defined on the dataset, and an ML pipeline (e.g., script or program code) configured to perform a sequence of operations to train an ML model for the ML task and use the ML model for new predictions. Some examples of large repositories of existing ML projects include, but are not limited to, Kaggle and GitHub. In some embodiments, each ML project in the OSS ML project databases 102a-102n may include a computational notebook. A computational notebook may be a computational structure used to develop and / or represent an ML pipeline, particularly during the development phase. One example of a computational notebook is a Jupyter notebook.

[0044] Each of the curation module 114, the search module 116, and the adaptation module 120 may include code and routines configured to enable a computing device to perform one or more operations. Additionally or alternatively, each of these modules may be implemented using hardware, including a processor, a microprocessor (e.g., that performs or controls the execution of one or more operations), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In some other examples, each module may be implemented using a combination of hardware and software. In this disclosure, the operations described as being performed by each of these modules may include operations that each of these modules may instruct a corresponding system to perform.

[0045] The curation module 114 may be configured to perform a series of operations on existing ML projects stored in the OSS ML project databases 102a-102n before or after storing the existing ML projects in the ML project corpus 104. For example, the curation module 114 may be configured to automatically mine raw ML projects from the OSS ML project databases 102a-102n to automatically curate the raw ML projects before or after storing them in the ML project corpus 104. The ML project corpus 104 may be a repository of existing ML projects curated from the OSS ML project databases 102a-102n. In some embodiments, the ML project corpus 104 may be a large corpus of cleaned, high-quality, indexed existing ML projects that can be utilized in an automated “search and adapt” style workflow. In this style of workflow, searching may include identifying existing ML projects that are related to new ML tasks and new datasets and that should be used as “seeds” for building new ML projects including new ML pipelines. Furthermore, in this style of workflow, adapting may include adapting related existing ML projects using an iterative composition approach to generate new ML pipelines for new ML projects.

[0046] In some embodiments, the curation module 114 may be configured to mine and curate existing ML projects so that only diverse, high-quality existing ML projects are stored in the ML project corpus 104. In some embodiments, the curation module 114 may also be configured to clean the ML pipelines of existing ML projects (e.g., using dynamic program slicing). In some embodiments, the curation module 114 may also be configured to calculate a feature set to capture the quality and diversity of each ML project and select an optimal number of existing ML projects from the OSS ML project databases 102a-102n to store in the ML project corpus 104. In some embodiments, the curation module 114 may also be configured to automatically identify and index function blocks in the ML pipelines of existing ML projects. Unlike traditional software programs, the ML pipelines of an ML project can be thought of as a sequence of function blocks, typically following a well-defined workflow based on dataset properties. Accordingly, the curation module 114 may be configured to automatically extract and label function blocks (with semantic labels such as "read data") in the ML pipeline and properly index them in the ML project corpus 104 so that they can be efficiently searched for new datasets 106 and new ML tasks 108 and efficiently synthesized into new ML pipelines 112. More specifically, the curation module 114 may be configured to abstract the ML pipeline at an appropriate level and utilize graph-based sequence mining algorithms to extract both custom and idiomatic function blocks. Finally, the curation module 114 may be configured to generate and assign semantic labels to each function block.

[0047] The search module 116 may be configured to perform a series of operations related to searching through existing ML projects stored in the ML project corpus 104. For example, the search module 116 may be configured to receive a new dataset 106 and a new ML task 108 for a new ML project, e.g., from a data scientist 118. Upon receipt, the search module 116 may be configured to first automatically synthesize a function block-level pipeline skeleton for the new ML project using an ML model using a hierarchical approach, and then explicitly search through the ML project corpus 104 to identify a related existing ML project 110 from which to instantiate this pipeline skeleton.

[0048] The adaptation module 120 may be configured to perform a series of operations related to synthesizing and adapting function blocks from the ML pipeline 111 of an associated existing ML project 110 into a new ML project 112. For example, the adaptation module 120 may be configured to automatically select function blocks from the ML pipeline 111 and concretize the pipeline skeleton into a new ML pipeline 112 for the new ML project (e.g., including a new dataset 106, a new ML task 108, and a new ML pipeline 112). Furthermore, the adaptation module 120 may be configured to adapt the function blocks of the new ML pipeline 112 so that the new ML pipeline 112 is executable to execute the new ML task 108 on the new dataset 106.

[0049] Thus, in some embodiments, a data scientist 118, who may be a non-expert, need only formulate a new dataset 106 and a new ML task 108 for a new ML project, and the curation module 114, search module 116, and adaptation module 120 can work together (e.g., by performing one or more of the methods disclosed herein) to ultimately produce a new ML pipeline 112 for the new ML project that is immediately executable to perform the new ML task 108 on the new dataset 106, without any modifications by the data scientist 118.

[0050] Modifications, additions, or omissions may be made to Figure 1 without departing from the scope of this disclosure. For example, environment 100 may include more or fewer elements than shown and described in this disclosure.

[0051] 2 illustrates an example environment 200 related to automatically curating existing ML projects into a corpus, configured in accordance with at least one embodiment described herein. Similar to the environment 100 of FIG. 1, the environment 200 may include OSS ML project databases 102a-102n, a curation module 114, and an ML project corpus 104. Further, as disclosed in the environment 200, after the data scientists 202a-202n store the existing ML projects 204 in the OSS ML project databases 102a-102n, the curation module 114 may be configured to crawl the OSS ML project databases 102a-102n to generate a set of existing ML projects 204. This set of existing ML projects 204 may then be further analyzed by the curation module 114.

[0052] While further analyzing the existing ML projects 204, the curation module 114 may be configured to filter 206 the existing ML projects 204 for quality and relevance and clean 208 the existing ML projects 204 to identify and / or remove irrelevant content. This filtering 206 and cleaning 208 may be configured to overcome various challenges in the existing ML projects 204. For example, some of the computational notebooks in some of the existing ML projects 204 may not have sufficiently high quality to build a high-quality ML project corpus. Therefore, this filtering 206 may automatically identify higher-quality computational notebooks (e.g., that use standard APIs rather than custom code, use appropriate classifiers, and have high accuracy) for inclusion in the ML project corpus 104. Furthermore, a high-quality ML project corpus should include existing ML projects with diverse computational notebooks. Therefore, this filtering 206 may automatically identify a greater diversity of computational notebooks for inclusion in the ML project corpus 104. Additionally, ML pipelines in notebooks of existing ML projects 204 can typically be noisy, such as Jupyter notebooks that may have a significant amount of extraneous code (e.g., debugging code, visualization code, and / or experimentation code) or deprecated APIs that render good quality code unexecutable. Therefore, this cleaning 208 may automatically clean the ML pipelines in the notebooks to resolve noise (e.g., extraneous code and / or deprecated APIs). Irrelevant code may be resolved by marking portions of code that do not programmatically contribute to the overall ML pipeline and thus may add noise to the overall technology. Deprecated APIs may be resolved by automatically replacing deprecated APIs with new APIs using API adaptation techniques.

[0053] Furthermore, the curation module 114 may be configured to abstract 210 the cleaned and filtered existing ML projects and generate project artifacts 212 of the existing ML projects 204 for indexing purposes. This abstraction 210 and the generated project artifacts 212 may be configured to overcome various challenges in existing ML projects 204. For example, it can be difficult to represent ML pipelines in the ML project corpus 104 to facilitate better search. Therefore, this abstraction 210 may automatically identify function blocks in the ML pipeline code and further identify mappings between specific metafeatures in the dataset and function blocks. Furthermore, it can be difficult to determine the appropriate level of abstraction to find function blocks so that they can be identified in any line of code. Therefore, this abstraction 210 may automatically identify function blocks based on the insight that ML pipelines often rely heavily on APIs, similar function blocks often include similar sets of APIs, and the structure of computational notebooks (e.g., Jupyter notebooks) can also provide important information about function blocks. Additionally, it can be difficult to extract the semantic purpose of each function block and to use the semantic labels to identify alternative implementations of a particular function. Therefore, abstraction 210 utilizes information derived from Markdown cells in computational notebooks (e.g., Jupyter notebooks), source code comments, and library API documentation to automatically generate semantic labels and provide them for later use to identify function blocks that are functionally equivalent even when the function blocks use different syntax. In this way, alternative implementations of function blocks can be identified and grouped together (e.g., this grouping may be referred to as "clustering").

[0054] Finally, before and / or after filtering 260, cleaning 208, and abstracting 210, the curation module 114 may be configured to store the curated existing ML projects 204 in the ML project corpus 104 to generate project artifacts 212 (e.g., by performing one or more of the methods disclosed herein). Thus, in some embodiments, the environment 200 may be utilized to automatically curate existing ML projects into the ML project corpus 104, allowing the existing ML projects to be later searched and adapted for new ML projects.

[0055] Modifications, additions, or omissions may be made to Figure 2 without departing from the scope of this disclosure. For example, environment 200 may include more or fewer elements than shown and described in this disclosure.

[0056] 3 is a diagram illustrating an example environment 300 related to automatically generating a pipeline for a new ML project from a pipeline for an existing ML project stored in a corpus. Similar to the environment 100 of FIG. 1, the environment 300 may include an ML project corpus 104, a new dataset 106, a new ML task 108, an associated existing ML project 110, a new ML pipeline 112, a search module 116, and an adaptation module 120. Further, as disclosed in the environment 300, after the existing ML project is stored in the ML project corpus 104, the search module 116 may be configured to receive the new dataset 106 and the new ML task 108 for the new ML project 310, for example, from a data scientist 118. The search module 116 may then be configured to synthesize a block-level pipeline skeleton 304 for the new ML task project 310 using a pipeline skeleton ML model 302 (which may have been pre-trained using training data derived from the ML project corpus 104).

[0057] The search module 116 may then be configured to generate a query 306 based on the pipeline skeleton 304 and search for related existing ML projects 110 through the curated and labeled ML project corpus 104. This query 306 may be configured to overcome various challenges. For example, formulating an effective query from a new dataset 106 and a new ML task 108 can be difficult. Thus, the query 306 may be formulated based on the insight that there is often a mapping between specific meta-features in a new dataset 106 with a new ML task 108 and a set of function blocks that an ML pipeline solution for this dataset should include. Thus, the set of function blocks included in the pipeline skeleton 304 can form the basis of the query 306.

[0058] Next, the search module 116 may be configured to search the ML project corpus 104 based on the query 306. This search may be configured to overcome various challenges. For example, it may be difficult to identify the best computational notebook among the existing ML projects in the ML project corpus 104 to adapt from among many other related computational notebooks. Therefore, the search may be organized based on the insight that there may be many computational notebooks that have the necessary functional blocks for the new ML pipeline 112. Thus, a small set of computational notebooks that have all the necessary semantic labels may be identified during the search, while ensuring quality.

[0059] Next, in some embodiments, the adaptation module 120 may be configured to perform a pipeline merge 308 of function blocks from the ML pipelines 111 of the associated existing ML project 110 to generate a new ML pipeline 212. This pipeline merge 308 may be configured to overcome various challenges. For example, merging entire computational notebooks can be difficult. Therefore, the resulting code is syntactically correct and is a suitable solution for the new dataset 106 and new ML task 108. Therefore, the pipeline merge 308 may be configured to utilize the pipeline skeleton 304 (as indicated by the arrow from the pipeline skeleton 304 to the pipeline merge 308), and program analysis may be utilized to make the code of the new ML pipeline 112 syntactically correct and executable without further modification.

[0060] Thus, in some embodiments, the data scientist 118, who may be a non-expert, need only formulate a new dataset 106 and a new ML task 108 for a new ML project, and the search module 116 and the adaptation module 120 can work together (e.g., by performing one or more of the methods disclosed herein) to ultimately generate a new ML pipeline 112 for the new ML project 310 that is immediately executable to run the new ML task 108 on the new dataset 106, in part without any further modifications by the data scientist 118.

[0061] Modifications, additions, or omissions may be made to Figure 3 without departing from the scope of this disclosure. For example, environment 300 may include more or fewer elements than shown and described in this disclosure.

[0062] 4 illustrates a block diagram of an exemplary computing system 402 in accordance with at least one embodiment of the present disclosure. The computing system 402 may be configured to perform or direct one or more operations associated with one or more modules (e.g., the curation module 114, the search module 116, or the adaptation module 120 of FIGS. 1-3, or some combination thereof). The computing system 402 may include a processor 450, a memory 452, and a data storage device 454. The processor 450, the memory 452, and the data storage device 454 may be communicatively coupled.

[0063] Generally, processor 450 may include any suitable special-purpose or general-purpose computer, computing entity, or processing device, including various computer hardware or software modules, and may be configured to execute instructions stored on any suitable computer-readable storage medium. For example, processor 450 may include a microprocessor, microcontroller, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or any other digital or analog circuitry configured to interpret and / or execute program instructions and / or process data. Although shown as a single processor in FIG. 4, processor 450 may include any number of processors configured to individually or collectively perform or direct the execution of any number of operations described in this disclosure. Furthermore, one or more of the processors may reside on one or more different electronic devices, such as different servers.

[0064] In some embodiments, processor 450 may be configured to interpret and / or execute program instructions and / or process data stored in memory 820, data storage device 454, or memory 452 and data storage device 454. In some embodiments, processor 450 may fetch program instructions from data storage device 454 and load the program instructions into memory 452. After the program instructions are loaded into memory 452, processor 450 may execute the program instructions.

[0065] For example, in some embodiments, one or more of the above-mentioned modules (e.g., curation module 114, search module 116, or adaptation module 120, or some combination thereof) may be included as program instructions in data storage 454. Processor 450 may fetch the program instructions of the corresponding module from data storage 454 and load the program instructions of the corresponding module into memory 452. After the program instructions of the corresponding module are loaded into memory 452, processor 450 may execute the program instructions, such that the computing system performs the operations associated with the corresponding module as directed by the instructions.

[0066] Memory 452 and data storage device 454 may include computer-readable storage media for carrying or having stored computer-executable instructions or data structures. Such computer-readable storage media may include any commercially available media that can be accessed by a general-purpose or special-purpose computer, such as processor 450. By way of example, and not limitation, such computer-readable storage media may include tangible or non-transitory computer-readable media including random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory devices (e.g., solid-state memory devices), or any other storage medium that can be used to carry or store specific program code in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer. Combinations of the above may also be included within the scope of computer-readable storage media. Computer-executable instructions may include, for example, instructions and data configured to cause processor 450 to perform a certain operation or group of operations.

[0067] Modifications, additions, or omissions may be made to computing system 402 without departing from the scope of the present disclosure. For example, in some embodiments, computing system 402 may include any number of other components that may not be explicitly shown or described.

[0068] 5 is a flowchart of an example method 500 for automatically curating existing ML projects into an adaptable corpus for use in new ML projects, according to at least one embodiment described in this disclosure. Method 500 may be performed by any suitable system, device, or apparatus. For example, curation module 114 of FIGS. 1-2 or (when directed by one or more modules) computing system 402 of FIG. 4 may perform one or more of the operations associated with method 500. Although shown by separate blocks, steps and operations associated with one or more of the blocks of method 500 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

[0069] In some embodiments, as shown in FIG. 2 , the method 500 may be used by the curation module 114 to filter 206 and clean 208 existing ML projects 204 before storing a subset of the existing ML projects 204 in the ML project corpus 104.

[0070] The method 500 may include, at least at block 502, collecting a set of ML projects from a repository of ML projects. In some embodiments, the collecting may be based on filtering criteria. For example, the curation module 114 may collect a set of existing ML projects 204 from the OSS ML project databases 102a-102n based on the filtering criteria. In some embodiments, the set of ML projects may be collected according to one or more operations of a method 600, which is further described below with reference to FIG. 6.

[0071] The method 500 may include ensuring the viability of ML pipelines in the set of ML projects at block 504. For example, the curation module 114 may ensure the viability of a set of existing ML projects 204. In some embodiments, the viability of the ML pipelines may be ensured according to one or more operations of a method 700, which is described further below with reference to FIG. 7. Additionally, in some embodiments, the viability of the ML pipelines may be ensured as shown in ML pipelines 800 and 850 in FIGS. 8A and 8B, which are described further below.

[0072] The method 500 may include identifying irrelevant portions of the ML pipeline in the set of ML projects at block 506. For example, the curation module 114 may identify and annotate irrelevant portions of the set of existing ML projects 204. In some embodiments, the irrelevant portions of the ML pipeline may be identified as shown in ML pipelines 800 and 850 in FIGS. 8A and 8B, which are described further below.

[0073] The method 500 may include collecting quality features for the set of ML projects at block 508. For example, the curation module 114 may generate quality features for the set of existing ML projects 204. In some embodiments, the quality features may be generated as shown in table 900 of FIG. 9, which is described further below.

[0074] The method 500 may include generating diversity features for the set of ML projects at block 510. For example, the curation module 114 may generate diversity features for the set of existing ML projects 204. In some embodiments, the diversity features may be generated as shown in table 1000 of FIG. 10, which is described further below.

[0075] The method 500 may include selecting a subset of ML projects from the set of ML projects based on the quality and diversity features at block 512. For example, the curation module 114 may select the subset of ML projects from the set of existing ML projects 204 based on the quality and diversity features. In some embodiments, the subset of ML projects may be selected from the set of ML projects according to one or more operations of a method 1100, described further below with reference to FIG.

[0076] The method 500 may include storing a subset of the ML projects in a corpus of ML projects that may be adapted for use in a new ML project, at block 514. For example, the curation module 114 may store a subset of the existing ML projects 204 in the ML project corpus 104 that may be adapted for a new ML project (such as the new ML project 310).

[0077] Modifications, additions, or omissions may be made to method 500 without departing from the scope of the present disclosure. For example, some of the operations of method 500 may be performed in a different order. Additionally, or alternatively, two or more operations may be performed simultaneously. Furthermore, the outlined steps and operations are provided as examples, and some of the steps and operations may be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the disclosed embodiments.

[0078] 6 is a flowchart of an example method 600 for collecting a set of ML projects from one or more repositories of ML projects based on filtering criteria, according to at least one embodiment described in this disclosure. In some embodiments, the operations of block 502 described above with respect to method 500 of FIG. 5 may be performed in accordance with method 600.

[0079] Method 600 may be performed by any suitable system, apparatus, or device. For example, curation module 114 of Figures 1-2 or (when directed by one or more modules) computing system 402 of Figure 4 may perform one or more of the operations associated with method 600. Although illustrated by separate blocks, the steps and operations associated with one or more of the blocks of method 600 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

[0080] The method 600 may include, at block 602, ranking all datasets of all ML projects from one or more repositories of ML projects based on the quality of the datasets. For example, the curation module 114 may rank all datasets of existing ML projects 204 from the OSS ML project databases 102a-102n based on the quality of the datasets. In some embodiments, the quality of a dataset may be determined based on votes by other users (e.g., votes on Kaggle), based on the dataset being part of a "featured set" of datasets hosted in the project database (e.g., on Kaggle), or based on how recent the dataset is.

[0081] The method 600 may include selecting a first predetermined number of top-ranked datasets at block 604. For example, the curation module 114 may select the first predetermined number of top-ranked datasets from existing ML projects 204.

[0082] The method 600 may include, at block 606, ranking corresponding ML projects for each selected dataset based on an importance criterion. For example, the curation module 114 may rank corresponding ML projects from existing ML projects 204 for each selected dataset based on an importance criterion. In some embodiments, the importance of a dataset may be determined based on votes by other users (e.g., votes on Kaggle). In other embodiments, the importance of a dataset may be determined based on the accuracy of the corresponding pipeline.

[0083] The method 600 may include selecting a second predetermined number of top-ranked ML projects for inclusion in the set of ML projects at block 608. For example, the curation module 114 may select the second predetermined number of top-ranked ML projects from the existing ML projects 204 for inclusion in the set of existing ML projects 20. For example, if the top 15 top-ranked ML projects (based on up-votes by other users of Kaggle) were selected for each of the top 150 datasets in Kaggle, this may generate 2250 ML projects.

[0084] Modifications, additions, or omissions may be made to method 600 without departing from the scope of the present disclosure. For example, the operations of method 600 may be performed in a different order. Furthermore, in some embodiments, method 600 may be performed repeatedly or simultaneously with respect to block 502 of FIG. 5.

[0085] 7 is a flowchart of an example method 700 for ensuring the viability of ML pipelines in a set of ML projects, according to at least one embodiment described in this disclosure. In some embodiments, the operations of block 504 described above with respect to method 500 of FIG. 5 may be performed in accordance with method 700.

[0086] Method 700 may be performed by any suitable system, apparatus, or device. For example, curation module 114 of Figures 1-2 or (when directed by one or more modules) computing system 402 of Figure 4 may perform one or more of the operations associated with method 600. Although shown by separate blocks, steps and operations associated with one or more of the blocks of method 700 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

[0087] The method 700 may include determining whether an ML pipeline in an ML project can be run without modification of the ML pipeline at block 702. If so (Yes at block 702), the method 700 may proceed to block 704; if not (No at block 702), the method 700 may proceed to block 706. For example, the curation module 114 may determine whether an ML pipeline in one of the existing ML projects 204 can be run without modification.

[0088] The method may include maintaining the ML pipeline in the set of ML projects at block 704. For example, the curation module 114 may maintain the ML project in the set of existing ML projects 204 due to the ML pipeline of the ML project being executable before or after performing API adaptation on the ML pipeline.

[0089] The method may include determining whether the ML pipeline in the ML project is executable after performing the API adaptation on the ML pipeline at block 706. If executable (Yes at block 706), the method 700 may proceed to block 704; if not executable (No at block 706), the method 700 may proceed to block 708. For example, the curation module 114 may determine whether the ML pipeline in one of the existing ML projects 204 is executable after performing the API adaptation on the ML pipeline.

[0090] The method may include removing an ML pipeline from the set of ML projects at block 708. For example, the curation module 114 may remove an ML project from the set of existing ML projects 204 due to the ML pipeline of the ML project not being executable, either before or after performing API adaptation on the ML pipeline.

[0091] Modifications, additions, or omissions may be made to method 700 without departing from the scope of the present disclosure. For example, the operations of method 700 may be performed in a different order. Furthermore, in some embodiments, method 700 may be performed repeatedly or simultaneously with respect to block 504 of FIG. 5.

[0092] FIG. 8A shows example ML pipeline code 800 for an existing ML project, and FIG. 8B shows example cleaned ML pipeline code 850 resulting from cleaning the example ML pipeline code 800 of FIG. 8A . ML pipeline code 800 may be part of a computational notebook (e.g., a Jupyter notebook) of one of the existing ML projects 204, where the ML task is to predict whether a person earns more than $50,000 per year based on a dataset including characteristics such as age, work class, education, occupation, family status, and race of multiple workers. FIGS. 8A and 8B are discussed here to provide an example of how blocks 504 and 506 of method 500 may be implemented with respect to an ML pipeline in a set of existing ML projects 204.

[0093] 8A and 8B, the illustrated API adaptations may be performed on ML pipeline code 800, resulting in ML pipeline code 850 to ensure the executableness of the ML pipeline code. Additionally, irrelevant portions of ML pipeline code 800 may be marked in bold and italics in ML pipeline code 850 to identify that execution of this irrelevant code should be avoided in certain environments. In some embodiments, these irrelevant portions may include debugging code, checking code, and graph drawing code.

[0094] In some embodiments, this identification of irrelevant portions of the ML pipeline code 800 is performed by the ML project P<D,L> Save Dataset Properties Sample D sample (e.g., reducing the time associated with processing the entire dataset but not sacrificing the range of properties of the dataset by smartly sampling the dataset), instrumenting the ML pipeline L to trace the variables and objects modified within each statement of the ML pipeline L, generating a sample D of the dataset sample ML pipeline instrumented with L instr to collect an execution trace E; identify a target statement T in the ML pipeline L (e.g., the target statement shown in the ML pipeline code 850); extract all variables and objects B from the target statement T; generate a backward slice B of the extracted variables and objects B from the execution trace E; (annotated ML pipeline L) N ML pipeline L annotated as relevant to statements in the ML pipeline from backward slice B (to generate N, and annotating all other statements in the target statement as irrelevant. In this manner, only statements in the ML pipeline code 800 that are relevant to the target statement are annotated as relevant. In some embodiments, smart sampling of the dataset may include one or more of: using stratified sampling to retain all categorical values ​​in their original proportions for categorical columns; uniformly sampling across a specified range for continuous (numeric) columns; randomly sampling instances for string columns; and, if a column had missing values ​​in the original data, having the missing values ​​in that column after dataset reduction. In some embodiments, this smart sampling of the dataset may result in a significant reduction of the dataset. For example, a 2 GB dataset may be reduced to 9 MB, resulting in a run time reduction of 10 minutes to 20 seconds.

[0095] In some embodiments, cleaning of the ML pipeline code 800, which may be done in parts of a Jupyter notebook, may result in cleaned ML pipeline code 850 that is more suitable for feature extraction (for selected projects) and for subsequent steps in the search and adaptation workflow (e.g., indexing, searching, and adapting the computational notebook).

[0096] Modifications, additions, or omissions may be made to ML pipeline code 800 and cleaned ML pipeline code 850 without departing from the scope of the present disclosure. For example, some statements in ML pipeline code 800 may be performed in a different order.

[0097] 9 is an example table 900 of quality features. FIG. 9 is discussed here to provide an example of how block 508 of method 500 may be performed with respect to a set of existing ML projects 204. In the example shown in FIG. 9, the quality features may include performance features, code quality features, and community metric features. These quality features may be generated for the set of existing ML projects 204. Each of these quality features may include measurements, metrics, and normalized metrics.

[0098] In some embodiments, as disclosed in table 900 of FIG. 9 , generating quality features for the set of existing ML projects 204 (e.g., at block 508 of method 500) may include generating, for each ML project in the set of existing ML projects 204, a numerical quality score for the ML project based on the accuracy of the ML project's trained ML model, the documentation quality of the ML project's ML pipeline, the modularity quality of the ML project's ML pipeline, the standard API usage quality of the ML project's ML pipeline, and the ML project's rating.

[0099] In some embodiments, the purpose of extracting quality features may be to calculate features that may form the basis for selecting existing ML projects to place in the ML project corpus 104. These quality features may represent the intrinsic value of an ML pipeline in terms of the quality of the trained model, the code structure, and its value to the community. These quality features may be evaluated individually for a given ML pipeline and may be calculated for each ML pipeline as a single numerical score (e.g., between 0 and 1.0) representing its quality. In some embodiments, this single numerical score may be calculated according to the following formula:

number

[0100] Modifications, additions, or omissions may be made to table 900 without departing from the scope of the present disclosure. For example, additional quality features may be added to the quality features disclosed in table 900.

[0101] FIG. 10 is a table 1000 of example diversity features. FIG. 10 is discussed here to provide an example of how block 510 of method 500 may be performed with respect to a set of existing ML projects 204. In the example shown in FIG. 10, the diversity features may include dataset features and computational notebook features. These diversity features may be generated for a set of existing ML projects 204. Each of these diversity features may include an intent, a metric, and a normalization.

[0102] In some embodiments, as disclosed in Table 1000, generating diversity features for the set of existing ML projects 204 (e.g., in block 510 of method 500) may include, for each ML project in the set of existing ML projects 204, extracting a feature vector for the ML project based on the size of the ML project's dataset, the presence or absence of missing values ​​in the ML project's dataset, the type of data in the ML project's dataset, the presence or absence of pre-defined library APIs in the ML project's ML pipeline, and the relative ranges of the component data streams in the ML project's dataset.

[0103] In some embodiments, the purpose of extracting diversity features may be to compute features that may form the basis for the selection of existing ML projects to place within the ML project corpus 104. These diversity features may capture the ability of ML pipelines to add to the range of solutions ultimately available within the ML project corpus 104. These diversity features may be evaluated with respect to the selection of other ML pipelines and may be computed as a feature vector for each ML project representing its contribution to diversity.

[0104] Modifications, additions, or omissions may be made to table 1000 without departing from the scope of the present disclosure. For example, additional diversity features may be added to the diversity features disclosed in table 1000.

[0105] 11 is a flowchart of an example method 1100 for selecting a subset of ML projects from a set of ML projects based on quality and diversity features, according to at least one embodiment described in this disclosure. In some embodiments, the operations of block 512 described above with respect to method 500 of FIG. 5 may be performed in accordance with method 1100.

[0106] Method 1100 may be performed by any suitable system, apparatus, or device. For example, curation module 114 of Figures 1-2 or (when directed by one or more modules) computing system 402 of Figure 4 may perform one or more of the operations associated with method 600. Although shown by separate blocks, steps and operations associated with one or more of the blocks of method 1100 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

[0107] The method 1100 may include, at block 1102, generating a quality indicator for each ML project in the set of ML projects based on the quality features of the ML project. For example, the curation module 114 may generate a quality indicator for each ML project in the set of existing ML projects 204 based on the quality features of the ML project. For example, given a cleaned set of ML projects as follows:

number

number

number

[0108] The method 1100 may include generating a weight from the ML project's quality indicators for each ML project in the set of ML projects at block 1104. For example, the curation module 114 may generate a weight from the ML project's quality indicators for each ML project in the set of existing ML projects 204. For example, for each project, the weight is:

number

[0109] The method 1100 may include, at block 1106, constructing a Weighted Set Cover (WSC) problem from the ML projects in the set of ML projects based on weights and diversity characteristics. For example, the curation module 114 may construct a WSC problem from the ML projects in the set of existing ML projects 204 based on weights and diversity characteristics. Calculating weights from the quality metrics of each ML pipeline may allow the original task of selecting a set of pipelines that maximizes the collective quality of the selected pipelines (i.e., the maximization objective) to be formulated as a problem of selecting a set of pipelines with minimal weights (i.e., the minimization objective), which can be naturally solved as a WSC problem. Furthermore, larger weights with respect to the quality values ​​(through the selection of W) motivate minimizing the cardinality of the selected set. Because WSC is an NP-Complete problem, well-known approximation algorithms for WSC may be used to solve it. Two possibilities include a greedy algorithm or an algorithm based on Linear Programming (LP) relaxation.

[0110] The method may include, at block 1108, solving a WSC problem to select a subset of the ML projects. For example, the curation module 114 may solve the WSC problem to select a subset of the existing ML projects 204. Solving the WSC problem may solve for a minimum-weight subset, which may indirectly maximize the aggregate quality of the selected pipeline. For example, the goal may be to select the following subsets that together exhibit all the features in U:

number

number

number

[0111] Modifications, additions, or omissions may be made to method 1100 without departing from the scope of the present disclosure. For example, the operations of method 1100 may be performed in a different order. Furthermore, in some embodiments, method 1100 may be performed repeatedly or simultaneously with respect to block 512 of FIG. 5.

[0112] 12 is a flowchart of an example method 1200 for automatically labeling functional blocks in a pipeline of an existing ML project in an adaptable corpus for use in a new ML project, according to at least one embodiment described in this disclosure. Method 1200 may be performed by any suitable system, device, or apparatus. For example, curation module 114 of FIGS. 1-2 or (when directed by one or more modules) computing system 402 of FIG. 4 may perform one or more of the operations associated with method 1200. Although shown by separate blocks, steps and operations associated with one or more of the blocks of method 1200 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

[0113] In some embodiments, as shown in FIG. 2 , the method 1200 may be utilized by the curation module 114 to abstract 210 and generate project artifacts 212 on existing ML projects 204 before storing a subset of the existing ML projects 204 in the ML project corpus 104.

[0114] The method 1200 may include normalizing ML pipelines of existing ML projects stored in a corpus of existing ML projects at block 1202. For example, the curation module 114 may normalize the ML pipelines of a subset of existing ML projects 204 stored in the ML project corpus 104 (possibly after the existing ML projects 204 have been filtered 206 and cleaned 208). In some embodiments, the ML pipelines may be normalized as shown in original ML pipeline code 1300 and normalized ML pipeline code 1350 in Figures 13A and 13B, which are described further below.

[0115] Method 1200 may include extracting function blocks from the normalized ML pipeline at block 1204. For example, curation module 114 may extract the function blocks from the normalized ML pipeline. In some embodiments, the function blocks may be extracted according to one or more operations of method 1400, which is further described below with reference to FIG.

[0116] Method 1200 may include assigning a label to each of the function blocks in the normalized ML pipeline at block 1206. For example, curation module 114 may assign a label to each of the function blocks in the normalized ML pipeline. In some embodiments, the labels may be assigned according to one or more operations of method 1600, which is further described below with reference to FIG.

[0117] The method 1200 may include indexing each of the ML pipelines in the corpus based on the labels assigned to the function blocks, at block 1208. For example, the curation module 114 may index each of the ML pipelines in the ML project corpus based on the labels assigned to the function blocks.

[0118] The method 1200 may include, at block 1210, generating a new ML pipeline for performing a new ML task on a new dataset for a new ML project using labels assigned to the function blocks in the corpus. For example, the search module 116 and the adaptation module 120 may generate a new ML pipeline 112 for performing a new ML task 108 on a new dataset 106 for a new ML project 310 using labels assigned to the function blocks in the ML project corpus 104.

[0119] Modifications, additions, or omissions may be made to method 1200 without departing from the scope of the present disclosure. For example, some of the operations of method 1200 may be performed in a different order. Additionally, or alternatively, two or more operations may be performed simultaneously. Furthermore, the outlined steps and operations are provided as examples, and some of the steps and operations may be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the disclosed embodiments.

[0120] FIG. 13A shows exemplary original ML pipeline code 1300 of an existing ML project before normalization, and FIG. 13B shows exemplary normalized ML pipeline code 1350 after normalization of the original ML pipeline code 1300. The original ML pipeline code 1300 may be part of a computational notebook (e.g., a Jupyter notebook) of one of the existing ML projects 204, where the ML task is to predict whether a person earns more than $50,000 per year based on a dataset including characteristics such as age, work class, education, occupation, family status, and race of multiple workers. FIGS. 13A and 13B are discussed here to provide an example of how block 1202 of method 1200 may be performed with respect to an ML pipeline in a set of existing ML projects 204.

[0121] 13A and 13B, the original ML pipeline code 1300 may be normalized in various ways, including one or more of: normalizing variable names, replacing column names with column data types, removing parameters from API statements, and collapsing repeated instances of API statements into a single instance of the API statement. For example, the variable names “dataset,” “array,” “X,” “Y,” “X_train,” “X_validation,” “Y_train,” “Y_validation,” “random_forest,” and “predictions” in the original ML pipeline code 1300 may each be normalized to “_var_” in the normalized ML pipeline code 1350. Additionally, the columns “workclass,” “occupation,” “native.country,” and “sex” in the original ML pipeline code 1300 may each be normalized to “_str_category_” in the normalized ML pipeline code 1350. Furthermore, each of the parameters “filna”, “map”, “train_test_split”, “RandomForestClassifier”, “fit”, and “predict” (e.g., parameters such as “X”, “United-States”, “Male”, “0”, “Female”, “1”, etc.) of the API statements in the original ML pipeline code 1300 may be normalized by respectively removing the parameters in the normalized ML pipeline code 1350. Also, the API statement repeated three times in the original ML pipeline code 1300:

number

[0122] Modifications, additions, or omissions may be made to the original ML pipeline code 1300 and the normalized ML pipeline code 1350 without departing from the scope of this disclosure. For example, some statements in the original ML pipeline code 1300 may be performed in a different order, and other normalizations may be performed on the original ML pipeline code 1300.

[0123] 14 illustrates a flowchart of an example method 1400 for identifying idiomatic and custom function blocks, in accordance with at least one embodiment described herein. In some embodiments, the operations of block 1204 described above with respect to method 1200 of FIG. 12 may be performed in accordance with method 1400.

[0124] Method 1400 may be performed by any suitable system, apparatus, or device. For example, curation module 114 of Figures 1-2 or (when directed by one or more modules) computing system 402 of Figure 4 may perform one or more of the operations associated with method 1400. Although illustrated by separate blocks, steps and operations associated with one or more of the blocks of method 1400 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

[0125] In some embodiments, method 1400 may be utilized to partition each ML pipeline in each existing ML project stored in ML project corpus 104 into code cells. In some embodiments, computational notebooks (e.g., Jupyter notebooks) are naturally structured, with all code in the computational notebook organized into a set of code cells, each of which may initially be assumed to be a distinct function block, although this assumption may be invalidated after additional analysis. Method 1400 may then be utilized to identify statement groups that are repeated across code cells as idiomatic function blocks, and to identify all other statement groups within the code cells as custom function blocks.

[0126] The method 1400 may include, at block 1402, generating a directed graph. For example, the curation module 114 may generate the directed graph (see, e.g., the directed graph illustrated in FIG. 15 ). In some embodiments, each node in the directed graph may represent a statement normalized by its occurrence in the ML project corpus 104, and each edge may represent the average probability score of co-occurrence of the statements corresponding to the two nodes the edge connects. There may not be a connection between the last statement in one cell and the first statement in the next cell. For example, given two nodes A and B, the value of their edge may be expressed as:

number

[0127] The method 1400 may include, for each statement in each of the code cells, at block 1404, adding the statement as a node in a directed graph or incrementing a count in an existing node in the directed graph for the statement; at block 1404b, calculating a co-occurrence score between the statement in the code cell and the statement immediately preceding the statement; and at block 1404c, generating an edge between the node of the statement and the node of the statement immediately preceding the statement if the co-occurrence score is greater than a co-occurrence score threshold. For example, for each statement in each of the code cells (e.g., each code cell in a computational notebook such as a Jupyter notebook), the curation module 114 may add the statement as a node in a directed graph or increment a count in an existing node in the statement's directed graph (e.g., see the nodes in the directed graph shown in FIG. 15 , which have a count inside each node), calculate a co-occurrence score between the statement in the code cell and the statement immediately preceding it (e.g., see the co-occurrence score in the directed graph of the multi-line idiom shown in FIG. 15 ), and generate an edge between the statement's node and the node of the statement immediately preceding it if the co-occurrence score is greater than a co-occurrence score threshold (e.g., see the edge between nodes in the directed graph of the multi-line idiom shown in FIG. 15 ). In some embodiments, this generation of edges may alternatively be performed by first generating all edges regardless of co-occurrence score, but then removing all edges with co-occurrence scores below a certain threshold. The threshold may be determined empirically. After edge removal, a set of connected components may remain, where each component represents an idiom (e.g., a set of statements / APIs that occur together to implement functionality).

[0128] The method 1400 may include marking all nodes connected by edges as idiomatic function blocks at block 1406. For example, the curation module 114 may mark all nodes connected by edges as idiomatic function blocks (e.g., see the nodes connected by edges in the multi-line idiomatic directed graph shown in FIG. 15 ).

[0129] Method 1400 may include marking each set of nodes corresponding to consecutive statements in any of the normalized ML pipelines, such as custom function blocks, that are not marked as one of the idiomatic function blocks, at block 1408. For example, curation module 114 may mark each set of nodes corresponding to consecutive statements in any of the normalized ML pipelines, such as custom function blocks, that are not marked as one of the idiomatic function blocks (e.g., see the single-line function blocks and multi-line function blocks shown in FIG. 15 ).

[0130] Modifications, additions, or omissions may be made to method 1400 without departing from the scope of the present disclosure. For example, the operations of method 1400 may be performed in a different order. Furthermore, in some embodiments, method 1400 may be performed repeatedly or simultaneously with respect to block 1204 of FIG. 12 .

[0131] FIG. 15 illustrates extracting function blocks from a normalized ML pipeline using a directed graph. FIG. 15 is discussed here to provide an example of how block 1204 of method 1200 and blocks 1402-1408 of method 1400 may be implemented with respect to normalized ML pipeline code 1500. In the example shown in FIG. 15, normalized ML pipeline code 1500 may be used to generate one or more directed graphs, where each node represents a statement and each edge represents a co-occurrence between statements with a score higher than a threshold. As shown in the single-line function block, across the entire normalized ML pipeline, statement 1 occurs 94 times, statement 2 occurs 6 times, and statement 3 occurs 6 times. Furthermore, as shown in the multiple-line function block, statements 4, 5, 6, and 7 occur only once. This indicates that these statements appear only in normalized ML pipeline code 1500 and not in any other normalized ML pipelines. Also, as disclosed in the multi-line idiom, statement 8 occurs 10 times, statement 9 occurs 10 times, and statement 11 occurs twice. Edges whose corresponding co-occurrence scores are higher than a threshold score (e.g., a threshold of 0.5) indicate that statement sequences 8, 9, 10 and 8, 9, 11 appear together in multiple normalized ML pipelines and should therefore be marked together as idiomatic function blocks in each normalized ML pipeline in which they appear. Furthermore, after marking idioms 8, 9, and 10 as idiomatic function blocks in normalized ML pipeline code 16500, the remaining statements in normalized ML pipeline code 1500 can be decomposed into single-line function blocks 1, 2, and 3 and multi-line custom function blocks including statements 4, 5, 6, and 7 according to the boundaries of the code cells in which they reside.

[0132] Modifications, additions, or omissions may be made to the normalized ML pipeline code 1500 and directed graphs of Figure 15 without departing from the scope of this disclosure. For example, additional directed graphs for additional normalized ML pipeline codes may be added.

[0133] 16 is a flowchart of an example method 1600 for assigning labels to each of the functional blocks in a normalized ML pipeline. In some embodiments, the operations of block 1206 described above with respect to method 1200 of FIG. 12 may be performed in accordance with method 1600.

[0134] Method 1600 may be performed by any suitable system, apparatus, or device. For example, curation module 114 of Figures 1-2 or (when directed by one or more modules) computing system 402 of Figure 4 may perform one or more of the operations associated with method 1600. Although illustrated by separate blocks, the steps and operations associated with one or more of the blocks of method 1600 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

[0135] At block 1602, method 1600 may include extracting text from any comments in the function block, or, when no comments are present in the function block, mining text from documentation of any API statements in the function block. For example, curation module 114 may extract text from any comments in the function block (e.g., statements beginning with “#” that precede each function block in normalized ML pipeline code 1500) or, when no comments are present in the function block, mine text from documentation of any API statements in the function block (e.g., documentation of API statements available at a repository of API code, such as a website hosting the API code). This extraction or mining may additionally or alternatively include preprocessing the extracted or mined text by applying standard preprocessing techniques (e.g., stop word removal, stemming, tokenization, etc.).

[0136] Method 1600 may include identifying first common verbs and nouns or noun phrases in the extracted or mined text at block 1604. For example, the curation module 114 may identify common verbs (e.g., “scale” or “apply”) and nouns or noun phrases (e.g., “data” or “standard” or “unit variance”) in the extracted or mined text. In the context of method 1600, the term “common” may refer to a verb, noun, or noun phrase that is common across multiple instances of an idiom function block or common across labels. In some embodiments, block 1604 may include extracting noun / verb candidates for each instance of the idiom function block from the extracted or mined text that may form labels for all instances of the idiom block.

[0137] Method 1600 may include, at block 1606, generating a label for the function block based on the first common verb and noun or noun phrase. For example, the curation module 114 may generate the label "scale date" from the comment "scale the data to be between -1 and 1." In some embodiments, generating this label may include taking the most important verbs and nouns or noun phrases and assigning these words as semantic labels for the function block and for other instances of the same idiomatic function block. In these embodiments, the most important words may be determined as the top N frequently used words, or may be determined through topic modeling, or may be determined in some other manner. In some embodiments, block 1606 may include performing a consensus operation among the label candidates contributed by each instance of the idiomatic function block to find the most important common noun / verb instance across these different candidates. This may form an initial label for all instances of the idiomatic function block. For example, the four labels from the idiomatic function block "scale data," "apply standard," "scale numerical column data," and "standard feature remove mean scale unit variance" may be relabeled with a single common label "scale standard data."

[0138] In some embodiments, blocks 1604 and 1606 may be performed for idiom function blocks, but may be modified for custom function blocks. In these embodiments, blocks 1604 and 1606 may be modified for each custom function block by incorporating the most important nouns and verbs from the custom function block's extracted or mined text, providing a starting point for block 1608 rather than performing a matching operation.

[0139] The method 1600 may include, at block 1608, generating a similarity score for each pair of function blocks in the normalized ML pipeline. For example, the curation module 114 may generate a similarity score (e.g., a score between 0 and 1.0) for each pair of function blocks in the normalized ML pipeline. In some embodiments, the similarity score may be calculated through simple word matching. For example, for two labels with word sets A and B, the similarity score may be calculated as follows:

number

[0140] At block 1610, the method 1600 may include generating groups of function blocks if the similarity score is greater than a similarity score threshold. For example, the curation module 114 may generate groups of function blocks if the similarity score is greater than a similarity score threshold (e.g., a threshold of 0.5). In some embodiments, for a given function block, other function blocks may be sorted based on similarity scores based on semantic labels, and the top K may be marked as different implementations of the same function. In some embodiments, the similarity score threshold may be empirically adjusted.

[0141] The method 1600 may include identifying a second common verb and a noun or noun phrase in the label of each of the function blocks in the function block group at block 1612. For example, the curation module 114 may identify a second common verb (e.g., "scale") and a noun or noun phrase (e.g., "data") for each of the function blocks in the function block group. This second identification may enable a second iteration after the first set of labels is generated to further enrich the function block group with labels that are sufficiently similar to be considered functionally equivalent.

[0142] At block 1614, the method 1600 may include generating a common label for the group of function blocks based on a second common verb and a noun or noun phrase. For example, the curation module 114 may generate the common label from the second common verb and a noun or noun phrase. In some embodiments, a common or frequent word in the semantic labels may be assigned as the common semantic label for the entire group. For example, the curation module 114 may update the labels assigned to each of the function blocks in each function block group to a common label. For example, two labels, “scale standard data” and “scale data feature,” from function blocks determined to be functionally equivalent may be relabeled with a single common label, “scale data.”

[0143] The method 1600 may include, at block 1616, updating the labels assigned to each of the function blocks in each function block group to a common label.

[0144] Modifications, additions, or omissions may be made to method 1600 without departing from the scope of the present disclosure. For example, the operations of method 1600 may be performed in a different order. Furthermore, in some embodiments, method 1600 may be performed repeatedly or simultaneously with respect to block 1206 of FIG. 12 .

[0145] FIG. 17 illustrates automatic labeling of function blocks in an ML pipeline. FIG. 17 is discussed herein to provide an example of how various blocks of method 1600 may be implemented. In the example shown in FIG. 17, function block Block-1 may include two normalized statements, namely, “_var_=StandardScaler()” and “_var1_=_var_.fit_transform().” This function block may be an idiomatic function block because it appears in four separate computational notebooks, namely, notebook-1, notebook-2, notebook-3, and notebook-4. Furthermore, a second function block Block-2 may include two normalized statements, namely, “_var_=MinMaxScaler()” and “_var1_=_var_.fit_transform().” Although these two function blocks are not identical, they may be determined to be functionally equivalent based on the similarity between their assigned labels, as described below.

[0146] For Block-1, at block 1602 of method 1600, text may be extracted from comments for notebook-1, notebook-2, and notebook-3, and there may be no comments available for notebook-4. Thus, text may be mined from an alternative source (e.g., API documentation) for notebook-4. Next, at block 1604 of method 1600, a common verb (e.g., “scale” or “apply”) and a noun or noun phrase (e.g., “data” or “standard” or “unit variance”) may be identified from the extracted or mined text. Next, at block 1606 of method 1600, a label “scale standard data” may be generated for Block-1 based on the first common verb and noun or noun phrase. Similarly, for Block-2, a label “scale data feature” may be generated at block 1602 and (as described above) at modified versions of blocks 1604 and 1606.

[0147] At block 1608 of method 1600, a similarity score of 0.67 may be generated for the pair of Block-1 and Block-2. At block 1610 of method 1600, Block-1 and Block-2 may be grouped together because their similarity score (0.67) is higher than a similarity score threshold (e.g., a threshold of 0.60). At block 1612 of method 1600, common verbs (e.g., "scale") and nouns (e.g., "data") may be identified among the labels of Block-1 and Block-2. At block 1614, method 1600 may include generating a common label ("scale data") for Block-1 and Block-2 based on the common verbs (e.g., "scale") and nouns (e.g., "data").

[0148] Modifications, additions, or omissions may be made to the functional blocks, extracted or mined text, similarity scores, and automatically assigned labels without departing from the scope of this disclosure.

[0149] 18 is a flowchart of an example method 1800 for automatically generating a pipeline of a new ML project from a pipeline of existing ML projects stored in a corpus, according to at least one embodiment described in this disclosure. Method 1800 may be performed by any suitable system, device, or apparatus. For example, curation module 114, search module 116, and adaptation module 120 of FIGS. 1-3, or (when directed by one or more modules) computing system 402 of FIG. 4, may perform one or more of the operations associated with method 1800. Although shown by separate blocks, steps and operations associated with one or more of the blocks of method 1800 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

[0150] In some embodiments, the method 1800 may be utilized by the curation module 114, the search module 116, and the adaptation module 120 to perform the operations disclosed in FIGS.

[0151] Method 1800 may include, at block 1802, storing existing ML projects in a corpus, where the existing ML projects include ML pipelines with function blocks. For example, curation module 114 may store existing ML projects 204 in ML project corpus 104. In some embodiments, existing ML projects 204 may include ML pipelines with function blocks. In some embodiments, these function blocks may be identified according to the operations of block 1204 of method 1200.

[0152] The method 1800 may include generating a search query for the new ML project based on the new dataset for the new ML project and the new ML task for the new ML project at block 1804. For example, the search module 116 may generate a query 306 from the new ML project 310 based on the new dataset 106 for the new ML project 310 and the new ML task 108 for the new ML project 310.

[0153] The method 1800 may include searching through the existing ML projects stored in the corpus for a set of relevant existing ML projects based on the search query, at block 1806. For example, the search module 116 may search through the existing ML projects stored in the ML project corpus 104 for relevant existing ML projects 110 based on the query 306.

[0154] The method 1800 may include, at block 1808, merging ML pipelines of the set of associated existing ML projects to generate a new ML pipeline for the new ML project, where the new ML pipeline has function blocks borrowed from the ML pipelines of the set of associated existing ML projects. For example, the adaptation module 120 may perform a pipeline merge 308 of the ML pipelines 111 of the associated existing ML projects 110 to generate a new ML pipeline 112 for the new ML project 310. In this example, the new ML pipeline 112 may have function blocks borrowed from the ML pipelines 111 of the associated existing ML projects 110.

[0155] The method 1800 may include adapting functional blocks of the new ML pipeline for the new ML project 310 so that the new ML pipeline is executable to perform the new ML task on the new dataset, at block 1810. For example, the adaptation module 120 may adapt functional blocks of the new ML pipeline 112 for the new ML project 310 so that the new ML pipeline 112 is executable to perform the new ML task 108 on the new dataset 106.

[0156] Modifications, additions, or omissions may be made to method 1800 without departing from the scope of the present disclosure. For example, some of the operations of method 1800 may be performed in a different order. Additionally, or alternatively, two or more operations may be performed simultaneously. Furthermore, the outlined steps and operations are provided as examples, and some of the steps and operations may be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the disclosed embodiments.

[0157] Figure 19 shows a sequence graph 1900 and a pipeline skeleton 2002 for a new ML project (e.g., new ML project 1310). Figure 20 shows the pipeline skeleton 2002 and a table 2050 of ML pipelines that may be searched for function blocks that match the pipeline skeleton 2002. Figures 19 and 20 are discussed here to provide an example of how blocks 1804, 1806, 1808, and 1810 of the method 1800 may be performed with respect to the ML pipeline corpus 104.

[0158] 19 and 20 , the pipeline skeleton 2002 may be an ordered set of function blocks for a new ML pipeline 112 of a new ML project 310 and may correspond to labels assigned to the function blocks of an ML pipeline of an existing ML project stored in the ML project corpus 104. In some embodiments, the pipeline skeleton 2002 may be generated by a pipeline skeleton ML model 302. The pipeline skeleton ML model 302 (or a set of ML models) may be trained to learn the mapping between dataset metafeatures and semantic labels. For example, given the metafeatures of a new dataset 1056, the pipeline skeleton ML model 302 may be trained to synthesize a pipeline skeleton 2002 that includes the desired semantic labels with their sequence.

[0159] In some embodiments, the pipeline skeleton ML model 302 may include a multivariate, multi-class classifier that is trained prior to generating the pipeline skeleton 2002. The multivariate, multi-class classifier may be configured to map dataset metafeatures to an unordered set of function blocks (indicated by corresponding semantic labels) that the pipeline skeleton 304 or 2002 should include. This training may include extracting dataset features from datasets of existing ML projects in the ML project corpus 104 associated with particular labels, identifying a set of full labels from the function blocks of the existing ML projects, preparing training data including input vectors having the dataset features and binary output tuples indicating the presence or absence of each of the set of full labels, and training the pipeline skeleton ML model 302 to learn the mapping between the dataset features and corresponding labels from the set of full labels. In some embodiments, training the pipeline skeleton ML model 302 may enable the pipeline skeleton ML model 302 to use salient characteristics of a new dataset 106 and a new ML task 108 (metafeatures) to predict an ordered set of function blocks (e.g., in a pipeline skeleton 304 or 2002) that can compose an ML pipeline for a new ML pipeline 112. The metafeatures of a dataset may include, but are not limited to, the number of rows, the number of features, the presence of numeric values, the presence of missing values, the presence of counts, the presence of numeric categories, the presence of string categories, the presence of text, and the type of target.

[0160] In some embodiments, pipeline skeleton ML model 302 may further include a sequence graph (similar to sequence graph 1900) representing a partial order among function blocks learned from training data. The sequence graph may be configured to map an unordered set of blocks (e.g., as shown in pipeline skeleton 2002) to an ordered set based on a partial order among the blocks learned from a training project corpus. The sequence graph may include a node for each label in the set of all labels from the function blocks of the existing ML projects. The sequence graph may also include a directed edge between each pair of a first node and a second node, where the first node precedes the second node in one of the existing ML projects.

[0161] Once the pipeline skeleton ML model 302 is trained, it may be utilized to generate queries 306 for new ML projects 310. In some embodiments, this generation of queries 306 may include mapping dataset features to an unordered set of labels for the new ML pipeline 112 of the new ML project 310, and may further include mapping the unordered set of labels to an ordered set of labels using a partial order represented in a sequence graph (e.g., sequence graph 1900). The queries 306 may include such ordered sequences of labels as pipeline skeletons 2002. For example, FIG. 19 illustrates an example of mapping an unordered set of labels generated by a pipeline skeleton ML model to an ordered sequence of labels using sequence graph 1900. The unordered set of labels may first be mapped to corresponding nodes in the sequence graph 1900, which is represented by the set of bolded nodes, namely, "Read Data," "Fill Missing Values," "Convert String to Int," "Split Train Test," and "Random Forest." Next, a subgraph of the sequence graph 1900 represented by these nodes may be extracted, and a topological order of the nodes may be computed based on this subgraph to provide an ordered sequence of these labels, which is represented in the pipeline skeleton 2002.

[0162] In some embodiments, query 306 may be utilized to search through existing ML projects stored in ML project corpus 104. This search may include generating a label vector for each existing ML project stored in ML project corpus 104 and generating a weight from the quality indicators of the existing ML projects. Next, this search may involve solving a Weighted Set Cover (WSC) problem based on the weights and label vectors from the existing ML projects stored in ML project corpus 104, and solving a WSC problem to select a set of existing ML projects that together contain all of the labels in the ordered label set. For example, given a domain: U={s1, s2,..., s n} and the set of required semantic labels: R = {s i ,s j ,...,s k} and a set of cleaned candidate computational notebooks collectively containing: J={J1,J2,...,J m}, the search may be formulated to select the following subsets that together contain all semantic labels in R:

number

number

[0163] After the search is complete, the search results, such as related existing ML projects 110, may be merged (e.g., in pipeline merge 308) to generate a new ML pipeline 112 for the new ML project 310. This pipeline merge 308 may include incorporating all of the function blocks of the new ML project (e.g., corresponding to an ordered set of labels) from the set of ML pipelines 111 of the related existing ML project 110. For example, as disclosed in FIG. 20 , if the related existing ML project 110 is represented by three ML projects labeled “Mushroom Classification,” “World Happiness Report 2019,” and “Cardio” in table 2050, each of the function blocks in pipeline skeleton 2002 may be incorporated from the function blocks of these three ML projects. The ML project labeled “Mushroom Classification” has most of the required function blocks and may be treated as the main ML project, while the remaining function blocks may be incorporated from the ML project “World Happiness Report 2019,” which may be treated as an auxiliary ML project. In some embodiments, if the same label exists in multiple auxiliary computation notebooks, one of the computation notebooks may be selected (e.g., based on quality, randomly, etc.). For example, Figure 20 illustrates a case where the ML project corpus 103 includes a total of three ML projects, and a search (e.g., solved through the WSC problem disclosed herein) retrieves the first two ML projects as being ten billion to fit the pipeline skeleton 2002.

[0164] The pipeline merge 308 may further include adapting the functional blocks of the new ML pipeline for the new ML project, which may include resolving conflicts in various names or object names (e.g., adapting names based on program analysis) to make the new ML pipeline 112 executable to execute the new ML task 108 on the new dataset 106.

[0165] Modifications, additions, or omissions may be made to sequence graph 1900, pipeline skeleton 2002, and table 2050 without departing from the scope of the present disclosure. For example, sequence graph 1900, pipeline skeleton 2002, and table 2050 may each include fewer or more components than those shown in FIGS. 19 and 20.

[0166] As mentioned above, the embodiments described herein may involve the use of special purpose or general purpose computers including various computer hardware or software modules, as discussed in more detail below. Further, as mentioned above, the embodiments described in this disclosure may be implemented using computer-readable media having computer-executable instructions or data structures stored thereon.

[0167] As used in this disclosure, the term "module" or "component" may refer to a specific hardware implementation configured to perform the operations of the module or component, and / or a software object or software routine that may be stored and / or executed by general-purpose hardware of a computing system (e.g., computer-readable medium, processing unit, etc.). In some embodiments, different components, modules, engines, and services than those described in this disclosure may be implemented as objects or processes (e.g., separate threads) executing on a computing system. While some of the systems and methods described in this disclosure have been generally described as being implemented in software (stored on and / or executed by general-purpose hardware), dedicated hardware implementations or combinations of software and dedicated hardware implementations are also possible and contemplated. In this description, a "computing entity" may be any computing system described above in this disclosure, or any module or combination of modules executing on a computing system.

[0168] Terms used in this disclosure and particularly in the appended claims (e.g., the body of the appended claims) are generally intended as "broad" terms (e.g., the term "including" should be interpreted as "including but not limited to," the term "having" should be interpreted as "having but not limited to," etc.).

[0169] Furthermore, where the recitation of a specific number of introduced claims is intended, such intention shall be expressly set forth in the claim; in the absence of such a recitation, no such intention exists. For example, as an aid to understanding, the following appended claims may include the use of the introductory phrases "at least one" and "one or more" to introduce claim recitations. However, the use of such phrases should not be construed as meaning that introducing a claim recitation with the indefinite article "a" or "an" limits any particular claim that includes such an introduced claim recitation to embodiments that include only one such recitation, even when the same claim also includes the introductory phrase "one or more" or "at least one" and the indefinite article "a" or "an" (e.g., "a" and / or "an" should be interpreted to mean "at least one" or "one or more"). Indeed, the same applies to the use of definite articles used to introduce claim recitations.

[0170] Furthermore, when the recitation of a specific number of incorporated claims is explicitly recited, those skilled in the art will understand that such recitation should be interpreted to mean at least the recited number (e.g., a recitation of "two enumerations" without other qualification means at least two enumerations, or two or more enumerations). Furthermore, in instances where recitations similar to "at least one of A, B, and C, etc." or "one or more of A, B, and C, etc." are used, such configurations are typically intended to include A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, etc. This interpretation of the phrase "A or B" remains applicable even when the term "A and / or B" is sometimes used to include the possibilities of "A" or "B" or "A and B."

[0171] Furthermore, any disjunctive word or phrase expressing two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibility of including one of the terms, either of the terms, or both terms. For example, the phrase "A or B" should be understood to include the possibilities of "A" or "B" or "A and B."

[0172] All examples and conditional language set forth in this disclosure are intended for instructional purposes to assist the reader in understanding the disclosure and the concepts it contributes to further developing the art, and should not be construed as being limited to such specifically recited examples and conditions. Although embodiments of the present disclosure have been described in detail, various modifications, substitutions, and alternatives may be made thereto without departing from the spirit and scope of the present disclosure.

[0173] In addition to the above-described embodiments, the following supplementary notes are also disclosed. (Appendix 1) 1. A method comprising: normalizing machine learning (ML) pipelines of existing ML projects stored in a corpus of existing ML projects; extracting a function block from the normalized ML pipeline; assigning a label to each of the function blocks in the normalized ML pipeline; indexing each of the ML pipelines in the corpus based on the labels assigned to the function blocks; generating a new pipeline for performing a new ML task on a new dataset for a new ML project, utilizing the labels assigned to the function blocks in the corpus; A method comprising: (Appendix 2) Normalizing the ML pipelines of the existing ML projects stored in the corpus includes: normalizing variable names in the ML pipeline of the existing ML project; Replacing column names with common data types in the ML pipeline of the existing ML project; removing parameters from application program interface (API) statements in the ML pipeline of the existing ML project; collapsing repeated instances of the API statement into a single instance of the API statement in the ML pipeline of the existing ML project; 2. The method of claim 1, comprising: (Appendix 3) The step of extracting the function blocks from the normalized ML pipeline comprises: dividing each ML pipeline for each of the existing ML projects stored in the corpus into code cells, each code cell being described by commented text; identifying repeated groups of statements across said code cells as idiomatic function blocks; identifying every other group of statements in the code cell as a custom function block; 2. The method of claim 1, comprising: (Appendix 4) The step of identifying the idiomatic function blocks and the custom function blocks comprises: generating a directed graph; For each statement in each of the code cells: adding the statement as a node in the directed graph or incrementing a count in an existing node in the directed graph for the statement; calculating a co-occurrence score between the statement and the statement immediately preceding it in the code cell; generating an edge between the node of a statement and the node of a statement immediately preceding the statement if the co-occurrence score is greater than a co-occurrence score threshold; marking all nodes connected by edges as said idiomatic function blocks; marking each set of nodes corresponding to consecutive statements in any of the normalized ML pipelines that are not marked as one of the idiomatic function blocks as one of the custom function blocks; 4. The method of claim 3, comprising: (Appendix 5) The step of assigning the label to each of the function blocks in the normalized ML pipeline includes, for each function block of the function blocks: extracting text from comments in the function blocks, or, if no comments are present in the function blocks, mining text from documentation of API statements in the function blocks; identifying first common verbs and nouns or noun phrases in the extracted or mined text; generating the labels for the function blocks based on the first common verb and noun or noun phrase; 2. The method of claim 1, comprising: (Appendix 6) The step of assigning the label to each of the function blocks in the normalized ML pipeline comprises: generating a similarity score for each pair of the function blocks in the normalized ML pipeline; generating a group of the function blocks if the similarity score of the function blocks is higher than a similarity score threshold; updating the labels assigned to each of the function blocks in each group of function blocks to a common label; 6. The method of claim 5, comprising: (Appendix 7) The step of updating the labels assigned to each of the function blocks in each group of function blocks to a common label comprises: identifying a second common verb and noun or noun phrase in the labels of each of the function blocks in the group of function blocks; generating the common label for the group of function blocks based on the second common verb and noun or noun phrase; 7. The method of claim 6, comprising: (Appendix 8) One or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed, cause the system to perform operations, including: normalizing machine learning (ML) pipelines of existing ML projects stored in a corpus of existing ML projects; extracting a function block from the normalized ML pipeline; assigning a label to each of the function blocks in the normalized ML pipeline; indexing each of the ML pipelines in the corpus based on the labels assigned to the function blocks; generating a new pipeline for performing a new ML task on a new dataset for a new ML project, utilizing the labels assigned to the function blocks in the corpus; one or more non-transitory computer-readable storage media, (Appendix 9) Normalizing the ML pipelines of the existing ML projects stored in the corpus includes: normalizing variable names in the ML pipeline of the existing ML project; Replacing column names with common data types in the ML pipeline of the existing ML project; removing parameters from application program interface (API) statements in the ML pipeline of the existing ML project; collapsing repeated instances of the API statement into a single instance of the API statement in the ML pipeline of the existing ML project; 9. One or more non-transitory computer-readable storage media as described in Appendix 8, including: (Appendix 10) The step of extracting the function blocks from the normalized ML pipeline comprises: dividing each ML pipeline for each of the existing ML projects stored in the corpus into code cells, each code cell being described by commented text; identifying repeated groups of statements across said code cells as idiomatic function blocks; identifying every other group of statements in the code cell as a custom function block; 9. One or more non-transitory computer-readable storage media as described in Appendix 8, including: (Appendix 11) The step of identifying the idiomatic function blocks and the custom function blocks comprises: generating a directed graph; For each statement in each of the code cells: adding the statement as a node in the directed graph or incrementing a count in an existing node in the directed graph for the statement; calculating a co-occurrence score between the statement and the statement immediately preceding it in the code cell; generating an edge between the node of a statement and the node of a statement immediately preceding the statement if the co-occurrence score is greater than a co-occurrence score threshold; marking all nodes connected by edges as said idiomatic function blocks; marking each set of nodes corresponding to consecutive statements in any of the normalized ML pipelines that are not marked as one of the idiomatic function blocks as one of the custom function blocks; 11. One or more non-transitory computer-readable storage media as described in claim 10, including: (Appendix 12) The step of assigning the label to each of the function blocks in the normalized ML pipeline includes, for each function block of the function blocks: extracting text from comments in the function blocks, or, if no comments are present in the function blocks, mining text from documentation of API statements in the function blocks; identifying first common verbs and nouns or noun phrases in the extracted or mined text; generating the labels for the function blocks based on the first common verb and noun or noun phrase; 9. One or more non-transitory computer-readable storage media as described in Appendix 8, including: (Appendix 13) The step of assigning the label to each of the function blocks in the normalized ML pipeline comprises: generating a similarity score for each pair of the function blocks in the normalized ML pipeline; generating a group of the function blocks if the similarity score of the function blocks is higher than a similarity score threshold; updating the labels assigned to each of the function blocks in each group of function blocks to a common label; 13. One or more non-transitory computer-readable storage media as described in Clause 12, comprising: (Appendix 14) The step of updating the labels assigned to each of the function blocks in each group of function blocks to a common label comprises: identifying a second common verb and noun or noun phrase in the labels of each of the function blocks in the group of function blocks; generating the common label for the group of function blocks based on the second common verb and noun or noun phrase; 14. One or more non-transitory computer-readable storage media as described in Appendix 13, including: (Appendix 15) 1. A system comprising: one or more processors; one or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed by the one or more processors, cause the system to perform operations, including: normalizing machine learning (ML) pipelines of existing ML projects stored in a corpus of existing ML projects; extracting a function block from the normalized ML pipeline; assigning a label to each of the function blocks in the normalized ML pipeline; indexing each of the ML pipelines in the corpus based on the labels assigned to the function blocks; generating a new pipeline for performing a new ML task on a new dataset for a new ML project, utilizing the labels assigned to the function blocks in the corpus; Including, the system. (Appendix 16) Normalizing the ML pipelines of the existing ML projects stored in the corpus includes: normalizing variable names in the ML pipeline of the existing ML project; Replacing column names with common data types in the ML pipeline of the existing ML project; removing parameters from application program interface (API) statements in the ML pipeline of the existing ML project; collapsing repeated instances of the API statement into a single instance of the API statement in the ML pipeline of the existing ML project; 16. The system of claim 15, comprising: (Appendix 17) The step of extracting the function blocks from the normalized ML pipeline comprises: dividing each ML pipeline for each of the existing ML projects stored in the corpus into code cells, each code cell being described by commented text; identifying repeated groups of statements across said code cells as idiomatic function blocks; identifying every other group of statements in the code cell as a custom function block; 16. The system of claim 15, comprising: (Appendix 18) The step of identifying the idiomatic function blocks and the custom function blocks comprises: generating a directed graph; For each statement in each of the code cells: adding the statement as a node in the directed graph or incrementing a count in an existing node in the directed graph for the statement; calculating a co-occurrence score between the statement and the statement immediately preceding it in the code cell; generating an edge between the node of a statement and the node of a statement immediately preceding the statement if the co-occurrence score is greater than a co-occurrence score threshold; marking all nodes connected by edges as said idiomatic function blocks; marking each set of nodes corresponding to consecutive statements in any of the normalized ML pipelines that are not marked as one of the idiomatic function blocks as one of the custom function blocks; 18. The system of claim 17, comprising: (Appendix 19) The step of assigning the label to each of the function blocks in the normalized ML pipeline includes, for each function block of the function blocks: extracting text from comments in the function blocks, or, if no comments are present in the function blocks, mining text from documentation of API statements in the function blocks; identifying first common verbs and nouns or noun phrases in the extracted or mined text; generating the labels for the function blocks based on the first common verb and noun or noun phrase; 16. The system of claim 15, comprising: (Appendix 20) The step of assigning the label to each of the function blocks in the normalized ML pipeline comprises: generating a similarity score for each pair of the function blocks in the normalized ML pipeline; generating a group of the function blocks if the similarity score is higher than a similarity score threshold; updating the labels assigned to each of the function blocks in each group of function blocks to a common label; Further comprising: The step of updating the labels assigned to each of the function blocks in each group of function blocks to a common label comprises: identifying a second common verb and noun or noun phrase in the labels of each of the function blocks in the group of function blocks; generating the common label for the group of the function blocks based on the second common verb and noun or noun phrase; 20. The system of claim 19, comprising: [Explanation of symbols]

[0174] 102 OSS ML Project Database 104 ML Project Corpus 106 new datasets 108 New ML Tasks 109 datasets 110 Related Existing ML Projects 111 ML Pipeline 112 New ML Pipelines 114 Curation Module 116 Search Module 120 Adaptation Module

Claims

1. A method implemented by a system, the system including a curation module, a search module, and an adaptation module, the method comprising: normalizing, by the curation module, existing machine learning (ML) pipelines of existing ML projects stored in a corpus of existing ML projects, each of the existing ML pipelines including program code organized into a corresponding set of code cells; Replacing unique variable names with common variable names, Replacing unique column names with common data types, Removing parameters from application program interface (API) statements; collapsing repeated instances of said API statement into a single instance; generating a normalized ML pipeline by a method including at least one of: extracting, by the curation module, function blocks from the normalized ML pipelines, for each of the normalized ML pipelines: Dividing program code corresponding to the normalized ML pipeline into code cells, each code cell being a set of one or more statements present after a comment included in the program code; identifying code cells that appear repeatedly across multiple of the normalized ML pipelines as function blocks; identifying and extracting function blocks by assigning, by the curation module, to each of the function blocks a label generated based on comments in the function block or text in API statements in the function block; generating, by the adaptation module, a new ML pipeline for executing a new ML task on a new dataset for a new ML project; Decomposing the new ML pipeline into functional blocks; indexing the labeled function block in the normalized ML pipeline with 1 if the labeled function block in the normalized ML pipeline is the same as a function block in the new ML pipeline; preferentially incorporating function blocks from the normalized ML pipeline that contain many function blocks indexed with 1 as function blocks of the new ML pipeline; generating the new ML pipeline by A method comprising:

2. The step of extracting a function block from the normalized ML pipeline comprises: identifying code cells that appear repeatedly across a plurality of said normalized ML pipelines as idiomatic function blocks; identifying all other code cells in the normalized ML pipeline other than the idiomatic function blocks as custom function blocks; The method of claim 1 , comprising:

3. The steps of identifying the idiomatic function block and identifying the custom function block include: generating a directed graph; For each statement in each of the code cells: adding the statement as a node in the directed graph or incrementing a count in an existing node in the directed graph for the statement; calculating a co-occurrence score between the statement and the statement immediately preceding it in the code cell; generating an edge between the node of a statement and the node of a statement immediately preceding the statement if the co-occurrence score is greater than a co-occurrence score threshold; marking all nodes connected by edges as said idiomatic function blocks; marking each set of nodes corresponding to consecutive statements in any of the normalized ML pipelines that are not marked as one of the idiomatic function blocks as one of the custom function blocks; The method of claim 2 , comprising:

4. The step of assigning a label to each of the function blocks includes, for each of the function blocks: extracting text from comments in the function blocks, or, if no comments are present in the function blocks, mining text from documentation of API statements in the function blocks; identifying first common verbs and nouns or noun phrases in the extracted or mined text; generating the labels for the function blocks based on the first common verb and noun or noun phrase; The method of claim 1 , comprising:

5. The step of assigning a label to each of the function blocks, generating a similarity score for each pair of said function blocks in said normalized ML pipeline; generating a group of the function blocks if the similarity score of the function blocks is higher than a similarity score threshold; updating the labels assigned to each of the function blocks in each group of function blocks to a common label; The method of claim 4, comprising:

6. The step of updating the labels assigned to each of the function blocks in each group of function blocks to a common label includes: identifying a second common verb and noun or noun phrase in the labels of each of the function blocks in the group of function blocks; generating the common label for the group of function blocks based on the second common verb and noun or noun phrase; The method of claim 5 , comprising:

7. One or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed, cause a system to perform operations, the system including a curation module, a search module, and an adaptation module, the operations including: normalizing, by the curation module, existing machine learning (ML) pipelines of existing ML projects stored in a corpus of existing ML projects, each of the existing ML pipelines including program code organized into a corresponding set of code cells; Replacing unique variable names with common variable names, Replacing unique column names with common data types, Removing parameters from application program interface (API) statements; collapsing repeated instances of said API statement into a single instance; generating a normalized ML pipeline by a method including at least one of: extracting, by the curation module, function blocks from the normalized ML pipelines, for each of the normalized ML pipelines: Dividing program code corresponding to the normalized ML pipeline into code cells, each code cell being a set of one or more statements present after a comment included in the program code; identifying code cells that appear repeatedly across multiple of the normalized ML pipelines as function blocks; identifying and extracting function blocks by assigning, by the curation module, to each of the function blocks a label generated based on comments in the function block or text in API statements in the function block; generating, by the adaptation module, a new ML pipeline for executing a new ML task on a new dataset for a new ML project; Decomposing the new ML pipeline into functional blocks; indexing the labeled function block in the normalized ML pipeline with 1 if the labeled function block in the normalized ML pipeline is the same as a function block in the new ML pipeline; preferentially incorporating function blocks from the normalized ML pipeline that contain many function blocks indexed with 1 as function blocks of the new ML pipeline; generating the new ML pipeline by [0023] 1. One or more non-transitory computer-readable storage media, including:

8. The step of extracting a function block from the normalized ML pipeline comprises: identifying code cells that appear repeatedly across a plurality of said normalized ML pipelines as idiomatic function blocks; identifying all other code cells in the normalized ML pipeline other than the idiomatic function blocks as custom function blocks; 8. One or more non-transitory computer-readable storage media according to claim 7, comprising:

9. The steps of identifying the idiomatic function block and identifying the custom function block include: generating a directed graph; For each statement in each of the code cells: adding the statement as a node in the directed graph or incrementing a count in an existing node in the directed graph for the statement; calculating a co-occurrence score between the statement and the statement immediately preceding it in the code cell; generating an edge between the node of a statement and the node of a statement immediately preceding the statement if the co-occurrence score is greater than a co-occurrence score threshold; marking all nodes connected by edges as said idiomatic function blocks; marking each set of nodes corresponding to consecutive statements in any of the normalized ML pipelines that are not marked as one of the idiomatic function blocks as one of the custom function blocks; 10. One or more non-transitory computer-readable storage media according to claim 8, comprising:

10. The step of assigning a label to each of the function blocks includes, for each of the function blocks: extracting text from comments in the function blocks, or, if no comments are present in the function blocks, mining text from documentation of API statements in the function blocks; identifying first common verbs and nouns or noun phrases in the extracted or mined text; generating the labels for the function blocks based on the first common verb and noun or noun phrase; 8. One or more non-transitory computer-readable storage media according to claim 7, comprising:

11. The step of assigning a label to each of the function blocks, comprising: generating a similarity score for each pair of said function blocks in said normalized ML pipeline; generating a group of the function blocks if the similarity score of the function blocks is higher than a similarity score threshold; updating the labels assigned to each of the function blocks in each group of function blocks to a common label; 11. One or more non-transitory computer-readable storage media according to claim 10, comprising:

12. The step of updating the labels assigned to each of the function blocks in each group of function blocks to a common label includes: identifying a second common verb and noun or noun phrase in the labels of each of the function blocks in the group of function blocks; generating the common label for the group of function blocks based on the second common verb and noun or noun phrase; 12. One or more non-transitory computer-readable storage media according to claim 11, comprising:

13. 1. A system comprising: one or more processors; one or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed by the one or more processors, cause the system to perform operations, the system including a curation module, a search module, and an adaptation module, the operations including: normalizing, by the curation module, existing machine learning (ML) pipelines of existing ML projects stored in a corpus of existing ML projects, each of the existing ML pipelines including program code organized into a corresponding set of code cells; Replacing unique variable names with common variable names, Replacing unique column names with common data types, Removing parameters from application program interface (API) statements; collapsing repeated instances of said API statement into a single instance; generating a normalized ML pipeline by a method including at least one of: extracting, by the curation module, function blocks from the normalized ML pipelines, for each of the normalized ML pipelines: Dividing program code corresponding to the normalized ML pipeline into code cells, each code cell being a set of one or more statements present after a comment included in the program code; identifying code cells that appear repeatedly across multiple of the normalized ML pipelines as function blocks; identifying and extracting function blocks by assigning, by the curation module, to each of the function blocks a label generated based on comments in the function block or text in API statements in the function block; generating, by the adaptation module, a new ML pipeline for executing a new ML task on a new dataset for a new ML project; Decomposing the new ML pipeline into functional blocks; indexing the labeled function block in the normalized ML pipeline with 1 if the labeled function block in the normalized ML pipeline is the same as a function block in the new ML pipeline; preferentially incorporating function blocks from the normalized ML pipeline that contain many function blocks indexed with 1 as function blocks of the new ML pipeline; generating the new ML pipeline by Including, the system.

14. The step of extracting a function block from the normalized ML pipeline comprises: identifying code cells that appear repeatedly across a plurality of said normalized ML pipelines as idiomatic function blocks; identifying all other code cells in the normalized ML pipeline other than the idiomatic function blocks as custom function blocks; The system of claim 13 , comprising:

15. The steps of identifying the idiomatic function block and identifying the custom function block include: generating a directed graph; For each statement in each of the code cells: adding the statement as a node in the directed graph or incrementing a count in an existing node in the directed graph for the statement; calculating a co-occurrence score between the statement and the statement immediately preceding it in the code cell; generating an edge between the node of a statement and the node of a statement immediately preceding the statement if the co-occurrence score is greater than a co-occurrence score threshold; marking all nodes connected by edges as said idiomatic function blocks; marking each set of nodes corresponding to consecutive statements in any of the normalized ML pipelines that are not marked as one of the idiomatic function blocks as one of the custom function blocks; The system of claim 14 , comprising:

16. The step of assigning a label to each of the function blocks includes, for each of the function blocks: extracting text from comments in the function blocks, or, if no comments are present in the function blocks, mining text from documentation of API statements in the function blocks; identifying first common verbs and nouns or noun phrases in the extracted or mined text; generating the labels for the function blocks based on the first common verb and noun or noun phrase; The system of claim 13 , comprising:

17. The step of assigning a label to each of the function blocks, generating a similarity score for each pair of said function blocks in said normalized ML pipeline; generating a group of the function blocks if the similarity score is higher than a similarity score threshold; updating the labels assigned to each of the function blocks in each group of function blocks to a common label; Further comprising: The step of updating the labels assigned to each of the function blocks in each group of function blocks to a common label comprises: identifying a second common verb and noun or noun phrase in the labels of each of the function blocks in the group of function blocks; generating the common label for the group of the function blocks based on the second common verb and noun or noun phrase; 17. The system of claim 16, comprising:

Images