Extending the machine learning pipeline corpus to synthesize new machine learning pipelines

Abstract

To provide augmentation of machine learning pipeline corpus for synthesizing new machine leaning pipelines.SOLUTION: Operations may include: receiving an ML project stored in an ML corpus database; and mutating a first ML pipeline, of a set of first ML pipelines associated with the received ML project, to determine a set of second ML pipelines. The mutation of the first ML pipeline may correspond to a substitution of a first ML model associated with the first ML pipeline with a second ML model associated with one of the set of predefined ML pipelines. The operations may further include: selecting one or more ML pipelines from the set of second ML pipelines based on a performance score associated with each of the determined set of ML pipelines; and augmenting the ML corpus database to include the selected one or more ML pipelines and the set of first ML pipeline.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] [Cross - References to Related Applications / Incorporation by Reference] This application claims priority to U.S. Provisional Patent Application No. 63 / 260908, filed on September 3, 2021, with the title “Using Data Augmentation With Learning From Human - Written Pipelines To Generate High - Quality ML Pipelines”, the entire content of which is hereby incorporated by reference into this application.

[0002] [Field] The embodiments discussed in this disclosure relate to the expansion of a machine learning pipeline corpus for the synthesis of new machine learning pipelines.

Background Art

[0003] Advances in the fields of Artificial Intelligence (AI) and Machine Learning (ML) have led to the application of AI / ML algorithms in various fields. Usually, an ML pipeline can be manually created by a data scientist for a given dataset. The manual creation of an ML pipeline can be a time - consuming task, and such tasks may require a significant amount of effort from expert users such as data scientists. In recent years, some techniques for the automatic generation of ML pipelines for datasets have been developed. Current techniques for the automatic generation of ML pipelines typically follow a search - based approach, where a vast space of possible ML pipelines can be repeatedly searched to find the optimal pipeline for a given dataset based on the instantiation and testing of multiple candidate ML pipelines. Thus, current techniques for the automatic generation of ML pipelines may not be able to generate accurate ML pipelines and may require significant computational time and resources.

[0004] The scope claimed in this disclosure is not limited to embodiments that operate only in the environments described above or embodiments that eliminate any defects. Rather, this background is provided solely to illustrate an example of the technical scope in which some of the embodiments described in this disclosure may be implemented. [Overview of the project]

[0005] Depending on the embodiment, the operation may include receiving ML projects from multiple ML projects stored in a machine learning (ML) corpus database, where each of the multiple ML projects may include a dataset and a set of ML pipelines applicable to the dataset. The operation may further include mutating the first ML pipeline in the first set of ML pipelines associated with the received ML project to determine a second set of ML pipelines based on a predefined set of ML pipelines, where the mutation of the first ML pipeline may correspond to replacing the first ML model associated with the first ML pipeline with a second ML model associated with one of the predefined sets of ML pipelines. The operation may further include selecting one or more ML pipelines from the determined second set of ML pipelines based on the performance scores associated with each of the determined second set of ML pipelines. The operation may further include extending the ML corpus database to include the first set of ML pipelines associated with the received ML project and the one or more selected ML pipelines.

[0006] The objectives and advantages of the embodiments are realized and achieved by at least the elements, features, and combinations specifically indicated in the claims.

[0007] Both the above summary and the following detailed description are illustrative and descriptive, and do not limit the claimed invention.

[0008] Exemplary embodiments are described and explained with further identification and detail through the use of the accompanying drawings. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram illustrates an example of an environment for extending a machine learning pipeline corpus. [Figure 2] This is a block diagram of the system for extending the machine learning pipeline corpus. [Figure 3] This diagram shows a flowchart illustrating an example of a method for extending a machine learning pipeline corpus for the synthesis of new machine learning pipelines. [Figure 4] This diagram shows a flowchart illustrating an example of how to mutate the first ML pipeline within a set of ML pipelines associated with a received ML project, in order to determine the set of the second ML pipeline. [Figure 5] This diagram illustrates an exemplary scenario for extending a machine learning pipeline corpus based on a predefined machine learning pipeline. [Figure 6A] This diagram illustrates an exemplary scenario for identifying a code snippet of a first ML model related to an exemplary first ML pipeline. [Figure 6B] This diagram illustrates an exemplary scenario for instantiation of a second ML model selected from a predefined set of models associated with a predefined set of ML pipelines. [Figure 7] This diagram illustrates a flowchart showing an example of how to instantiate a second ML model selected from a predefined set of models associated with a predefined set of ML pipelines. [Figure 8] This diagram shows a flowchart illustrating an example of a method for identifying one or more statements related to the first ML model. [Figure 9] This diagram shows an example flowchart of how to obtain a model slice to identify a code snippet of the first ML model. [Figure 10]This diagram shows a flowchart illustrating an example of a method for training a meta-learning model. [Figure 11] This represents an exemplary scenario for training a meta-learning model. [Modes for carrying out the invention]

[0010] All figures are represented in accordance with at least one embodiment described in this disclosure.

[0011] Some embodiments described in this disclosure relate to methods and systems for extending a machine learning pipeline corpus for the synthesis of new machine learning pipelines. In this disclosure, a machine learning (ML) project may be received from multiple ML projects stored in an ML corpus database. Furthermore, a first ML pipeline in a first set of ML pipelines associated with the received ML project may be mutated to determine a second set of ML pipelines based on a predefined set of ML pipelines. Here, the mutation of the first ML pipeline may correspond to replacing a first ML model associated with the first ML pipeline with a second ML model associated with one of the predefined sets of ML pipelines. Subsequently, one or more ML pipelines may be selected from the determined set of second ML pipelines based on the performance scores associated with each of the determined sets of second ML pipelines. Furthermore, the ML corpus database may be extended to include the first set of ML pipelines associated with the received ML project and one or more selected ML pipelines.

[0012] In accordance with one or more embodiments of this disclosure, the field of artificial intelligence (AI) / machine learning (ML) can be improved by configuring a computing system so that it can extend an ML pipeline corpus for the synthesis of new ML pipelines. The computing system may receive ML projects from a plurality of ML projects stored in an ML corpus database, where each of the plurality of ML projects may include a dataset and a set of ML pipelines applicable to the dataset. The computing system may mutate a first ML pipeline in the first set of ML pipelines associated with the received ML project to determine a second set of ML pipelines based on a predefined set of ML pipelines, where the mutation of the first ML pipeline may correspond to replacing a first ML model associated with the first ML pipeline with a second ML model associated with one of the predefined sets of ML pipelines. Furthermore, the first set of ML pipelines may correspond to each ML pipeline associated with the received ML project. The first ML pipeline may be selected from the first set of ML pipelines. Subsequently, the computing system may select one or more ML pipelines from the determined set of second ML pipelines based on the performance scores associated with each of them. Furthermore, the computing system may extend the ML corpus database to include the set of first ML pipelines associated with the received ML project and the one or more selected ML pipelines.

[0013] Conventional methods used to generate ML pipelines may require explicit exploration of a large space of possible ML pipelines to determine the optimal ML pipeline for an ML project. Therefore, conventional techniques for automated ML pipeline generation may not be able to generate accurate ML pipelines and may require considerable computation time and resources. On the other hand, the disclosed technique (executed by a computing system) may include mutating a first ML pipeline in a set of first ML pipelines associated with a received ML project to determine a set of second ML pipelines. Here, the mutation of the first ML pipeline may correspond to replacing a first ML model associated with the first ML pipeline with a second ML model associated with one of the predefined sets of ML pipelines. Furthermore, the disclosed technique may include selecting one or more ML pipelines from the determined set of second ML pipelines based on the performance scores associated with each of the determined sets of second ML pipelines. The ML corpus database may be extended to include the selected one or more ML pipelines and the sets of first ML pipelines. Therefore, the disclosed technology allows for the selection of such mutated ML pipelines that may have high performance scores, thereby ensuring good accuracy of the final set of ML pipelines in the augmented ML corpus database.

[0014] The electronic devices of this disclosure may follow a generative approach in which a meta-learning model learns from a corpus of existing ML pipelines created by data scientists on other datasets and uses it to efficiently and optimally synthesize ML pipelines for new datasets. This disclosure may substantially resolve key challenges of automated machine learning approaches based on generative learning. The electronic devices of this disclosure may augment an ML corpus database using data augmentation techniques that can systematically mutate a given ML pipeline by replacing ML models used in existing ML pipelines in a corpus with other viable choices to generate a population of new ML pipelines. The population of new ML pipelines may be used to provide the meta-learning model with higher quality and more consistent learning features, so that the meta-learning model can learn from the augmented ML corpus database and subsequently synthesize new, higher quality ML pipelines for the user database. Mutation may employ a novel Abstract Syntax Tree (AST) level analysis of the human-written pipeline to extract the necessary program elements while considering all syntactic and stylistic variations of the human-written program that an ML pipeline might typically contain, and to mutate them in a syntactically well-formed manner. Thus, new ML pipelines can be synthesized efficiently.

[0015] Embodiments of this disclosure will be described with reference to the accompanying drawings.

[0016] Figure 1 is a diagram illustrating an example of an environment related to the extension of a machine learning pipeline corpus according to at least one embodiment described herein. Referring to Figure 1, Environment 100 is shown. Environment 100 may include an electronic device 102, a server 104, a user end device 106, a database 108, and a communication network 110. The electronic device 102, the server 104, and the user end device 106 may be coupled to each other so as to be able to communicate with each other via the communication network 110. The electronic device 102 may have a meta-learning model 102A. Figure 1 further shows a user 112 who is associated with or can operate the electronic device 102 (or user end device 106). The database 108 may include a plurality of ML projects. For example, the plurality of ML projects may include 'n' ML projects, such as ML project-1 114, ML project-2 116, ... and ML project-n 118. Each of the plurality of ML projects may include a dataset and a set of ML pipelines applicable to the dataset. For example, ML project-1 114 may include dataset 114A and ML pipeline set 114B. Furthermore, ML project-2 116 may include dataset 116A and ML pipeline set 116B. Similarly, ML project-n 118 may include dataset 118A and ML pipeline set 118B.

[0017] The number "n" of ML projects shown in Figure 1 is for illustrative purposes only. Multiple ML projects may include as few as two ML projects or more than "n" ML projects without departing from the scope of this disclosure. For simplicity, Figure 1 simply shows "n" ML projects. However, in some embodiments, there may be more than "n" ML projects without limiting the scope of this disclosure.

[0018] The electronic device 102 may include appropriate logic, circuitry, and interfaces configured to receive a machine learning (ML) project (e.g., ML project-1 114) from a plurality of ML project stored in an ML corpus database (e.g., database 108). The electronic device 102 may further be configured to mutate a first ML pipeline in a first set of ML pipelines associated with the received ML project to determine a second set of ML pipelines based on a predefined set of ML pipelines. The electronic device 102 may further be configured to select one or more ML pipelines from the determined second set of ML pipelines based on performance scores associated with each of the determined second set of ML pipelines. The electronic device 102 may further be configured to expand the ML corpus database to include the first set of ML pipelines and the selected one or more ML pipelines associated with the received ML project. Examples of the electronic device 102 may include, but are not limited to, computing devices, smartphones, cellular phones, mobile phones, gaming consoles, mainframe machines, servers, computer workstations, and / or consumer electronic (CE) devices.

[0019] The meta-learning model 102A may include appropriate logic, circuitry, interfaces, and / or code that can be configured to use a meta-learning algorithm to generate a predictive model based on a previously trained model (e.g., an ML pipeline) and dataframes or features. The meta-learning model 102A may learn from the outputs of other learning algorithms. For example, for prediction, the meta-learning model 102A may learn based on the outputs of other learning algorithms. In other examples, the meta-learning model 102A may be supplied with the parameters of other ML models (e.g., a neural network model, a polynomial regression model, a random forest classifier, a logistic regression model, or an ensemble learning model) and dataframes / features corresponding to each ML algorithm. The meta-learning model 102A may learn meta-features and meta-heuristics based on the parameters associated with the dataframes / features and the respective input ML models. In the example, once the meta-learning model 102A is trained, it may be used to generate an ML pipeline based on input features or dataframes related to an ML project.

[0020] Server 104 may include appropriate logic, circuitry, and interfaces, and / or code configured to receive a machine learning (ML) project from a plurality of ML projects stored in an ML corpus database (e.g., database 108). Server 104 may further be configured to mutate a first ML pipeline in a set of first ML pipelines associated with the received ML project to determine a set of second ML pipelines. Server 104 may further be configured to select one or more ML pipelines from the determined set of second ML pipelines based on performance scores associated with each in the determined set of second ML pipelines. Server 104 may further be configured to expand the ML corpus database to include the set of first ML pipelines and the selected one or more ML pipelines associated with the received ML project. Server 104 may be implemented as a cloud server and may execute operations through a web application, a cloud application, an HTTP request, a repository operation, a file transfer, etc. Other examples of server 104 may include, but are not limited to, a database server, a file server, a web server, a media server, an application server, a mainframe server, or a cloud computing server.

[0021] In at least one embodiment, server 104 may be implemented as a plurality of distributed cloud-based resources by use of some techniques well known to those skilled in the art. Those skilled in the art will understand that the scope of the present disclosure cannot be limited to the implementation of server 104 and electronic device 102 as two separate entities. In certain embodiments, the functions of server 104 may be incorporated, in whole or at least in part, into electronic device 102 without departing from the scope of the present disclosure. In certain embodiments, server 104 may host database 108. Alternatively, server 104 may be separate from database 108 and may be communicatively coupled to database 108.

[0022] The user end device 106 may have appropriate logic, circuitry, interfaces, and / or code that can be configured to store real-time applications on which new ML pipelines can be synthesized based on extensions of an ML corpus. In some embodiments, the user end device 106 may receive a first user input from a user (e.g., a data scientist such as user 112) and generate a predefined set of ML pipelines based on the received first user input. In other embodiments, the user end device 106 may receive one or more second user inputs from a user (e.g., a data scientist such as user 112) and generate a set of ML pipelines associated with each of a plurality of ML projects based on one or more second user inputs. Furthermore, the user end device 106 may receive multiple datasets associated with the plurality of projects from various sources such as online dataset repositories, code repositories, and online open-source projects. The user end device 106 may be configured to upload the predefined set of ML pipelines, the plurality of ML pipelines and the plurality of datasets associated with the plurality of projects to the server 104. Multiple uploaded ML pipelines and datasets may be stored as multiple ML projects in the ML corpus database within database 108. A set of predefined uploaded ML pipelines may also be stored in database 108 along with multiple ML projects. Examples of user end devices 106 may include, but are not limited to, mobile devices, desktop computers, laptops, computer workstations, computing devices, mainframe machines, servers, such as cloud servers, and groups of servers.

[0023] Database 108 may include appropriate logic, interfaces, and / or code that can be configured to store multiple ML projects, each ML project may include a dataset and a set of ML pipelines applicable to the dataset. Database 108 may further store predefined sets of ML pipelines. Database 108 may be derived from data from relational or non-relational databases, or from comma-separated values ​​(csv) files in conventional or big data storage. Database 108 may be stored or cached on a server (e.g., server 104) or a device such as electronic device 102. The device storing database 108 may be configured to receive queries from electronic device 102 for machine learning (ML) projects from multiple ML projects. In response, database 108 may be configured to read the queried machine learning ML project, including the dataset and the set of ML pipelines applicable to the dataset, based on the received query, and supply it to electronic device 102.

[0024] In some embodiments, the database 108 may be hosted on multiple servers stored in the same or different locations. The operation of the database 108 may be performed using hardware including a processor, a microprocessor (e.g., for performing or controlling one or more operations), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In some other cases, the database 108 may be implemented using software.

[0025] The communication network 110 may include a communication medium through which the electronic device 102, server 104, and user end device 106 can communicate with each other. The communication network 110 may be either a wired or wireless connection. Examples of the communication network 110 may include, but are not limited to, the Internet, a cloud network, a cellular or wireless mobile network (e.g., Long-Term Evolution and 5G New Radio), a Wireless Fidelity (Wi-Fi) network, a Personal Area Network (PAN), a Local Area Network (LAN), or a Metropolitan Area Network (MAN). Various devices in environment 100 may be configured to connect to the communication network 110 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of the following: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Zig Bee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocols, and Bluetooth® (BT) communication protocols.

[0026] In operation, the electronic device 102 may receive ML projects (e.g., ML project-1 114) from multiple ML projects stored in an MP corpus database (e.g., database 108). Here, each of the multiple ML projects may include a dataset and a set of ML pipelines applicable to the dataset. An ML project may help an application perform a task such as a prediction task (e.g., classification or regression), even if it is not programmed to do so. A dataset may include historical data corresponding to a specific ML task defined with respect to that dataset. An ML pipeline or script or program may be a sequence of actions to train an ML model for a specific ML prediction task. It can be understood that a data scientist or developer may generate several ML pipelines and datasets and upload such ML pipelines and datasets to the internet as a knowledge base in various online source code or ML repositories. The knowledge base may be downloaded from the internet and stored in database 108. The electronic device 102 may receive ML projects from database 108 from multiple ML projects. For example, electronic device 102 may receive ML project-1 114. Here, electronic device 102 may receive a dataset 114A and a set of ML pipelines 114B corresponding to ML project-1 114. Details of multiple ML projects are further given, for example, in Figure 5.

[0027] It should be noted that human-written ML pipelines, while constituting training corpora for learning-based machine learning approaches, may not contain the best representative ML pipeline solutions for each dataset. Even worse, pipelines can vary in quality, potentially lacking any learnable patterns regarding which components can be used for which types of datasets. This problem can be particularly serious when ML models are used in ML pipelines as ML model choices, potentially severely impacting the accuracy of the ML pipeline. Human-written ML pipelines can be instantiated in many ways, such as under cross-validation and hyperparameter optimization. Safe refactoring of syntactically and semantically correct human-written ML pipelines can be critical and difficult. Therefore, human-written ML pipelines may need to be mutated.

[0028] The electronic device 102 may mutate the first ML pipeline in the set of ML pipelines associated with the received ML project so as to determine the second ML pipeline set based on a predefined set of ML pipelines. Here, the mutation of the first ML pipeline may correspond to replacing the first ML model associated with the first ML pipeline with a second ML model associated with one of the predefined sets of ML pipelines. It should be noted that the mutation may improve the accuracy of the ML pipeline by systematically modifying the ML model of the original ML pipeline. As will be explained, the first ML pipeline may be written by a human and may not be optimal for various reasons. In one example, the data scientist who wrote the set of ML pipelines associated with the received ML project may not be an expert, and therefore the optimal ML model may not be used. In another example, the set of ML pipelines associated with the received ML project may not be optimal due to the unavailability of the optimal model for the dataset. Therefore, the set of ML pipelines applicable to the dataset corresponding to the received ML project may need to be mutated to improve its accuracy. Human-written ML pipelines can be syntactically and semantically diverse. Therefore, there can be no common pattern for replacing an original ML pipeline with another. The electronic device 102 of this disclosure may mutate a first ML pipeline associated with a received ML project to determine a second set of ML pipelines based on a predefined set of ML pipelines. In an embodiment, the first ML pipeline may be mutated to determine a second ML pipeline. In an alternative embodiment, the first ML pipeline may be mutated to determine a second set of ML pipelines. In an embodiment, the electronic device 102 may mutate each ML pipeline or a selected group of ML pipelines within the set of ML pipelines associated with a received ML project. Details of the mutation of the first ML pipeline are further given, for example, in Figure 3.

[0029] The electronic device 102 may select one or more ML pipelines from the determined set of second ML pipelines based on the performance score associated with each of the determined sets of second ML pipelines. Each of the sets of second ML pipelines may be ranked based on the performance score associated with each of the determined sets of second ML pipelines. The performance score may be an F1 score, an R2 score, etc., associated with the corresponding ML pipeline in the set of second ML pipelines. For a given dataset and original ML pipeline, the electronic device 102 may execute each mutated ML pipeline on the training dataset in a manner similar to how a data scientist would execute an ML pipeline on the dataset. Since the objective of this disclosure may be to extend the machine learning pipeline corpus with robust ML pipelines, only the best mutated ML pipeline may be retained for meta-learning model training along with existing ML pipelines. One or more ML pipelines selected from the determined set of second ML pipelines may be the best ML pipelines that may be retained for meta-learning model training. Further details of the selection of one or more ML pipelines are given, for example, in Figure 3.

[0030] The electronic device 102 may extend the ML corpus database in database 108 to include a set of first ML pipelines associated with an received ML project (e.g., ML project-1 114) and one or more selected ML pipelines. Extensions can be standard techniques for improving the quality of the ML corpus database. Systematic data extensions may be used to improve the ML corpus database. If one or more ML pipelines can be selected, the electronic device 102 may add the selected ML pipelines to database 108 to improve the quality of database 108. In one embodiment, the electronic device 102 may add the selected ML pipelines to database 108 along with the set of first ML pipelines. In another embodiment, the electronic device 102 may replace the set of first ML pipelines in database 108 with the selected ML pipelines. Further details of the extension of the ML corpus database are given, for example, in Figure 5.

[0031] The electronic device 102 can help improve the training of the meta-learning model 102A based on an extension of an ML corpus database that uses high-quality ML pipelines that can function well on a given dataset. The meta-learning model 102A can generate an abstract version of the ML pipeline. Therefore, if the accuracy of the meta-learning model 102A falls below a certain quality threshold, the generated ML pipeline may not be optimal. On the other hand, if the quality of the meta-learning model 102A is good or above a certain quality threshold, the generated ML pipeline may be better. Consequently, if the training of the meta-learning model 102A is based solely on the original ML pipeline that can be obtained directly from the internet, the quality of the meta-learning model 102A may be unacceptable, even if care was taken during the download of the original ML pipeline.

[0032] Without departing from the scope of this disclosure, modifications, additions, or omissions may be made to Figure 1. For example, the environment 100 may include more or fewer elements than those illustrated and described in this disclosure. For example, in some embodiments, the environment 100 may include an electronic device 102 but not a database 108. Furthermore, in some embodiments, the respective functions of the database 108 and the server 104 may be incorporated into the electronic device 102 without departing from the scope of this disclosure.

[0033] Figure 2 is a block diagram of a system for extending a machine learning pipeline corpus according to at least one embodiment described herein. Figure 2 is described in relation to elements from Figure 1. Referring to Figure 2, a block diagram 200 of system 202 including electronic device 102 is shown. Electronic device 102 may include a processor 204, memory 206, meta-learning model 102A, input / output (I / O) devices 208 (including display device 208A), and network interface 210.

[0034] The processor 204 may include appropriate logic, circuitry, and interfaces that can be configured to execute sets of instructions stored in memory 206. The processor 204 may be configured to execute program instructions related to different operations performed by the electronic device 102. For example, some operations may include receiving a machine learning (ML) project, mutating a first ML pipeline, selecting one or more ML pipelines from a determined set of second ML pipelines, and extending an ML corpus database. The processor 204 may be implemented based on a number of processor technologies known in the art. Examples of processor technologies may include, but are not limited to, central processing units (CPUs), x86-based processors, reduced instruction set computing (RISC) processors, application-specific integrated circuit (ASIC) processors, complex instruction set computing (CISC) processors, graphical processing units (GPUs), and other processors.

[0035] Although represented as a single processor in Figure 2, processor 204 may include any number of processors configured to perform or direct any number of operations of electronic device 102 individually or collectively, as described in this disclosure. Furthermore, one or more processors may reside in one or more different electronic devices, such as different servers. In some embodiments, processor 204 may be configured to interpret and / or execute program instructions stored in memory 206 and / or process stored data. After program instructions are loaded into memory 206, processor 204 may execute the program instructions. Some examples of processor 204 may be graphics processing units (GPUs), central processing units (CPUs), reduced instruction set computer (RISC) processors, ASIC processors, multiple instruction set computer (CISC) processors, coprocessors, and / or combinations thereof.

[0036] Memory 206 may include appropriate logic, circuitry, and interfaces that can be configured to store one or more instructions executed by the processor 204. One or more instructions stored in memory 206 can be executed by the processor 204 to perform different operations of the processor 204 (and electronic device 102). Memory 206 may be configured to store multiple ML projects, a predefined set of ML pipelines, a first set of ML pipelines, a second set of ML pipelines, and one or more selected ML pipelines. Examples of memory implementations may include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), hard disk drives (HDDs), solid-state drives (SSDs), CPU caches, and / or secure digital (SD) cards.

[0037] The I / O device 208 may include appropriate logic, circuitry, and interfaces that can be configured to receive input from user 112 and supply output based on the received input. For example, the I / O device 208 may receive user input from user 112 related to ML pipelines, datasets associated with ML pipelines, or the generation of an ML project. Furthermore, the I / O device 208 may render predicted outputs of a meta-learning model 102A that can be trained based on a determined set of second ML pipelines, one or more selected ML pipelines, and / or an extended ML corpus database. The I / O device 208 may include various input and output devices and may be configured to communicate with the processor 204. Examples of the I / O device 208 may include, but are not limited to, a touchscreen, keyboard, mouse, joystick, microphone, display device (e.g., display device 208A), and speaker.

[0038] The display device 208A may include appropriate logic, circuitry, and interfaces that can be configured to display a set of second ML pipelines and / or one or more selected ML pipelines. The display device 208A may be a touchscreen that allows a user (e.g., user 112) to provide user input through the display device 208A. The touchscreen may be at least one of resistive, capacitive, or thermal touchscreens. The display device 208A may be implemented through several known technologies or other display devices, such as liquid crystal display (LCD) displays, light-emitting diode (LED) displays, plasma displays, or organic LED (OLED) display technologies, without limitation. Depending on the embodiment, the display device 208A may refer to a display screen of a head-mounted device (HMD), smart glasses device, see-through display, projection display, electrochromic display, or transparent display.

[0039] The network interface 210 may include appropriate logic, circuitry, and interfaces that can be configured to facilitate communication between the processor 204, the server 104, and the user end device 106 (or any other device in the environment 100) via the communication network 110. The network interface 210 may be implemented by using various known techniques to support wired or wireless communication of the electronic device 102 with the communication network 110. The network interface 210 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identification module (SIM) card, or a local buffer circuit. The network interface 210 may be configured to communicate wirelessly with the Internet, an intranet, or a wireless network, such as a cellular telephone network, a wireless local area network (LAN), and a metropolitan area network (MAN).Wireless communication may be configured to use one or more of several communication standards, protocols, and technologies, such as GSM (Global System for Mobile communications), EDGE (Enhanced Data GSM Environment, EDGE), Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), 5G New Radio (NR), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, or IEEE 802.11n), VoIP (Voice over Internet Protocol), Light Fidelity (Li-Fi), Wi-MAX (Worldwide Interoperability for Microwave Access), protocols for email, instant messaging, and Short Message Service (SMS).

[0040] Without departing from the scope of this disclosure, modifications, additions, or omissions may be made to the exemplary electronic device 102. For example, in some embodiments, the exemplary electronic device 102 may include any number of other components that are not explicitly illustrated or described for the sake of brevity.

[0041] Figure 3 is a flowchart illustrating an example of a method for extending a machine learning pipeline corpus for synthesizing a new machine learning pipeline, according to embodiments of the present disclosure. Figure 3 is described in relation to elements from Figures 1 and 3. Referring to Figure 3, flowchart 300 is shown. The method represented in flowchart 300 can begin from block 302 and may be performed by any suitable system, apparatus, or device, for example, by the electronic device 102 in the example of Figure 1 or the processor 204 in Figure 2. Steps and operations related to one or more blocks of flowchart 300, although represented in separate blocks, may be divided into additional blocks, combined into fewer blocks, or deleted, depending on the particular implementation.

[0042] In block 302, machine learning (ML) projects (e.g., ML project-1 114) may be received from multiple ML projects stored in an ML corpus database (e.g., database 108). Here, each of the multiple ML projects may include a dataset and a set of ML pipelines applicable to the dataset. Processor 204 may be configured to receive ML projects from multiple ML projects stored in an ML corpus database (e.g., database 108). An ML project may help an application perform a task, such as a prediction task (e.g., classification or regression), even if it is not programmed to do so. A dataset may include historical data corresponding to a specific ML task defined with respect to that dataset. An ML pipeline may include a sequence of actions that can be used to train an ML model for a specific ML prediction task. It may be understood that a data scientist or developer may generate several ML pipelines and datasets and upload such ML pipelines and datasets to the internet as a knowledge base in various online source code or ML repositories. The knowledge base may be downloaded from the internet and stored in database 108. The electronic device 102 can receive ML projects from multiple ML projects in the database 108. For example, the electronic device 102 can receive ML project-1 114. Here, the electronic device 102 can receive a dataset 114A and a set of ML pipelines 114B corresponding to ML project-1 114.

[0043] In block 304, the first ML pipeline in the set of first ML pipelines associated with the received ML project may be mutated based on a predefined set of ML pipelines to determine the set of second ML pipelines. Here, the mutation of the first ML pipeline may correspond to replacing the first ML model associated with the first ML pipeline with a second ML model associated with one of the predefined sets of ML pipelines. Furthermore, the set of first ML pipelines may correspond to each ML pipeline associated with the received ML project. The first ML pipeline may be selected from the set of first ML pipelines. The processor 204 may be configured to mutate the first ML pipeline associated with the received ML project based on one of the predefined sets of ML pipelines to determine the set of second ML pipelines. It should be noted that the mutation may improve the accuracy of the ML pipeline by systematically changing the ML model of the original ML pipeline. As described, the first ML pipeline may be written by a human and may not be optimal for various reasons. In one example, the data scientist who can write the first set of ML pipelines associated with the received ML project may not be an expert, and therefore the optimal ML model may not be used. In another example, the first set of ML pipelines associated with the received ML project may be suboptimal due to the unavailability of the optimal model for the dataset. Therefore, the first set of ML pipelines associated with the received ML project may need to be mutated to improve its accuracy. Human-written ML pipelines can be syntactically and semantically diverse. Therefore, there can be no common pattern that can be used to replace an original ML pipeline with another ML pipeline. The processor 204 may mutate the first ML pipeline associated with the received ML project to determine a second set of ML pipelines based on a predefined set of ML pipelines. In one embodiment, the first ML pipeline may be mutated to determine a second ML pipeline.In an alternative embodiment, the first ML pipeline may be mutated to determine the set of second ML pipelines. In an embodiment, the electronic device 102 may mutate each ML pipeline in the set of ML pipelines associated with the received ML project one by one. Further details of the mutation of the first ML pipeline are given, for example, in Figure 5.

[0044] In block 306, one or more ML pipelines may be selected from the determined set of second ML pipelines based on the performance score associated with each of them in the determined set of second ML pipelines. The processor 204 may be configured to select one or more ML pipelines from the determined set of second ML pipelines based on the performance score associated with each of them in the determined set of second ML pipelines. Each of the second ML pipelines in the set may be ranked based on the performance score associated with each of them in the determined set of second ML pipelines. For a given dataset and original ML pipelines, the electronic device 102 may automatically execute each mutated ML pipeline on the training dataset in a manner similar to how a data scientist would execute an ML pipeline on the dataset. Since the objective of this disclosure may be to extend the machine learning pipeline corpus with robust ML pipelines, only the best mutated ML pipelines may be retained along with existing ML pipelines. The one or more ML pipelines selected from the determined set of second ML pipelines may be the best ML pipelines that may be retained for meta-learning model training.

[0045] In embodiments, the performance scores associated with each of the determined sets of second ML pipelines may, but are not limited to, the F1 score or R2 score associated with the corresponding ML pipeline. In an example, the performance score of an ML pipeline may correspond to the ratio of the F1 score and R2 score of the ML pipeline. It can be understood that the F1 score (F1-score) can determine the accuracy of a given ML project for a given dataset, based on the harmonic mean of the precision and recall of a given ML model. Here, the precision of the model may be determined based on the ratio of the number of true positive results to the total number of positive results, including true positives and false positives. The recall or sensitivity of a given ML model may be determined based on the ratio of the number of true positive results to the total number of true positives and false negatives. The F1 score can be determined based on equation (1) as follows: F1 score = 2 × ((precision × recall) / (precision + recall)) (1)

[0046] The F1 score can range from "0" to "1". Here, the F1 score is close to "1" when the precision and recall of a given ML model are close to "1". Similarly, the F1 score is close to "0" when the precision or recall of a given ML model is close to "0". In an embodiment, the F1 scores associated with each of the determined second set of ML pipelines may be determined to select one or more ML pipelines. In an example, the determined set of ML pipelines may include five ML pipelines such as ML pipeline-1 with an F1 score of "0.2", ML pipeline-2 with an F1 score of "0.27", ML pipeline-3 with an F1 score of "0.55", ML pipeline-4 with an F1 score of "0.65", and ML pipeline-5 with an F1 score of "0.72". Here, ML pipeline-5 may be the most accurate ML pipeline among the five ML pipelines. Thus, ML pipeline-5 may be selected as one or more ML pipelines.

[0047] The R² score, also known as the R-squared score, can be a coefficient that indicates how well a given ML model can fit a dataset. The R² score can represent how the variation in the independent variables may affect the dependent variables of a given ML model. The R² score can be determined based on the ratio of the regression variation or residual sum of squares (SSR) to the total sum of squares (TSS), where SSR can be the total variation of the predicted values ​​from the mean of all dependent variables, and TSS can be the total variation of the actual values ​​from the mean. Similar to the F1 score, the R² score can range between "0" and "1". When the R² score is "1", the change in the dependent variable can be fully explained by the change in the independent variable. In other words, the closer the R² score is to "1", the better the given ML model can fit the dataset. In embodiments, the R² scores associated with each of the determined second set of ML pipelines may be determined to select one or more ML pipelines. In the example, the determined set of second ML pipelines may include five ML pipelines, such as ML pipeline-1 with an R2 score of "0.4", ML pipeline-2 with an R2 score of "0.5", ML pipeline-3 with an R2 score of "0.75", ML pipeline-4 with an R2 score of "0.65", and ML pipeline-5 with an R2 score of "0.6". Here, since ML pipeline-3 may have the highest R2 score among the five ML pipelines, ML pipeline-3 may be the best fit among the five ML pipelines. Therefore, ML pipeline-3 may be selected as one or more ML pipelines.

[0048] In an embodiment, one or more ML pipelines selected from the determined set of second ML pipelines may include, but are not limited to, an ML pipeline associated with the highest performance score (from the determined set of second ML pipelines), a first group of ML pipelines that can correspond to performance scores above a threshold (from the determined set of second ML pipelines), or a second group of ML pipelines that can correspond to a predefined number of top ML pipelines based on performance scores (from the determined set of second ML pipelines).

[0049] For example, one or more selected ML pipelines may include ML pipelines (from the determined set of second ML pipelines) associated with the highest performance score (e.g., F1 score or R2 score). Here, the ML pipelines with the highest performance score among the determined set of second ML pipelines may be selected as one or more ML pipelines. In the example, the determined set of ML pipelines may include seven ML pipelines such as ML pipeline-1 with an R2 score of "0.4", ML pipeline-2 with an R2 score of "0.5", ML pipeline-3 with an R2 score of "0.75", ML pipeline-4 with an R2 score of "0.65", ML pipeline-5 with an R2 score of "0.6", ML pipeline-6 ​​with an R2 score of "0.8", and ML pipeline-7 with an R2 score of "0.49". Here, ML pipeline-6 ​​has the highest R2 score of 0.8. Therefore, ML pipeline-6 ​​may be selected as one or more ML pipelines.

[0050] In another example, the selected ML pipeline may include a first group of ML pipelines (from the determined second set of ML pipelines) that may have performance scores above a threshold, where the threshold may be a performance score (e.g., F1 score or R2 score) that can be used to select one or more ML pipelines. Consider the above scenario of a determined second set of ML pipelines containing seven ML pipelines, and a threshold of "0.7". Here, ML pipeline-3 (R2 score of "0.75") and ML pipeline-4 (R2 score of "0.8") may be selected as one or more ML pipelines that may have performance scores greater than the threshold (e.g., "0.7").

[0051] In another example, one or more selected ML pipelines may include a second group of ML pipelines (from the determined second set of ML pipelines) that can correspond to a predefined number of top ML pipelines based on their performance scores. The processor 204 may select the top K (e.g., the top 3) ML pipelines based on the performance scores of each in the determined second set of ML pipelines. Consider the above scenario of a determined second set of ML pipelines containing 7 ML pipelines, and the predefined number (i.e., K) as 3. In such a case, the top 3 ML pipelines (based on their respective performance scores) may be selected as one or more ML pipelines. For example, the one or more selected ML pipelines may include ML pipeline-2 with an R2 score of "0.69", ML pipeline-3 with an R2 score of "0.75", and ML pipeline-4 with an R2 score of "0.8".

[0052] In block 308, the ML corpus database (e.g., database 108) may be expanded to include a set of first ML pipelines associated with the received ML project and one or more selected ML pipelines. The processor 204 may be configured to expand the ML corpus database (e.g., database 108) to include a set of first ML pipelines associated with the received ML project and one or more selected ML pipelines. Once one or more ML pipelines are selected, the electronic device 102 may add the selected ML pipelines to database 108 to improve the quality of the ML corpus database stored in database 108. In one embodiment, the electronic device 102 may store the selected ML pipelines in database 108 along with the set of first ML pipelines. In another embodiment, the electronic device 102 may replace the set of first ML pipelines in database 108 with the selected ML pipelines. Control may be passed to termination.

[0053] Although flowchart 300 is represented as separate operations such as 302, 304, 306, and 308, in certain embodiments, such separate operations may be further divided into additional operations, combined into fewer operations, or deleted, depending on the particular implementation, without deviating from the essence of the disclosed embodiments.

[0054] Figure 4 is a flowchart illustrating an example of a method for mutation of a first ML pipeline in a set of ML pipelines associated with an received ML project in order to determine a set of second ML pipelines according to embodiments of the present disclosure. Figure 4 is described in relation to elements from Figures 1, 2, and 3. Referring to Figure 4, flowchart 400 is shown. The method represented in flowchart 400 can begin from block 402 and may be performed by any suitable system, apparatus, or device, for example, by the electronic device 102 in the example of Figure 1 or the processor 204 in Figure 2. Steps and operations related to one or more blocks of flowchart 400, although represented in separate blocks, may be divided into additional blocks, combined into fewer blocks, or deleted, depending on the particular implementation.

[0055] In block 402, a code snippet of the first ML model associated with the first ML pipeline may be identified. Here, the code snippet may be a portion of the code of the first ML pipeline that can implement the first ML model. The processor 204 may be configured to identify the code snippet of the first ML model associated with the first ML pipeline. Since only a portion of the code of the first ML pipeline can implement the first ML model, that particular portion may be identified and mutated to obtain the second ML model. Further details of the identification of the code snippet of the first ML model are given, for example, in Figure 9.

[0056] In block 404, one or more input parameters associated with the identified code snippet may be determined. Processor 204 may be configured to determine one or more input parameters associated with the identified code snippet. In order to maintain the functionality of the second ML model the same as that of the first ML model, one or more input parameters corresponding to the identified code snippet of the first ML model may be supplied to the second ML model. Thus, one or more input parameters of the identified code snippet may be determined and subsequently supplied to the second ML model to obtain an equivalent output from the second ML model, which may be determined based on the mutation of the first ML model. Specific details of the one or more input parameters associated with the identified code snippet are further given, for example, in Figure 6A.

[0057] In an embodiment, one or more determined input parameters associated with an identified code snippet may have at least one of a training dataset, a test dataset, and a set of hyperparameters associated with a first ML model. Here, the training dataset may be used to train the first ML model based on weight updates associated with the first ML model. The test dataset may be used to test the trained first ML model to check whether the accuracy of the trained first ML model is within acceptable limits. The set of hyperparameters may correspond to a set of parameters that can control the learning of the first ML model. For example, the learning rate, the number of neural network layers, the number of neurons in each neural network layer, etc., may correspond to a set of hyperparameters. The processor 204 may collect all read accesses for one or more input parameters of each application programming interface (API) in the identified code snippet.

[0058] In block 406, the second ML model may be selected from a set of predefined models associated with a set of predefined ML pipelines. The processor 204 may be configured to select the second ML model from a set of predefined ML models associated with a set of predefined ML pipelines. The set of predefined models may be ML models that can already be generated. For example, the set of predefined models may correspond to human-written ML models that are generated as template ML models for a particular application scenario and dataset. The set of predefined models may be stored in database 108. The processor 204 may receive the set of predefined models associated with a set of predefined ML pipelines from database 108. Alternatively, the set of predefined models may be pre-stored in memory 206, and the processor 204 may read the set of predefined models from memory 206. Once the set of predefined models is received / read, the processor 204 may select the second ML model from the set of predefined models. Further details of the selection of the second ML model are given, for example, in Figure 5.

[0059] In block 408, the selected second ML model may be instantiated based on the replacement of the first ML model by the selected second ML model in the first ML pipeline. The processor 204 may be configured to instantiate the selected second ML model based on the replacement of the first ML model by the selected second ML model in the first ML pipeline. Here, the identified code snippet may be replaced by the selected second ML model. One or more input parameters associated with the identified code snippet may be supplied to the selected second ML model. Since only the identified code snippet corresponding to the first ML model in the first ML pipeline may be replaced by the selected second ML model, and one or more input parameters may remain the same, the functionality of the instantiated second ML model may remain the same as that of the first ML model. Further details of the instantiation of the second ML model are given, for example, in Figure 7. Control may be passed to termination.

[0060] While flowchart 400 is represented as separate operations such as 402, 404, 406, and 408, in certain embodiments, such separate operations may be further divided into additional operations, combined into fewer operations, or deleted, depending on the particular implementation, without deviating from the essence of the disclosed embodiments.

[0061] Figure 5 is a diagram illustrating an exemplary scenario for extending a machine learning pipeline corpus based on a predefined machine learning pipeline, according to at least one embodiment described herein. Figure 5 is described in relation to elements from Figures 1, 2, 3 and 4. Referring to Figure 5, an exemplary scenario 500 is shown. The exemplary scenario 500 may include a database 108, a first ML pipeline 502, a code snippet identification unit 504, a code snippet 506, an input parameter discovery unit 508, a first ML model 510, a set of predefined ML models 512, an instantiation block 514, a set of second ML pipelines 516, an ML pipeline evaluation unit 518, one or more ML pipelines 520, and an extension block 522. The database 108 may include a plurality of ML projects, each containing 'n' ML projects, such as ML project-1 114, ML project-2 116, ... and ML project-n 118. Each of the plurality of ML projects may include a dataset and a set of ML pipelines applicable to the dataset. ML Project-1 114 may include dataset 114A and ML pipeline set 114B, and ML Project-2 116 may include dataset 116A and ML pipeline set 116B. Similarly, ML Project-n 118 may include dataset 118A and ML pipeline set 118B.

[0062] The number "n" of ML projects shown in Figure 5 is for illustrative purposes only. Multiple ML projects may include as few as two ML projects or more than "n" ML projects without departing from the scope of this disclosure. For simplicity, Figure 5 simply shows "n" ML projects. However, in some embodiments, there may be more than "n" ML projects without limiting the scope of this disclosure.

[0063] Processor 204 may be configured to receive machine learning ML projects from multiple ML projects stored in an ML corpus database (e.g., database 108). Here, each of the multiple ML projects may include a dataset and a set of ML pipelines applicable to the dataset. A machine learning ML project may help an application perform tasks such as prediction, even if it is not programmed to do so. A dataset may include a training dataset, a validation dataset, and a test dataset for the corresponding machine learning (ML) project. The training dataset may be used to train the ML model corresponding to the machine learning (ML) project. The validation dataset may be used to validate the trained ML model corresponding to the machine learning (ML) project. The validation dataset can further improve the accuracy of the training data and prevent overfitting and underfitting of a given ML model. The test dataset may be used to test the trained ML model to check whether the accuracy of the trained ML model is within acceptable limits. An ML pipeline may be a script or program containing a sequence of actions for training an ML model for a particular ML prediction task. It can be understood that a data scientist or developer may generate several ML pipelines and datasets and upload such ML pipelines and datasets to the internet as a knowledge base in various online source code or ML repositories. The knowledge base may be downloaded from the internet and stored in database 108. Electronic device 102 may receive ML projects from multiple ML projects from database 108. For example, electronic device 102 may receive ML project-1 114, where electronic device 102 may receive dataset 114A and a set of ML pipelines 114B corresponding to ML project-1 114. Each ML pipeline in the set of ML pipelines 114B may be mutated one by one.For example, the first ML pipeline 502 may be selected for mutation.

[0064] The processor 204 may be configured to identify a code snippet 506 of a first ML model associated with a first ML pipeline 502, where the code snippet 506 may be a portion of the code of the first ML pipeline 502 that can implement the first ML model. Since only a portion of the code of the first ML pipeline 502 can implement the first ML model, that particular portion may be mutated to obtain a second ML model. To identify the code snippet 506 of the first ML model associated with the first ML pipeline 502, the first ML pipeline 502 may be supplied to the code snippet identification unit 504. Further details of the identification of the code snippet 506 of the first ML model are given, for example, in Figure 6A.

[0065] The processor 204 may be configured to determine one or more input parameters associated with the identified code snippet 506 by using the input parameter discovery unit 508. The identified code snippet 506 may be supplied to the input parameter discovery unit 508 to determine one or more input parameters associated with the identified code snippet 506. The one or more input parameters may include a training dataset, a test dataset, and hyperparameters. In order to maintain the functionality of the second ML model the same as the first ML model, one or more input parameters corresponding to the identified code snippet 506 of the first ML model may also be supplied to the second ML model. Thus, one or more input parameters of the identified code snippet 506 may be determined. The first ML model associated with one or more input parameters (i.e., the ML model with input parameters) is represented as the first ML model 510 in Figure 5. Specific details of the one or more input parameters associated with the identified code snippet 506 are further given, for example, in Figure 6A.

[0066] The processor 204 may be configured to select a second ML model from a set of predefined models 512 associated with a set of predefined ML pipelines. The set of predefined models 512 may be ML models that have already been created. The set of predefined models 512 associated with a set of predefined ML pipelines may be stored in the database 108 or pre-stored in memory 206. Therefore, if the set of predefined models 512 is stored in the database 108, the processor 204 may receive the set of predefined models 512 from the database 108, or alternatively, if the set of predefined models 512 is pre-stored in memory 206, the processor 204 may read the set of predefined models 512 from memory 206.

[0067] The processor 204 may be configured to instantiate a selected second ML model using an instantiation block 514, based on the substitution of the first ML model 510 with the selected second ML model in the first ML pipeline 502. The selected second ML model may be instantiated based on the substitution of the first ML model 510 with the selected second ML model in the first ML pipeline 502. Here, the identified code snippet 506 may be replaced with the code snippet of the selected second ML model. One or more input parameters associated with the identified code snippet 506 may be supplied to the selected second ML model. Since only the identified code snippet 506 corresponding to the first ML model 510 in the first ML pipeline 502 can be replaced with the selected second ML model, the functionality of the first ML pipeline 502 after the mutation can remain the same. After the substitution of the first ML model 510 with the second ML model, the set 516 of the second ML pipeline may be determined. For example, referring to Figure 5, the second ML pipeline set 516 may include three ML pipelines. Further details of the substitution of the second ML model are given, for example, in Figure 9.

[0068] The processor 204 may be configured to select one or more ML pipelines from the determined set of second ML pipelines 516 based on the performance score associated with each of the determined set of second ML pipelines 516. The ML pipeline evaluation unit 518 may determine the performance score associated with each of the determined set of second ML pipelines 516. The performance score associated with each of the determined set of second ML pipelines 516 may correspond to at least one of the F1 score or R2 score associated with the corresponding ML pipeline. In an embodiment, the one or more ML pipelines selected from the determined set of second ML pipelines 516 may include, but are not limited to, an ML pipeline from the determined set of second ML pipelines 516 associated with the maximum performance score, a first group of ML pipelines from the determined set of second ML pipelines 516 corresponding to performance scores above a threshold, or a second group of ML pipelines from the determined set of second ML pipelines 516 that are a predefined number of top ML pipelines based on performance scores. Referring to Figure 5, the determined set of second ML pipelines 516 includes three ML pipelines. The selected one or more ML pipelines 520 may include two ML pipelines (e.g., the top two ML pipelines) selected from the determined set of second ML pipelines 516 based on their respective performance scores. The selection of one or more ML pipelines from the determined set of second ML pipelines 516 is further described, for example, in Figure 3.

[0069] Processor 204 may extend database 108 using extension block 522 to include a set of first ML pipelines 502 associated with the received ML project-1 114 and one or more selected ML pipelines 520. Once one or more ML pipelines 520 are selected, processor 204 may add the selected ML pipelines 520 to database 108 to improve the quality of database 108. In one embodiment, electronic device 102 may add the selected ML pipelines 520 to database 108 along with the set of first ML pipelines 502. In another embodiment, electronic device 102 may replace the first ML pipeline 502 in database 108 with the selected ML pipelines 520. Similarly, each of the first ML pipelines associated with each of several ML projects, such as ML project-1 114, ML project-2 116, and ML project-n 118, may be mutated, and their corresponding selected ML pipelines may be added to extend database 108. Referring to Figure 5, after expansion, the set of ML pipelines 114B corresponding to ML project-1 114 may be changed to set of ML pipelines 524, the set of ML pipelines 116B corresponding to ML project-2 116 may be changed to set of ML pipelines 526, and the set of ML pipelines 118B corresponding to ML project-n 118 may be changed to set of ML pipelines 528. The expansion of database 108 may then be used to feed a higher quality and more consistent learning pipeline / features to the meta-learning model 102A so that it learns from the corpus and then synthesizes new, higher quality ML pipelines for other databases.

[0070] It should be noted that Scenario 500 in Figure 5 is for illustrative purposes only and should not be construed as limiting the scope of this disclosure.

[0071] Figure 6A is a diagram illustrating an exemplary scenario for identifying code snippets of a first ML model associated with an exemplary first ML pipeline, according to embodiments of the present disclosure. Figure 6A is described in relation to elements from Figures 1, 2, 3, 4, and 5. Referring to Figure 6A, exemplary scenario 600A is shown. Exemplary scenario 600A may include a first ML pipeline 602. The first ML pipeline 602 may include statements-1 604, 2 606, 3 608, 4 610, 5 612, and 6 614. An electronic device 102 or processor 204 may identify code snippets of a first ML model associated with the first ML pipeline 602, as described herein.

[0072] The processor 204 may determine one or more input parameters corresponding to the first ML pipeline 602. For example, referring to Figure 6A, the determined one or more input parameters may include “X_train”, “X_test”, “y_train”, and “y-test”. All assignment representations may correspond to key-value pairs associated with sets of hyperparameters. For example, “family=sm.families.Binomial()” may correspond to a hyperparameter key-value pair that can define the first ML model based on a binomial distribution. API-level information may be used to map existing variables to “X_train”, “X_test”, “y_train”, and “y-test”. Referring to Figure 6A, the first ML pipeline 602 may be divided into code snippets 616 and 618. Code snippet 616 may correspond to an irrelevant portion of the first ML pipeline 602 related to the first ML model. Code snippet 618 may correspond to a first ML model associated with the first ML pipeline 602. Processor 204 may identify code snippet 618 as corresponding to the first ML model associated with the first ML pipeline 602, and identify code snippet 616 as an unrelated part associated with the first ML pipeline 602. Thus, processor 204 may extract only code snippet 618 corresponding to the first ML model associated with the first ML pipeline 602 for instantiation. For example, referring to Figure 6A, the first ML model may be identified based on statements 3 608 and 4 610 in code snippet 618. Further details on identifying the code snippet of the first ML model are given, for example, in Figure 8.

[0073] Figure 6B is a diagram illustrating an exemplary scenario for instantiation of a selected second ML model from a predefined set of models associated with a predefined set of ML pipelines, according to embodiments of the present disclosure. Figure 6B is described in relation to elements from Figures 1, 2, 3, 4, 5 and 6A. Referring to Figure 6B, exemplary scenario 600B is shown. Exemplary scenario 600B may include code snippets 620 and 622. Code snippet 620 may include statements 7, 624 and 8, 616. Code snippet 620 may include statements 9, 628, 10, 630 and 11, 632. An electronic device 102 or processor 204 may instantiate a selected second ML model from a predefined set of models associated with a predefined set of ML pipelines, as described herein.

[0074] Referring to Figure 6B, the selected second ML model may be instantiated based on the replacement of the first ML model by the selected second ML model. Code snippet 620 may correspond to the first ML model associated with the first ML pipeline. Code snippet 622 may correspond to the selected second ML model. Processor 204 may instantiate the selected second ML model based on the replacement of code snippet 620 (including statements 7, 624 and 8, 626) by code snippet 622 (including statements 9, 628, 10, 630 and 11, 632) in the first ML pipeline. Thus, the first ML model may be replaced by the second ML model in the first ML pipeline for instantiation. It should be noted that one or more input parameters of the first ML model that can be determined for code snippet 620 may be supplied as input parameters to the second ML model, as defined by code snippet 622. For example, referring to Figure 6B, one or more input parameters may be “y_train” and “X_train_sm”. The instantiation of the selected second ML model may result in the determination of the second ML pipeline set. Further details of the instantiation of the selected second ML model are given, for example, in Figure 7.

[0075] Referring to Figures 6A and 6B, while exemplary scenarios 600A and 600B are represented in a high-level programming language (e.g., the "Python" programming language) using a web-based compute notebook, it should be noted that the teachings of this disclosure may be valid for other ML pipelines written in different languages ​​and development platforms. Furthermore, it should be noted that web-based compute notebooks may correspond to compute structures that can be used to develop / represent ML pipelines, particularly during the development phase. For example, a web-based compute notebook may be used to develop the script portion of an ML project (e.g., an ML pipeline).

[0076] It should be noted that Scenario 600A in Figure 6A and Scenario 600B in Figure 6B are for illustrative purposes only and should not be interpreted as limiting the scope of this disclosure.

[0077] Figure 7 is a flowchart illustrating an example of a method for instantiating a selected second ML model from a predefined set of models associated with a predefined set of ML pipelines, according to at least one embodiment described herein. Figure 7 is described in relation to elements from Figures 1, 2, 3, 4, 5, 6A, and 6B. Referring to Figure 7, flowchart 700 is shown. The method represented in flowchart 700 can begin from block 702 and may be performed by any suitable system, apparatus, or device, for example, by the electronic device 102 in the example of Figure 1 or the processor 204 in Figure 2. Steps and operations related to one or more blocks of flowchart 700, although represented in separate blocks, may be divided into additional blocks, combined into fewer blocks, or deleted, depending on the particular implementation.

[0078] In block 702, a predefined template associated with the second ML model may be selected. The selected predefined template may be annotated with one or more input parameters of an identified code snippet of the first ML model. The processor 204 may be configured to select a predefined template associated with the second ML model, where the predefined template associated with the second ML model may be different from the template associated with the first ML model. In order to maintain the functionality of the second ML model as that of the first ML model, one or more input parameters of an identified code snippet of the first ML model may be supplied to the predefined template associated with the second ML model. For example, referring to Figure 6B, code snippet 620 may be an identified code snippet of the first ML model, where one or more input parameters may include "y_train" and "X_train_sm". Code snippet 620 may further include hyperparameters that can define a family of distributions (e.g., a binomial family) associated with the first ML model, as given in statement 7 624. The template associated with the first ML model may perform regression based on a binomial distribution. The predefined template associated with the second ML model may include a "RandomFrestRegressor" function call (as shown in statements 9-628 and 10-630 of code snippet 622) instead of a function call associated with the "binomial" family of distributions to perform regression. Furthermore, the input parameters "y_train" and "X_train_sm" may be passed as parameters to the predefined template associated with the second ML model in statement 632 in order to maintain the functionality of the second ML model as it is that of the first ML model.

[0079] In block 704, a code snippet of the second ML model may be constructed based on the parameterization of one or more function calls in a selected predefined template using one or more annotated input parameters. Processor 204 may be configured to construct a code snippet of the second ML model constructed based on the parameterization of one or more function calls in a selected predefined template using one or more annotated input parameters. Here, parameterization may be used to pass the values ​​of one or more annotated input parameters to one or more function calls in a selected predefined template. In other words, previously collected variable names pointing to the appropriate holes in a selected predefined template may be inserted to construct a new model snippet, such as code snippet 622 of the second ML model. For example, referring to Figure 6B, code snippet 622 of the second ML model may include “RandomFrestRegressor” as a function call, where statement-9 628 may call the “RandomFrestRegressor” function which may be defined in the library “sklearn.ensemble”. Statement-10 630 in code snippet 622 allows the “RandomFrestRegressor” function to be assigned to the variable “logm3”. As shown in statement-11 632, one or more annotated input parameters, such as “y_train” and “X_train_sm”, can be passed to the “fit” function call associated with the variable “logm3” to constitute code snippet 622 of the second ML model.

[0080] In block 706, the identified code snippet 620 of the first ML model may be replaced with the configured code snippet 622 of the second ML model to instantiate the second ML model. The processor 204 may be configured to replace the identified code snippet 620 of the first ML model with the configured code snippet 622 of the second ML model to instantiate the second ML model. The functionality of the second ML model can be the same as that of the first ML model, since the input parameters may be the same for both the first and second ML models. However, one or more functions called in the second ML model may differ from the corresponding functions called in the first ML model. In this way, the code snippet 620 of the first ML model may be replaced with the code snippet 622 associated with the second ML model to instantiate the second ML model. Control may be passed to termination.

[0081] Although flowchart 700 is represented as separate operations such as 702, 704, and 706, in certain embodiments, such separate operations may be further divided into additional operations, combined into fewer operations, or deleted, depending on the particular implementation, without deviating from the essence of the disclosed embodiments.

[0082] Figure 8 is a flowchart illustrating an example of a method for identifying one or more statements related to a first ML model, according to at least one embodiment described herein. Figure 8 is described in relation to elements from Figures 1, 2, 3, 4, 5, 6A, 6B and 7. Referring to Figure 8, flowchart 800 is shown. The method represented in flowchart 800 can begin in block 802 and may be performed by any suitable system, apparatus, or device, for example, by the electronic device 102 in the example of Figure 1 or the processor 204 in Figure 2. Steps and operations related to one or more blocks of flowchart 800, although represented in separate blocks, may be divided into additional blocks, combined into fewer blocks, or deleted, depending on the particular implementation.

[0083] In block 802, an abstract syntax tree (AST) related to the first ML pipeline may be constructed. Processor 204 may be configured to construct an abstract syntax tree (AST) related to the first ML pipeline. It can be understood that the AST of code may be a tree representation of the abstract syntax framework of programming language code in a formal language. The AST may not contain any details of the code or the syntax of the code; rather, the AST may only contain abstract syntax in a formal language, such as “while” blocks, “if” statements, “conditional branches”, “compare” statements, “assignment” statements, “variable names”, etc. Each node of the tree may represent the syntactic structure of the code. The AST related to the first ML pipeline may thus be constructed based on a tree representation of the abstract syntax of statements in the first ML pipeline in formal language form. The AST may be useful for the easy manipulation and representation of statements related to the first ML model in the first ML pipeline. For example, referring to Figure 6A, the AST of the first ML pipeline 602 may be constructed.

[0084] In block 804, the last Application Programming Interface (API) call related to the prediction function in the first ML pipeline may be determined based on the configured AST. The processor 204 may be configured to determine the last Application Programming Interface (API) call related to the prediction function in the first ML pipeline based on the configured AST. It can be understood that the API may be used to read data based on API endpoints that may be exposed by the API function. To receive data, a request (also called an API call) may be sent to an address associated with an API endpoint that may be exposed by the API function. The prediction function may predict values ​​based on the training of a given ML model. The prediction function in the first ML pipeline may make predictions based on the training of the first ML model. Referring to Figure 6A, the prediction function in the first ML pipeline 602 may make predictions using the input parameter "X_test" in statement 614. The processor 204 may determine the last API call related to the prediction function in the first ML pipeline 602 based on the configured AST. For example, referring to Figure 6A, statement-6 614 may correspond to the last API call related to the prediction function (i.e., “predict(X_test)”) in the first ML pipeline 602.

[0085] In block 806, the last determined API call may be assigned as the target line. The processor 204 may be configured to assign the last determined API call as the target line. The target line may be a line in the first ML pipeline that contains a prediction function. The target line may be a line from which the prediction output of the first ML model can be extracted. For example, referring to Figure 6A, statement-6 614 of the first ML pipeline 602 may contain a prediction function and therefore may be set as the target line.

[0086] In block 808, one or more statements related to the first ML model may be identified based on the assigned target line. The processor 204 may be configured to identify one or more statements related to the first ML model based on the assigned target line. As described, only code snippets containing statements related to the first ML model may be replaced in order to instantiate the second ML model. Thus, statements related to the first ML model may need to be identified in the first ML pipeline. For example, referring to Figure 6A, statements 3 608, 4 610, and 5 612 may be identified together with statement 6 614 in the first ML pipeline 602 as one or more statements related to the first ML model based on the assigned target line which may be statement 6 614. Thus, code snippet 618 may contain one or more statements related to the first ML model. The remaining statements, such as statement 1 604 and statement 2 606, may not be related to the first ML model and may be grouped together as code snippet 616. When code snippet 618 containing one or more statements related to the first ML model is identified, the second ML model may be instantiated by replacing one or more statements of the first ML model with equivalent statements of the second ML model using the same input parameters, where the statements in code snippet 616 may remain unchanged.

[0087] In an embodiment, one or more statements associated with a first ML model may be identified based on the application of backward program slicing from an assigned target line up to a model declaration associated with the first ML model. Here, backward program slicing may be used to obtain a slice of the program based on the addition of one associated statement at a time from the last statement. In other words, backward program slicing may be used to obtain a portion of the program based on backward traversal from the last statement. The processor 204 may identify one or more statements associated with a first ML model by using backward program slicing, which may involve backward traversal of code statements of the first ML model from a target line up to a model declaration of the first ML model. For example, referring to Figure 6A, statement-6 614 may be the target line. Backward program slicing may traverse one or more statements of the first ML model from statement-6 614 until it reaches the model declaration of the first ML model in the first ML pipeline 602, so as to obtain code snippet 618. Here, the model declaration of the first ML model may correspond to statement-3 608. For example, the function call "sm.GLM" present in statement-3 608 may declare the first ML model.

[0088] In an embodiment, the processor 204 may further be configured to store the line count of each of one or more statements associated with the first ML model. Here, one or more statements may, but are not limited to, a model definition, a fit function call, or a predictive function call. A model definition statement may define the first ML model associated with the first ML pipeline. It may be understood that an ML model may need to be defined first in order to generate an ML model. For example, referring to Figure 6A, the first ML model associated with the first ML pipeline 602 may be defined in statement-3 608. The function “sm.GLM” present in statement-3 608 may define the first ML model. A fit function call may accept input parameters (e.g., a training dataset containing examples of features and outputs, or labels / scores associated with the training dataset) and train the first ML model based on the supplied input parameters. Referring to Figure 6A, the fit function call associated with the first ML pipeline 602 is present in statement-4 610. Here, one or more input parameters corresponding to the training dataset, such as “y_train” and “X_train”, may be supplied. The prediction function call may make predictions based on the training of a given ML model. Here, the prediction function call may supply predicted values ​​using the trained first ML model. Referring to Figure 6A, the prediction function call associated with the first ML pipeline 602 is located in statement-6 614. Here, the first ML model associated with the first ML pipeline 602 may make predictions with respect to input parameters associated with the test dataset, such as “X_test”. The processor 204 may store the line counts of each of the one or more statements associated with the first ML model. For example, the processor 204 may store the line counts of statement-3 608, which contains the model definition call, statement-4 610, which contains the fit function call, and statements-5 612 and 6 614, which contain the prediction function for the first ML pipeline 602.Remembering the line counts of one or more statements associated with the first ML model can be useful for identifying code snippet 618 and replacing code snippet 618 with an equivalent code snippet of the second ML model to instantiate the second ML model. Control can then be passed to termination.

[0089] While flowchart 800 is represented as separate operations such as 802, 804, 806, and 808, in certain embodiments, such separate operations may be further divided into additional operations, combined into fewer operations, or deleted, depending on the particular implementation, without deviating from the essence of the disclosed embodiments.

[0090] Figure 9 is a flowchart illustrating an example of a method for obtaining a model slice to identify a code snippet of a first ML model, according to at least one embodiment described herein. Figure 9 is described in relation to elements from Figures 1, 2, 3, 4, 5, 6A, 6B, 7, and 8. Referring to Figure 9, flowchart 900 is shown. The method represented in flowchart 900 can begin from block 902 and may be performed by any suitable system, apparatus, or device, for example, by the electronic device 102 in example Figure 1 or the processor 204 in Figure 2. Steps and operations related to one or more blocks of flowchart 900, although represented in separate blocks, may be divided into additional blocks, combined into fewer blocks, or deleted, depending on the particular implementation.

[0091] In block 902, the assigned target line may be taken out of the first ML pipeline. The processor 204 may be configured to take the assigned target line out of the first ML pipeline. As described, the assigned target line may be a line in the first ML pipeline that may contain a prediction function. Referring to Figure 6A, statement-6 614 of the first ML pipeline 602 may correspond to the assigned target line, in that statement-6 614 may contain a prediction function. The assigned target line (i.e., statement-6 614) may be taken out of the first ML pipeline 602.

[0092] In block 904, the retrieved target line may be added to a queue containing a set of statements associated with the first ML pipeline. Here, the set of statements associated with the first ML pipeline may be one or more statements associated with the first ML model, which can be identified based on the assigned target line. The processor 204 may be configured to add the retrieved target line to a queue that may contain a set of statements associated with the first ML pipeline. Referring to Figure 6A, the set of statements associated with the first ML pipeline 602 in code snippet 618 may include statement-3 608, statement-4 610, and statement-5 612. The queue may contain a set of statements. The retrieved target line, such as statement-6 614, may be added to the queue.

[0093] In block 906, the first statement may be popped from the queue, where the first statement may be an allocated target line. The processor 204 may be configured to pop the first statement from the queue. For example, referring to Figure 6A, statement-6 614 may be the first statement that can be popped from the queue.

[0094] In block 908, the execution of the first set of operations 910 may be controlled to retrieve a model slice associated with the first ML model from the first ML pipeline. The processor 204 may be configured to control the execution of the first set of operations 910 to retrieve a model slice associated with the first ML model from the first ML pipeline. The model slice may be retrieved based on the extraction of one or more statements associated with the first ML model from the first ML pipeline. For example, referring to Figure 6A, the model slice may include statements 3 608, 4 610, 5 612, and 6 614.

[0095] The first set of operations 910 may include a first operation 910A, a second operation 910B, a third operation 910C, a fourth operation 910D, a fifth operation 910E, and a sixth operation 910F. The first set of operations 910 may be repeatedly executed by the processor 204 based on a check to see if the queue is empty. If it is determined that the queue is empty, the execution of the first set of operations 910 may be stopped, and the model slice may be acquired in operation 912. The first set of operations 910 acquires the model slice as described herein.

[0096] In block 910A (i.e., the first operation), one or more variables and objects may be extracted from the first statement. Processor 204 may be configured to extract one or more variables and objects from the first statement, where one or more variables and objects may be extracted from the extracted target line. For example, referring to Figure 6A, statement-6 614 may be the first statement that can be popped from the queue. One or more variables and objects extracted from statement-6 614 may include "X_test".

[0097] In block 910B (i.e., second operation), a set of second statements that occurred before the first statement in the first ML pipeline and contain at least one of the extracted variables and objects may be identified. The processor 204 may be configured to identify a set of second statements that occurred before the first statement in the first ML pipeline and contain at least one of the extracted variables and objects. Here, all statements in the first ML pipeline that occur before the first statement and contain at least one of the extracted variables and objects may be identified as a set of second statements. For example, referring to Figure 6A, statement-5 612 may contain one or more extracted variables and objects such as “X_test” in the first ML pipeline 602. Thus, statement-5 612 may be identified as a set of second statements.

[0098] In block 910C (i.e., the third operation), a check may be performed to determine whether the third statement in the identified set of second statements occurs before the model definition associated with the first ML model. The processor 204 may be configured to determine whether the third statement in the identified set of second statements occurs before the model definition associated with the first ML model. Here, one of the statements in the identified set of second statements may be assigned as the third statement, and the processor 204 may determine whether the third statement occurs before the model definition associated with the first ML model. If it is determined that the third statement in the identified set of second statements occurs before the model definition associated with the first ML model, then control may then be transferred to operation 910, otherwise the processor 204 may select another statement as the third statement and repeat operation 910C. For example, referring to Figure 6A, statement-5 612 may be identified as the set of second statements. Since the second set of statements contains only statement-5 612, statement-5 612 can be assigned as the third statement. Also, in the present case, processor 204 may determine that the first ML model can be defined in statement-3 608 in the first ML pipeline 602.

[0099] In block 910D (i.e., fourth operation), the third statement may be added to the queue based on the determination that the third statement occurs before the model definition. The processor 204 may be configured to add the third statement to the queue based on the determination that the third statement occurs before the model definition. Since only statements in the first ML pipeline corresponding to the first ML model can be identified as one or more statements, the third statement may be related to the first ML model if it appears before the model definition, otherwise the third statement cannot be related to the first ML model. Thus, the third statement may be added to the queue if it appears before the model definition. For example, referring to Figure 6A, statement-5 612, which is assigned as the third statement, occurs before the model definition. Since the processor 204 performs a backward trace of each statement in code snippet 618 from the target line, i.e., statement-6 614, the processor 204 can reach statement-5 612. Thus, statement-5 612 may be added to the queue. The third action 910C and the fourth action 910D may be repeated for each of the pairs of second statements.

[0100] In block 910E (i.e., operation 5), the first statement may be added to the model slice. Processor 204 may be configured to add the first statement to the model slice. Here, the identified target statement may be added to the model slice. For example, referring to Figure 6A, statement-6 614 may be added to the model slice.

[0101] In block 910F (i.e., operation 6), the fourth statement may be popped from the queue as the first statement, based on the determination that the queue is not empty. The processor 204 may be configured to pop the fourth statement from the queue as the first statement, based on the determination that the queue is not empty. Once the first statement may be added to the model slice, the queue may be checked to determine whether the queue contains further statements and is not empty. If the queue is determined to be empty, the model slice may be retrieved to identify a code snippet of the first model. The model slice may then be displayed on the display device 208A. However, if the queue is not empty, the fourth statement may then be popped from the queue as the first statement.

[0102] In block 912, a model slice may be obtained to identify a code snippet of a first ML model based on the iterative execution of a first set of operations 910. The processor 204 may be configured to obtain a model slice to identify a code snippet of a first ML model (e.g., code snippet 618 in Figure 6A) based on the iterative execution of a first set of operations 910. The iterative execution may depend on the number of statements in the queue. The first set of operations 910 may relate to operations corresponding to blocks 910A through 910F. In the example, the queue may contain three statements, where the first set of operations may be executed three times to obtain a model slice. The model slice may represent a code snippet of a first ML model. Control may be passed to termination.

[0103] While flowchart 900 is represented as separate operations such as 902, 904, 906, 908, 910A through 910F, and 912, in certain embodiments, such separate operations may be further divided into additional operations, combined into fewer operations, or deleted, depending on the particular implementation, without deviating from the essence of the disclosed embodiments.

[0104] It should be noted that the purpose of automatic generation of ML pipelines may be to learn how to write ML pipelines for a given dataset through the meta-learning model 102A, which may be a completely offline process. In an online configuration, if user 112 can provide the meta-learning model 102A with a new dataset, the meta-learning model 102A may automatically generate a machine learning pipeline.

[0105] It should be noted that the quality of the meta-learning model 102A may depend on the quality of the ML corpus database, more specifically, on the ML models used in individual ML pipelines. However, ML pipelines written by data scientists may not be the best or optimal ML models for a number of reasons, including the unavailability of suitable ML models and the lack of knowledge about good ML models on the part of some data scientists. Low-quality ML models in the ML corpus database may negatively impact the training of the meta-learning model 102A, which may not be able to identify any learnable patterns regarding which models should be used for which types of datasets. To mitigate the above problems, the meta-learning model 102A (or the disclosed electronic device 102) may be trained on an extended ML corpus database that may contain mutated ML pipelines that may perform better than the original ML pipelines handwritten by data scientists.

[0106] Figure 10 is a diagram showing a flowchart of an example method for training a meta-learning model according to at least one embodiment described herein. Figure 10 is described in relation to elements from Figures 1, 2, 3, 4, 5, 6A, 6B, 7, 8, and 9. Referring to Figure 10, flowchart 1000 is shown. The method represented in flowchart 1000 can begin from block 1002 and may be performed by any suitable system, apparatus, or device, for example, by the electronic device 102 in the example of Figure 1 or the processor 204 in Figure 2. Steps and operations related to one or more blocks of flowchart 1000, although represented in separate blocks, may be divided into additional blocks, combined into fewer blocks, or deleted, depending on the particular implementation.

[0107] In block 1002, sets of metafeatures may be extracted from datasets associated with each of several ML projects stored in an augmented ML corpus database. Processor 204 may be configured to extract sets of metafeatures extracted from datasets associated with each of several ML projects stored in an augmented ML corpus database. It can be understood that features may be independent variables supplied to a given ML model that may need to be learned. Features may include columns of a tabular dataset associated with a given ML model. Metafeatures may be used to estimate the performance of a given ML model. Metafeatures may be predefined metafeatures that can be commonly used to learn relationships between metafeatures and ML components. Sets of metafeatures may be extracted from datasets associated with each of several ML projects stored in an augmented ML corpus database. Sets of metafeatures may be extracted based on the injection of metafeature extractor code in the ML pipeline, such as a metafeature method call. For example, a dataset may be passed to a metafeature method to extract sets of metafeatures. Details of the meta-feature set are given, for example, in Figure 11.

[0108] In block 1004, sets of ML pipeline components may be extracted from sets of ML pipelines associated with each of several ML projects stored in an extended ML corpus database. Processor 204 may be configured to extract sets of ML pipeline components from sets of ML pipelines associated with each of several ML projects stored in an extended ML corpus database. It can be understood that ML components may include functions used in the ML pipeline of a given ML model. ML components from sets of ML pipelines associated with each of several ML projects that can be stored in an extended ML corpus database may be extracted using an ML component extractor. Details of the extraction of sets of ML pipeline components are given, for example, in Figure 11.

[0109] In block 1006, the meta-learning model 102A may be trained based on the extracted set of meta-features and the extracted set of ML pipeline components. The processor 204 may be configured to train the meta-learning model 102A based on the extracted set of meta-features and the extracted set of ML pipeline components. The meta-learning model 102A may use a meta-learning algorithm that can be trained on a learning algorithm that has already been trained. Here, the output of another learning algorithm for a given dataset may be supplied to the meta-learning model 102A along with a learning algorithm applicable to that dataset. The meta-learning model 102A may learn from the output of other learning algorithms. For example, the meta-learning model 102A may learn and predict based on the output of another learning algorithm as input. Thus, the meta-learning model 102A may learn to make predictions from predictions already made by other learning algorithms.

[0110] It should be noted that the meta-learning model 102A does not have to be a single ML model internally, but may contain multiple ML models. For simplicity, the meta-learning model 102A may be considered a black box that can take data as input and generate an abstract pipeline as output. Here, the abstract pipeline may be a sequence of labels that can be translated into code. It should be understood that the labels may be names given to a sequence of functions, modules, or statements to perform a particular task. It should be noted that each of the sets of ML pipelines associated with each of the multiple ML projects stored in the extended ML corpus database may contain several components that can take the form of code within the corresponding ML pipeline, so that developers do not have to write the functionality of each component. Users may not be able to infer anything from the corresponding ML pipeline unless the ML pipeline is broken down into components that can be associated with unique labels. Expansion of explanations may be used as a technique to provide natural language explanations of the components used in the corresponding ML pipeline.

[0111] The meta-learning model 102A may be trained on a set of extracted meta-features and a set of extracted ML pipeline components. Since the meta-learning model 102A can be trained on a set of extracted meta-features and a set of extracted ML pipeline components from an augmented ML corpus database, the quality of the meta-learning model 102A may directly depend on the quality and robustness of the augmented ML corpus database. Details of the meta-learning model 102A are given, for example, in Figure 11. Control may be passed to termination.

[0112] While flowchart 1000 is represented as separate operations such as 1002, 1004, and 1006, in certain embodiments, such separate operations may be further divided into additional operations, combined into fewer operations, or deleted, depending on the particular implementation, without deviating from the essence of the disclosed embodiments.

[0113] Figure 11 illustrates an exemplary scenario for training a digit learning model according to at least one embodiment described herein. Figure 11 is described in relation to elements from Figures 1, 2, 3, 4, 5, 6A, 6B, 7, 8, 9, and 10. Referring to Figure 11, an exemplary scenario 1100 is shown. The exemplary scenario 1100 may include a database 108, a set of meta-features 1102, a set of ML pipeline components 1104, a meta-learning block 1106, a set of meta-features for a subject dataset (represented by 1108), a meta-learning model 102A, and a set of components for a subject ML pipeline (represented by 1110). The database 108 may include “n” ML projects, such as ML project-1 114, ML project-2 116, ..., and ML project-n 118. Each of the multiple ML projects may include a dataset and a set of ML pipelines applicable to the dataset. For example, ML project-1 114 may include dataset 114A and ML pipeline set 114B, while ML project-2 116 may include dataset 116A and ML pipeline set 116B. Similarly, ML project-n 118 may include dataset 118A and ML pipeline set 118B.

[0114] The number "n" of ML projects shown in Figure 11 is for illustrative purposes only. Multiple ML projects may include as few as two ML projects or more than "n" ML projects without departing from the scope of this disclosure. For simplicity, Figure 11 simply shows "n" ML projects. However, in some embodiments, there may be more than "n" ML projects without limiting the scope of this disclosure.

[0115] For example, referring to Figure 11, the processor 204 may extract a set of metafeature sets 1102 from datasets associated with each of several ML projects stored in an extended ML corpus database (which may be stored in database 108). The set of metafeature sets 1102 may be extracted based on the injection of a metafeature extractor, such as a metafeature method call, in the corresponding ML pipeline code. The dataset may be passed to the metafeature method for extracting the set of metafeature sets 1102. In the example, the set of metafeature sets 1102 may include rows, columns, missing values, and a flag indicating the presence of text.

[0116] Processor 204 may extract a set of ML pipeline components 1104 from a set of ML pipelines associated with each of several ML projects stored in an extended ML corpus database (e.g., database 108). For example, processor 204 may extract a set of ML pipeline components 1104 from a set of ML pipelines 524 associated with the first ML project-1 114, a set of ML pipelines 526 associated with the second ML project-2 116, and a set of ML pipelines 528 corresponding to the nth ML project-n 118. In this example, the set of ML pipeline components 1104 may include “fillna”, “TfidfVectorizer”, and “logsiticregression”, where “fillna” may be a function used to fill in missing values ​​in rows of the dataset. "TfdfVectorizer" may be a term-frequency inverse-document frequency function that can convert text into meaningful numerical values ​​based on the number of occurrences of a term in a given document compared to a large number of documents containing the term. "logsiticregression" may be a function that predicts values ​​based on logistic regression techniques.

[0117] The meta-learning block 1106 may supply a set of extracted meta-features 1102 and a set of extracted ML pipeline components 1104 for training the meta-learning model 102A. The set of ML pipeline components 1104 can be extracted from an extended ML corpus database (e.g., database 108) which may contain high-quality mutated ML pipelines, so that the meta-learning model 102A can be adequately trained.

[0118] Once the meta-learning model 102A can be trained, the processor 204 may supply the meta-learning model 102A with a set of meta-features (represented by 1108) of the subject dataset, such as rows, columns, missing values, and flags indicating the presence of text. Based on the set of meta-features (represented by 1108) of the subject dataset, the meta-learning model 102A may generate a set of components (represented by 1110) of the subject ML pipeline, such as “fillna”, “TfdfVectorizer”, and “logsiticregression”. Since the meta-learning model 102A can be trained on high-quality mutated ML pipelines from an extended ML corpus database (e.g., database 108), the generated set of components (represented by 1110) of the subject ML pipeline can also be of high quality. Thus, the generated set of components (represented by 1110) of the subject ML pipeline can function well for the subject dataset associated with the subject ML pipeline.

[0119] An exemplary experimental setup for this disclosure is shown in Table 1 below: [Table 1]

[0120] It should be noted that the data given in Table 1 can be interpreted simply as experimental data and should not be interpreted as limiting this disclosure.

[0121] Exemplary experimental data for validating the improvement in training data performance is presented in Table 2 below: [Table 2]

[0122] Table 2 shows that, out of a total of 170 ML pipelines, 62 percent showed improved accuracy based on the proposed mutation framework. For 17 percent of the ML pipelines, performance improvements of over 5 percent are possible, which is significant. Furthermore, for 13 percent of the ML pipelines, performance improvements of over 1 percent are possible.

[0123] It should be noted that the data given in Table 2 can be interpreted simply as experimental data and should not be interpreted as limiting this disclosure.

[0124] Exemplary experimental data regarding the impact on test data are shown in Table 3 below: [Table 3]

[0125] It should be noted that the data given in Table 3 can be interpreted simply as experimental data and should not be interpreted as limiting this disclosure.

[0126] Various embodiments of this disclosure may provide one or more non-temporary computer-readable storage media configured to store instructions causing a system (e.g., an example electronic device 102) to perform an action in response to being executed. The action may include receiving machine learning (ML) projects from a plurality of ML projects stored in an ML corpus database, where each of the plurality of ML projects may include a dataset and a set of ML pipelines applicable to the dataset. The action may further include mutating a first ML pipeline in the first set of ML pipelines associated with the received ML project to determine a second set of ML pipelines based on a predefined set of ML pipelines, where the mutation of the first ML pipeline may correspond to replacing a first ML model associated with the first ML pipeline with a second ML model associated with one of the predefined sets of ML pipelines. The action may further include selecting one or more ML pipelines from the determined second set of ML pipelines based on the performance scores associated with each of the determined second set of ML pipelines. The operation may further include extending the ML corpus database to include a set of first ML pipelines associated with the received ML project and one or more selected ML pipelines.

[0127] As used in this disclosure, the terms “module” or “component” may refer to a specific hardware implementation configured to perform the operation of a module or component, and / or a software object or software routine stored in and / or executed by general-purpose hardware of a computing system (e.g., computer-readable media, processing devices, etc.). In some embodiments, different components, modules, engines, and services described in this disclosure may be implemented as objects or processes that run on a computing system (e.g., as separate threads). While some of the systems and methods described in this disclosure are generally described as being implemented in software (stored in and / or executed by general-purpose hardware), specific hardware implementations, or combinations of software and specific hardware implementations, are also possible and conceivable. **In this specification, a “computation entity” may be any computing system as previously defined in this disclosure, or any module or combination of modules that runs on a computing system.**

[0128] In this disclosure, the terms used in particular in the attached claims (e.g., the text of the attached claims) are generally intended to be “open” terms (for example, the word “including” should be interpreted as “including but not limited to,” the word “having” should be interpreted as “having at least,” and the word “includes” should be interpreted as “including but not limited to,” etc.).

[0129] Furthermore, if a specific number is intended in an introduced claim recitation, that intention must be clearly stated in the claim; if there is no such statement, then no such intention exists. For example, to facilitate understanding, subsequent appended claims may use introductory phrases such as "at least one" and "one or more" to introduce a claim. However, the use of such phrases should not be interpreted as suggesting that a particular claim containing the introduced claim recitation is limited to cases that include only one instance of that item, even if the claim includes both an introductory phrase such as "one or more" or "at least one" and an indefinite article such as "a" or "an" (for example, "a" and / or "an" should be interpreted as meaning "at least one" or "one or more"). The same applies when introducing a claim recitation using a definite article.

[0130] Furthermore, even if a specific number is explicitly stated in the introduced claim description, it will be understood by those skilled in the art that such description should generally be interpreted to mean at least the number stated (for example, if there is a description of only "two descriptions" without any other modifiers, this description means at least two descriptions, or two or more descriptions). Furthermore, when a notation similar to "at least one of A, B, and C, etc." or "one or more of A, B, and C, etc." is used, such a structure is generally intended to include A only, B only, C only, both A and B, both A and C, both B and C, and / or all of A, B, and C, etc.

[0131] Furthermore, any disjunctions and / or disjunctions representing two or more selectable terms, whether in the specification, claims, or drawings, should be understood as intended to include the possibility of including one of those terms, either of those terms, or both of those terms. For example, the phrase "A or B" should be understood to include the possibility of "A or B" or "A and B".

[0132] All examples and conditional language cited herein are intended for educational purposes to help readers understand the concepts and inventions to which the inventors have contributed to the advancement of the art, and should be construed as not being limited to such specifically cited examples and conditions. While embodiments of this disclosure have been described in detail, various modifications, substitutions, and alternatives may be made without departing from the spirit and scope of this disclosure.

[0133] In addition to the embodiments described above, the following additional information is disclosed. (Note 1) A method that is executed by a processor, The process involves receiving ML projects from multiple ML projects stored in a machine learning (ML) corpus database, where each of the multiple ML projects includes a dataset and a set of ML pipelines applicable to that dataset. The process involves mutating the first ML pipeline in the set of first ML pipelines associated with the received ML project so as to determine a set of second ML pipelines based on a set of predefined ML pipelines, wherein the mutation of the first ML pipeline corresponds to replacing the first ML model associated with the first ML pipeline with a second ML model associated with one of the predefined sets of ML pipelines. Selecting one or more ML pipelines from the determined set of second ML pipelines based on the performance scores associated with each of the determined sets of second ML pipelines, To extend the ML corpus database to include the set of first ML pipelines associated with the received ML project and one or more selected ML pipelines. A method of having. (Note 2) Extracting sets of metafeatures from the database related to each of the multiple ML projects stored in the expanded ML corpus database, Extracting sets of ML pipeline components from sets of ML pipelines associated with each of the multiple ML projects stored in the expanded ML corpus database, The meta-learning model is trained based on the extracted set of meta-features and the extracted set of ML pipeline components. The method described in Appendix 1, further comprising the above. (Note 3) The performance score associated with each of the determined sets of second ML pipelines corresponds to at least one of the F1 score or R2 score associated with the corresponding ML pipeline. The method described in Appendix 1. (Note 4) The selected one or more ML pipelines from the set of second ML pipelines determined above are: ML pipelines from the determined set of second ML pipelines, related to the maximum performance score, The first group of ML pipelines from the determined set of second ML pipelines corresponding to performance scores exceeding the threshold, or A second group of ML pipelines from the determined set of second ML pipelines, corresponding to a predefined number of top ML pipelines based on the performance score. Includes one of the following: The method described in Appendix 1. (Note 5) Identifying the code snippet of the first ML model related to the first ML pipeline, Determining one or more input parameters associated with the identified code snippet, Selecting the second ML model from a set of predefined models associated with the aforementioned set of predefined ML pipelines, Based on replacing the first ML model with the selected second ML model in the first ML pipeline, the selected second ML model is instantiated. The method described in Appendix 1, further comprising the above. (Note 6) The one or more input parameters determined in relation to the identified code snippet include at least one of the training dataset, test dataset, and set of hyperparameters related to the first ML model. The method described in Appendix 5. (Note 7) This involves selecting a predefined template associated with the second ML model, wherein the selected predefined template is annotated with one or more input parameters of the identified code snippet of the first ML model. The code snippet of the second ML model is constructed based on the parameterization of one or more function calls in the selected predefined template that use one or more input parameters annotated as described above, To instantiate the second ML model, replace the identified code snippet of the first ML model with the configured code snippet of the second ML model. The method described in Appendix 5, further comprising the above. (Note 8) Constructing an Abstract Syntax Tree (AST) related to the aforementioned 1 ML pipeline, Based on the configured AST, the last application programming interface (API) call related to the prediction function in the first ML pipeline is determined, Assign the last API call determined above as the target line, To identify one or more statements related to the first ML model based on the assigned target line. The method described in Appendix 5, further comprising the above. (Note 9) The one or more statements associated with the first ML model are identified based on the application of backward program slicing from the assigned target line up to the model declaration associated with the first ML model. The method described in Appendix 8. (Note 10) The system further includes storing the line numbers of each of the one or more statements related to the first ML model, The one or more statements mentioned above correspond to at least one of the following: a model definition, a fit function call, or a predictive function call. The method described in Appendix 8. (Note 11) Extracting the assigned target line from the first ML pipeline, Adding the retrieved target line to a queue containing a set of statements related to the first ML pipeline, Popping the first statement from the aforementioned queue, This involves controlling the execution of a first set of operations to obtain model slices related to the first ML model from the first ML pipeline, wherein the first set of operations is: Extracting one or more variables and objects from the first statement, Identifying a set of second statements that occur in the first ML pipeline prior to the first statement and include at least one of the extracted variables and objects, To determine whether the third statement in the identified set of second statements occurs before the model definition associated with the first ML model, Based on the determination that the third statement occurs before the model definition, the third statement is added to the queue, Adding the first statement to the aforementioned model slice, Based on the determination that the queue is not empty, the fourth statement is popped from the queue as the first statement. Including the control and Based on the iterative execution of the first set of operations, the model slice is obtained to identify the code snippet of the first ML model. The method described in Appendix 8, further comprising the above. (Note 12) In response to being executed, the electronic device, The process involves receiving ML projects from multiple ML projects stored in a machine learning (ML) corpus database, where each of the multiple ML projects includes a dataset and a set of ML pipelines applicable to that dataset. The process involves mutating the first ML pipeline in the set of first ML pipelines associated with the received ML project so as to determine a set of second ML pipelines based on a set of predefined ML pipelines, wherein the mutation of the first ML pipeline corresponds to replacing the first ML model associated with the first ML pipeline with a second ML model associated with one of the predefined sets of ML pipelines. Selecting one or more ML pipelines from the determined set of second ML pipelines based on the performance scores associated with each of the determined sets of second ML pipelines, To extend the ML corpus database to include the set of first ML pipelines associated with the received ML project and one or more selected ML pipelines. One or more non-temporary computer-readable storage media configured to store instructions for performing an action having [a certain characteristic]. (Note 13) The aforementioned operation is, Extracting sets of metafeatures from the database related to each of the multiple ML projects stored in the expanded ML corpus database, Extracting sets of ML pipeline components from sets of ML pipelines associated with each of the multiple ML projects stored in the expanded ML corpus database, The meta-learning model is trained based on the extracted set of meta-features and the extracted set of ML pipeline components. It further has, One or more non-temporary computer-readable storage media as described in Appendix 12. (Note 14) The performance score associated with each of the determined sets of second ML pipelines corresponds to at least one of the F1 score or R2 score associated with the corresponding ML pipeline. One or more non-temporary computer-readable storage media as described in Appendix 12. (Note 15) The selected one or more ML pipelines from the set of second ML pipelines determined above are: ML pipelines from the determined set of second ML pipelines, related to the maximum performance score, The first group of ML pipelines from the determined set of second ML pipelines corresponding to performance scores exceeding the threshold, or A second group of ML pipelines from the determined set of second ML pipelines, corresponding to a predefined number of top ML pipelines based on the performance score. Includes one of the following: One or more non-temporary computer-readable storage media as described in Appendix 12. (Note 16) The aforementioned operation is, Identifying the code snippet of the first ML model related to the first ML pipeline, Determining one or more input parameters associated with the identified code snippet, Selecting the second ML model from a set of predefined models associated with the aforementioned set of predefined ML pipelines, Based on replacing the first ML model with the selected second ML model in the first ML pipeline, the selected second ML model is instantiated. It further has, One or more non-temporary computer-readable storage media as described in Appendix 12. (Note 17) The one or more input parameters determined in relation to the identified code snippet include at least one of the training dataset, test dataset, and set of hyperparameters related to the first ML model. One or more non-temporary computer-readable storage media as described in Appendix 16. (Note 18) The aforementioned operation is, This involves selecting a predefined template associated with the second ML model, wherein the selected predefined template is annotated with one or more input parameters of the identified code snippet of the first ML model. The code snippet of the second ML model is constructed based on the parameterization of one or more function calls in the selected predefined template that use one or more input parameters annotated as described above, To instantiate the second ML model, replace the identified code snippet of the first ML model with the configured code snippet of the second ML model. It further has, One or more non-temporary computer-readable storage media as described in Appendix 16. (Note 19) The aforementioned operation is, Constructing an Abstract Syntax Tree (AST) related to the aforementioned 1 ML pipeline, Based on the configured AST, the last application programming interface (API) call related to the prediction function in the first ML pipeline is determined, Assign the last API call determined above as the target line, To identify one or more statements related to the first ML model based on the assigned target line. It further has, One or more non-temporary computer-readable storage media as described in Appendix 16. (Note 20) The memory that stores the instructions, It is coupled to the aforementioned memory and executes the aforementioned instruction, The process involves receiving ML projects from multiple ML projects stored in a machine learning (ML) corpus database, where each of the multiple ML projects includes a dataset and a set of ML pipelines applicable to that dataset. The process involves mutating the first ML pipeline in the set of first ML pipelines associated with the received ML project so as to determine a set of second ML pipelines based on a set of predefined ML pipelines, wherein the mutation of the first ML pipeline corresponds to replacing the first ML model associated with the first ML pipeline with a second ML model associated with one of the predefined sets of ML pipelines. Selecting one or more ML pipelines from the determined set of second ML pipelines based on the performance scores associated with each of the determined sets of second ML pipelines, To extend the ML corpus database to include the set of first ML pipelines associated with the received ML project and one or more selected ML pipelines. A processor that executes a process having An electronic device having [Explanation of Symbols]

[0134] 102 Electronic Devices 102A Meta-learning model 104 Servers 106 User End Devices 108 Databases 110 Communication Network 114,116,118 ML Project 204 Processors 206 memory 502,602 1st ML Pipeline 504 Code Snippet Identification Unit 506 Code Snippets 508 Input parameter discovery unit 510 1ML Model 512 A set of predefined ML models 514 instantiation blocks 516 Second ML pipeline set 518 ML Pipeline Evaluation Department 520 One or more ML pipelines 522 Expansion Blocks 1102 Meta-feature set 1104 ML Pipeline Component Set 1106 Meta-learning block

Claims

1. A method that is executed by a processor, The process involves receiving ML projects from multiple ML projects stored in a machine learning (ML) corpus database, where each of the multiple ML projects includes a dataset and a set of ML pipelines applicable to that dataset. The first ML pipeline in the set of first ML pipelines associated with the received ML project is mutated so as to determine a set of second ML pipelines based on a set of predefined ML pipelines, wherein the mutation of the first ML pipeline corresponds to replacing the first ML model associated with the first ML pipeline with a second ML model associated with one of the predefined sets of ML pipelines. Based on the performance scores associated with each of the determined sets of second ML pipelines, one or more ML pipelines are selected from the determined sets of second ML pipelines. To extend the ML corpus database to include the set of first ML pipelines associated with the received ML project and one or more selected ML pipelines. A method of having.

2. Extracting sets of metafeatures from the database related to each of the multiple ML projects stored in the expanded ML corpus database, Extracting sets of ML pipeline components from sets of ML pipelines associated with each of the multiple ML projects stored in the expanded ML corpus database, The meta-learning model is trained based on the extracted set of meta-features and the extracted set of ML pipeline components. The method according to claim 1, further comprising:

3. The performance score associated with each of the determined sets of second ML pipelines corresponds to at least one of the F1 score or R2 score associated with the corresponding ML pipeline. The method according to claim 1.

4. The selected one or more ML pipelines from the set of second ML pipelines determined above are: ML pipelines from the determined set of second ML pipelines, related to the maximum performance score, The first group of ML pipelines from the determined set of second ML pipelines corresponding to a performance score exceeding the threshold, or A second group of ML pipelines from the determined set of second ML pipelines, corresponding to a predefined number of top ML pipelines based on the performance score. Includes one of the following: The method according to claim 1.

5. Identifying the code snippet of the first ML model associated with the first ML pipeline, Determining one or more input parameters associated with the identified code snippet, Selecting the second ML model from a set of predefined models associated with the set of predefined ML pipelines, Based on replacing the first ML model with the selected second ML model in the first ML pipeline, the selected second ML model is instantiated. The method according to claim 1, further comprising:

6. The one or more input parameters determined in relation to the identified code snippet include at least one of the training dataset, test dataset, and set of hyperparameters related to the first ML model. The method according to claim 5.

7. This involves selecting a predefined template associated with the second ML model, wherein the selected predefined template is annotated with one or more input parameters of the identified code snippet of the first ML model. The code snippet of the second ML model is constructed based on the parameterization of one or more function calls in the selected predefined template that use one or more input parameters annotated as described above, To instantiate the second ML model, replace the identified code snippet of the first ML model with the configured code snippet of the second ML model. The method according to claim 5, further comprising:

8. Constructing an abstract syntax tree (AST) related to the first ML pipeline, Based on the configured AST, the last application programming interface (API) call related to the prediction function in the first ML pipeline is determined, Assign the previously determined last API call as the target line, Identifying one or more statements related to the first ML model based on the assigned target line. The method according to claim 5, further comprising:

9. The one or more statements associated with the first ML model are identified based on the application of backward program slicing from the assigned target line up to the model declaration associated with the first ML model. The method according to claim 8.

10. The system further includes storing the line numbers of each of the one or more statements related to the first ML model, The one or more statements mentioned above correspond to at least one of a model definition, a fit function call, or a predictive function call. The method according to claim 8.

11. Extracting the assigned target line from the first ML pipeline, Adding the retrieved target line to a queue containing a set of statements related to the first ML pipeline, Popping the first statement from the aforementioned queue, This involves controlling the execution of a first set of operations to obtain model slices related to the first ML model from the first ML pipeline, wherein the first set of operations is: Extracting one or more variables and objects from the first statement above, Identifying a set of second statements that occur in the first ML pipeline before the first statement and include at least one of the extracted variables and objects, To determine whether the third statement in the identified set of second statements occurs before the model definition associated with the first ML model, Based on the determination that the third statement occurs before the model definition, the third statement is added to the queue, Adding the first statement to the aforementioned model slice, Based on the determination that the queue is not empty, the fourth statement is popped from the queue as the first statement. Including the control and Obtaining the model slice to identify the code snippet of the first ML model based on the iterative execution of the first set of operations: The method according to claim 8, further comprising:

12. In response to being executed, the electronic device, The process involves receiving ML projects from multiple ML projects stored in a machine learning (ML) corpus database, where each of the multiple ML projects includes a dataset and a set of ML pipelines applicable to that dataset. The first ML pipeline in the set of first ML pipelines associated with the received ML project is mutated so as to determine a set of second ML pipelines based on a set of predefined ML pipelines, wherein the mutation of the first ML pipeline corresponds to replacing the first ML model associated with the first ML pipeline with a second ML model associated with one of the predefined sets of ML pipelines. Based on the performance scores associated with each of the determined sets of second ML pipelines, one or more ML pipelines are selected from the determined sets of second ML pipelines. To extend the ML corpus database to include the set of first ML pipelines associated with the received ML project and one or more selected ML pipelines. One or more non-temporary computer-readable storage media configured to store instructions for performing an action having [a certain characteristic].

13. The aforementioned operation is, Extracting sets of metafeatures from the database related to each of the multiple ML projects stored in the expanded ML corpus database, Extracting sets of ML pipeline components from sets of ML pipelines associated with each of the multiple ML projects stored in the expanded ML corpus database, The meta-learning model is trained based on the extracted set of meta-features and the extracted set of ML pipeline components. It further has, One or more non-temporary computer-readable storage media according to claim 12.

14. The performance score associated with each of the determined sets of second ML pipelines corresponds to at least one of the F1 score or R2 score associated with the corresponding ML pipeline. One or more non-temporary computer-readable storage media according to claim 12.

15. The selected one or more ML pipelines from the set of second ML pipelines determined above are: ML pipelines from the determined set of second ML pipelines, related to the maximum performance score, The first group of ML pipelines from the determined set of second ML pipelines corresponding to a performance score exceeding the threshold, or A second group of ML pipelines from the determined set of second ML pipelines, corresponding to a predefined number of top ML pipelines based on the performance score. Includes one of the following: One or more non-temporary computer-readable storage media according to claim 12.

16. The aforementioned operation is, Identifying the code snippet of the first ML model associated with the first ML pipeline, Determining one or more input parameters associated with the identified code snippet, Selecting the second ML model from a set of predefined models associated with the set of predefined ML pipelines, Based on replacing the first ML model with the selected second ML model in the first ML pipeline, the selected second ML model is instantiated. It further has, One or more non-temporary computer-readable storage media according to claim 12.

17. The one or more input parameters determined in relation to the identified code snippet include at least one of the training dataset, test dataset, and set of hyperparameters related to the first ML model. One or more non-temporary computer-readable storage media according to claim 16.

18. The aforementioned operation is, This involves selecting a predefined template associated with the second ML model, wherein the selected predefined template is annotated with one or more input parameters of the identified code snippet of the first ML model. The code snippet of the second ML model is constructed based on the parameterization of one or more function calls in the selected predefined template that use one or more input parameters annotated as described above, To instantiate the second ML model, replace the identified code snippet of the first ML model with the configured code snippet of the second ML model. It further has, One or more non-temporary computer-readable storage media according to claim 16.

19. The aforementioned operation is, Constructing an abstract syntax tree (AST) related to the first ML pipeline, Based on the configured AST, the last application programming interface (API) call related to the prediction function in the first ML pipeline is determined, Assign the previously determined last API call as the target line, Identifying one or more statements related to the first ML model based on the assigned target line. It further has, One or more non-temporary computer-readable storage media according to claim 16.

20. The memory that stores the instructions, It is coupled to the aforementioned memory and executes the aforementioned instruction, The process involves receiving ML projects from multiple ML projects stored in a machine learning (ML) corpus database, where each of the multiple ML projects includes a dataset and a set of ML pipelines applicable to that dataset. The first ML pipeline in the set of first ML pipelines associated with the received ML project is mutated so as to determine a set of second ML pipelines based on a set of predefined ML pipelines, wherein the mutation of the first ML pipeline corresponds to replacing the first ML model associated with the first ML pipeline with a second ML model associated with one of the predefined sets of ML pipelines. Based on the performance scores associated with each of the determined sets of second ML pipelines, one or more ML pipelines are selected from the determined sets of second ML pipelines. To extend the ML corpus database to include the set of first ML pipelines associated with the received ML project and one or more selected ML pipelines. A processor that executes a process having An electronic device having

Images