Automatic noise reduction for machine learning projects

Abstract

To provide a method of automatic denoising of machine learning projects, a program, and an electronic device.SOLUTION: A method includes: receiving a machine learning (ML) project; determining one or more atomic steps corresponding to an ML pipeline to determine an atomized ML pipeline; instrumenting the atomized ML pipeline to determine an instrumented ML pipeline including one or more operations corresponding to the received ML project; executing the instrumented ML pipeline to capture one or more data-frame snapshots based on each of the one or more operations; constructing a feature provenance graph (FPG) on the basis of the ML pipeline and the captured one or more data-frame snapshots; and identifying one or more discarded features, from a plurality of features corresponding to the received ML project, based on the constructed FPG.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] [Cross - Reference to Related Applications / Incorporation by Reference] This application claims priority to U.S. Provisional Patent Application No. 63 / 260,909, filed on September 3, 2021, entitled "DE - NOISING PIPELINE DATA USING DYNAMIC DATA FLOW ANALYSIS FOR ACCURATE META - LEARNING", the entire content of which is hereby incorporated by reference herein.

[0002] [Technical Field] The embodiments discussed in this disclosure relate to automatic noise removal in machine learning projects.

Background Art

[0003] Advances in the fields of artificial intelligence (AI) and machine learning (ML) have led to the application of AI / ML algorithms in various areas. Typically, a data scientist may manually create an ML pipeline for a given dataset. Manually creating an ML pipeline can be a time - consuming task, which may require a significant amount of effort from expert users such as data scientists. Further, human data scientists often discard some features of the input dataset for a given ML project. In recent years, certain techniques have been developed for the automatic generation of ML pipelines for datasets. Typically, the discarded features can constitute a source of noise for learning about the automatic generation of ML pipelines. However, there can be multiple reasons and tricks for pruning features. Therefore, simple analysis or heuristics may not be appropriate for identifying discarded features in order to improve the automatic generation of ML pipelines.

[0004] The subject matter claimed in this disclosure is not limited to embodiments that resolve any shortcomings or operate only in the aforementioned environment. Rather, this background art is provided solely to illustrate one exemplary technical field in which some of the embodiments described in this disclosure may be implemented. [Overview of the Initiative]

[0005] According to one embodiment, the operation may include receiving a machine learning (ML) project, the ML project including a data frame and an ML pipeline containing a plurality of code statements associated with a plurality of features corresponding to the received ML project. The operation may further include determining one or more atomic steps corresponding to the ML pipeline to determine an atomicized ML pipeline. The operation may further include instrumenting the atomicized ML pipeline to determine an instrumented ML pipeline containing one or more operations corresponding to the received ML project. The operation may further include executing the instrumented ML pipeline and capturing one or more data frame snapshots based on each of the one or more operations. The operation may further include constructing a feature origin graph (FPG) based on the ML pipeline and the captured one or more data frame snapshots. The operation may further include identifying one or more discarded features from the plurality of features corresponding to the received ML project based on the constructed FPG.

[0006] The objectives and advantages of this embodiment are realized and achieved by at least the elements, features, and combinations specifically indicated in the claims.

[0007] Both the general description above and the detailed description below are provided as examples, are explanatory, and do not limit the claimed invention. [Brief explanation of the drawing]

[0008] Exemplary embodiments are described and illustrated with further specificity and detail through the use of the accompanying drawings. [Figure 1] This figure shows an example environment related to automated noise reduction in a machine learning project. [Figure 2] This is a block diagram of a system for automated noise reduction in a machine learning project. [Figure 3] This diagram shows the execution pipeline for automated noise reduction in a machine learning project. [Figure 4A] This figure shows an exemplary scenario of an exemplary ML pipeline corresponding to an exemplary ML project. [Figure 4B] This figure shows an example feature origin graph (FPG) constructed based on the ML pipeline in Figure 4A. [Figure 5] This diagram shows a flowchart illustrating an example method for automated noise reduction in a machine learning project. [Figure 6A] Figures 6A and 6B are diagrams that collectively show flowcharts of an exemplary method for constructing a feature origin graph (FPG). [Figure 6B] Figures 6A and 6B are diagrams that collectively show flowcharts of an exemplary method for constructing a feature origin graph (FPG). [Figure 7] This diagram shows a flowchart illustrating an exemplary method for identifying one or more discarded features from multiple features corresponding to an ML project based on a constructed FPG. [Figure 8] This figure shows a flowchart illustrating an exemplary method for determining explicit feature mapping in an ML pipeline. [Figure 9] This figure shows an exemplary scenario of an exemplary ML pipeline corresponding to an exemplary ML project. [Figure 10] This figure shows a flowchart illustrating an exemplary method for determining the implicit feature mapping of an ML pipeline. [Figure 11] This figure shows an exemplary scenario for extracting a second mapping.

[0009] All of these are subject to at least one embodiment described herein. [Modes for carrying out the invention]

[0010] Some embodiments of this disclosure relate to systems and methods for automatic denoising of machine learning projects. In this disclosure, ML projects can be received from multiple ML projects stored in a machine learning (ML) corpus database. An ML project may include data frames and ML projects. This ML pipeline may include multiple code statements associated with multiple features corresponding to the received ML project. Furthermore, one or more atomic steps corresponding to the ML pipeline can be determined to determine an atomicized ML pipeline. Subsequently, the atomicized ML pipeline can be instrumented to determine an instrumented ML pipeline including one or more operations corresponding to the ML project. Furthermore, the instrumented ML pipeline can be executed to capture one or more data frame snapshots based on each of one or more operations. A feature origin graph (FPG) can be constructed based on the ML pipeline and the captured one or more data frame snapshots. Subsequently, one or more discarded features from multiple features corresponding to the received ML project can be identified based on the constructed FPG.

[0011] According to one or more embodiments of this disclosure, the field of artificial intelligence (AI) / machine learning (ML) can be improved by configuring a computing system, such as an electronic device, so that the computing system can automatically denoise ML projects. The computing system can receive ML projects from multiple ML projects stored in a machine learning (ML) corpus database. An ML project may include data frames and ML projects. This ML pipeline may include multiple code statements associated with multiple features corresponding to the received ML project. The computing system can further determine an atomicized ML pipeline by determining one or more atomic steps corresponding to the ML pipeline. The computing system can then instrument the atomicized ML pipeline to determine an instrumented ML pipeline that includes one or more operations corresponding to the ML project. The computing system can further execute the instrumented ML pipeline and capture one or more data frame snapshots based on each of the one or more operations. Furthermore, the computing system can construct a feature provenance graph (FPG) based on the ML pipeline and the one or more captured data frame snapshots. The computing system can then identify one or more discarded features from multiple features corresponding to the received ML project based on the constructed FPG.

[0012] Traditional methods of ML pipeline generation can require considerable manual effort and time from expert users such as data scientists. Human data scientists often discard certain features from the input dataset (i.e., the data frame of the ML project). In recent years, specific techniques have been developed for the automated generation of ML pipelines for datasets. Typically, deleted features can constitute a source of noise for learning about automated ML pipeline generation. However, there may be multiple reasons and techniques for cutting out features. Therefore, simple analysis or heuristics may not be appropriate for identifying discarded features in order to improve automated ML pipeline generation.

[0013] The electronic device of this disclosure can automatically denoise ML projects. The electronic device of this disclosure can accurately identify data features that may not be used by the ML pipeline of an ML project or that may be effectively discarded, by utilizing dynamic data flow analysis of the ML pipeline. The disclosed electronic device can systematically track the execution of the ML pipeline operation by operation and can automatically record the addition, removal, or modification of data features in the data frame, thereby clearly identifying features that may be used by the ML pipeline. Furthermore, the disclosed electronic device can label the ML pipeline of an ML project that has discarded features to obtain a cleaner ML pipeline and data frame, which may be used for automated generation of ML pipelines with higher accuracy. Accurate identification of discarded features can further result in the generation of higher quality meta-learning models from the ML project based on the obtained cleaner ML pipeline and data frame.

[0014] Embodiments of this disclosure will be described with reference to the accompanying drawings.

[0015] Figure 1 shows an exemplary environment related to automated denoising of a machine learning project according to at least one embodiment described herein. Referring to Figure 1, Environment 100 is shown. Environment 100 may include an electronic device 102, a server 104, a user end device 106, a database 108, and a communication network 110. The electronic device 102 and the server 104, as well as the user end device 106, may be communicatively coupled to each other via the communication network 110. The electronic device 102 may include a meta-learning model 102A. A user 112 is further shown in Figure 1, who can be associated with or interact with the electronic device 102 or the user end device 106. The database 108 may include multiple ML projects. For example, the multiple ML projects may include "n" ML projects, such as ML project 1 114A, ML project 2 114B, ..., and ML project n 114N. Each of the multiple ML projects may include a dataset (also called a data frame) and a set of ML pipelines applicable to the dataset. For example, ML Project 1 114A may include data frame 116A and ML pipeline 118A. Furthermore, ML Project 2 114B may include data frame 116B and ML pipeline 118B. Similarly, ML Project n 114N may include data frame 116N and ML pipeline 118N.

[0016] The "n" ML projects shown in Figure 1 are presented merely as an example. Multiple ML projects may include only two ML projects or more than two ML projects, without deviating from the scope of the disclosure. For brevity, only "n" ML projects are shown in Figure 1. However, in some embodiments, there may be more than "n" ML projects without limiting the scope of the disclosure.

[0017] The electronic device 102 can include appropriate logic, circuitry, and interfaces configured to receive an ML project (e.g., ML project 1 114A) that includes a data frame (e.g., data frame 1 116A) and an ML pipeline (e.g., ML pipeline 118A). The ML pipeline can include a plurality of code sentences associated with a plurality of features corresponding to the received ML project. The electronic device 102 may further be configured to determine one or more atomic steps corresponding to the ML pipeline to determine an atomized ML pipeline. The electronic device 102 may further be configured to instrument the atomized ML pipeline to determine an instrumented ML pipeline that includes one or more operations corresponding to the received ML project. The electronic device 102 may further be configured to execute the instrumented ML pipeline to capture one or more data frame snapshots based on each of the one or more operations. The electronic device 102 may further be configured to construct a feature provenance graph (FPG) based on the ML pipeline and the one or more captured data frame snapshots. The electronic device 102 may further be configured to identify one or more discarded features from the plurality of features corresponding to the received ML project based on the constructed FPG. Examples of the electronic device 102 may include, but are not limited to, computing devices, smartphones, cellular phones, mobile phones, gaming devices, mainframe machines, servers, computer workstations, data analysis machines, and / or consumer electronics (CE) devices.

[0018] The meta-learning model 102A may include appropriate logic, circuits, interfaces, and / or code configured to use a meta-learning algorithm to generate a predictive model based on a previously trained model (e.g., an ML pipeline) and a data frame or features. The meta-learning model 102A can learn from the output of other learning algorithms. For example, for prediction, the meta-learning model 102A may be learned based on the output of other learning algorithms. In another example, the meta-learning model 102A may be supplied with the parameters of other ML models (e.g., a neural network model, a polynomial regression model, a random forest classifier, a logistic regression model, an ensemble learning model) and data frames / features corresponding to each ML algorithm. The meta-learning model 102A can learn meta-features and meta-heuristics based on the data frames / features and parameters associated with each of the input ML models. In one example, once the meta-learning model 102A is trained, it can be used to generate an ML pipeline based on input features or data frames associated with an ML project.

[0019] Server 104 can include appropriate logic, circuitry, and interfaces, and / or code configured to receive an ML project that includes a data frame and an ML pipeline. The ML pipeline can include a plurality of code statements associated with a plurality of features corresponding to the ML project. Server 104 may further be configured to determine one or more atomic steps corresponding to the received ML pipeline to determine an atomized ML pipeline. Server 104 may further be configured to instrument the atomized ML pipeline to determine an instrumented ML pipeline that includes one or more operations corresponding to the received ML project. Server 104 may further be configured to execute the instrumented ML pipeline to capture one or more data frame snapshots based on each of the one or more operations. Server 104 may further be configured to construct a feature provenance graph (FPG) based on the received ML pipeline and the captured one or more data frame snapshots. Server 104 may further be configured to identify one or more discarded features from the plurality of features corresponding to the received ML project based on the constructed FPG. Server 104 may be implemented as a cloud server and can execute operations through web applications, cloud applications, HTTP requests, repository operations, file transfers, etc. Other exemplary implementations of Server 104 may include, but are not limited to, database servers, file servers, web servers, media servers, application servers, mainframe servers, data analysis servers, or cloud computing servers.

[0020] In at least one embodiment, the server 104 may be implemented as a plurality of distributed cloud-based resources using some techniques well known to those skilled in the art. Those skilled in the art will understand that the scope of the disclosure does not have to be limited to implementing the server 104 and the electronic device 102 as two separate entities. In certain embodiments, without departing from the scope of the disclosure, the functionality of the server 104 may be incorporated in whole or at least in part into the electronic device 102.

[0021] The user end device 106 may include appropriate logic, circuitry, interfaces, and / or code that can be configured to store a real-time application in which automated denoising of machine learning pipelines may be performed. In some embodiments, the user end device 106 may receive a first user input from a user (e.g., a data scientist, such as user 112) and generate a set of ML pipelines based on the received first user input. In another embodiment, the user end device 106 may receive one or more second user inputs from a user (e.g., a data scientist, such as user 112) and generate a set of ML pipelines associated with each of a plurality of ML projects based on one or more second user inputs. Furthermore, the user end device 106 may receive multiple datasets (or dataframes) associated with the plurality of projects from various sources such as online dataset repositories, code repositories, and online open-source projects. The user end device 106 may be configured to upload the set of ML pipelines associated with the plurality of projects and the multiple datasets (i.e., dataframes) to the server 104. Multiple uploaded ML pipelines and datasets can be stored as multiple ML projects in the ML corpus database in database 108. In one embodiment, a user end device 106 can receive another user input associated with the identification of one or more features that can be removed from each of the multiple ML projects. Examples of user end devices 106 include, but are not limited to, mobile devices, desktop computers, laptops, computer workstations, computing devices, mainframe machines, data analysis systems, servers such as cloud servers, and groups of servers.

[0022] Database 108 may contain appropriate logic, interfaces, and / or code that can be configured to store multiple ML projects, and each ML project may contain a data frame for each of the multiple ML projects and a set of ML pipelines applicable to this data frame. Database 108 may be derived from data in relational or non-relational databases, or from a set of comma-separated value (CSV) files in conventional or big data storage. Database 108 may be stored or cached on a device such as a server (e.g., server 104) or an electronic device 102. The device storing database 108 may be configured to receive queries from the electronic device 102 for ML projects from multiple machine learning (ML) projects. In response, the database 108 device may be configured to retrieve the queried ML project, including the relevant data frame and the associated set of ML pipelines, based on the received query, and provide it to the electronic device 102.

[0023] In some embodiments, the database 108 may be hosted on multiple servers stored in the same or different locations. The operation of the database 108 may be performed using hardware including a processor, a microprocessor (for example, to perform or control one or more operations), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In some other examples, the database 108 may be implemented using software.

[0024] The communication network 110 may include a communication medium that enables electronic devices 102, a server 104, and user end devices 106 to communicate with each other. The communication network 110 may be either a wired or wireless connection. Examples of the communication network 110 include, but are not limited to, the Internet, a cloud network, a cellular or wireless mobile network (such as Long-Term Evolution and 5G New Radio), a Wireless Fidelity (Wi-Fi) network, a Personal Area Network (PAN), a Local Area Network (LAN), or a Metropolitan Area Network (MAN). Various devices within the environment 100 may be configured to connect to the communication network 110 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols include, but are not limited to, at least one of the following: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee®, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, Multihop Communication, Wireless Access Point (AP), Device-to-Device Communication, Cellular Communication Protocol, and Bluetooth® (BT) Communication Protocol.

[0025] In operation, the electronic device 102 may be configured to receive an ML project (e.g., ML project 114A) which includes a data frame (e.g., data frame 116A) and an ML pipeline (e.g., ML pipeline 118A). The ML pipeline may contain multiple code statements associated with multiple features corresponding to the received ML project. The ML project can help an application perform tasks such as prediction and regression without being programmed to do so. The data frame may contain historical data corresponding to a specific ML task defined on that data frame. The ML pipeline, or script or program, is a set of operations for training an ML model for a specific ML task and for using the trained model for prediction on a new data instance. Multiple code statements may correspond to a set of operations. Multiple features may also be independent variables provided as input to the ML model for training. Each of the multiple features may correspond to a column in a data frame which may contain a tabular dataset.

[0026] The electronic device 102 may be configured to determine an atomicized ML pipeline by determining one or more atomic steps corresponding to an ML pipeline (e.g., ML pipeline 118A). It can be understood that each atomic step may perform one operation. The received ML pipeline may perform one or more atomic steps, such as variable assignment, feature renaming, or feature deletion. The electronic device 102 may convert the ML pipeline into one or more atomic statements to determine an atomicized ML pipeline.

[0027] The electronic device 102 may be configured to instrument an atomicized ML pipeline to determine an instrumented ML pipeline that includes one or more operations corresponding to an received ML project. To instrument an atomicized ML pipeline, one or more code statements can be injected into the atomicized ML pipeline to generate logs such as events, warnings, and used dataframes. In this specification, an instrumented ML pipeline may include one or more operations corresponding to an received ML project. Thus, an atomicized ML pipeline can be instrumented based on the injection of specific additional lines of code after each of one or more atomic steps, such that the current state of the dataframes can be collected as dataframe snapshots.

[0028] The electronic device 102 may be configured to run an instrumented ML pipeline and capture one or more dataframe snapshots based on each of one or more operations. In this specification, the instrumented ML pipeline may run to generate a log of actions such as feature insertion, feature renaming, feature deletion, and value assignment, or it may run for each of one or more operations. Based on the capture of snapshots of the generated action logs, one or more dataframe snapshots can be obtained. For example, feature "A" may be renamed to "A*" using a rename operation. In this specification, this is a dataframe snapshot of the corresponding rename operation. Details of one or more operations are further provided, for example, in Figure 4A.

[0029] The electronic device 102 may be configured to construct a feature origin graph (FPG) based on an ML pipeline and one or more captured data frame snapshots. The FPG can represent the evolution of each feature as a directed acyclic graph (DAG) through various feature engineering steps associated with the received ML project. In this specification, a DAG may include multiple nodes and directed edges. Each node may be associated with a feature, and each edge may indicate the relationship between feature nomenclature at different steps of the feature engineering process associated with the received ML project. To construct an FPG, the FPG may begin with a first layer containing one node for each of the multiple features corresponding to the received ML project. Subsequently, nodes for each of the multiple features corresponding to the received ML project can be added, deleted, or modified in each layer based on one or more captured data frame snapshots. For example, if feature "A" is renamed to "A*" in a first operation, the node corresponding to feature "A" in the first layer of the FPG may be assigned as "A*" in the second layer of the FPG, based on the dataframe snapshot corresponding to the first operation. Thus, an FPG can be constructed by analyzing the dataframe snapshots corresponding to each of one or more operations. In one embodiment, the FPG can represent how each of several features corresponding to an incoming ML project may be modified, deleted, or renamed in the ML pipeline. Details of the FPG are provided, for example, in Figure 4B.

[0030] The electronic device 102 may be configured to identify one or more discarded features from a plurality of features corresponding to the received ML project based on the constructed FPG. In this specification, discarded features may correspond to features among the plurality of features that are deleted or ignored in the received ML project. In one example, the ML project may be associated with an ML task to predict the price per night of rental accommodations at a certain location. Here, the plurality of features of the ML project may include "neighbourhood_group", "minimum_nights", "longitude", "calculated_host_listings_count", "host_id", "id", "name", "host_name", "last_review", "number_of_reviews", "availability_365", "neighbourhood", "latitude", "room_type", "reviews_per_month", and "price". The ML pipeline corresponding to the ML project may be as follows:

number

[0031] Based on the ML pipeline described above, it can be seen that features such as "id", "name", "host_name", and "host_id" will be discarded. Therefore, nodes in the second layer of the FPG corresponding to such features may be removed. Thus, discarded features can be identified from the FPG based on features corresponding to nodes that may not exist in the final layer of the FPG. Details of one or more discarded features are further provided, for example, in Figures 8 and 10.

[0032] The electronic device 102 of this disclosure can identify one or more discarded features and can provide cleaner ML pipelines and dataframes associated with ML projects for learning-based automated machine learning (AutoML) techniques. Cleaner ML pipelines and dataframes that can be generated based on the identification of one or more discarded features may lead to improved accuracy of AutoML techniques. For example, meta-learning model 102A may be trained and used for an AutoML task. Meta-learning model 102A can learn meta-features and meta-heuristics based on dataframes / features (including one or more discarded features) and ML pipelines associated with multiple ML projects. Since one or more discarded features may include certain features that may be associated or that may be accidentally deleted during data preprocessing stages associated with ML projects, controlled inclusion or removal of such initially discarded features may lead to better feature engineering results, thereby improving the overall quality of meta-learning model 102A. Once the meta-learning model 102A is trained, it can be used to generate an ML pipeline based on input features or data frames associated with the ML project.

[0033] Without departing from the scope of this disclosure, modifications, additions, or omissions may be made to Figure 1. For example, the environment 100 may include more or fewer elements than those exemplified and described in this disclosure. For example, in some embodiments, the environment 100 may include an electronic device 102 but not a database 108. Furthermore, in some embodiments, without departing from the scope of the disclosure, the respective functions of the database 108 and the server 104 may be incorporated into the electronic device 102.

[0034] Figure 2 is a block diagram of a system for automated denoising of machine learning projects according to at least one embodiment described herein. Figure 2 is described in relation to the elements of Figure 1. Referring to Figure 2, a block diagram 200 of system 202 including electronic device 102 is shown. Electronic device 102 may include a processor 204, memory 206, meta-learning model 102A, input / output (I / O) devices 208 (including display device 208A), and network interface 210.

[0035] The processor 204 may include appropriate logic, circuitry, and interfaces that can be configured to execute instruction sets stored in memory 206. The processor 204 may be configured to execute program instructions associated with different operations performed by the electronic device 102. For example, part of the operation may include receiving an ML project. The operation may further include determining one or more atomic steps corresponding to the ML pipeline to determine an atomicized ML pipeline. The operation may further include instrumenting the atomicized ML pipeline to determine an instrumented ML pipeline that includes one or more operations corresponding to the received ML project. The operation may further include executing the instrumented ML pipeline and capturing one or more data frame snapshots based on each of the one or more operations. The operation may further include constructing a feature origin graph (FPG) based on the received ML pipeline and the one or more captured data frame snapshots. The operation may further include identifying one or more discarded features from a plurality of features corresponding to the received ML project based on the constructed FPG. Examples of processor technologies may include, but are not limited to, central processing units (CPUs), x86-based processors, reduced instruction set computing (RISC) processors, application-specific integrated circuit (ASIC) processors, composite instruction set computing (CISC) processors, graphical processing units (GPUs), and other processors.

[0036] Although shown as a single processor in Figure 2, the processor 204 may include any number of processors configured to individually or collectively perform or direct the performance of any number of operations of the electronic device 102 described herein. Furthermore, one or more processors may reside in one or more different electronic devices, such as different servers. In some embodiments, the processor 204 may be configured to interpret and / or execute program instructions stored in memory 206 and / or process data. After the program instructions are loaded into memory 206, the processor 204 can execute the program instructions. Some examples of the processor 204 may be a graphics processing unit (GPU), a central processing unit (CPU), a reduced instruction set computer (RISC) processor, an ASIC processor, a composite instruction set computer (CISC) processor, a coprocessor, and / or a combination thereof.

[0037] Memory 206 may include appropriate logic, circuitry, and interfaces that can be configured to store one or more instructions executed by the processor 204. One or more instructions stored in memory 206 may be executed by the processor 204 to perform different operations of the processor 204 (and electronic device 102). Memory 206 may be configured to store received ML projects, including associated data frames and associated ML pipelines. Memory 206 may further be configured to store atomicized ML pipelines, instrumented ML pipelines, one or more captured data frame snapshots, constructed FPGs, and one or more identified discarded features. Examples of memory implementations, but not limited to, include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), hard disk drives (HDDs), solid-state drives (SSDs), CPU caches, and / or secure digital (SD) cards.

[0038] The I / O device 208 may include appropriate logic, circuitry, and interfaces that can be configured to receive input from user 112 and provide output based on the received input. For example, the I / O device 208 may receive from user 112 a user input associated with the generation of an ML pipeline, a data frame associated with the ML pipeline, or an ML project. Furthermore, the I / O device 208 may receive another user input associated with the selection or deletion of features associated with the ML project. In addition, the I / O device 208 may render one or more of the following: an atomicized ML pipeline, an instrumented ML pipeline, one or more captured data frame snapshots, a constructed FPG, one or more identified discarded features, and / or a predicted output of a trained meta-learning model 102A. The I / O device 208, which may include various input / output devices, may be configured to communicate with the processor 204. Examples of the I / O device 208 may include, but are not limited to, a touchscreen, keyboard, mouse, joystick, microphone, display device (such as display device 208A), and speaker.

[0039] The display device 208A may include appropriate logic, circuitry, and interfaces that can be configured to display one or more discarded features identified from multiple features corresponding to an ML project. The display device 208A may also display the predictive output of the meta-learning model 102A. The display device 208A may also be a touchscreen, which may allow a user (e.g., user 112) to provide user input via the display device 208A. The touchscreen may be at least one of resistive, capacitive, or thermal touchscreens. The display device 208A may be implemented through several known technologies, such as at least one of liquid crystal display (LCD) displays, light-emitting diode (LED) displays, plasma displays, or organic LED (OLED) display technologies, or other display devices. According to one embodiment, the display device 208A may refer to a display screen of a head-mounted device (HMD), smart glasses device, see-through display, projection-based display, electrochromic display, or transparent display.

[0040] The network interface 210 may include appropriate logic, circuitry, and interfaces that can be configured to facilitate communication between the processor 204, the server 104, and the user end device 106 (or any other device in the environment 100) over the communication network 110. The network interface 210 may be implemented using various known techniques to support wired or wireless communication of the electronic device 102 with the communication network 110. The network interface 210 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identification module (SIM) card, or a local buffer circuit. The network interface 210 may be configured to communicate wirelessly with networks such as the Internet, an intranet, or wireless networks such as a cellular telephone network, a wireless local area network (LAN), and a metropolitan area network (MAN).Wireless communication technologies include Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), wideband code division multiple access (W-CDMA), Long Term Evolution (LTE®), 5G New Radio (NR), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, or IEEE 802.11n, etc.), Voice over Internet Protocol (VoIP), Light Fidelity (Li-Fi), and Worldwide Interoperability for Microwave Access. It may be configured to use one or more of several communication standards, protocols, and technologies, such as Access, Wi-MAX, email, instant messaging, and protocols for Short Message Service (SMS).

[0041] Without departing from the scope of this disclosure, the exemplary electronic device 102 may be modified, added to, or omitted. For example, in some embodiments, the exemplary electronic device 102 may include any number of other components that are not explicitly illustrated or described for the sake of brevity.

[0042] Figure 3 shows an execution pipeline for automated denoising of a machine learning project according to one embodiment of the present disclosure. Figure 3 is described in relation to the elements of Figures 1 and 2. Referring to Figure 3, the execution pipeline 300 is shown. An exemplary execution pipeline 300 may include a set of operations that can be performed by one or more components of Figure 1, such as an electronic device 102. The operations may include a transformation operation 304, an instrumentation operation 308, a data frame snapshot capture operation 312, a snapshot analysis operation 316, an abstract syntax tree (AST) analysis operation 320, a graph construction operation 324, and a path analysis operation 328. The exemplary execution pipeline 300 may further include an ML project 302 (including an ML pipeline 302A and a data frame 302B), an atomicized ML pipeline 306, an instrumented ML pipeline 310, a data frame snapshot 314, an implicit feature mapping 318, an explicit column mapping 322, an FPG 326, a list of relevant features 330, and a meta-learning model 102A. The set of operations may be performed by the electronic device 102 or processor 204 to automatically denoise ML projects (such as ML project 302) as described herein.

[0043] In 304, a transformation operation may be performed. In the transformation operation, the processor 204 may be configured to receive an ML project 302. The ML project 302 may include a data frame 302B and an ML pipeline 302A (containing multiple code statements associated with multiple features corresponding to the ML project 302). In the transformation operation, the processor 204 may further be configured to determine one or more atomic steps corresponding to the ML pipeline 302A to determine an atomicized ML pipeline 306. In this specification, each atomic step may correspond to a unitary operation such as variable assignment, feature renaming, or feature deletion. The processor 204 can transform the ML pipeline 302A into an atomicized ML pipeline 306 such that each statement in the atomicized ML pipeline 306 corresponds to one operation.

[0044] In 308, an instrumentation operation may be performed. In the instrumentation operation, the processor 204 may be configured to instrument the atomicized ML pipeline 306 pipeline and determine an instrumented ML pipeline 310 that includes one or more operations corresponding to the ML project 302. To instrument the atomicized ML pipeline 306, one or more code statements may be injected into the atomicized ML pipeline 306 to generate logs such as events, warnings, and used data frames. In this specification, the instrumented ML pipeline 310 may include one or more operations corresponding to the ML project 302.

[0045] In 312, a data frame snapshot capture operation may be performed. In the data frame snapshot capture operation, the processor 204 may be configured to execute an instrumented ML pipeline 310 to capture one or more data frame snapshots (e.g., data frame snapshot 314) based on each of one or more operations. In this specification, the instrumented ML pipeline 310 may be executed to generate a log of actions, such as inserting features, renaming features, deleting features, or assigning values, for each of the one or more operations. The data frame snapshot 314 may be obtained by taking a snapshot of each of the generated action logs.

[0046] In 316, a snapshot analysis operation may be performed. In the snapshot analysis operation, the processor 204 may be configured to analyze the data frame snapshot 314 to determine the implicit feature mapping 318. In this specification, feature renaming can be tracked through each atomic statement to identify the feature mapping between the original data frame (containing multiple features corresponding to ML project 302) and the final data frame (containing one or more features from multiple features present in each of one or more atomic steps in the atomicized ML pipeline 306). Details of the analysis of the data frame snapshot 314 to determine the implicit feature mapping 318 are further provided, for example, in Figures 10 and 11.

[0047] An AST analysis operation may be performed at 320. In the AST analysis operation, processor 204 may be configured to retrieve the abstract syntax tree (AST) corresponding to the atomicized ML pipeline 306 and determine the explicit feature mapping 322. It should be noted that some features may be explicitly renamed by the developer in the ML pipeline 302A. Furthermore, in certain cases, the new feature name (changed by the developer) may differ significantly from the original feature name and therefore may not be captured using the text similarity metric. Such features may correspond to explicit features. Explicit features can be captured as explicit feature mappings 322 based on AST traversal. Further details of the AST analysis for determining the explicit feature mappings 318 are provided, for example, in Figure 8.

[0048] In 324, a graph construction operation may be performed. In the graph construction operation, the processor 204 may be configured to construct a feature origin graph (FPG) 326 based on the ML pipeline 302A and one or more captured data frame snapshots (e.g., data frame snapshot 314). The FPG 326 may be a directed acyclic graph (DAG) that can represent the evolution of the nomenclature of each feature through various feature engineering steps in the ML pipeline 302A. In this specification, a DAG may include a plurality of nodes and a set of directed edges between sets of nodes among these plurality of nodes. Each node may be associated with one feature, and each edge may represent the relationship between feature nomenclature at different steps of the feature engineering process associated with the received ML project. The FPG 326 may be constructed based on implicit feature mappings 318 and explicit feature mappings 322. Further details of the FPG 326 are provided, for example, in Figure 4B.

[0049] In 328, a path analysis operation may be performed. In the path analysis operation, the processor 204 may be configured to identify one or more discarded features from a set of features corresponding to the ML project 302, based on the constructed FPG 326. Once one or more discarded features have been determined, the processor 204 may be configured to determine a list 330 of related features. The determined list 330 of related features can be used to train the meta-learning model 102A. For example, the processor 204 may iterate through each node in the initial data frame. For each node, the processor 204 may perform a depth-first search traverse (e.g., forward or backward depth-first search) to find out if any path exists from the initial data frame to the final data frame. If a path exists, the processor 204 may add the source node feature name to the list 330 of related features. Features not included in the list 330 of related features may correspond to one or more discarded features. Further details on the identification of one or more discarded features are provided, for example, in Figure 5.

[0050] In one embodiment, the processor 204 may be configured to determine one or more relevant features from a plurality of features corresponding to the ML project 302 based on one or more identified discarded features. The processor 204 may also be configured to train a meta-learning model 102A associated with the ML project 302 based on the determined one or more relevant features.

[0051] Typically, human data scientists may discard some features from an input dataframe during the feature engineering phase associated with an ML project. These discarded features can constitute a source of noise for learning about the automated generation of the ML pipeline. However, there may be multiple reasons and techniques for trimming features. Therefore, simple analysis or heuristics may not be appropriate for identifying discarded features in order to improve the automated generation of the ML pipeline.

[0052] The electronic device 102 of this disclosure can utilize dynamic data flow analysis of an ML pipeline to accurately identify data features that may not be used by the ML pipeline of an ML project or that may be effectively discarded. The disclosed electronic device 102 can systematically track the execution of the ML pipeline operation by operation and can automatically record the addition, removal, or modification of data features in the data frame, thereby clearly identifying features that may be used by the ML pipeline. Furthermore, the disclosed electronic device 102 can label the ML pipeline of an ML project that has discarded features to obtain a cleaner ML pipeline and data frame, which may be used for automated generation of ML pipelines with higher accuracy. The accurate identification of discarded features may further result in the generation of a higher quality meta-learning model (e.g., meta-learning model 102A) from the ML project based on the obtained cleaner ML pipeline and data frame.

[0053] For example, processor 204 can extract a set of metafeatures from dataframes associated with each of multiple ML projects. In this specification, the set of metafeatures may also include metafeatures corresponding to one or more discarded features. The set of metafeatures may be extracted in each of the multiple ML projects based on the injection of a metafeature extractor, such as a metafeature method call. The dataframe can then be passed to the metafeature method to extract a set of metafeatures containing columns, missing values, and indicators of text presence.

[0054] The processor 204 can extract a set of ML pipeline components from a set of ML pipelines associated with each of multiple ML projects. For example, a set of ML pipeline components may include the "fillna", "TfidfVectorizer", and "logisticregression" ML pipeline components. In this specification, "fillna" may be a function used to fill in missing values ​​in rows of a data frame. "TfidfVectorizer" may be a term-frequency inverse-document-frequency function that can transform text into meaningful numbers. "logisticregression" may be a function that predicts values ​​based on regression.

[0055] The processor 204 can provide a set of extracted meta-features and a set of extracted ML pipeline components for training the meta-learning model 102A. Since the set of extracted meta-features and the set of extracted ML pipeline components may include one or more discarded features, the meta-learning model 102A trained in this manner may be better.

[0056] Once the meta-learning model 102A is trained, the processor 204 can use the trained meta-learning model 102A to automatically synthesize a new ML pipeline and / or denoise an ML project.

[0057] Figure 4A shows an exemplary scenario of an exemplary ML pipeline corresponding to an exemplary ML project according to one embodiment of the disclosure. Figure 4A is described in relation to the elements of Figures 1, 2, and 3. Referring to Figure 4A, exemplary scenario 400A is shown. Scenario 400A may include an exemplary ML pipeline 402. The ML pipeline 402 may include a first statement 404, a second statement 406, a third statement 408, a fourth statement 410, a fifth statement 412, a sixth statement 414, and a seventh statement 416. An electronic device 102 or processor 204 may construct a feature origin graph (FPG), for example, FPG 400B in Figure 4B, based on the ML pipeline 402, as described herein.

[0058] Based on the first statement 404, the processor 204 can identify multiple features, such as feature "A*", feature "B*", feature "C", feature "D", and feature "E". Based on the second statement 406, the processor 204 can drop (i.e., delete / discard) feature "C". Based on the third statement 408, the processor 204 can rename feature "A*" to "A" and feature "B*" to "B". Based on the fourth statement 410, the processor 204 can call the pd.get_dummies function on features "A" and "D" to convert them into corresponding numeric features. Based on the fifth statement 412, the processor 204 can convert feature "A" into features "A_a" and "A_b". Furthermore, the processor 204 can convert feature "D" into features "D_x" and "D_y". Furthermore, the features "A_a", "A_b", "B", "D_x", and "D_y" can be assigned to the variable "X". Based on the sixth statement 414, the processor 204 can assign the feature "E" to the variable "y". In the seventh statement 416, the processor 204 can call the logistic regression function and perform regression based on the variables "X" and "y".

[0059] Figure 4B shows an exemplary feature origin graph (FPG) constructed based on the ML pipeline of Figure 4A according to one embodiment of the present disclosure. Figure 4B is described in relation to the elements of Figures 1, 2, 3, and 4A. Referring to Figure 4B, an exemplary FPG 400B is shown. FPG 400B may be constructed based on the ML pipeline 402 of Figure 4A. FPG 400B may include a first node layer 418, a second node layer 420, a third node layer 422, a fourth node layer 424, and a fifth node layer 426. FPG 400B may further include a first set of directed edges (including edges 428A-428E) and a second set of directed edges (including edges 432A-432E). FPG 400B may further include a node referred to as a null successor 430 for deleted features. The electronic device 102 or processor 204 may be configured to construct an exemplary FPG400B based on the ML pipeline 402, as described herein.

[0060] Referring to Figure 4A, based on the first statement 404, several features such as "A*", "B*", C, D, and E can be identified. Referring to Figure 4B, the processor 204 can initialize FPG400B as a directed acyclic graph containing one node for each of the multiple features, as shown in the first node layer 418.

[0061] Referring to Figure 4A, feature "C" may be dropped based on the second statement 406. Thus, referring to Figure 4B, processor 204 may drop feature "C" from the second node layer 420 of FPG 400B. The second node layer 420 may contain five nodes, one for each of the multiple features as "A*", "B*", "C", "D", and "E". However, the node corresponding to feature "C" may be marked as "X" to indicate that feature "C" is being removed.

[0062] Referring to Figure 4A, based on the third statement 408, features "A*" and "B*" can be renamed to "A" and "B", respectively. Thus, referring to Figure 4B, processor 204 can assign the label "A" to feature "A*" in the third node layer 422. Processor 204 can further assign the label "B" to feature "B*" in the third node layer 422. Feature "C" cannot be carried over to the third node layer 422. The nodes for features "D" and "E" may remain unchanged in the third node layer 422.

[0063] Referring to Figure 4A, based on the fifth statement 412, feature "A" can be transformed into features "A_a" and "A_b". Furthermore, feature "D" can be transformed into features "D_x" and "D_y". Thus, referring to Figure 4B, processor 204 can split feature "A" in the third node layer 422 into two nodes. In the fourth node layer 424, one node can be assigned the label "A_a" and the other node can be assigned the label "A_b". Similarly, processor 204 can split feature "D" in the third node layer 422 into two nodes, and in the fourth node layer 424, one node can be labeled "D_x" and the other node can be labeled "D_y". Feature "D" can be further split into another feature "D_z" in the fourth node layer 424. The processor 204 can finally obtain FPG400B by dropping the feature "D_z" in the fifth node layer 426. The fifth node layer 426 of FPG400B may include nodes representing the features "A_a", "A_b", "B", "D_x", "D_y", and "E", as shown in Figure 4B. Further details of FPG construction are provided, for example, in Figures 6A and 6B.

[0064] Please note that scenarios 400A and FPG400B in Figures 4A and 4B are illustrative and should not be interpreted as limiting the scope of this disclosure.

[0065] Figure 5 is a flowchart illustrating an exemplary method for automated denoising of a machine learning project according to at least one embodiment described herein. Figure 5 is described in relation to the elements of Figures 1, 2, 3, 4A, and 4B. Referring to Figure 5, flowchart 500 is shown. The method shown in flowchart 500 can begin in block 502 and can be performed by any suitable system, apparatus, or device, for example, by the exemplary electronic device 102 in Figure 1 or the processor 204 in Figure 2. Although shown in separate blocks, steps and operations associated with one or more blocks in flowchart 500 may be divided into further blocks, combined into fewer blocks, or removed, depending on the particular implementation.

[0066] In block 502, an ML project may be received, which includes a data frame and an ML pipeline. The ML pipeline may contain multiple code statements associated with multiple features corresponding to the received ML project. The processor 204 may be configured to receive an ML project (e.g., ML project 1 302) which includes a data frame (e.g., data frame 302B) and an ML pipeline (e.g., ML pipeline 302A). The ML pipeline 302A may contain multiple code statements associated with multiple features corresponding to the ML project 302. The ML project 302 can help an application perform ML tasks such as prediction and regression without being programmed to do so. The ML pipeline 302A may contain a set of operations required to train an ML model for a particular ML task and to use the trained ML model for prediction on a new data instance. Multiple code statements may correspond to a set of operations. Multiple features may also be independent variables provided as input to the ML model for training.

[0067] In block 504, one or more atomic steps corresponding to an ML pipeline may be determined to determine an atomicized ML pipeline. Processor 204 may be configured to determine an atomicized ML pipeline by determining one or more atomic steps corresponding to an ML pipeline. It can be understood that each atomic step may perform one operation. An incoming ML pipeline may perform one or more atomic steps, such as assigning variables, renaming features, or deleting features. For example, processor 204 may determine an atomicized ML pipeline 306 by converting ML pipeline 302A into one or more atomic statements.

[0068] In one embodiment, the determination of one or more atomic steps corresponding to an ML pipeline may be based on the application of a source-code lifting technique to multiple code statements. In this specification, the source-code lifting technique can correspond to a code refactoring technique that can be implemented using a library, such as the "python-pl" library, to lift the source code. The source-code lifting technique can capture atomic changes in a data frame to identify the precise mapping of features between different steps in an ML pipeline. Therefore, based on the application of the source-code lifting technique, each of multiple code statements can be transformed into its corresponding atomic step. For example, a given set of code statements among multiple code statements may be as follows:

number

number

[0069] In block 506, an atomicized ML pipeline may be instrumented to determine an instrumented ML pipeline containing one or more operations corresponding to an incoming ML project. The processor 204 may be configured to instrument an atomicized ML pipeline (e.g., atomicized ML pipeline 306) to determine an instrumented ML pipeline (e.g., instrumented ML pipeline 310) containing one or more operations corresponding to an incoming ML project (e.g., ML project 302). One or more code statements may be included in the atomicized ML pipeline for instrumentation. Performing instrumentation of an atomicized ML pipeline may generate logs such as events, warnings, and used dataframes. In this specification, an instrumented ML pipeline may contain one or more operations corresponding to an ML project.

[0070] In one embodiment, the instrumentation of an atomicized ML pipeline may be based on the application of a method call injection technique to the atomicized ML pipeline. In this specification, the source code instrumentation technique can be used to inject predefined function calls to obtain feature information associated with an incoming ML project. For example, the "ast" package in the Python language may be used to inject method calls after each of one or more atomic steps of an atomicized ML pipeline. While the exemplary code, libraries, and packages provided in this disclosure correspond to a high-level programming language (e.g., the "Python" programming language) using a web-based compute notebook, it should be noted that the teachings in this disclosure may also be valid for other ML pipelines written in different languages ​​and development platforms. Furthermore, it should be noted that a web-based compute notebook may correspond to a compute structure that can be used to develop / represent an ML pipeline, particularly during the development phase. For example, a web-based compute notebook may be used to develop the script portion of an ML project (e.g., an ML pipeline).

[0071] In block 508, an instrumented ML pipeline may be executed to capture one or more dataframe snapshots based on each of one or more operations. Processor 204 may be configured to execute an instrumented ML pipeline (e.g., instrumented ML pipeline 310) to capture one or more dataframe snapshots (e.g., dataframe snapshot 314) based on each of one or more operations. For example, once a method call is injected after each of one or more atomic steps of an atomicized ML pipeline, the instrumented ML pipeline may be executed to generate an action log (e.g., feature insertion, feature renaming, feature deletion, value assignment) for each of one or more operations. In one example, since the execution of the instrumented ML pipeline may be time-consuming, a dataframe snapshot 314 may be obtained based on the capture of a snapshot of the generated action log and subsequent storage of the captured snapshot in a JSON file. Storage of the captured snapshot may be useful because, based on this storage, it may not be necessary to execute the instrumented ML pipeline multiple times. Therefore, the processor 204 may programmatically execute the entire instrumented ML pipeline associated with the received ML project and store a summary of one or more captured data frame snapshots in a separate JSON file.

[0072] In one embodiment, each of the one or more captured dataframe snapshots may include at least one of the following: a row number, inputs and outputs for each variable, and a set of feature names associated with a dataframe type. In this specification, the row number may also be the row of each atomic step in one or more atomic steps of an atomicized ML pipeline. The inputs and outputs for each variable may also be the input and output values ​​for each variable in each of one or more operations. The set of feature names associated with a dataframe type may also be the names of features of a particular dataframe type.

[0073] In block 510, a feature origin graph (FPG) may be constructed based on the ML pipeline and one or more captured data frame snapshots. The processor 204 may be configured to construct an FPG (e.g., FPG400B) based on the ML pipeline and one or more captured data frame snapshots. The FPG (e.g., FPG400B) can represent the evolution of each of several features of the received ML project through various feature engineering steps (e.g., including ML pipeline 402) associated with the received ML project (e.g., ML pipeline 402). The FPG may also be a directed acyclic graph (DAG) containing multiple nodes and directed edges. Each node can be associated with a feature, and each edge can represent the relationship between feature nomenclature at different steps of the feature engineering process associated with the received ML project. Further details of constructing the FPG are provided, for example, in Figures 6A and 6B.

[0074] In block 512, one or more discarded features may be identified from multiple features corresponding to the received ML project based on the constructed FPG. The processor 204 may be configured to identify one or more discarded features from multiple features corresponding to the received ML project based on the constructed FPG. For example, referring to Figure 4B, the processor 204 may identify one or more discarded features as features "C" and "D_z". The processor 204 can determine one or more discarded features by tracking multiple features of the ML project received through FPG 400B. Details related to the identification of one or more discarded features are described, for example, in Figures 7, 8, 9, 10, and 11. Control may proceed to termination.

[0075] Flowchart 500 is shown as individual operations such as 502, 504, 506, 508, 510, and 512; however, in certain embodiments, such individual operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation, without impairing the essence of the disclosed embodiments.

[0076] Figures 6A and 6B are diagrams that collectively show flowcharts of exemplary methods for constructing a feature origin graph (FPG) according to at least one embodiment described herein. Figures 6A and 6B are described in relation to elements of Figures 1, 2, 3, 4A, 4B, and 5. Referring to Figures 6A and 6B, flowchart 600 is shown. The method shown in flowchart 600 can be started at 602 and can be performed by any suitable system, apparatus, or device, for example, by the exemplary electronic device 102 in Figure 1 or the processor 204 in Figure 2. Although shown in separate blocks, steps and operations associated with one or more blocks in flowchart 600 may be divided into further blocks, combined into fewer blocks, or removed, depending on the particular implementation.

[0077] In block 602, the feature origin graph (FPG) may be initialized as a directed acyclic graph containing one node for each feature associated with one or more captured dataframe snapshots. The processor 204 may be configured to initialize the feature origin graph (FPG) as a directed acyclic graph containing one node for each feature associated with one or more captured dataframe snapshots. For example, based on one or more captured dataframe snapshots (such as one or more dataframe snapshots 314), a first node layer in the DAG can be initialized for each feature associated with one or more captured dataframe snapshots. In this specification, each node may be labeled with the name of the corresponding feature.

[0078] Referring to Figure 4B, the features associated with one or more captured dataframe snapshots may be features "A*", "B*", "C", "D", and "E". Here, the processor 204 can initialize FPG400B as a directed acyclic graph containing one node for each feature associated with one or more captured dataframe snapshots, as shown in the first node layer 418. The first node layer 418 may contain five nodes, one for each of the multiple features such as "A*", "B*", "C", "D", and "E". Each node may be labeled according to the corresponding feature.

[0079] In block 604, an abstract syntax tree (AST) associated with the atomicized ML pipeline (e.g., atomicized ML pipeline 306) may be retrieved. The processor 204 may be configured to retrieve the abstract syntax tree (AST) associated with the atomicized ML pipeline. It can be understood that the AST of the code may also be a tree representation of the abstract syntactic framework of the code in a formal language. The AST does not have to contain all the details or syntax of the code. Rather, the AST may contain only the abstract syntax in a formal language, such as "while" statements, "if" statements, "branch" statements, "comparison" statements, "assignment" statements, and variable names. Each node in the tree may represent a construct of the code. Once the ML pipeline has been atomicized and the atomicized ML pipeline has been determined, the processor 204 can determine the AST associated with the atomicized ML pipeline and store the determined AST in memory, for example, 206. The processor 204 may be configured to retrieve the AST associated with the atomicized ML pipeline from memory 206 in 604.

[0080] In block 606, a first operation from one or more operations may be selected to analyze a data frame snapshot from one or more captured data frame snapshots. In this specification, a data frame snapshot can correspond to the selected first operation and may include input and output information. The processor 204 may be configured to select a first operation from one or more operations to analyze a data frame snapshot from one or more captured data frame snapshots. The input information may be input values, and the output information may be output values ​​of the data frame corresponding to the data frame snapshot being analyzed. The first operation may need to be performed based on each of the one or more operations. Each of the one or more operations can be selected one by one as the first operation. For example, referring to Figure 4A, the input information associated with the third statement 408 may include features "A*" and "B*". The output information associated with the third statement 408 may be features "A" and "B" corresponding to features "A*" and "B*", respectively. Here, the selected first operation may be a rename operation.

[0081] In block 608, a node layer may be added to the FPG for each feature associated with the output information of the dataframe snapshot. The processor 204 may be configured to add a node layer to the FPG for each feature associated with the output information of the dataframe snapshot. For example, referring to Figure 4A, the input information of the second statement 406 may be features "A*", "B*", "C", "D", and "E". The output information of the second statement 406 may be features "A*", "B*", "D", and "E". Feature "C" may be dropped. Here, the selected first operation may be a drop (or delete) operation. Based on the output information, a second node layer 420 may be added for each feature in the output information of the dataframe snapshot. Since feature "C" may have been dropped and not be present in the output information, the node for feature "C" may be marked as "X" in the second node layer 420. Similarly, referring to Figures 4A and 4B, the input information for the third statement 408 may be features "A*" and "B*". The output information for the third statement 408 may be "A" for feature "A*", "B" for feature "B*", "D" for feature "D", and "E" for feature "E". Here, the selected first operation may be a rename operation. Therefore, based on the output information, a third node layer 422 may be added to the FPG 400B for each feature. Here, the third node layer 422 may include four nodes, namely a node labeled "A", a node labeled "B", a node labeled "D", and a node labeled "E".

[0082] In block 610, directed edges can be added to the FPG from a first node associated with input information to a second node associated with output information, based on the correspondence between the first name of the first feature associated with the first node and the second name of the second feature associated with the second node. The processor 204 may be configured to add directed edges in the FPG from a first node associated with input information to a second node associated with output information, based on the correspondence between the first name of the first feature associated with the first node and the second name of the second feature associated with the second node. As discussed, directed edges can provide relationships between input and output information. For example, referring to Figures 4A and 4B, the input information of the second statement 406 may be features "A*", "B*", "C", "D", and "E". The output information of the second statement 406 may be features "A*", "B*", "D", and "E". Feature "C" may be dropped. Here, edge 428A can connect the node corresponding to feature "A*" in the first node layer 418 to the node corresponding to feature "A*" in the second node layer 420. Furthermore, edge 428B can connect the node corresponding to feature "B*" in the first node layer 418 to the node corresponding to feature "B*" in the second node layer 420. Edge 428C can connect the node corresponding to feature "C" in the first node layer 418 to the node marked "X" (i.e., null trail 430), indicating that feature "C" is dropped in the second node layer 420. Furthermore, edge 428D can connect the node corresponding to feature "D" in the first node layer 418 to the node corresponding to feature "D" in the second node layer 420. Here, edge 428E can connect the node corresponding to feature "E" in the first node layer 418 to the node corresponding to feature "E" in the second node layer 420.

[0083] In block 612, one or more features and operation names associated with the selected first operation may be identified based on the retrieved AST. The processor 204 may be configured to identify one or more features and operation names associated with the selected first operation based on the retrieved AST. In this specification, the operation name may be an operation such as a rename operation or a delete operation that corresponds to the selected first operation. For example, referring to Figures 4A and 4B, if the second statement 406 is the selected first operation, one or more features may be identified as feature "C", and the operation name may be identified as a drop (or delete) operation.

[0084] In block 614, it may be determined whether the operation name corresponds to a delete operation. The processor 204 may be configured to determine whether the operation name corresponds to a delete operation. It may be understood that a delete operation can be used to discard a feature to which the delete operation may apply. For example, referring to Figures 4A and 4B, since the second statement 406 may correspond to a delete operation, the corresponding feature to which the second statement 406 applies (i.e., feature "C") may be discarded. Thus, the second statement 406 may lead to the deletion of feature "C".

[0085] In block 616, a null follower may be added to the FPG for each node in the layer of nodes associated with the input information of the data frame snapshot, based on the determination that the operation name corresponds to a delete operation. The processor 204 may be configured to add a null follower to the FPG for each node in the layer of nodes associated with the input information of the data frame snapshot, based on the determination that the operation name corresponds to a delete operation. A null follower can indicate that the corresponding feature may be deleted and not carried over in the FPG. For example, referring to Figures 4A and 4B, the second statement 406 may drop feature "C". Here, the selected first operation in the second statement 406 may be identified as a delete operation, and the corresponding feature may be identified as feature "C". Therefore, the input information may include feature "C" which can be deleted based on the second statement 406. Here, a null follower 430, which may be marked "X", may be added for feature "C" in the second node layer 420 of the FPG 400B. Similarly, if the input information contains multiple features that could be subject to deletion, one null successor may be added to each node in the layer of nodes associated with the input information.

[0086] In block 618, it may be determined whether the operation name corresponds to a rename operation. The processor 204 may be configured to determine whether the operation name corresponds to a rename operation. The rename operation can be used to rename features. For example, referring to Figures 4A and 4B, the third statement 408 may rename feature "A*" to "A" and feature "B*" to "B" using the rename function. Here, the input information may be features "A*" and feature "B*", and the selected first operation may be identified as a rename operation.

[0087] In block 620, based on the determination that the operation name corresponds to a rename operation, a directed edge may be added in the FPG from a node associated with the input information of a data frame snapshot to a corresponding renamed successor node associated with the output information of a data frame snapshot. The processor 204 may be configured to add a directed edge in the FPG from a node associated with the input information of a data frame snapshot to a corresponding renamed successor node associated with the output information of a data frame snapshot, based on the determination that the operation name corresponds to a rename operation. For example, referring to Figures 4A and 4B, the third statement 408 may rename feature "A*" to "A" and feature "B*" to "B" using a rename function. Here, the input information may be features "A*" and "B*", and the selected first operation may be identified as a rename operation. Therefore, the directed edge 432A can connect the node labeled "A*" in the second node layer 420 to the node labeled "A" in the third node layer 422. Similarly, the directed edge 432B can connect the node labeled "B*" in the second node layer 420 to the node labeled "B" in the third node layer 422.

[0088] In block 622, based on the determination that the operation name is other than a rename operation or a delete operation, a directed edge may be added in the FPG from each node in the layer of nodes associated with the input information of the data frame snapshot to the corresponding subsequent node associated with the output information of the data frame snapshot. The processor 204 may be configured to add a directed edge in the FPG from each node in the layer of nodes associated with the input information of the data frame snapshot to the corresponding subsequent node associated with the output information of the data frame snapshot, based on the determination that the operation name is other than a rename operation or a delete operation. For example, referring to Figures 4A and 4B, a directed edge 432D may be added in FPG 400B to connect a node labeled "D" in the second node layer 420 to a node labeled "D" in the third node layer 422. Similarly, a directed edge 432E may connect a node labeled "E" in the second node layer 420 to a node labeled "E" in the third node layer 422. Control may proceed to termination.

[0089] Flowchart 600 is shown as individual operations such as 602, 604, 606, 608, 610, 612, 614, 616, 618, 620, and 622; however, in a particular embodiment, such individual operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation, without impairing the essence of the disclosed embodiment.

[0090] Figure 7 is a flowchart illustrating an exemplary method for identifying one or more discarded features from a set of features corresponding to an ML project based on a constructed FPG, according to at least one embodiment described herein. Figure 7 is described in relation to elements of Figures 1, 2, 3, 4A, 4B, 5, 6A, and 6B. Referring to Figure 7, flowchart 700 is shown. The method shown in flowchart 700 can begin in block 702 and can be performed by any suitable system, apparatus, or device, for example, by the exemplary electronic device 102 in Figure 1 or the processor 204 in Figure 2. Although shown in separate blocks, steps and operations associated with one or more blocks in flowchart 700 may be divided into further blocks, combined into fewer blocks, or removed, depending on the particular implementation.

[0091] In block 702, the constructed feature origin graph (FPG) can be retrieved. The processor 204 can be configured to retrieve the constructed feature origin graph (FPG). For example, the constructed FPG 400B may be stored in memory 206, for example. The processor 204 can retrieve the constructed FPG 400B from memory 206.

[0092] In block 704, a node may be selected from a data frame snapshot in the retrieved FPG. The processor 204 may be configured to select a node from a data frame snapshot in the retrieved FPG, where the data frame snapshot may be either the final or initial data frame snapshot in the FPG. For traversing the FPG, one node may be selected from the data frame snapshot.

[0093] In block 706, a depth-first search of the extracted FPG may be performed by marking each visited node in the extracted FPG from a selected node. The processor 204 may be configured to perform a depth-first search of the extracted FPG by marking each visited node in the extracted FPG from a selected node. It can be understood that a depth-first search (DFS) of a tree may correspond to traversing the tree from the root node to leaf nodes along branches until all paths in the tree are explored and traversed, followed by backtracking to unexplored paths, and similar traversing to leaf nodes along newly explored paths. DFS can explore paths in a tree using backtracking. The DFS of this disclosure can traverse the FPG from a selected node (which may be considered the root node of the resulting tree associated with the FPG) by marking each visited node in the extracted FPG.

[0094] In one embodiment, the depth-first search can correspond to at least one of a forward depth-first search or a reverse depth-first search. A forward depth-first search may be performed based on a forward traverse from the root node in the tree. The forward depth-first search of this disclosure can traverse the FPG from selected nodes that may correspond to an initial data frame snapshot. For example, referring to Figure 4B, a first node layer 418 may be associated with an initial data frame snapshot. Here, for example, a node labeled "C" may be selected and a forward depth-first search may be performed. From the node labeled "C" in the first node layer 418, the forward depth-first search can proceed to a null trail point 430 in a second node layer 420 of the FPG 400B.

[0095] A reverse depth-first search can be performed based on a reverse traverse from leaf nodes in the tree. The reverse depth-first search of this disclosure can traverse the FPG from selected nodes that can correspond to the final data frame snapshot. For example, referring to Figure 4B, a fifth node layer 426 can be associated with the final data frame snapshot. Here, for example, a node labeled "D_x" may be selected and a reverse depth-first search may be performed. From the node labeled "C" in the fifth node layer 426, the reverse depth-first search can proceed to the node labeled "D" in the first node layer 418. In this way, each node in the fifth node layer 426 can be selected one by one, and a reverse depth-first search can be performed by traversing backward from the selected nodes. Each node encountered during the traverse can be marked to indicate that a particular node is covered and appears in the fifth node layer 426.

[0096] In block 708, one or more unmarked nodes may be determined in the retrieved FPG based on the depth-first search performed. In this specification, one or more discarded features may be identified based on the determined one or more unmarked nodes. Processor 204 may be configured to determine one or more unmarked nodes in the retrieved FPG based on the depth-first search performed. Processor 204 can identify one or more discarded features based on the determined one or more unmarked nodes. Discarded features may be one or more features that have been discarded from a group of features corresponding to an ML project (e.g., ML project 302). Features corresponding to marked nodes in the FPG (e.g., FPG400B) may be features used to train the ML model in the ML project, while features corresponding to unmarked nodes in the FPG may be features that are not used to train the ML model (or may be discarded). Referring to Figure 4B, it can be seen that the node corresponding to feature "C" in the first node layer 418 and the node corresponding to feature "D_z" in the fourth node layer 424 may remain unmarked by the reverse depth-first search because the nodes corresponding to features "C" and "D_z" do not appear in the fifth node layer 426 (i.e., the final data frame snapshot). Therefore, features "C" and "D_z" may be discarded features. Thus, if a reverse depth-first search is used to traverse FPG400B, all unmarked nodes in the initial data frame of FPG400B may correspond to discarded features. Similarly, if a forward depth-first search is used to traverse FPG400B, all unmarked nodes in the final data frame of FPG400B may correspond to discarded features. Control may proceed to termination.

[0097] While flowchart 700 is shown as individual operations such as 702, 704, 706, and 708, in certain embodiments, such individual operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation, without impairing the essence of the disclosed embodiments.

[0098] It should be noted that features can be ignored or discarded in many ways within an ML pipeline, for example, through deletion or selection by constructing a new dataframe, either explicitly or dynamically, using feature names. Furthermore, features can be renamed, one feature can be transformed into multiple sub-features, or multiple features can be merged into one. Therefore, static analysis of an ML pipeline may not be sufficient to capture discarded features because features can be dynamically deleted or selected. Moreover, dynamic analysis alone may not capture explicit renaming. Thus, a combination of static and dynamic analysis may be necessary to adequately capture discarded features in an ML project.

[0099] Figure 8 is a flowchart illustrating an exemplary method for determining explicit feature mapping of an ML pipeline according to at least one embodiment described herein. Figure 8 is described in relation to elements of Figures 1, 2, 3, 4A, 4B, 5, 6A, 6B, and 7. Referring to Figure 8, flowchart 800 is shown. The method shown in flowchart 800 can begin in block 802 and can be performed by any suitable system, apparatus, or device, for example, by the exemplary electronic device 102 in Figure 1 or the processor 204 in Figure 2. Although shown in separate blocks, steps and operations associated with one or more blocks in flowchart 800 may be divided into further blocks, combined into fewer blocks, or removed, depending on the particular implementation.

[0100] Figure 9 shows an exemplary scenario of an exemplary ML pipeline corresponding to an exemplary ML project according to one embodiment of the present disclosure. Figure 9 is described in relation to elements of Figures 1, 2, 3, 4A, 4B, 5, 6A, 6B, 7, and 8. Referring to Figure 9, an exemplary scenario 900 is shown. The exemplary scenario 900 may include an exemplary ML pipeline 902. The ML pipeline 902 may include a number of statements, such as statements 904, 906, 908, and 910. An electronic device 102 or processor 204 may process the ML pipeline 902 to determine the explicit feature mapping of the ML pipeline 902, as described herein.

[0101] Referring to Figure 8, in block 802, an abstract syntax tree (AST) of the atomicized ML pipeline may be constructed. The processor 204 may be configured to construct the abstract syntax tree (AST) of the atomicized ML pipeline. The processor 204 can atomicize the ML pipeline 902 to determine the atomicized ML pipeline (e.g., atomicized ML pipeline 306). The processor 204 can construct the AST associated with atomicized ML pipeline 306. Once the AST of atomicized ML pipeline 306 is determined, the processor 204 can store the constructed AST associated with atomicized ML pipeline 306 in memory 206, for example. Details related to the construction of the AST of atomicized ML pipeline 306 are, for example, 604 (in Figure 6A).

[0102] In block 804, the constructed AST of an atomicized ML pipeline (e.g., atomicized ML pipeline 306) can be traversed to identify rename operations. Processor 204 may be configured to traverse the constructed AST of an atomicized ML pipeline to identify rename operations. As discussed, the AST of a given code may also be a tree representation of the abstract syntactic framework of the code in a formal language. The constructed AST of an atomicized ML pipeline does not have to contain all the details or syntax of the atomicized ML pipeline; rather, the AST may contain only the abstract syntax of the atomicized ML pipeline in a formal language (e.g., "while" statements, "if" statements, "branching" statements, "comparison" statements, "assignment" statements, variable names, etc.). The constructed AST of an atomicized ML pipeline can be traversed to identify rename operations. Renaming operations may be used to rename features. For example, referring to Figures 4A and 4B, the third statement 408 can use the rename function to rename feature "A*" to "A" and feature "B*" to "B". The constructed AST of the atomicized ML pipeline can be traversed from either a selected node, such as a leaf node of the AST, or the root node, to identify rename operations. For example, the AST can be traversed forward from the root node to check the rename application programming interface (API).

[0103] Referring to Figure 9, a rename operation may be defined in statement 904. In this specification, a rename function may be called to rename a feature. For example, an AST can be analyzed using AST analysis with a Python program such as LibCST to identify a rename operation.

[0104] Referring to Figure 8, in block 806, a parameter map associated with the identified rename operation may be determined. Processor 204 may be configured to determine the parameter map associated with the identified rename operation. The parameter map can map the initial name of a feature to the final name of the feature after the rename operation. It should be noted that the new feature name may be quite different from the original feature name and may not be captured through the text similarity metric. Therefore, a parameter map can be determined.

[0105] Referring to Figure 9, the rename operation may be defined in statement 904. In this specification, the rename function is called to rename a feature. From statements 906, 908, and 910, it can be seen that a feature named "Track.Name" can be changed to "track_name", a feature named "Beats.Per.Minute" can be changed to "bpm", and a feature named "Loudness..dB" can be changed to "loudness". Thus, the parameter map can map "Track.Name" to "track_name", "Beats.Per.Minute" to "bpm", and "Loudness..dB" to "loudness".

[0106] Referring to Figure 8, in block 808, a first mapping can be extracted between a set of features corresponding to an received ML project and a set of first features associated with the ML pipeline, based on a determined parameter map. In this specification, one or more discarded features can be identified further based on the extracted first mapping. Processor 204 may be configured to extract a first mapping between a set of features corresponding to an received ML project and a set of first features associated with the ML pipeline, based on a determined parameter map. Processor 204 can identify one or more discarded features further based on the extracted first mapping. The extracted first mapping can be analyzed to check whether any features are not mapped. Features not mapped by the first mapping can be identified as one or more discarded features.

[0107] In one embodiment, the first set of features can correspond to one or more features that can be renamed based on user input from a user (such as user 112). In other words, the first set of features may also be explicit features that can be renamed based on a name provided to the rename function by user 112. Thus, the processor 204 can rename the first set of features based on user input. For example, referring to Figure 9, from statements 906, 908, and 910, it can be seen that, based on user input associated with the ML pipeline 902, a feature named "Track.Name" can be changed to "track_name", a feature named "Beats.Per.Minute" can be changed to "bpm", and a feature named "Loudness..dB" can be changed to "loudness". Thus, such features may also be the first set of features. Control may then proceed to termination.

[0108] While flowchart 800 is shown as individual operations such as 802, 804, 806, and 808, in a particular embodiment, such individual operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation, without impairing the essence of the disclosed embodiment.

[0109] Figure 10 is a flowchart illustrating an exemplary method for determining implicit feature mapping of an ML pipeline according to at least one embodiment described herein. Figure 10 is described in relation to elements of Figures 1, 2, 3, 4A, 4B, 5, 6A, 6B, 7, 8, and 9. Referring to Figure 10, flowchart 1000 is shown. The method shown in flowchart 1000 can begin in block 1002 and can be performed by any suitable system, apparatus, or device, for example, by the exemplary electronic device 102 in Figure 1 or the processor 204 in Figure 2. Although shown in separate blocks, steps and operations associated with one or more blocks in flowchart 1000 may be divided into further blocks, combined into fewer blocks, or removed, depending on the particular implementation.

[0110] In block 1002, each of one or more atomic steps corresponding to an ML pipeline may be traversed to track name updates associated with each of the second set of features associated with the ML pipeline. Processor 204 may be configured to traverse each of one or more atomic steps corresponding to an incoming ML pipeline to track name updates associated with each of the second set of features associated with the ML pipeline. In this specification, the second set of features may differ from a first set of features that can be renamed based on a rename operation. The second set of features does not have to be explicitly renamed based on user input from a user such as user 112. Since the renaming of the second set of features may not be based on user input, each atomic step may need to be tracked to monitor name updates associated with each of the second set of features associated with an incoming ML pipeline such as the incoming ML pipeline 302A.

[0111] In one embodiment, the second set of features may correspond to one or more features that can be renamed based on a predefined set of rules. It should be noted that the second set of features may also be implicit features. In this specification, implicit features do not have to be renamed based on user input from a user such as user 112. Rather, the user can pass the second set of features to an application programming interface (API), such as the get_dummies function of the pandas library in the Python programming language. The user does not have to pass a new name to be assigned to the renamed features; instead, the API can assign a new name to the renamed features. In one example, the API can use the old name and add a suffix such as a number (e.g., 1, 2, etc.) or an alphabet. Further details on renaming the second set of features are provided, for example, in Figure 11.

[0112] In block 1004, a second mapping can be extracted between a set of features corresponding to a received ML project and a set of second features. The processor 204 may be configured to extract a second mapping between a set of features corresponding to a received ML project and a set of second features. In this specification, the second mapping can be determined between a set of features corresponding to a received ML project and a set of second features based on tracking updates to the names associated with each of the set of second features associated with an ML pipeline, such as ML pipeline 302A. The second mapping can also be called an implicit feature mapping. In one example, feature "B" among the multiple features corresponding to a received ML project can be converted to features "B1" and "B2" using an API, where feature "B" can be mapped to features "B1" and "B2".

[0113] The second mapping may be a one-to-one mapping, a one-to-many mapping, a many-to-one mapping, or a many-to-many mapping. In one-to-one, one-to-many, and many-to-one mappings, input features and output features can be mapped directly. In other words, in one-to-one, one-to-many, and many-to-one mappings, multiple features corresponding to the received ML project may be mapped directly to a second set of features. In many-to-many mappings, two basic heuristics may be used. The first heuristic allows for the direct mapping of features whose feature names remain the same. Furthermore, the second heuristic allows for the mapping of features where the original feature name is a prefix to the new feature name, or where the value of the original feature is the new feature name.

[0114] In one embodiment, multiple features can correspond to the initial data frame of one or more captured data frame snapshots, and a second set of features can correspond to the final data frame of one or more captured data frame snapshots. In other words, multiple features can correspond to the initial data frame of one or more captured data frame snapshots, such as a captured data frame snapshot 314 of an ML pipeline (ML pipeline 302A, for example). It can be noted that the second set of features may be obtained by multiple variations of one or more of the multiple features. Thus, the second set of features can correspond to the final data frame of one or more captured data frame snapshots. That is, the second set of features can be the final features. For example, if feature "A" corresponding to the initial data frame of one or more captured data frame snapshots is first renamed to "A_1", and then transformed into features "A_11" and "A_12", then feature "A" can be included in multiple features, and features "A_11" and "A_12" can be included in the second set of features. Here, the second mapping can map feature "A" to features "A_11" and "A_12". An exemplary scenario for extracting the second mapping is illustrated in Figure 11, for example.

[0115] In block 1006, one or more discarded features may be identified based further on the extracted second mapping. The processor 204 may be configured to identify one or more discarded features based further on the extracted second mapping. Based on the second mapping, one or more discarded features can be identified by checking for unmapped features. Thus, features that may be excluded from the second mapping can be identified as discarded features. In one embodiment, one or more discarded features may include a set of features that may not be included in the first mapping and / or the second mapping. Control may proceed to termination.

[0116] Flowchart 1000 is shown as individual operations such as 1002, 1004, and 1006; however, in certain embodiments, such individual operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation, without impairing the essence of the disclosed embodiments.

[0117] Figure 11 shows an exemplary scenario for extracting a second mapping according to at least one embodiment described herein. Figure 11 is described in relation to elements of Figures 1, 2, 3, 4A, 4B, 5, 6A, 6B, 7, 8, 9, and 10. Referring to Figure 11, an exemplary scenario 1100 is shown. Scenario 1100 may include extracted second mappings 1102, 1104, 1106, and 1108. The electronic device 102 in Figure 1 or the processor 204 in Figure 2 can extract the second mappings 1102, 1104, 1106, and 1108 as described herein.

[0118] A second mapping 1102 can be created using the "LabelEncoder" function, which allows mapping features in scenarios where the feature names remain the same. For example, in Figure 11, it can be seen that the name of feature "A" remains unchanged. The second mapping 1102 can be a one-to-one mapping.

[0119] The second mapping 1104 can map feature "A" to features "A1" and "A2". It can be seen that feature "A" can be split into two columns. By calling the "get_dummies" function, the first column can be renamed to "A1" and the second column can be renamed to "A2". The second mapping 1104 can be a one-to-many mapping.

[0120] Similarly, the second mapping 1106 can map feature "B" to features "B1" and "B2". Here, feature "B" can be split into two columns. By calling the "get_dummies" function, the first column can be renamed "B1" and the second column can be renamed "B2". The second mapping 1106 can be a one-to-many mapping.

[0121] The second mapping 1108 can map feature "A" and feature "B" to feature "A" through a feature aggregation function. Here, the second mapping 1108 can be a many-to-one mapping.

[0122] Please note that Scenario 1100 in Figure 11 is for illustrative purposes only and should not be interpreted as limiting the scope of disclosure.

[0123] Traditional methods of ML pipeline generation can require considerable manual effort and time from expert users such as data scientists. Human data scientists often discard certain features from the input dataset (i.e., the data frame of the ML project). In recent years, specific techniques have been developed for the automated generation of ML pipelines for datasets. Typically, deleted features can constitute a source of noise for learning about automated ML pipeline generation. However, there may be multiple reasons and techniques for cutting out features. Therefore, simple analysis or heuristics may not be appropriate for identifying discarded features in order to improve automated ML pipeline generation.

[0124] The electronic device 102 of this disclosure can automatically denoise ML projects. The disclosed electronic device 102 can accurately identify data features that may not be used by the ML pipeline of an ML project or that may be effectively discarded, by utilizing dynamic data flow analysis of the ML pipeline. The disclosed electronic device 102 can systematically track the execution of the ML pipeline operation by operation, and can record the addition, removal, or modification of data features in the data frame, thereby clearly identifying features that may be used by the ML pipeline. Furthermore, the disclosed electronic device 102 can label the ML pipeline of an ML project that has discarded features to obtain a cleaner ML pipeline and data frame, which may be used for automated generation of ML pipelines with higher accuracy. Accurate identification of discarded features may further result in the generation of higher quality meta-learning models (such as meta-learning model 102A) from the ML project based on the obtained cleaner ML pipeline and data frame.

[0125] An example experimental setup for this disclosure is shown in Table 1 below. [Table 1]

[0126] For example, referring to Table 1, the experimental setup may include 1094 ML pipelines and 208 datasets in an ML corpus database (e.g., database 108). The meta-learning model 102A in the experimental setup can correspond to a logistic regression model, and there may be five components in the search space associated with the logistic regression model. To validate the training of the meta-learning model 102A, four-fold cross-validation can be used based on the F1 macro evaluation metric. Note that the data provided in Table 1 may be understood only as experimental data and should not be interpreted as limiting this disclosure.

[0127] Table 2 shows example experimental data regarding noise present in the ML pipeline within the ML corpus. [Table 2]

[0128] Table 2 shows that, of the 1,094 ML pipelines in total, 60 percent of them may have one or more features discarded. For 40 percent of the ML pipelines, all of the multiple features may be used. Based on experiments, it was found that the macro F1 score of the skeleton predictor improved by an average of 5% by including one or more discarded features in the training of the meta-learning model 102A, with the maximum improvement being 9% for the text processing component. Therefore, the disclosed electronic device 102 may provide a significant improvement in the existing technology space of machine learning, including, but not limited to, the training of meta-learning models. The significant improvement may be provided by efficiently identifying discarded features associated with the ML project and using relevant features from the identified discarded features to train the meta-learning model.

[0129] Please note that the data provided in Table 2 should be understood solely as experimental data and should not be interpreted as limiting this disclosure.

[0130] Various embodiments of this disclosure can provide one or more non-temporary computer-readable storage media configured to store instructions causing a system (such as an exemplary electronic device 102) to perform an action in response to being executed. The action may include receiving a machine learning (ML) project, the ML project including a data frame and an ML pipeline which may include a plurality of code statements associated with a plurality of features corresponding to the received ML project. The action may further include determining one or more atomic steps corresponding to the ML pipeline to determine an atomicized ML pipeline. The action may further include instrumenting the atomicized ML pipeline to determine an instrumented ML pipeline which includes one or more operations corresponding to the received ML project. The action may further include executing the instrumented ML pipeline and capturing one or more data frame snapshots based on each of the one or more operations. The action may further include constructing a feature origin graph (FPG) based on the ML pipeline and the captured one or more data frame snapshots. The action may further include identifying one or more discarded features from the plurality of features corresponding to the received ML project based on the constructed FPG.

[0131] When used in this disclosure, the terms “module” or “component” may refer to a specific hardware implementation configured to perform the actions of a module or component, and / or a software object or software routine that can be stored and / or executed on general-purpose hardware of a computing system (e.g., computer-readable media, processing devices, etc.). In some embodiments, different components, modules, engines, and services described in this disclosure may be implemented as objects or processes that run on a computing system (e.g., as separate threads). While some of the systems and methods described in this disclosure are generally described as being implemented in software (stored and / or executed on general-purpose hardware), specific hardware implementations, or combinations of software and specific hardware implementations, are also possible and intended. In this description, “computation entity” may be any computing system as previously defined in this disclosure, or any combination of modules or modulates that operate on a computing system.

[0132] In this disclosure, the terms used in particular in the claims set forth in the appendix (for example, in the text of the claims set forth in the appendix) are generally intended to be "open" terms (for example, the term "contains" should be interpreted as "contains but not limited to," the term "has" should be interpreted as "has at least," and the term "includes" should be interpreted as "contains but not limited to," etc.).

[0133] Furthermore, if a specific number of introduced claim descriptions are intended, such intention is explicitly stated in the claim; if such statement is absent, such intention does not exist. For example, for the sake of understanding, the claims listed below may include the use of the introductory phrases “at least one” and “one or more” to introduce claim descriptions. However, the use of such phrases should not be seen as implicitly indicating that the introduction of a claim description by the indefinite article “a” or “an” limits any particular claim containing such introduced claim descriptions to embodiments containing only one such description, even when the same claim includes the introductory phrase “one or more” or “at least one” and an indefinite article such as “a” (for example, “a” and / or “an” should be interpreted as meaning “at least one” or “one or more”). The same applies to the use of definite articles used to introduce claim descriptions.

[0134] Furthermore, even when a claim description introducing a specific number is explicitly stated, a person skilled in the art will recognize that such description should be interpreted to mean at least the number stated (for example, the statement “two descriptions” without other modifiers means at least two descriptions, or two or more descriptions). Moreover, in examples where a similar provision is used, such as “at least one of A, B, and C, etc.” or “one or more of A, B, and C, etc.”, such structure is generally intended to include A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, etc.

[0135] Furthermore, any separate word or phrase presenting two or more alternative terms, whether in the specification, claims, or drawings, should be understood to include the possibility of containing one of the terms, either one of the terms, or both. For example, the phrase "A or B" should be understood to include the possibility of "A" or "B" or "A and B".

[0136] All examples and conditional language described herein are intended for educational purposes to assist readers in understanding the disclosure and the concepts to which the inventors have contributed to advancing the art, and should be considered as not being limited to the examples and conditions described herein. While embodiments of this disclosure have been described in detail, various changes, substitutions, and modifications can be made to them without departing from the spirit and scope of this disclosure.

[0137] The following note is left regarding the above embodiment. (Note 1) A method executed by a processor, A step of receiving an ML project which includes a data frame and an ML pipeline, wherein the ML pipeline includes multiple code statements associated with multiple features corresponding to the ML project, A step of determining one or more atomic steps corresponding to the ML pipeline and determining the atomicized ML pipeline, The steps include determining an instrumented ML pipeline that instrumentes the atomicized ML pipeline and includes one or more operations corresponding to the received ML project, A step of running an instrumented ML pipeline to capture one or more dataframe snapshots based on each of one or more operations, The steps include constructing a feature origin graph (FPG) based on the ML pipeline and one or more captured data frame snapshots, Based on the constructed FPG, the steps include identifying one or more discarded features from the multiple features corresponding to the received ML project, A method that includes this. (Note 2) The determination of the one or more atomic steps corresponding to the ML pipeline is the method described in Appendix 1, based on the application of a source code lifting technique to the plurality of code statements. (Note 3) The instrumentation of the atomicized ML pipeline is as described in Appendix 1, based on the application of a method call injection technique to the atomicized ML pipeline. (Note 4) The method according to Appendix 1, wherein each of the one or more captured data frame snapshots includes at least one of a row number, inputs and outputs for each variable, and a set of feature names associated with the data frame type. (Note 5) The steps include initializing the feature origin graph (FPG) as a directed acyclic graph containing one node for each feature associated with one or more captured data frame snapshots, The steps include: extracting the abstract syntax tree (AST) associated with the atomicized ML pipeline; The step is to select a first operation from the one or more operations described above and analyze the data frame snapshot from the one or more captured data frame snapshots, The data frame snapshot corresponds to the selected first operation and includes a step and an output information, The steps include adding a layer of nodes to the FPG for each feature associated with the output information of the data frame snapshot, A step of adding directed edges in the FPG from a first node associated with the input information to a second node associated with the output information, based on the correspondence between the first name of the first feature associated with the first node and the second name of the second feature associated with the second node, The method described in Appendix 1, further including the method described in Appendix 1. (Note 6) Based on the extracted AST, the steps include identifying one or more features and operation names associated with the selected first operation, The steps include determining whether the aforementioned operation name corresponds to a delete operation, Based on the determination that the operation name corresponds to the deletion operation, the steps include adding a null trail to the FPG for each node in the node layer associated with the input information of the data frame snapshot, The steps include determining whether the operation name corresponds to a rename operation, The steps include adding a directed edge in the FPG from the node associated with the input information of the data frame snapshot to the corresponding renamed successor node associated with the output information of the data frame snapshot, based on the determination that the operation name corresponds to the rename operation, A step of adding a directed edge in the FPG from each node in the node layer associated with the input information of the data frame snapshot to the corresponding subsequent node associated with the output information of the data frame snapshot, based on the determination that the operation name is other than the rename operation or the delete operation. The method described in Appendix 5, which further includes the method described in Appendix 5. (Note 7) The steps include extracting the constructed feature origin graph (FPG), The steps include selecting a node from the data frame snapshot in the extracted FPG, The steps include performing a depth-first search of the extracted FPG by marking each visited node in the extracted FPG from the selected node, The step is to determine one or more unmarked nodes in the retrieved FPG based on the depth-first search performed above, Steps include: identifying the one or more discarded features based on the one or more unmarked nodes determined; The method described in Appendix 1, further including the method described in Appendix 1. (Note 8) The depth-first search is the method described in Appendix 7, which corresponds to at least one of forward depth-first search or reverse depth-first search. (Note 9) The steps include constructing an abstract syntax tree (AST) of the atomicized ML pipeline, The steps include traversing the constructed AST of the atomicized ML pipeline to identify a rename operation, The steps include determining the parameter map associated with the identified rename operation, The step is to extract a first mapping between the multiple features corresponding to the received ML project and a first set of features associated with the ML pipeline, based on the determined parameter map. The one or more discarded features are identified based on the extracted first mapping, in the step of The method described in Appendix 1, further including the method described in Appendix 1. (Note 10) The method according to Appendix 9, wherein the set of the first features corresponds to one or more features that are renamed based on user input. (Note 11) A step of traversing each of the one or more atomic steps corresponding to the ML pipeline to track updates to the names associated with each of the second set of features associated with the ML pipeline, The step is to extract a second mapping between the multiple features corresponding to the received ML project and the second set of features, The one or more discarded features are identified based on the extracted second mapping, step, The method described in Appendix 1, further including the method described in Appendix 1. (Note 12) The method described in Appendix 11, wherein the set of the second set of features corresponds to one or more features that are renamed based on a set of predefined rules. (Note 13) The method according to Appendix 11, wherein the aforementioned set of features corresponds to the initial data frame of the captured data frame snapshot, and the second set of features corresponds to the final data frame of the captured data frame snapshot. (Note 14) The steps include determining one or more relevant features from the plurality of features corresponding to the received ML project based on the identified discarded features, A step of training a meta-learning model associated with the ML project based on one or more relevant features determined above, The method described in Appendix 1, further including the method described in Appendix 1. (Note 15) One or more non-temporary computer-readable storage media configured to store instructions that cause an electronic device to perform an action in response to being executed, wherein the action is: The process involves receiving an ML project that includes a data frame and an ML pipeline, wherein the ML pipeline includes multiple code statements associated with multiple features corresponding to the ML project. Determining one or more atomic steps corresponding to the aforementioned ML pipeline to determine the atomicized ML pipeline, Instrumenting the atomicized ML pipeline to determine an instrumented ML pipeline that includes one or more operations corresponding to the received ML project, This involves running an instrumented ML pipeline to capture one or more dataframe snapshots based on each of one or more operations, The ML pipeline and the captured data frame snapshots are used to construct a feature origin graph (FPG), Based on the constructed FPG, one or more discarded features are identified from the multiple features corresponding to the received ML project. One or more non-temporary computer-readable storage media, including [the specified text]. (Note 16) The aforementioned operation is, Initializing the feature origin graph (FPG) as a directed acyclic graph containing one node for each feature associated with one or more captured data frame snapshots, Extracting the Abstract Syntax Tree (AST) associated with the atomicized ML pipeline, Selecting a first operation from the one or more operations described above to analyze the data frame snapshot from the one or more captured data frame snapshots, The aforementioned data frame snapshot corresponds to the selected first operation and includes input and output information. For each feature associated with the output information of the data frame snapshot, a layer of nodes is added to the FPG, A directed edge in the FPG is added from the first node associated with the input information to the second node associated with the output information, based on the correspondence between the first name of the first feature associated with the first node and the second name of the second feature associated with the second node. One or more non-temporary computer-readable storage media as described in Appendix 15, further including the above. (Note 17) The aforementioned operation is, Based on the extracted AST, identify one or more features and operation names associated with the selected first operation, To determine whether the aforementioned operation name corresponds to a delete operation, Based on the determination that the operation name corresponds to the deletion operation, a null trail is added to the FPG for each node in the node layer associated with the input information of the data frame snapshot, To determine whether the aforementioned operation name corresponds to a rename operation, Based on the determination that the operation name corresponds to the rename operation, a directed edge is added in the FPG from the node associated with the input information of the data frame snapshot to the corresponding renamed successor node associated with the output information of the data frame snapshot, Based on the determination that the operation name is other than the rename operation or the delete operation, a directed edge is added in the FPG from each node in the layer of nodes associated with the input information of the data frame snapshot to the corresponding subsequent node associated with the output information of the data frame snapshot, One or more non-temporary computer-readable storage media as described in Appendix 16, further including the following: (Note 18) The aforementioned operation is, Extracting the aforementioned constructed feature origin graph (FPG), Selecting a node from the data frame snapshot in the extracted FPG, The depth-first search of the extracted FPG is performed by marking each visited node in the extracted FPG from the selected node, Based on the depth-first search performed, determine one or more unmarked nodes in the retrieved FPG. The one or more discarded features are identified based on the one or more unmarked nodes that have been determined, One or more non-temporary computer-readable storage media as described in Appendix 15, further including the above. (Note 19) The aforementioned operation is, Based on the identified one or more discarded features, determine one or more relevant features from the multiple features corresponding to the received ML project, Training a meta-learning model associated with the ML project based on one or more relevant features determined above, One or more non-temporary computer-readable storage media as described in Appendix 15, further including the above. (Note 20) It is an electronic device, Memory for storing instructions, The system comprises a processor coupled to the memory and executing the stored instructions to perform processing, wherein the processing is performed The process involves receiving an ML project that includes a data frame and an ML pipeline, wherein the ML pipeline includes multiple code statements associated with multiple features corresponding to the ML project. Determining one or more atomic steps corresponding to the aforementioned ML pipeline to determine the atomicized ML pipeline, Instrumenting the atomicized ML pipeline to determine an instrumented ML pipeline that includes one or more operations corresponding to the received ML project, This involves running an instrumented ML pipeline to capture one or more dataframe snapshots based on each of one or more operations, The ML pipeline and the captured data frame snapshots are used to construct a feature origin graph (FPG), Based on the constructed FPG, one or more discarded features are identified from the multiple features corresponding to the received ML project. Electronic devices, including those mentioned above.

Claims

1. A method executed by a processor, A step of receiving an ML project including a data frame and an ML pipeline, wherein the ML pipeline includes multiple code statements associated with multiple features corresponding to the ML project. A step of determining one or more atomic steps corresponding to the ML pipeline and determining the atomicized ML pipeline, The steps include determining an instrumented ML pipeline that instrumentes the atomicized ML pipeline and includes one or more operations corresponding to the received ML project, A step of running an instrumented ML pipeline and capturing one or more dataframe snapshots based on each of one or more operations, The steps include constructing a feature origin graph (FPG) based on the ML pipeline and one or more captured data frame snapshots, Based on the constructed FPG, the step of identifying one or more discarded features from the plurality of features corresponding to the received ML project, A method that includes this.

2. The method according to claim 1, wherein the determination of the one or more atomic steps corresponding to the ML pipeline is based on the application of a source code lifting technique to the plurality of code statements.

3. The method according to claim 1, wherein the instrumentation of the atomicized ML pipeline is based on the application of a method call injection technique to the atomicized ML pipeline.

4. The method according to claim 1, wherein each of the one or more captured data frame snapshots includes at least one of a row number, inputs and outputs for each variable, and a set of feature names associated with a data frame type.

5. The steps include initializing the feature origin graph (FPG) as a directed acyclic graph containing one node for each feature associated with one or more captured data frame snapshots, The steps include: extracting the abstract syntax tree (AST) associated with the atomicized ML pipeline; The step is to select a first operation from the one or more operations described above and analyze the data frame snapshot from the one or more captured data frame snapshots, The data frame snapshot corresponds to the selected first operation and includes a step and an output information, The steps include adding a layer of nodes to the FPGA for each feature associated with the output information of the data frame snapshot, A step of adding directed edges in the FPGA from a first node associated with the input information to a second node associated with the output information, based on the correspondence between a first name of a first feature associated with the first node and a second name of a second feature associated with the second node, The method according to claim 1, further comprising:

6. Based on the retrieved AST, the step of identifying one or more features and operation names associated with the selected first operation, The steps include determining whether the aforementioned operation name corresponds to a delete operation, Based on the determination that the operation name corresponds to the deletion operation, the steps include adding a null trail to the FPG for each node in the node layer associated with the input information of the data frame snapshot, The steps include determining whether the operation name corresponds to a rename operation, The steps include adding a directed edge in the FPGA from the node associated with the input information of the data frame snapshot to the corresponding renamed successor node associated with the output information of the data frame snapshot, based on the determination that the operation name corresponds to the rename operation, Based on the determination that the operation name is other than the rename operation or the delete operation, the steps of adding a directed edge in the FPGA from each node in the node layer associated with the input information of the data frame snapshot to the corresponding subsequent node associated with the output information of the data frame snapshot, The method according to claim 5, further comprising:

7. The steps include: extracting the constructed feature origin graph (FPG), The steps include selecting a node from the data frame snapshot in the extracted FPGA, The steps include performing a depth-first search of the extracted FPGA by marking each visited node in the extracted FPGA from the selected node, The step is to determine one or more unmarked nodes in the retrieved FPG based on the depth-first search performed above, Steps include: identifying the one or more discarded features based on the one or more unmarked nodes determined; The method according to claim 1, further comprising:

8. The method according to claim 7, wherein the depth-first search corresponds to at least one of forward depth-first search or reverse depth-first search.

9. The steps include constructing an abstract syntax tree (AST) of the atomicized ML pipeline, The steps include traversing the constructed AST of the atomicized ML pipeline to identify a rename operation, The steps include determining the parameter map associated with the identified rename operation, The step is to extract a first mapping between the plurality of features corresponding to the received ML project and a first set of features associated with the ML pipeline, based on the parameter map determined above. The one or more discarded features are identified based on the extracted first mapping, in the step of The method according to claim 1, further comprising:

10. The method according to claim 9, wherein the set of first features corresponds to one or more features that are renamed based on user input.

11. A step of traversing each of the one or more atomic steps corresponding to the ML pipeline to track updates to the names associated with each of the second set of features associated with the ML pipeline, The step is to extract a second mapping between the plurality of features corresponding to the received ML project and the second set of features, The one or more discarded features are identified based on the extracted second mapping, in the step of The method according to claim 1, further comprising:

12. The method according to claim 11, wherein the second set of features corresponds to one or more features that are renamed based on a set of predefined rules.

13. The method according to claim 11, wherein the plurality of features correspond to the initial data frame of the captured data frame snapshot, and the second set of features corresponds to the final data frame of the captured data frame snapshot.

14. The steps include determining one or more relevant features from the plurality of features corresponding to the received ML project based on the identified discarded features, A step of training a meta-learning model associated with the ML project based on one or more relevant features determined above, The method according to claim 1, further comprising:

15. A computer program that causes an electronic device to perform an action, wherein the action is: The process involves receiving an ML project that includes a data frame and an ML pipeline, wherein the ML pipeline includes multiple code statements associated with multiple features corresponding to the ML project. Determining one or more atomic steps corresponding to the aforementioned ML pipeline and determining the atomicized ML pipeline, Instrumenting the atomicized ML pipeline to determine an instrumented ML pipeline that includes one or more operations corresponding to the received ML project, Execute an instrumented ML pipeline to capture one or more dataframe snapshots based on each of one or more operations, Constructing a feature origin graph (FPG) based on the ML pipeline and one or more captured data frame snapshots, Based on the constructed FPG, one or more discarded features are identified from the multiple features corresponding to the received ML project. A computer program that includes [this].

16. The aforementioned operation is, The feature origin graph (FPG) is initialized as a directed acyclic graph containing one node for each feature associated with one or more captured data frame snapshots, Extracting the Abstract Syntax Tree (AST) associated with the atomicized ML pipeline, The first operation from the one or more operations described above is selected to analyze the data frame snapshot from the one or more captured data frame snapshots, The aforementioned data frame snapshot corresponds to the selected first operation and includes input and output information. For each feature associated with the output information of the aforementioned data frame snapshot, a node layer is added to the FPGA, A directed edge in the FPGA is added from the first node associated with the input information to the second node associated with the output information, based on the correspondence between the first name of the first feature associated with the first node and the second name of the second feature associated with the second node. The computer program according to claim 15, further comprising:

17. The aforementioned operation is, Based on the retrieved AST, one or more features and operation names associated with the selected first operation are identified. To determine whether the aforementioned operation name corresponds to a delete operation, Based on the determination that the operation name corresponds to the deletion operation, a null trail is added to the FPG for each node in the node layer associated with the input information of the data frame snapshot, To determine whether the aforementioned operation name corresponds to a rename operation, Based on the determination that the operation name corresponds to the rename operation, a directed edge is added in the FPGA from the node associated with the input information of the data frame snapshot to the corresponding renamed successor node associated with the output information of the data frame snapshot, Based on the determination that the operation name is other than the rename operation or the delete operation, a directed edge is added in the FPGA from each node in the node layer associated with the input information of the data frame snapshot to the corresponding subsequent node associated with the output information of the data frame snapshot, The computer program according to claim 16, further comprising:

18. The aforementioned operation is, Extracting the aforementioned constructed feature origin graph (FPG), Selecting a node from the data frame snapshot in the extracted FPGA, The depth-first search of the extracted FPGA is performed by marking each visited node in the extracted FPGA from the selected node, Based on the depth-first search performed, determine one or more unmarked nodes in the retrieved FPGA. The one or more discarded features are identified based on the one or more unmarked nodes determined, The computer program according to claim 15, further comprising:

19. The aforementioned operation is, Based on the identified one or more discarded features, determine one or more relevant features from the multiple features corresponding to the received ML project, Based on one or more relevant features determined above, train a meta-learning model associated with the ML project, The computer program according to claim 15, further comprising:

20. It is an electronic device, Memory for storing instructions, The system comprises a processor coupled to the memory and executing the stored instructions to perform processing, wherein the processing is performed The process involves receiving an ML project that includes a data frame and an ML pipeline, wherein the ML pipeline includes multiple code statements associated with multiple features corresponding to the ML project. Determining one or more atomic steps corresponding to the aforementioned ML pipeline and determining the atomicized ML pipeline, Instrumenting the atomicized ML pipeline to determine an instrumented ML pipeline that includes one or more operations corresponding to the received ML project, Execute an instrumented ML pipeline to capture one or more dataframe snapshots based on each of one or more operations, Constructing a feature origin graph (FPG) based on the ML pipeline and one or more captured data frame snapshots, Based on the constructed FPG, one or more discarded features are identified from the multiple features corresponding to the received ML project. Electronic devices, including those mentioned above.

Images