Industry-Specific Machine Learning Applications

Abstract

The machine learning application is selected from a plurality of machine learning applications. Each machine learning application corresponds to a different industrial problem and includes standard features and a machine learning pipeline specific to the corresponding industrial problem. The machine learning application receives a dataset for generating a model for making predictions for the industrial problem corresponding to the selected machine learning application. The standard features are provided for display to a user to map variables in the dataset to the standard features. The user mapping is received through a user interface. The machine learning pipeline is applied to the dataset to train a plurality of models based at least on the mapping. The trained models are ranked and one of the trained models is selected based on the ranking. The selected trained model is used to make predictions based on new data.

Description

[Technical Field]

[0001] The described embodiments relate generally to processing data streams, and more particularly to training models for making predictions based on data streams using industry-specific machine learning applications. [Background technology]

[0002] This application claims the benefit of U.S. Patent Application No. 17 / 242,927, filed April 28, 2021, which is incorporated herein by reference.

[0003] Automated machine learning tools automate the process of applying machine learning to real-world problems. Current automated machine learning tools enable faster and more efficient creation of deployable machine learning models. However, automated machine learning tools often produce suboptimal models because they do not incorporate domain knowledge about the dataset. As a result, the models generated by currently available automated machine learning tools are not good enough to make predictions based on the data.

[0004] The above and other problems are addressed by a method, a non-transitory computer-readable memory, and a system. One embodiment of the method is for generating a model for making predictions on an industrial problem. The method includes receiving, by a machine learning application, a dataset for generating the model. The machine learning application is selected from a plurality of machine learning applications based on the industrial problem. Each of the plurality of machine learning applications corresponds to a different industrial problem and includes standard features specific to the corresponding industrial problem and a machine learning pipeline specific to the corresponding industrial problem. The method further includes providing the standard features in the machine learning application for display to a client device associated with a user. The method further includes receiving, from the client device and in response to providing the standard features, a mapping of variables in the dataset to the standard features of the selected machine learning application. The method further includes applying the machine learning pipeline of the selected machine learning application to the dataset to train a plurality of models based at least on the mapping. The method further includes ranking the plurality of trained models. The method also includes selecting a generated model from the plurality of trained models based on the ranking.

[0005] An embodiment of a non-transitory computer-readable memory stores executable computer program instructions. The instructions are executable to perform operations for generating a model for making predictions on an industrial problem. The operations include receiving, by a machine learning application, a dataset for generating the model. The machine learning application is selected from a plurality of machine learning applications based on the industrial problem. Each of the plurality of machine learning applications corresponds to a different industrial problem and includes standard features specific to the corresponding industrial problem and a machine learning pipeline specific to the corresponding industrial problem. The operations further include providing the standard features in the machine learning application for display to a client device associated with a user. The operations further include receiving, from the client device and in response to providing the standard features, a mapping of variables in the dataset to the standard features in the selected machine learning application. The operations further include applying the machine learning pipeline of the selected machine learning application to the dataset to train a plurality of models based at least on the mapping. The operations further include ranking the plurality of trained models. The operations also include selecting a generated model from the plurality of trained models based on the ranking.

[0006] One embodiment of the system includes a computer processor for executing computer program instructions. The system also includes a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations for generating a model for making predictions on an industrial problem. The operations include receiving a dataset for generating the model by a machine learning application. The machine learning application is selected from a plurality of machine learning applications based on the industrial problem. Each of the plurality of machine learning applications corresponds to a different industrial problem and includes standard features specific to the corresponding industrial problem and a machine learning pipeline specific to the corresponding industrial problem. The operations further include providing the standard features of the machine learning application for display to a client device associated with a user. The operations further include receiving, from the client device and in response to providing the standard features, a mapping of variables in the dataset to the standard features in the selected machine learning application. The operations further include applying the machine learning pipeline of the selected machine learning application to the dataset to train a plurality of models based at least on the mapping. The operations further include ranking the plurality of trained models. The operations also include selecting a generated model from the plurality of trained models based on the ranking. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram illustrating a machine learning environment including a machine learning server according to one embodiment. [Figure 2] FIG. 2 is a block diagram illustrating an application generation engine for generating industry-specific machine learning applications according to one embodiment. [Figure 3] FIG. 3 is a block diagram illustrating an industry-specific machine learning application according to one embodiment. [Figure 4A]FIG. 4A is a diagram illustrating training a model from a dataset using the industry-specific machine learning application of FIG. 3 according to one embodiment. [Figure 4B] FIG. 4B is a diagram illustrating training a model from a dataset using the industry-specific machine learning application of FIG. 3 according to one embodiment. [Figure 4C] FIG. 4C illustrates training a model from a dataset using the industry-specific machine learning application of FIG. 3 according to one embodiment. [Figure 5] FIG. 5 is a flowchart illustrating a method for training a model using an industry-specific machine learning application according to one embodiment. [Figure 6] FIG. 6 is a high-level block diagram illustrating a functional view of a typical computer system for use as the machine learning server of FIG. 1 according to one embodiment.

[0008] The figures depict various embodiments for purposes of illustration only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods shown herein may be used without departing from the principles of the embodiments described herein. Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE INVENTION

[0009] 1 is a block diagram illustrating a machine learning environment 100 including a machine learning server 110 according to one embodiment. The environment 100 further includes multiple data sources 120 and client devices 130 connected to the machine learning server 110 via a network 140. Although the illustrated environment 100 includes only one machine learning server 110 coupled to multiple data sources 120 and client devices 130, embodiments can have multiple machine learning servers, a single data source, and a single client device, or other variations thereof.

[0010] The machine learning server 110 is a computer-based system utilized to build machine learning models and deploy the models to make predictions based on data. Data is collected, aggregated, or otherwise accessed via a network 140 from one or more of a plurality of data sources 120 or one or more of a plurality of client devices 130. The machine learning server 110 may implement scalable software tools and hardware resources used in accessing, preparing, blending, and analyzing data from a wide variety of data sources 120 or client devices 130.

[0011] The machine learning server 110 implements industry-specific machine learning processes. engine 150, and Machine Learning Server 110 The system includes a plurality of industry-specific machine learning applications 160 (also referred to as “machine learning applications 160,” and individually, as “industry-specific machine learning application 160” or “machine learning application 160”) generated by the system. An industry-specific machine learning application 160 is an application that can be used to train a model to make predictions within a specific industry problem. An industry problem is a problem in an industry or business domain. An industry / domain can be, for example, information technology (IT) operations, healthcare, industrial manufacturing, retail, sales and marketing, insurance, banking, etc. An industry problem can be, for example, application monitoring, service level agreement violation detection, user behavior prediction, etc.

[0012] The industry-problem-specific machine learning application 160 includes machine learning tools (e.g., labeling functions, standard features, machine learning pipelines, etc.) generated by the machine learning server 110 for a particular industry. Such machine learning tools may be generated and / or selected based on domain knowledge of the industry problem, knowledge of past training of models associated with the industry problem, other types of knowledge about the industry problem, or a combination thereof. These industry-specific machine learning tools make the machine learning process more efficient compared to traditional machine learning techniques. For example, standard features can be used as features for training a model (e.g., by simply mapping variables in training data to standard features), which can save the time and computational resources required to extract these features from a dataset. As another example, the exploration and optimization of pipelines used in the machine learning process can be limited to the pipelines of selected machine learning applications, making the search and optimization more efficient compared to traditional machine learning processes. The machine learning tools allow the machine learning application 160 to perform automated industry-specific machine learning.

[0013] In some embodiments, the industry-specific machine learning application 160 may allow a user to provide input to the machine learning process. For example, it may allow a user to map variables in a training dataset to standard features. It may also allow a user to define values ​​for specific parameters in a labeling function to customize the labeling function for the specific prediction sought by the user. In this way, the industry-specific machine learning application 160 leverages both domain knowledge of the industry problem and the user's special knowledge of the dataset and the specific prediction. Therefore, compared to traditional machine learning techniques, the industry-specific machine learning application 160 is able to train models that better fit the needs of the industry and the user.

[0014] In some embodiments, the machine learning server 110 provides a variety of industry-specific machine learning applications 160 for display on a client device associated with a user. The machine learning server 110 allows the user to select one of the industry-specific machine learning applications 160 for training a machine learning model. The user is a person (e.g., a machine learning engineer, a development engineer, etc.) who has knowledge associated with the machine learning model to be trained, such as the predictions made by the model, the data used to train the model, and the data used to make the predictions. The user selects a machine learning application 160 specific to an industry problem related to the predictions made by the model, e.g., predictions that fall within the scope of the industry problem.

[0015] In some embodiments, the machine learning server 110 presents the machine learning applications 160 in a user interface. The machine learning applications 160 may be associated with labels that indicate the industry problem corresponding to the machine learning applications 160, so that a user can rely on the labels to determine whether the machine learning applications are suitable for training a model that the user needs. In some embodiments, the machine learning server 110 supports one or more user interfaces, e.g., graphical user interfaces (GUIs), that allow a user to interact with the machine learning applications. For example, the user interfaces provide the user with options to view the machine learning applications, download the machine learning applications, interact with online versions of the machine learning applications, upload datasets to the machine learning applications, map variables in the datasets to standard features of the machine learning applications, etc.

[0016] Data sources 120 provide electronic data to machine learning server 110. Data sources 120 may be storage devices such as hard disk drives (HDDs) or solid-state drives (SSDs), computers that manage and provide access to multiple storage devices, storage area networks (SANs), databases, or cloud storage systems. Data sources 120 may also be computer systems that can obtain data from other sources. Data sources 120 may be remote from machine learning server 110 and provide data over network 140. Additionally, some or all of data sources 120 may be directly coupled to and provide data to the data analysis system without passing the data over network 140.

[0017] The data provided by data source 120 includes data used to train machine learning models to solve industry problems and / or to input into models trained to make predictions within industry problems. The data may be organized into data records (e.g., rows). Each data includes one or more values. For example, a data record provided by data source 120 may include a series of comma-separated values. The data may include: Machine Learning Server The data from data sources 120 describes information related to businesses using 110. For example, data from data sources 120 may describe computer-based interactions with content accessible on a website and / or in an application (e.g., click tracking data). As another example, data from data sources 120 may describe customer transactions online and / or in stores. The businesses may be in one or more of a variety of industries, such as computer technology, manufacturing, etc.

[0018] The client device 130 is one or more computing devices capable of receiving user input and transmitting and / or receiving data over the network 140. In one embodiment, the client device 130 is a conventional computer system such as a desktop or laptop computer. Alternatively, the client device 130 may be a device with computer capabilities, such as a personal digital assistant (PDA), a mobile phone, a smartphone, or another suitable device. The client device 130 is configured to communicate with one or more data sources 120 and the machine learning server 110 over the network 140. In one embodiment, the client device 130 executes an application that allows a user of the client device 130 to interact with the machine learning server 110. For example, the client device 130 executes an application that allows interaction between the client device 130 and the machine learning application 160 over the network 140, for example, by executing a GUI supported by the machine learning server 110. The client device 130 includes or is otherwise associated with a display device that displays the GUI. The client device 130 is also associated with an input device, e.g., a keyboard, a mouse, etc., that allows a user to interact with the GUI, such as by providing input to the GUI. In another embodiment, the client device 130 interacts with the machine learning server 110 through an application programming interface (API) running on the client device's 130 native operating system, such as IOS® or ANDROID™. The client device 130 may interact with one or more data sources 120 to send data to or retrieve data from the data sources 120.

[0019] Network 140 represents a communication path between machine learning server 110 and data source 120. In one embodiment, network 140 is the Internet and uses standard communication technologies and / or protocols. Thus, network 140 may include links using technologies such as Ethernet, 802.11, Worldwide Interoperability for Microwave Access (WiMAX), 3G, Long Term Evolution (LTE), Digital Subscriber Line (DSL), Asynchronous Transfer Mode (ATM), InfiniBand, PCI Express Advanced Switching, etc. Similarly, networking protocols used in network 140 may include Multiprotocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transport Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP), etc.

[0020] Data exchanged over network 140 may be represented using technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc. Additionally, all or portions of the links may be encrypted using conventional encryption technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. In alternative embodiments, the entities may use custom and / or proprietary data communication technologies instead of or in addition to those described above.

[0021] FIG. 2 illustrates an application generation method for generating an industry-specific machine learning application according to one embodiment. engine2 is a block diagram illustrating application generation engine 200. Application generation engine 200 is one embodiment of application generation engine 150 of FIG. 1. Application generation engine 200 generates industry-specific machine learning applications that can be used to train models. Application generation engine 200 includes labeling function module 210, standard feature module 220, pipeline module 230, user interface module 240, and database 250. Those skilled in the art will recognize that other embodiments may have different and / or other components than those described herein, and that functionality may be distributed among the components in different ways.

[0022] The labeling function module 210 obtains a labeling function specific to the industry problem. The labeling function is a function that, when applied to a dataset, creates a label time from the dataset. The label time may be provided in a table (also called a "label time table"). The label time includes a cutoff time and a label associated with the cutoff time. The cutoff time is the time at which a prediction is made. Data associated with timestamps before the cutoff time can be used to extract label features. However, data associated with timestamps after the cutoff time should not be used to extract label features. The label associated with the cutoff time is a past example (e.g., true or false) of the prediction target associated with the cutoff time. Labels are generated based on data associated with timestamps on and / or beyond the cutoff time by using the labeling function. For example, for predictions about user behavior on a specific day, such as the first day of each month, the cutoff time is the first day of each month. Data associated with the first day of each month timestamp is applied to the labeling function to generate labels, but cannot be used to generate features. All features should be generated using data before the cutoff time, such as data from the previous month.

[0023] The labeling function includes customizable parameters. Examples of parameters include the prediction date / time (i.e., cutoff date / time), the prediction window (the period for making the prediction), the number of days or months (the future period to be predicted), etc. In some embodiments, the parameter values ​​are customized, for example, by a user with domain knowledge of the prediction and / or industry problem, to create label times for various predictions within the scope of the industry problem. For example, to make a prediction about user behavior on the first day of each month, the prediction date can be the first day of each month and the prediction window can be January.

[0024] The standard feature module 220 generates industry-specific standard features. For example, for each machine learning application, the standard feature module 220 generates one or more standard features specific to the industry domain of the machine learning application, e.g., based on knowledge associated with the industry problem domain. In some embodiments, the standard feature module 220 generates standard features based on typical variables in a dataset that a user inputs to a model trained to solve an industry problem. For example, for a machine learning application specific to predicting next purchases, the standard feature module 220 generates standard features associated with the user (e.g., user ID, gender, birth date, zip code) and past transactions (e.g., transaction ID, transaction date, transaction amount, product purchase). In some embodiments, the standard feature module 220 selects standard features from a pool of features. The standard feature module 220 may select standard features from the pool based on the performance of the standard features in past training of models associated with the industry problem.

[0025] In some embodiments, the standard feature module 220 generates standard primitives that are applied to a dataset to generate features. The standard primitives include algorithms that, when applied to the data, perform calculations on the data and generate corresponding standard features with associated values. In one example, the standard primitives are default primitives for the industry domain of the machine learning application. In another example, the standard primitives are selected from a pool of candidate primitives. For example, the candidate primitives are ranked based on the ranking of features generated from the candidate primitives. A candidate primitive that generates a higher-ranked feature (e.g., higher than features generated from other candidate primitives) is selected as the standard primitive. The algorithms of the standard primitives can be used to apply to different datasets with different variables. Therefore, the standard primitives can be reused with different datasets to train different machine learning models in the industry domain. More details about primitive and feature ranking are described below in conjunction with FIG. 3.

[0026] The pipeline module 230 generates one or more pipelines specific to an industry problem. A pipeline is a workflow of a machine learning process executed by a machine learning application to train a model and specifies a series of steps for training the model. A machine learning pipeline may also specify tools (e.g., algorithms) used in the machine learning process, such as tools for data imputation, feature scaling, classification, etc. In one example, pipeline steps include data composing, feature engineering, model training, model validation, and model deployment. Steps may include substeps. For example, a data preparation step may include data type setting, data encoding, and data imputation, a feature engineering step may include feature selection and ranking, and a model training step may include hyperparameter tuning and algorithm selection. Different pipelines include different orders of steps and / or different steps.

[0027] In some embodiments, the pipeline module 230 selects a pipeline from the pool of pipelines based on an objective function. The objective function is a function to be optimized (e.g., minimized versus maximized). It measures how well a predicted goal / target is reached. It can be a loss function or a cost function. The pipeline module 230 may select an objective function from the pool of objective functions based on the industry problem domain. The objective function is domain-specific. The pipeline module 230 applies the objective function to the pool of pipelines to select multiple pipelines. For example, the pipeline module 230 ranks the pool of pipelines based on how well each pipeline optimizes the objective function and selects multiple pipelines from the pool of pipelines based on the ranking.

[0028] In some embodiments, the pipeline module 230 obtains a pipeline template and generates an industry problem-specific pipeline from the pipeline template. Each template includes a set of components. A component is a tool for performing a step in a machine learning process. Examples of components include a data submission tool, a data type setting tool, a data encoding tool, a data imputation tool, a feature selection tool, a feature ranking tool, an algorithm selection tool, etc. A component is associated with one or more parameters. The value of the parameter can be changed or customized. For example, in the case of a feature ranking tool, the parameter of the feature ranking tool is the number of decision trees used to rank features. The value of the parameter can be, for example, 100, 200, 300, etc.

[0029] In some embodiments, the pipeline module 230 determines values ​​for parameters of components in the pipeline template. In one example, the pipeline module 230 uses default values ​​for the industry challenge. In another example, the pipeline module 230 determines values ​​for parameters of components using a machine learning model. The machine learning model is trained to determine values ​​for parameters of components in the machine learning pipeline. For example, the pipeline module 230 inputs relevant information to the machine learning model, and the machine learning model outputs values ​​for parameters of one or more components in the pipeline template. The relevant information may include information about the pipeline template (e.g., information about the components in the pipeline template, information about other components in the pipeline template, etc.), information about the machine learning application, information about the industry challenge, information received from a user of the machine learning application (e.g., expected accuracy of a model trained using the pipeline template, expected time required to train a model using the pipeline template, etc.), etc.

[0030] The pipeline module 230 selects a pipeline from the pipeline templates by ranking the pipeline templates. For example, the pipeline module 230 ranks the pipeline templates based on the accuracy of machine learning models trained using each pipeline template and selects a higher-ranked pipeline template. The pipeline module 230 may rank the pipeline templates before and / or after determining values ​​for parameters of components in the pipeline templates.

[0031] The user interface module 240 generates a user interface (e.g., a graphical user interface (GUI)) for the industry-specific machine learning application. The user interface includes elements used by a user to interact with the machine learning application. Examples of elements include icons, tabs, checkboxes, buttons, drop-down lists, list boxes, radio buttons, switches, or other types of elements that a user can use to select or deselect options, input fields that a user can use to enter numbers, symbols, and / or text, presentation areas for presenting information to a user for user confirmation, etc. More details about the user interface are described below in conjunction with FIG. 3.

[0032] Database 250 stores data associated with application generation engine 200, such as data received, used, or generated by application generation engine 200. In some embodiments, database 250 stores data associated with application generation engine 200, such as data received, used, or generated by application generation engine 200. ring Stores functions, standard features, objective functions, machine learning pipelines, etc.

[0033] FIG. 3 is a block diagram illustrating an industry-specific machine learning application 300 according to one embodiment. The machine learning application 300 is one embodiment of the machine learning application 160 of FIG. 1. The machine learning application 300 includes industry-specific machine learning tools and is used to train models for making predictions within a range of industry problems. The machine learning application 300 includes a user interface module 310, a labeling module 320, a feature engineering module 330, a training module 340, a ranking module 350, and a database 360. Those skilled in the art will recognize that other embodiments may have different and / or other components than those described herein, and that functionality may be distributed among the components in different ways.

[0034] The user interface module 310 supports a user interface (e.g., a GUI) that allows a user to access and interact with the machine learning application 300. For example, the user interface allows a user to load a dataset into the machine learning application, for example, from a client device or a data source. The user interface allows a user to select a portion of the dataset for training a model, for example, by allowing a user to specify a time range before a cutoff time, such that data falling within that time range will be used to train the model.

[0035] The user interface allows a user to provide values ​​for customizable parameters of the labeling function to the machine learning application 300. The values ​​received from the user can be used to customize the machine learning process for the particular prediction sought by the user. In some embodiments, the user interface presents the customizable parameters to the user. The user interface may include one or more input fields for the customizable parameters for the user to enter values ​​for the customizable parameters. The user interface may also provide a drop-down list from which the user can select values ​​for the customizable parameters. The user interface module 310 transmits the values ​​of the customizable parameters received from the user to the labeling module 320 to customize the labeling function.

[0036] The user interface also presents standard features in the machine learning application 300 to the user and allows the user to map variables in the dataset to the standard features. In some embodiments, after receiving the dataset, the user interface module 310 identifies the variables in the dataset. The user interface module 310 presents all or some of the variables in the user interface for display to the user, allowing the user to select variables and map the variables to the standard features. The user interface receives the user's mapping and sends the mapping to the feature engineering module 300. 3 Send to 0.

[0037] The user interface may also allow the user to make other selections that affect the machine learning process, such as editing datasets, selecting data types for variables, defining and / or adjusting hyperparameters, providing other guidance to the machine learning process, or combinations thereof. In some embodiments, the user interface provides a visual representation of the machine learning process, such as, for example, a visual representation of a machine learning pipeline, for presentation to the user.

[0038] The labeling module 320 generates label times from the dataset by applying the labeling function of the machine learning application 300 to the dataset. Each label time includes a label and a cutoff time associated with the label. The label is a past example of a target to be predicted. The label will be used as a target in the supervised machine learning process performed by the training module 340. The cutoff time indicates when to stop using data to make features for the label. In the example where the prediction is whether a customer will churn on the first of each month, the cutoff time is the first of each month, as shown in the label time table. All features for each label should use data prior to this time to prevent data leakage.

[0039] In some embodiments, the labeling module 320 customizes the labeling function based on values ​​of customizable parameters of the labeling function, which may be received from a user through a user interface. The values ​​provided by the user may be specific to the particular prediction within the industry problem for which the model is to be trained. Because the user has knowledge of the particular prediction for which the model is trained, the labeling module 320 incorporates such knowledge of the user so that the trained model is tailored to the particular prediction desired by the user.

[0040] The feature engineering module 330 generates features based on data in the dataset associated with timestamps before the cutoff time. The features may be standard features in the machine learning application 300 that have been mapped to variables in the dataset by a user, for example, through a user interface. Features may also be extracted from the dataset by the feature engineering module 330. For example, the feature engineering module 330 identifies variables in the dataset that have not been mapped to one of the standard features by a user and generates features from the variables.

[0041] To extract features, the feature engineering module 330 may select one or more primitives from a pool of primitives maintained by the machine learning application 300. The pool of primitives includes a large number of primitives, such as hundreds or thousands of primitives. Each primitive includes an algorithm that, when applied to data, performs a calculation on the data and generates a feature having an associated value. A primitive is associated with one or more attributes. A primitive's attribute may be a description of the primitive (e.g., a natural language description specifying the calculation performed by the primitive when applied to data), an input type (i.e., the type of input data), a return type (i.e., the type of output data), primitive metadata indicating how useful the primitive was in previous feature engineering processes, or other attributes.

[0042] In some embodiments, the pool of primitives includes a variety of different types of primitives. One type of primitive is an aggregation primitive. When applied to a dataset, an aggregation primitive identifies related data in the dataset, performs a decision on the related data, and creates a value that summarizes and / or aggregates the decisions. For example, the aggregation primitive "count" identifies values ​​in related rows of the dataset, determines whether each of the values ​​is a non-null value, and returns (outputs) a count of the number of non-null values ​​in the rows of the dataset. Another type of primitive is a transformation primitive. When applied to a dataset, a transformation primitive creates a new variable from one or more existing variables in the dataset. For example, the transformation primitive "weekend" evaluates a timestamp in the dataset and returns a binary value (e.g., true or false) indicating whether the date indicated by the timestamp falls on a weekend. Another exemplary transformation primitive evaluates a timestamp and returns a count value indicating the number of days until a specified date (e.g., the number of days until a particular holiday).

[0043] The feature engineering module 330 selects a set of primitives based on the dataset. In some embodiments, the feature engineering module 330 selects the primitives using a skim view approach, a summary view approach, or both. In the skim view approach, the feature engineering module 330 identifies one or more semantic representations of the dataset. The semantic representation of the dataset describes characteristics of the dataset and is obtained without performing calculations on the data in the dataset. Examples of semantic representations of a dataset include the presence of one or more specific variables (e.g., column names) in the dataset, the number of columns, the number of rows, the output type of the dataset, other attributes of the dataset, and combinations thereof. To select primitives using the skim view approach, the feature engineering module 330 determines whether the identified semantic representation of the dataset matches the attributes of the primitives in the pool. If so, the feature engineering module 330 selects the primitives.

[0044] The SkimView approach is a rule-based analysis. The determination of whether the identified semantic representation of a dataset matches the attributes of a primitive is done through feature engineering. Module 330The feature engineering module 330 may select primitives with an input type of "Birthdate" that match the semantic representation of the dataset. In another example, the semantic representation of the dataset is a column named "Timestamp," and the feature engineering module 330 may select primitives with an attribute indicating that the primitive is appropriate for use with data that indicates a timestamp.

[0045] In the summary view approach, the feature engineering module 330 generates representative vectors from the dataset. The representative vectors encode data describing the dataset, such as data indicating the number of tables in the dataset, the number of columns per table, the average number of each column, and the average number of each row. The representative vectors therefore serve as fingerprints of the dataset. A fingerprint is a compact representation of the dataset and may be generated by applying one or more fingerprinting functions, such as a hash function, Rabin's fingerprinting algorithm, or other types of fingerprinting functions, to the dataset.

[0046] The feature engineering module 330 selects primitives for a dataset based on the representative vectors. For example, the feature engineering module 330 inputs the representative vectors for a dataset into a machine-learned model. The machine-learned model outputs primitives for the dataset. The machine-learned model is trained, for example, by the feature engineering module 330 to select primitives for a dataset based on the representative vectors. It may be trained based on training data including multiple representative vectors for multiple training datasets and a set of primitives for each of the multiple training datasets. The set of primitives for each of the multiple training datasets is used to generate features determined to be useful for making predictions based on the corresponding training dataset. In some embodiments, the machine-learned model is trained continuously. For example, the feature engineering module 330 may further train the machine-learned model based on at least some of the representative vectors for the dataset and the selected primitives.

[0047] The feature engineering module 330 synthesizes multiple features based on the selected primitives and the dataset. In some embodiments, the feature engineering module 330 applies each of the selected primitives to at least a portion of the dataset to synthesize one or more features. For example, the feature engineering module 330 applies the "weekend" primitive to a column named "timestamp" in the dataset to synthesize a feature that indicates whether a date is a weekend. The feature engineering module 330 can synthesize a large number of features of a dataset, such as hundreds or millions of features.

[0048] The feature engineering module 330 evaluates the features and removes some of the features based on the evaluation to obtain a group of features. In some embodiments, the feature engineering module 330 evaluates the features through an iterative process. In each round of iteration, the feature engineering module 330 applies features not removed by the previous iteration (also referred to as "remaining features") to a different portion of the dataset and determines a utility score for each of the features. The feature engineering module 330 removes some of the features with the lowest utility scores from the remaining features. In some embodiments, the feature engineering module 330 uses a random forest to determine the utility scores of the features.

[0049] The feature engineering module 330 ranks features (mapped standard features and / or features generated from unmapped variables) and determines a ranking score for each feature. The feature ranking score indicates how important the feature is in predicting the target variable, or in other words, how good the feature is as a predictor. In some embodiments, the feature engineering module 330 constructs a random forest based on the features and the dataset. The feature engineering module 330 determines a feature ranking score based on each decision tree in the random forest and takes the average of the individual ranking scores as the feature ranking score. The feature engineering module 330 may use GINI impurity as part of each decision tree to measure how much a feature contributes to the overall predictive model. The feature ranking score determined by using the random forest indicates how important the feature is with respect to other features and is called a "relative ranking score." In one example, Feature Engineering The module 330 determines that the highest ranked selected feature has a relative ranking score of one. Feature EngineeringThe module 330 then determines the ratio of the ranking score of each of the remaining features to the ranking score of the highest ranked feature as the relative ranking score of the corresponding selected feature.

[0050] The feature engineering module 330 may determine an absolute ranking score for each selected feature, for example, based on the Goodman-Kruskal Tau (GKT) measure. The GKT measure is a measure of topographical or absolute relevance that indicates how well a feature predicts a subject. The feature engineering module 330 may select a subset of the group of features based on their relative and / or absolute ranking scores as features for training the model.

[0051] The feature engineering module 330 also determines an importance factor for each selected feature, e.g., based on the relative and / or absolute ranking scores of the selected feature. The importance factor indicates how important / relevant it is to the target prediction. The feature engineering module 330 also generates values ​​for each selected feature, e.g., by applying transformers to corresponding data in the dataset associated with a timestamp before the cutoff time. The feature engineering module 330 sends the selected features, their importance factors, and their values ​​(collectively referred to as a “feature matrix”) to the training module 340 for training the model.

[0052] The training module 340 uses the respective machine learning pipeline of the machine learning application 300 to train a model based on the labels from the labeling module 320 and the feature matrix from the feature engineering module 330.

[0053] In the process of training a model, the training module 340 may detect missing values ​​and perform data imputation to supply values. In some embodiments, the training module 340 determines new values ​​to replace missing values ​​based on current values. For example, for each feature or label that has a missing value, the training module 340 replaces the missing value with the mean or median of the current values, the most frequent value, or a value from a new data sample. The training module 340 may use other imputation methods such as K-Nearest Neighbor (kNN) imputation, hot deck imputation, cold deck imputation, regression imputation, Stochastic regression imputation, extrapolation and interpolation, single imputation, multiple imputation, Multivariate Imputation by Chained Equation (MICE), imputation using Deep Neural Networks, etc.

[0054] The training module 340 may also perform feature scaling, for example, by normalizing or standardizing the feature values. In some embodiments, the training module 340 scales the value range of a feature based on the feature's importance factor. For example, the value range of a feature with a higher importance factor is scaled higher than the value range of another feature with a lower importance factor. If a feature has a relatively higher value range than other features, the training module may reduce the value range of the feature to prevent the feature from dominating other features during the training process. The training module 340 can use various methods for scaling features, such as, for example, a Min Max Scaler, a Standard Scaler, a Max Abs Scaler, a Robust Scaler, a Quantile Transformer Scaler, a Power Transformer Scaler, or a Unit Vector Scaler.

[0055] The training module 340 also obtains an algorithm that implements the classification. The training module 340 may select an algorithm from a pool of candidate algorithms. Examples of candidate algorithms include, for example, decision trees, logistic regression, random forests, XGBoost, linear support vector machines (linear SVMs), AdaBoost, neural networks, naive Bayes, and memory-based learning. 、Examples of candidate algorithms include bagged trees, boosted trees, boosted stumps, etc. In some embodiments, the training module 340 may constrain the number of candidate algorithms in the pool based on available information, such as time limits for training the model, computational resource limits (e.g., processor limits, memory usage limits, etc.), the prediction problem to be solved, characteristics of the dataset, selected features, etc. The training module 340 may test each candidate algorithm and select the best one.

[0056] The training module 340 trains the model by using a classification algorithm, and since a machine learning application has multiple machine learning pipelines, the training module 340 trains multiple models.

[0057] The ranking module 350 ranks the multiple trained models. In some embodiments, the ranking module 350 defines a test harness associated with a performance measure (e.g., classification accuracy) to evaluate the performance of the trained models. For example, the ranking module 350 applies the trained models to a test set to quantify the accuracy of the trained models. The test set includes data that differs from the data used to train the models. In some embodiments, the machine learning application 300 divides the label and feature matrix into a training set and a test set. The training set is provided to the training module 340 to train the models, and the test set is provided to the ranking module 350 to rank the models.

[0058] Common metrics applied to measure accuracy include: Precision = TP / (TP+FP) and Recall = TP / (TP+FN), where Precision is the number of outcomes (TP or true positives) that the model correctly predicted out of the total number that the model predicted (TP+FP or false positives), and Recall is the number of outcomes (TP) that the model correctly predicted out of the total number that actually occurred (TP+FN or false negatives). F-score (F-score=2 * PR / (P+R)) combines precision and recall into one measure.

[0059] The results of testing the trained models against the test harness estimate how the trained models performed on the prediction task against the performance measures. The ranking module 350 may determine a ranking score for each trained model, where the ranking score indicates the measured performance and / or accuracy of the trained model. Based on the ranking, the ranking module 350 may select one of the trained models, e.g., the training model with the best performance, as the best performing model. was Select your model.

[0060] Ranking Module 3 5 0 then deploys the selected trained model, which can be used to make predictions based on the new values. In some embodiments, ranking module 3 5 The system may further include a computer system, such as a server of an industrial organization associated with the industrial problem, that transmits the artifacts to a database of the computer system. The artifacts are outputs produced by the machine learning process, such as the selected trained model, other trained models, model checkpoints, features, labels, etc. The computer system may further provide the selected trained model to other computer systems that use the selected trained model to make predictions based on new values.

[0061] Database 360 ​​stores data associated with machine learning application 300, such as data received, used, and generated by machine learning application 300. For example, database 360 ​​stores datasets, standard features, feature matrices, transformers, label times, training sets, test sets, machine learning pipelines, decisions made in each machine learning pipeline step, algorithms, hyperparameters, trained models, ranking scores of trained models, etc.

[0062] 4A-C illustrate training a model from a dataset using an industry-specific machine learning application 300 according to one embodiment. In FIG. 4A, a dataset 410 is input to a labeling module 320, which outputs a label timetable 420. The label timetable 420 includes labels, each associated with a cutoff time.

[0063] In FIG. 4B, feature generation data 430 is passed to feature engineering module 3 3 0 and outputs a feature matrix 440. Feature generation data 430 includes some or all of the data in the dataset associated with timestamps before the cutoff time. Feature matrix 440 includes a plurality of features, feature values, and feature importance factors. Some of the plurality of features are standard features included in machine learning application 300, provided by machine learning application 300 to the user, and mapped by the user to variables in the dataset.

[0064] In FIG. 4C , a feature matrix 440, label values ​​425 from a label timetable 420, and a machine learning pipeline 450 are input to a training module 340. The machine learning pipeline 450 includes an imputer 453, a scaler 455, and a classifier 457. The machine learning pipeline 450 is one of multiple machine learning pipelines in the machine learning application 300. The multiple machine learning pipelines are specific to the industry problem. The training module 340 uses the machine learning pipeline 450 to train a model 460: the training module 340 uses the imputer 453 to detect missing values ​​and provide new values ​​for the missing values; the training module 340 uses the scaler 455 to scale the value range of the features; and the training module 340 uses the classifier 457 to perform supervised machine learning.

[0065] The training module 340 also generates trained models using each of the other machine learning pipelines of the machine learning application 300. In some embodiments, the trained models are ranked based on their predictive performance, and the trained model determined to have the best performance is deployed and used to make predictions based on new data.

[0066] 5 is a flowchart illustrating a method 500 for generating a model for making predictions for an industry problem, according to one embodiment. In some embodiments, the method is performed by machine learning application 160, while in other embodiments, some or all of the operations of the method may be performed by other entities. In some embodiments, the operations in the flowchart are performed in a different order and include different and / or additional steps.

[0067] The machine learning application 160 receives 510 a dataset for generating a model. The dataset can be received from a client device associated with a user or from a data source, for example, one of the data sources 120 of FIG. 1 . The machine learning application 160 is selected from a plurality of machine learning applications based on an industry problem. Each of the plurality of machine learning applications corresponds to a different industry problem and includes standard features specific to the corresponding industry problem and a machine learning pipeline specific to the corresponding industry problem. Exemplary industry problems include application monitoring, service level agreement violation detection, user behavior prediction, etc.

[0068] In some embodiments, the standard features are pre-generated and / or selected based on the industry problem by the machine learning server 110. The standard features can be features that have proven important in past training of models that solve the industry problem, common variables in datasets used to train models that solve the industry problem, features that are logically related to the industry problem, or a combination thereof.

[0069] In some embodiments, the machine learning pipelines are pre-generated by the machine learning server 110 based on the domain of the industry problem. For example, the machine learning server 110 identifies the domain of the industry problem based on the description of the industry problem. The domain is associated with a business. The machine learning server 110 selects an objective function from a plurality of objective functions based on the identified domain. Each of the plurality of objective functions is specific to a respective domain and is used to select an optimal machine learning pipeline for prediction in the respective domain. The machine learning server 110 then applies the objective function to a pool of machine learning pipelines to select a machine learning pipeline for each machine learning application from the pool of machine learning pipelines.

[0070] The machine learning application 160 provides 520 the standard features in the machine learning application for display on a client device associated with a user. In some embodiments, the machine learning application 160 provides 520 the standard features to a user interface. The user interface allows a user associated with the client device to map variables in the dataset to standard features in the selected machine learning application. In some embodiments, the user interface allows a user to map one variable in the dataset to one standard feature, map multiple variables in the dataset to one standard feature, and / or map one variable in the dataset to multiple standard features.

[0071] The machine learning application 160 receives (530) from the client device and in response to providing the standard features, a mapping of variables in the dataset to standard features in the selected machine learning application. For example, the machine learning server 110 receives a mapping of a variable "username" to a standard feature "ID" of a user. After the machine learning server 110 receives the mapping, the machine learning server 110 may convert the value of the variable to a new value as the value of the standard feature. For example, if the variable "username" includes multiple text strings representing the user's name, the machine learning server converts the text strings to numeric values ​​as the values ​​of the standard feature "ID."

[0072] The machine learning application 160 applies its machine learning pipelines to the dataset to train 540 multiple models based on at least the mapping. Each machine learning pipeline specifies steps in the training process. In some embodiments, the training process includes data imputation, feature scaling, and classification.

[0073] In some embodiments, machine learning application 160 generates a plurality of features, including one or more standard features in the selected machine learning application that are mapped to one or more variables in the dataset, and one or more other features. Machine learning application 160 extracts one or more other features from variables in the dataset that are not mapped to any of the standard features. To extract such features, machine learning application 160 may identify variables in the dataset that are not mapped to any of the pool of standard features, select primitives from a pool of primitives based on the identified variables, and apply the primitives to the variables.

[0074] The machine learning application 160 ranks 550 the multiple trained models. In some embodiments, the machine learning application 160 ranks 5 5 The trained models are ranked by defining a test harness associated with a performance measure (e.g., classification accuracy) and ranking the trained models based on their performance. The performance of each trained model can be measured by inputting a test set to the trained model and comparing the trained model's output with known predictions associated with the test set.

[0075] The machine learning application 160 selects 560 a generated model from the multiple trained models based on the rankings. The selected trained model is used to make predictions on new data.

[0076] FIG. 6 is a high-level block diagram illustrating a functional view of an exemplary computer system 600 for use as the machine learning server 110 of FIG. 1, according to one embodiment.

[0077] The illustrated computer system includes at least one processor 602 coupled to a chipset 604. The processor 602 may include multiple processor cores on the same die. The chipset 604 includes a memory controller hub 620 and an input / output (I / O) controller hub 622. The memory 606 and the graphics adapter 612 are coupled to the memory controller hub 620, and the display 618 is coupled to the graphics adapter 612. The storage device 608, the keyboard 610, the pointing device 614, and the network adapter 616 may be coupled to the I / O controller hub 622. In some other embodiments, the computer system 600 may have additional, fewer, or different components, and the components may be coupled separately. For example, an embodiment of the computer system 600 may lack a display and / or a keyboard. Additionally, the computer system 600 may be instantiated as a rack-mounted blade server or as a cloud server instance in some embodiments.

[0078] The memory 606 holds instructions and data used by the processor 602. In some embodiments, the memory 606 is a random access memory. The storage device 608 is a non-transitory computer-readable storage medium. The storage device 608 can be an HDD, an SSD, or another type of non-transitory computer-readable storage medium. Data processed and analyzed by the machine learning server 110 can be stored in the memory 606 and / or the storage device 608.

[0079] Pointing device 614 may be a mouse, trackball, or other type of pointing device and may be used in combination with keyboard 610 to input data into computer system 600. Graphics adapter 612 displays images and other information on display 618. In some embodiments, display 618 includes touch screen capabilities for receiving user inputs and selections. Network adapter 616 connects computer system 600 to a network 1 4 Connect to 0.

[0080] The computer system 600 is adapted to execute computer modules for providing the functionality described herein. As used herein, the term "module" refers to computer program instructions and other logic for providing a particular function. A module can be implemented in hardware, firmware, and / or software. A module can include one or more processes and / or be provided by only a portion of a process. A module is typically stored in the storage device 608, loaded into the memory 606, and executed by the processor 602.

[0081] The particular naming of components, term capitalization, attributes, data structures, or any other programming or structural aspect is not required or important, and mechanisms for implementing the described embodiments may have different names, formats, or protocols. Furthermore, the system may be implemented through a combination of hardware and software as described, or entirely with hardware elements. Also, the particular division of functionality among various system components described herein is merely exemplary and not required; functions performed by a single system component may instead be performed by multiple components, and functions performed by multiple components may instead be performed by a single component.

[0082] Some portions of the above description are presented in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. These operations, while described in functional or logical terms, will be understood to be implemented by computer programs. Further, it has proven convenient at times to refer to arrangements of these operations as modules or functional names, without loss of generality.

[0083] Unless otherwise stated as is clear from the above discussion, throughout the description, discussions utilizing terms such as "processing" or "computing" or "calculating" or "determining" or "displaying" will be understood to refer to the operations and processing of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities in the computer system memory or registers or other such information storage, transmission or display device.

[0084] Certain embodiments described herein include process steps and instructions written in the form of algorithms. It should be noted that the process steps and instructions of the embodiments may be implemented in software, firmware, or hardware, and if implemented in software, may be downloaded to reside on and be operated from different platforms used by the real-time network operating system.

[0085] Finally, it should be noted that the language used herein has been chosen primarily for ease of reading and educational purposes, and may not be chosen to delineate or limit the subject matter of the present invention. Accordingly, the disclosure of embodiments is intended to be illustrative, not limiting.

Claims

1. 1. A computer-implemented method for generating a model for making predictions on an industry problem, comprising: receiving, by a machine learning application, a dataset for generating a model, the machine learning application being selected from a plurality of machine learning applications based on the industry problem, each of the plurality of machine learning applications corresponding to a different industry problem and including standard features specific to the corresponding industry problem and a machine learning pipeline specific to the corresponding industry problem; providing the standard features within the machine learning application for display on a client device associated with a user; receiving from the client device and in response to providing the standard features, a mapping of variables in the dataset to standard features in the selected machine learning application; applying the machine learning pipeline of the selected machine learning application to the dataset to train a plurality of models based on at least the mapping; ranking the trained models; selecting a generated model from the plurality of trained models based on the ranking; and A computer-implemented method comprising:

2. providing, by the machine learning application, the standard features within the machine learning application for display on the client device associated with the user, providing, by the machine learning application, the standard features in the machine learning application for display in a user interface, the user interface enabling the user to map variables in the dataset to the standard features in the machine learning application; The computer-implemented method of claim 1 , comprising:

3. applying the machine learning pipeline of the selected machine learning application to the dataset to train a plurality of models based on at least the mapping, generating a plurality of features including one or more standard features in the selected machine learning application mapped to one or more of the variables in the dataset and one or more other features, wherein the one or more other features are extracted from one or more other variables in the dataset that are not mapped to any of the standard features of the selected machine learning application; The computer-implemented method of claim 1 , comprising:

4. generating a plurality of features including one or more standard features and one or more other features mapped to one or more of the variables in the dataset, identifying variables in the dataset that do not map to any of the standard features of the selected machine learning application; selecting a primitive from a pool of primitives based on the identified variables, the primitive comprising a function that transforms the variables into features; applying the primitive to the variable to generate one of the one or more other features; The computer-implemented method of claim 3 , comprising:

5. applying a labeling function to the data set to generate label times, each label time including a label and a cutoff time associated with the label; The computer-implemented method of claim 1 further comprising:

6. the labeling function is included in the machine learning application and is specific to the industry problem corresponding to the selected machine learning application; The computer-implemented method of claim 5 .

7. The labeling function comprises customizable parameters, and the method comprises: receiving values ​​of the customizable parameters from the client device, the values ​​being specific to a prediction within the industry problem; and customizing the labeling function based on the received values; The computer-implemented method of claim 5 further comprising:

8. A non-transitory computer-readable memory storing executable computer program instructions, the executable instructions performing operations to generate a model for making predictions for an industrial problem, the instructions comprising: receiving, by a machine learning application, a dataset for generating the model, wherein the machine learning application is selected from a plurality of machine learning applications based on the industry problem, each of the plurality of machine learning applications corresponding to a different industry problem and including standard features specific to the corresponding industry problem and a machine learning pipeline specific to the corresponding industry problem; providing the standard features within the machine learning application for display on a client device associated with a user; receiving from the client device and in response to providing the standard features, a mapping of variables in the dataset to standard features in the selected machine learning application; applying the machine learning pipeline of the selected machine learning application to the dataset to train a plurality of models based on at least the mapping; ranking the trained models; selecting a generated model from the plurality of trained models based on the ranking; and 1. A non-transitory computer-readable memory comprising:

9. providing, by the machine learning application, the standard features within the machine learning application for display on the client device associated with the user, providing, by the machine learning application, the standard features in the machine learning application for display in a user interface, the user interface enabling the user to map variables in the dataset to the standard features in the machine learning application; 9. The non-transitory computer-readable memory of claim 8, comprising:

10. applying the machine learning pipeline of the selected machine learning application to the dataset to train a plurality of models based on at least the mapping, generating a plurality of features including one or more standard features in the selected machine learning application mapped to one or more of the variables in the dataset and one or more other features, wherein the one or more other features are extracted from one or more other variables in the dataset that are not mapped to any of the standard features of the selected machine learning application; 9. The non-transitory computer-readable memory of claim 8, comprising:

11. generating a plurality of features including one or more standard features and one or more other features mapped to one or more of the variables in the dataset, identifying variables in the dataset that do not map to any of the standard features of the selected machine learning application; selecting a primitive from a pool of primitives based on the identified variables, the primitive comprising a function that transforms the variables into features; applying the primitive to the variable to generate one of the one or more other features; 11. The non-transitory computer-readable memory of claim 10, comprising:

12. The operation is applying a labeling function to the data set to generate label times, each label time including a label and a cutoff time associated with the label; 10. The non-transitory computer-readable memory of claim 8, further comprising:

13. the labeling function is included in the machine learning application and is specific to the industry problem corresponding to the selected machine learning application; 13. The non-transitory computer-readable memory of claim 12.

14. The labeling function has customizable parameters, and the operation comprises: receiving values ​​of the customizable parameters from the client device, the values ​​being specific to a prediction within the industry problem; and customizing the labeling function based on the received values; 13. The non-transitory computer-readable memory of claim 12, further comprising:

15. a computer processor for executing computer program instructions; a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations to generate a model for making predictions on an industrial problem, the executable instructions comprising: receiving, by a machine learning application, a dataset for generating the model, wherein the machine learning application is selected from a plurality of machine learning applications based on the industry problem, each of the plurality of machine learning applications corresponding to a different industry problem and including standard features specific to the corresponding industry problem and a machine learning pipeline specific to the corresponding industry problem; providing the standard features within the machine learning application for display on a client device associated with a user; receiving from the client device and in response to providing the standard features, a mapping of variables in the dataset to standard features in the selected machine learning application; applying the machine learning pipeline of the selected machine learning application to the dataset to train a plurality of models based on at least the mapping; ranking the trained models; selecting a generated model from the plurality of trained models based on the ranking; and A system comprising:

16. providing, by the machine learning application, the standard features within the machine learning application for display on the client device associated with the user, providing, by the machine learning application, the standard features in the machine learning application for display in a user interface, the user interface enabling the user to map variables in the dataset to the standard features in the machine learning application; The system of claim 15, comprising:

17. applying the machine learning pipeline of the selected machine learning application to the dataset to train a plurality of models based on at least the mapping, generating a plurality of features including one or more standard features in the selected machine learning application mapped to one or more of the variables in the dataset and one or more other features, wherein the one or more other features are extracted from one or more other variables in the dataset that are not mapped to any of the standard features of the selected machine learning application; The system of claim 15, comprising:

18. generating a plurality of features including one or more standard features and one or more other features mapped to one or more of the variables in the dataset, identifying variables in the dataset that do not map to any of the standard features of the selected machine learning application; selecting a primitive from a pool of primitives based on the identified variables, the primitive comprising a function that transforms the variables into features; applying the primitive to the variable to generate one of the one or more other features; 20. The system of claim 17, comprising:

19. The operation is applying a labeling function to the data set to generate label times, each label time including a label and a cutoff time associated with the label; The system of claim 15 further comprising:

20. The labeling function has customizable parameters, and the operation comprises: receiving values ​​of the customizable parameters from the client device, the values ​​being specific to a prediction within the industry problem; and customizing the labeling function based on the received values; 20. The system of claim 19, further comprising:

Images