Industry-specific machine learning applications

Abstract

The machine learning application is selected from a plurality of machine learning applications. Each machine learning application corresponds to a different industry problem and includes standard features and a machine learning pipeline specific to the respective industry problem. The machine learning application receives a dataset for generating a model for making predictions for the industry problem corresponding to the selected machine learning application. The standard features are provided for display for a user to map variables in the dataset to the standard features. The mapping by the user is received through the user interface. The machine learning pipeline is applied to the dataset to train a plurality of models based at least on the mapping. The trained models are ranked and one of the trained models is selected based on the ranking. The selected trained model will be used to make predictions based on new data.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Application No. 17 / 242,927, filed April 28, 2021, which is incorporated herein by reference. Technical Field

[0003] The described embodiments generally relate to processing data streams, and more specifically to using industry-specific machine learning applications to train models for making predictions based on data streams. Background Technology

[0004] Automated machine learning tools automate the process of applying machine learning to real-world problems. Current automated machine learning tools can quickly and efficiently create deployable machine learning models. However, these tools often produce suboptimal models because they lack domain knowledge relevant to the dataset. Consequently, models generated by currently available automated machine learning tools are less effective at making predictions based on data than expected. Summary of the Invention

[0005] The above and other problems are addressed by methods, non-transitory computer-readable storage, and systems. An embodiment of the method is a method for generating a model for predicting an industry problem. The method includes: receiving a dataset for generating the model through a machine learning application. The machine learning application is selected from multiple machine learning applications based on the industry problem. Each of the multiple machine learning applications corresponds to a different industry problem and includes standard features specific to the respective industry problem and a machine learning pipeline specific to the respective industry problem. The method further includes: providing the standard features from the machine learning applications for display to a client device associated with a user. The method further includes: in response to providing the standard features, receiving a mapping from the client device of variables in the dataset to standard features in the selected machine learning applications. The method further includes: applying the machine learning pipeline in the selected machine learning applications to the dataset to train multiple models at least based on the mapping. The method further includes: ranking the multiple trained models. The method further includes: selecting a generated model from the multiple trained models based on the ranking.

[0006] Embodiments of the non-transitory computer-readable storage medium store executable computer program instructions. The instructions are executable to perform operations for generating a model for predicting an industry problem. The operations include: receiving a dataset for generating the model via a machine learning application. The machine learning application is selected from multiple machine learning applications based on the industry problem. Each of the multiple machine learning applications corresponds to a different industry problem and includes standard features specific to the respective industry problem and a machine learning pipeline specific to the respective industry problem. The operations also include: providing the standard features from the machine learning applications for display to a client device associated with a user. The operations further include: receiving a mapping from the client device of variables in the dataset to standard features in the selected machine learning applications in response to providing the standard features. The operations also include: applying the machine learning pipeline in the selected machine learning applications to the dataset to train multiple models at least based on the mapping. The operations further include: ranking the multiple trained models. The operations also include: selecting a generated model from the multiple trained models based on the ranking.

[0007] Embodiments of the system include a computer processor for executing computer program instructions. The system also includes a non-transitory computer-readable storage device storing computer program instructions executable by the computer processor to perform operations for generating a model for making predictions for an industry problem. The operations include: receiving a dataset for generating the model via a machine learning application. The machine learning application is selected from multiple machine learning applications based on the industry problem. Each of the multiple machine learning applications corresponds to a different industry problem and includes standard features specific to the respective industry problem and a machine learning pipeline specific to the respective industry problem. The operations also include: providing the standard features from the machine learning applications for display to a client device associated with a user. The operations further include: receiving a mapping from the client device of variables in the dataset to standard features in the selected machine learning applications in response to providing the standard features. The operations also include: applying the machine learning pipeline in the selected machine learning applications to the dataset to train multiple models at least based on the mapping. The operations further include: ranking the multiple trained models. The operations also include: selecting a generated model from the multiple trained models based on the ranking. Attached Figure Description

[0008] Figure 1 This is a block diagram illustrating a machine learning environment including a machine learning server according to one embodiment.

[0009] Figure 2 This is a block diagram illustrating an application generation engine for generating industry-specific machine learning applications according to one embodiment.

[0010] Figure 3 This is a block diagram illustrating an industry-specific machine learning application according to one embodiment.

[0011] Figures 4A-4C It is shown that, according to one embodiment, the use of... Figure 3 Industry-specific machine learning applications use datasets to train models.

[0012] Figure 5 This is a flowchart illustrating a method for training a model using an industry-specific machine learning application, according to one embodiment.

[0013] Figure 6 This illustrates the use of, according to one embodiment, as Figure 1 A high-level block diagram of the functional view of a typical computer system for machine learning servers.

[0014] The accompanying drawings depict various embodiments for illustrative purposes only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods shown herein can be employed without departing from the principles of the embodiments described herein. The same reference numerals and names in the various drawings indicate the same elements. Detailed Implementation

[0015] Figure 1 This is a block diagram illustrating a machine learning environment 100 including a machine learning server 110 according to one embodiment. Environment 100 also includes multiple data sources 120 and client devices 130 connected to the machine learning server 110 via a network 140. While the illustrated environment 100 includes only one machine learning server 110 coupled to multiple data sources 120 and client devices 130, embodiments may have multiple machine learning servers, a single data source and a single client device, or other variations thereof.

[0016] Machine learning server 110 is a computer-based system used to build machine learning models and deploy those models for data-driven predictions. Data is collected, aggregated, or otherwise accessed via network 140 from one or more of multiple data sources 120 or one or more of multiple client devices 130. Machine learning server 110 enables the use of scalable software tools and hardware resources to access, prepare, mix, and analyze data from multiple data sources 120 or client devices 130.

[0017] Machine learning server 110 implements industry-specific machine learning processes. Machine learning server 110 includes application generation engine 150 and multiple industry-specific machine learning applications 160 (also referred to as "machine learning applications 160"; individually referred to as "industry-specific machine learning applications 160" or "machine learning applications 160") generated by application generation engine 150. Industry-specific machine learning applications 160 are applications that can be used to train models to make predictions within a specific industry problem. The industry problem is a problem in an industry or enterprise domain. This industry / domain can be, for example, information technology (IT) operations, healthcare, industrial manufacturing, retail, sales and marketing, insurance, banking, etc. For example, industry problems could be application monitoring, service level agreement violation detection, user action prediction, etc.

[0018] The industry-specific machine learning application 160 includes machine learning tools (e.g., labeled functions, standard features, machine learning pipelines, etc.) already generated by the machine learning server 110 for a specific industry problem. Such machine learning tools can be generated and / or selected based on domain knowledge of the industry problem, knowledge of the historical training of models associated with the industry problem, other types of knowledge related to the industry problem, or some combination thereof. Utilizing these industry-specific machine learning tools, the machine learning process is more efficient than traditional machine learning techniques. For example, standard features can be used as features for training the model (e.g., by simply mapping variables in the training data to standard features), which saves the time and computational resources required to extract these features from the dataset. As another example, the search and optimization of the pipeline used in the machine learning process can be limited to the pipelines in the selected machine learning application, making the search and optimization more efficient compared to traditional machine learning processes. Using these machine learning tools, the machine learning application 160 performs automated and industry-specific machine learning.

[0019] In some embodiments, the industry-specific machine learning application 160 can allow users to provide inputs to the machine learning process. For example, it can allow users to map variables in the training dataset to standard features. It can also allow users to define the values ​​of certain parameters in the labeling function to customize the labeling function for the specific predictions sought by the user. In this way, the industry-specific machine learning application 160 leverages both domain knowledge of the industry problem and the user's expertise in the dataset and the specific predictions. Therefore, compared to traditional machine learning techniques, the industry-specific machine learning application 160 can train models that are more suitable for industry and user needs.

[0020] In some embodiments, the machine learning server 110 provides a plurality of industry-specific machine learning applications 160 for display to a client device associated with a user. The machine learning server 110 allows a user to select one of the industry-specific machine learning applications 160 to train a machine learning model. The user may be a person with knowledge associated with the machine learning model to be trained (e.g., a machine learning engineer, development engineer, etc.), such knowledge as predictions to be made by the model, data used to train the model, data used to make predictions, etc. The user selects a machine learning application 160 specific to the industry problem to which the model is to make predictions, for example, predictions falling within the scope of the industry problem.

[0021] In some embodiments, the machine learning server 110 presents a machine learning application 160 in a user interface. The machine learning application 160 may be associated with labels indicating an industry problem corresponding to it, allowing a user to rely on these labels to determine if the machine learning application is suitable for training a model required by the user. In some embodiments, the machine learning server 110 supports one or more user interfaces, such as a graphical user interface (GUI), that allow users to interact with the machine learning application. For example, the user interface provides users with options such as viewing the machine learning application, downloading the machine learning application, interacting with an online version of the machine learning application, uploading datasets to the machine learning application, and mapping variables in the dataset to standard features in the machine learning application.

[0022] Data source 120 provides electronic data to machine learning server 110. Data source 120 can be a storage device such as a hard disk drive (HDD) or solid-state drive (SSD), a computer that manages and provides access to multiple storage devices, a storage area network (SAN), a database, or a cloud storage system. Data source 120 can also be a computer system capable of retrieving data from another source. Data source 120 can be located remotely from machine learning server 110 and provide data via network 140. Furthermore, some or all of data sources 120 can be directly coupled to the data analysis system and provide data without needing to transfer data through network 140.

[0023] The data provided by data source 120 includes data used to train a machine learning model for solving an industry problem and / or data used as input to a trained model to make predictions within the scope of the industry problem. The data may be organized into data records (e.g., rows). Each data record includes one or more values. For example, a data record provided by data source 120 may include a series of comma-separated values. This data describes relevance information about the business using machine learning server 110. For example, data from data source 120 may describe computer-based interactions with accessible content on a website and / or with applications (e.g., click tracking data). As another example, data from data source 120 may describe customer transactions online and / or in a store. The business may belong to one or more different industries, such as computer technology, manufacturing, etc.

[0024] Client device 130 is one or more computing devices capable of receiving user input and sending and / or receiving data via network 140. In one embodiment, client device 130 is a conventional computer system, such as a desktop computer or laptop computer. Alternatively, client device 130 may be a computer-enabled device, such as a personal digital assistant (PDA), mobile phone, smartphone, or other suitable device. Client device 130 is configured to communicate with one or more data sources 120 and machine learning server 110 via network 140. In one embodiment, client device 130 executes an application that allows a user of client device 130 to interact with machine learning server 110. For example, client device 130 executes an application to enable interaction between client device 130 and machine learning application 160 via network 140, such as by running a GUI supported by machine learning server 110. Device 130 includes a display device for displaying the GUI or is otherwise associated with a display device for displaying the GUI. Client device 130 is also associated with an input device, such as a keyboard, mouse, etc., which allows the user to interact with the GUI, such as providing input to the GUI. In another embodiment, client device 130 interacts with machine learning server 110 via an application programming interface (API) running on client device 130's native operating system (such as iOS® or Android™). Client device 130 may interact with one or more data sources 120 to send data to or receive data from data sources 120.

[0025] Network 140 represents the communication path between machine learning server 110 and data source 120. In one embodiment, network 140 is the Internet and uses standard communication technologies and / or protocols. Therefore, network 140 may include links using technologies such as Ethernet, 802.11, WiMAX, 3G, LTE, DSL, ATM, InfiniBand, PCI Fast Advanced Switching, etc. Similarly, network protocols used on network 140 may include Multiprotocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP), etc.

[0026] Data exchanged over network 140 can be represented using technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc. Furthermore, conventional encryption techniques can be used to encrypt all or some of the links, such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. In another embodiment, the entity may use custom and / or dedicated data communication technologies instead of the aforementioned technologies, or use custom and / or dedicated data communication technologies in addition to the aforementioned technologies.

[0027] Figure 2 This is a block diagram illustrating an application generation engine 200 for generating industry-specific machine learning applications according to one embodiment. The application generation engine 200 is... Figure 1 An embodiment of application generation engine 150 is described below. Application generation engine 200 generates industry-specific machine learning applications that can be used to train models. Application generation engine 200 includes a labeling function module 210, a standard feature module 220, a pipeline module 230, a user interface module 240, and a database 250. Those skilled in the art will recognize that other embodiments may have different components and / or other components besides those described herein, and functionality may be distributed among the components in different ways.

[0028] The annotation function module 210 obtains an industry-specific annotation function. An annotation function is a function that creates annotation timestamps based on the dataset when applied to it. Annotation timestamps (referred to as "annotation schedules") can be provided in a table. Annotation timestamps include the deadline and the annotations associated with that deadline. The deadline is the time when a prediction is made. Data associated with timestamps before the deadline can be used to extract features from the annotations. However, data associated with timestamps after the deadline should not be used to extract features from the annotations. Annotations associated with the deadline are historical examples of the target of the prediction (e.g., true or false) associated with that deadline. Annotations can be generated based on data associated with timestamps on and / or beyond the deadline using an annotation function. For example, for a prediction of user actions on a specific date (e.g., the first day of each month), the deadline is the first day of that month. Data associated with the timestamp of the first day of each month is applied to the annotation function to generate annotations but cannot be used to generate features. All features must be generated using data prior to the deadline, such as data from the previous month.

[0029] The annotation function includes customizable parameters. Examples of parameters include the prediction date / time (i.e., the deadline / time), the prediction window (the period of time for which the prediction is made), the number of days or months (the future period of time to be predicted), and so on. In some embodiments, the values ​​of the parameters are customized, for example, by a user with domain knowledge of the prediction and / or industry problem, to create annotation times for different predictions across a range of industry problems. For example, to predict user actions on the first day of each month, the prediction date could be the first day of that month, and the prediction window could be one month.

[0030] The standard feature module 220 generates industry-specific standard features. For example, for each machine learning application, the standard feature module 220 generates one or more industry-specific standard features for the machine learning application, such as based on knowledge associated with the industry problem. In some embodiments, the standard feature module 220 generates standard features based on typical variables input by the user into a dataset trained on a model used to solve the industry problem. For example, for a machine learning application specific to predicting the next purchase, the standard feature module 220 generates standard features associated with the user (e.g., user ID, gender, birthday, postal code, etc.) and standard features associated with historical transactions (e.g., transaction ID, transaction date, transaction amount, purchased product, etc.). In some embodiments, the standard feature module 220 selects standard features from a feature pool. The standard feature module 220 may select standard features from the pool based on the performance of the standard features in the historical training of the model related to the industry problem.

[0031] In some embodiments, the standard feature module 220 generates standard primitives to be applied to the dataset to generate features. Standard primitives include algorithms that, when applied to the data, perform computations on the data and generate corresponding standard features with associated values. In one example, the standard primitives are industry-specific primitives by default for machine learning applications. In another example, standard primitives are selected from a pool of candidate primitives. For example, candidate primitives are ranked based on the ranking of features generated from them. Candidate primitives that generate higher-ranked features (e.g., higher than features generated from other candidate primitives) are selected as standard primitives. The algorithms for standard primitives can be applied to different datasets with different variables. Therefore, standard primitives can be reused on different datasets to train different machine learning models in different industry domains. The following is combined with... Figure 3 Provide more information about primitive features and ordering features.

[0032] Pipeline module 230 generates one or more pipelines specific to an industry problem. A pipeline is a workflow for a machine learning process executed by a machine learning application to train a model, specifying a series of steps for training the model. Machine learning pipelines can also specify tools (e.g., algorithms) used in the machine learning process, such as tools for data imputation, feature scaling, classification, etc. In one example, the steps in a pipeline include data composition, feature engineering, model training, model validation, and model deployment. Steps may include sub-steps. For example, data preparation steps may include data type setting, data encoding, and data imputation; feature engineering steps may include feature selection and feature sorting; and model training steps may include hyperparameter tuning and algorithm selection. Different pipelines may include steps in different orders and / or different steps.

[0033] In some embodiments, pipeline module 230 selects pipelines from a pipeline pool based on an objective function. The objective function is the function to be optimized (e.g., minimized to maximize). It measures the effectiveness of achieving a predicted goal / objective. It can be a loss function or a cost function. Pipeline module 230 may select an objective function from the pool based on the domain of the industry problem. The objective function is domain-specific. Pipeline module 230 applies the objective function to the pipeline pool to select multiple pipelines. For example, pipeline module 230 ranks the pipeline pool based on the effectiveness of each pipeline in optimizing the objective function, and selects multiple pipelines from the pipeline pool based on this ranking.

[0034] In some embodiments, pipeline module 230 obtains a pipeline template and generates an industry-specific pipeline based on the template. Each template contains a series of components. Components are tools used to perform steps in the machine learning process. Examples of components include data transformation tools, data type setting tools, data encoding tools, data imputation tools, feature selection tools, feature ranking tools, algorithm selection tools, etc. Components are associated with one or more parameters. The values ​​of the parameters can be changed or customized. Taking a feature ranking tool as an example, the parameter of the feature ranking tool is the number of decision trees used to rank the features. The value of the parameter can be, for example, 100, 200, 300, etc.

[0035] In some embodiments, pipeline module 230 determines the values ​​of parameters for components in a pipeline template. In one example, pipeline module 230 uses default values ​​for an industry problem. In another example, pipeline module 230 uses a machine learning model to determine the values ​​of the component parameters. The machine learning model has been trained to determine the values ​​of parameters for components in a machine learning pipeline. For example, pipeline module 230 feeds relevant information into the machine learning model, and the machine learning model outputs the values ​​of parameters for one or more components in the pipeline template. The relevant information may include information about the pipeline template (information about the component in the pipeline template, information about other components in the pipeline template, etc.), information about the machine learning application, information about the industry problem, information received from the user of the machine learning application (e.g., the expected accuracy of training the model using the pipeline template, the expected duration required to train the model using the pipeline template, etc.), and so on.

[0036] Pipeline module 230 selects pipelines from pipeline templates by sorting them. For example, pipeline module 230 sorts pipeline templates based on the accuracy of the machine learning model trained using each pipeline template and selects the pipeline template with the higher ranking. Pipeline module 230 may sort pipeline templates before and / or after determining the values ​​of the parameters of the components in the pipeline template.

[0037] User interface module 240 generates user interfaces (e.g., graphical user interfaces (GUIs)) for industry-specific machine learning applications. The user interface includes elements that users use to interact with the machine learning application. Examples of elements include: icons, tabs, checkboxes, buttons, dropdown lists, list boxes, radio buttons, switches, or other types of elements that users can use to select or deselect options; input fields that users can use to enter numbers, symbols, and / or text; presentation areas that present information to the user for review; and so on. (The following section combines...) Figure 3 Describe more details about the user interface.

[0038] Database 250 stores data associated with application generation engine 200, such as data received, used, or generated by application generation engine 200. In some embodiments, database 250 stores annotation functions, standard features, objective functions, machine learning pipelines, and so on.

[0039] Figure 3 This is a block diagram illustrating an industry-specific machine learning application 300 according to one embodiment. Machine learning application 300 is... Figure 1 An embodiment of machine learning application 160 is described below. Machine learning application 300 includes industry-specific machine learning tools and is used to train a model to make predictions within the scope of an industry problem. Machine learning application 300 includes a user interface module 310, a labeling module 320, a feature engineering module 330, a training module 340, a ranking module 350, and a database 360. Those skilled in the art will recognize that other embodiments may have different components and / or other components than those described herein, and functionality may be distributed among the components in different ways.

[0040] User interface module 310 supports a user interface (e.g., GUI) that allows users to access and interact with machine learning application 300. For example, the user interface allows users to load datasets into the machine learning application, such as from a client device or from a data source. The user interface can allow users to select a portion of the dataset to train a model, for example, by allowing users to specify a time range before a deadline, so that data falling within that time range will be used to train the model.

[0041] The user interface allows a user to provide values ​​for customizable parameters of a labeling function to the machine learning application 300. The values ​​received from the user can be used to customize the machine learning process for a specific prediction sought by the user. In some embodiments, the user interface presents customizable parameters to the user. The user interface may include one or more input fields for the customizable parameters, allowing the user to input values ​​for the customizable parameters. The user interface may also provide a dropdown list from which the user can select values ​​for the customizable parameters. The user interface module 310 sends the values ​​of the customizable parameters received from the user to the labeling module 320 to customize the labeling function.

[0042] The user interface also presents standard features from the machine learning application 300 to the user and allows the user to map variables in the dataset to these standard features. In some embodiments, after receiving the dataset, the user interface module 310 identifies the variables in the dataset. The user interface module 310 can provide all or some of the variables to display to the user in the user interface, allowing the user to select a variable and map it to a standard feature. The user interface receives the user's mapping and transmits it to the feature engineering module 330.

[0043] The user interface can also allow users to make other choices to influence the machine learning process, such as editing the dataset, choosing the data type of variables, defining and / or tuning hyperparameters, providing additional guidance to the machine learning process, or some combination thereof. In some embodiments, the user interface provides a visual representation of the machine learning process, such as a visual representation of a machine learning pipeline, to be presented to the user.

[0044] The annotation module 320 generates annotation times from the dataset by applying the annotation function in the machine learning application 300 to the dataset. Each annotation time includes a label and a deadline associated with that label. A label is a historical example of the target to be predicted. The labels will be used as targets in the supervised machine learning process executed by the training module 340. The deadline indicates when to stop using the data to create features for annotation. In the example of predicting whether a customer will churn on the first day of the month, the deadline is the first day of the month, as shown in the annotation schedule. All features for each annotation must use data prior to that time to prevent data leakage.

[0045] In some embodiments, the annotation module 320 customizes the annotation function based on the values ​​of customizable parameters of the annotation function received from the user through a user interface. The user-provided values ​​can be specific to a particular prediction that will be made for the model trained on it and falls within the scope of an industry problem. Because the user has knowledge of the specific predictions that will be made for the model trained on it, the annotation module 320 incorporates this user knowledge, making the trained model suitable for the specific predictions sought by the user.

[0046] The feature engineering module 330 generates features based on data in a dataset associated with timestamps prior to the deadline. Features can be standard features in the machine learning application 300, mapped by the user, for example, through a user interface, to variables in the dataset. The feature engineering module 330 can also extract features from the dataset. For example, the feature engineering module 330 identifies variables in the dataset that have not been mapped by the user to any standard features and generates features based on those variables.

[0047] To extract features, the feature engineering module 330 can select one or more primitives from a primitive pool maintained by the machine learning application 300. The primitive pool comprises a large number of primitives, such as hundreds or thousands. Each primitive includes an algorithm that performs computations on the data and generates features with associated values ​​when applied. A primitive is associated with one or more attributes. The attributes of a primitive can be a description of the primitive (e.g., a natural language description specifying the computations performed by the primitive when applied to the data), an input type (i.e., the type of input data), a return type (i.e., the type of output data), metadata of the primitive indicating its usefulness in previous feature engineering processes, or other attributes.

[0048] In some embodiments, the primitive pool includes several different types of primitives. One type of primitive is the aggregation primitive. When applied to a dataset, an aggregation primitive identifies relevant data in the dataset, performs determinations on the relevant data, and creates a summary and / or aggregate of the determined values. For example, the aggregation primitive "count" identifies values ​​in relevant rows of the dataset, determines whether each value is non-null, and returns (outputs) a count of the number of non-null values ​​in the rows of the dataset. Another type of primitive is the transformation primitive. When applied to a dataset, a transformation primitive creates new variables based on one or more existing variables in the dataset. For example, the transformation primitive "weekend" evaluates timestamps in the dataset and returns a binary value (e.g., true or false) indicating whether the date indicated by the timestamp occurred on a weekend. Another exemplary transformation primitive evaluates timestamps and returns a count indicating the number of days up to a specified date (e.g., the number of days up to a specific holiday).

[0049] Feature engineering module 330 selects a set of primitives based on a dataset. In some embodiments, feature engineering module 330 uses a browse view method, a summary view method, or both to select primitives. In the browse view method, feature engineering module 330 identifies one or more semantic representations of the dataset. A semantic representation of a dataset describes the characteristics of the dataset and can be obtained without performing computations on the data in the dataset. Examples of semantic representations of a dataset include the presence of one or more specific variables in the dataset (e.g., column names), the number of columns, the number of rows, the input type of the dataset, other attributes of the dataset, and certain combinations thereof. To select primitives using the browse view method, feature engineering module 330 determines whether the identified semantic representation of the dataset matches the attributes of primitives in the pool. If a match exists, feature engineering module 330 selects that primitive.

[0050] The browsing view method is rule-based analysis. Determining whether the identified semantic representations of the dataset match the attributes of primitives is based on rules maintained by the application generation engine 200. Rules specify which semantic representations of the dataset match which attributes of the primitives, for example, based on matching keywords in the semantic representation of the dataset with keywords in the attributes of the primitives. In one example, the semantic representation of the dataset is the column name "Date of Birth," and the feature engineering module 330 selects primitives with the input type "Date of Birth" that match the semantic representation of the dataset. In another example, the semantic representation of the dataset is the column name "Timestamp," and the feature engineering module 330 selects primitives with attributes indicating that the primitive is suitable for use with data indicating timestamps.

[0051] In the overview view method, feature engineering module 330 generates representation vectors based on the dataset. The representation vector encodes data describing the dataset, such as the number of tables in the dataset, the number of columns in each table, the average of each column, and the average of each row. Therefore, the representation vector acts as a fingerprint of the dataset. A fingerprint is a compact representation of the dataset and can be generated by applying one or more fingerprint functions (such as hash functions, Rabin fingerprint algorithms, or other types of fingerprint functions) to the dataset.

[0052] Feature engineering module 330 selects primitives for the dataset based on representation vectors. For example, feature engineering module 330 inputs representation vectors of the dataset into a machine learning model. The machine learning model outputs primitives for the dataset. For example, the machine learning model is trained by feature engineering module 330 to select primitives for the dataset based on representation vectors. This training can be based on training data comprising multiple representation vectors of multiple training datasets and a set of primitives for each of the multiple training datasets. The set of primitives for each of the multiple training datasets has been used to generate features determined to be useful for predictions based on the respective training dataset. In some embodiments, the machine learning model is trained sequentially. For example, feature engineering module 330 can further train the machine learning model based on the representation vectors of the dataset and at least some of the selected primitives.

[0053] The feature engineering module 330 synthesizes multiple features based on selected primitives and the dataset. In some embodiments, the feature engineering module 330 applies each of the selected primitives to at least a portion of the dataset to synthesize one or more features. For example, the feature engineering module 330 applies the "weekend" primitive to a column named "timestamp" in the dataset to synthesize a feature indicating whether a date occurs on a weekend. The feature engineering module 330 can synthesize a large number of features from the dataset, such as hundreds or even millions of features.

[0054] Feature engineering module 330 evaluates features and removes some features based on the evaluation to obtain a feature set. In some embodiments, feature engineering module 330 evaluates features through an iterative process. In each iteration, feature engineering module 330 applies the features not removed in previous iterations (also referred to as "residual features") to different parts of the dataset and determines a usefulness score for each feature. Feature engineering module 330 removes some features with the lowest usefulness scores from the residual features. In some embodiments, feature engineering module 330 determines the usefulness scores of features by using a random forest.

[0055] Feature engineering module 330 ranks the features (including mapped standard features and / or features generated from unmapped variables) and determines a ranking score for each feature. The ranking score of a feature indicates its importance in predicting the target variable; in other words, how well the feature performs as a predictor. In some embodiments, feature engineering module 330 constructs a random forest based on features and a dataset. Feature engineering module 330 determines the ranking score of a feature based on each decision tree in the random forest and obtains the average of the individual ranking scores as the feature's ranking score. Feature engineering module 330 can use Gini impurity as part of each decision tree to measure how much a feature contributes to the overall predictive model. The ranking score of a feature determined using the random forest indicates its importance relative to other features and is referred to as a "relative ranking score." In one example, ranking module 350 determines the relative ranking score of the highest-ranking selected feature as 1. Ranking module 350 then determines the ratio of the ranking score of each of the remaining features to the ranking score of the highest-ranking feature as the relative ranking score of the corresponding selected feature.

[0056] The feature engineering module 330 can determine the absolute ranking score of each selected feature, for example, based on the Goodman-Kruskal Tau (GKT) metric. The GKT metric is a measure of local or absolute correlation and indicates how well the feature predicts the target. The feature engineering module 330 can select a subset of the feature group as features for training the model based on the relative and / or absolute ranking scores of the feature group.

[0057] Feature engineering module 330 also determines, for example, an importance factor for each selected feature based on its relative and / or absolute ranking score. The importance factor indicates the feature's importance / relevance to the target prediction. Feature engineering module 330 also generates values ​​for each selected feature, for example, by applying a transformer to the corresponding data in the dataset associated with timestamps prior to the deadline. Feature engineering module 330 transmits the selected features, their importance factors, and their values ​​(collectively referred to as the "feature matrix") to training module 340 to train the model.

[0058] The training module 340 trains the model using each machine learning pipeline in the machine learning application 300, based on annotations from the annotation module 320 and feature matrices from the feature engineering module 330.

[0059] During model training, training module 340 can detect missing values ​​and perform data imputation to provide these values. In some embodiments, training module 340 determines new values ​​to replace missing values ​​based on current values. For example, for each feature or label with missing values, training module 340 replaces the missing value with the mean or median of the current values, the most frequent value, or a value from a new data sample. Training module 340 can use other imputation methods, such as k-nearest neighbor (kNN) imputation, hot-deck imputation, cold-deck imputation, regression imputation, stochastic regression imputation, extrapolation and interpolation, single imputation, multiple imputation, chain equation multiple imputation (MICE), imputation using deep neural networks, etc.

[0060] The training module 340 can also perform feature scaling, for example, by normalizing or standardizing the values ​​of features. In some embodiments, the training module 340 scales the range of feature values ​​based on the importance factor of the feature. For example, the range of values ​​of a feature with a higher importance factor is scaled to be higher than the range of values ​​of another feature with a lower importance factor. For a feature that has a relatively high range of values ​​compared to other features, the training module can reduce the range of values ​​of that feature to prevent that feature from overwhelming other features during the training process. The training module 340 can use various methods for feature scaling, such as min-max scaler, standard scaler, max-absscaler, robust scaler, quantile transformer scaler, power transformer scaler, unit vector scaler, and so on.

[0061] The training module 340 also obtains algorithms that implement classification. The training module 340 can select algorithms from a pool of candidate algorithms. Examples of candidate algorithms include, for example, decision trees, logistic regression, random forests, XGBoost, linear support vector machines (linear SVMs), AdaBoost, neural networks, Naive Bayes, memory-based learning, random forests, bagged trees, boosted trees, boosted stumps, etc. In some embodiments, the training module 340 can limit the number of candidate algorithms in the pool based on available information, such as time constraints for training the model, computational resource constraints (e.g., processor limitations, memory usage limitations, etc.), the prediction problem to be solved, the characteristics of the dataset, the selected features, etc. The training module 340 can test each candidate algorithm and select the best candidate algorithm.

[0062] Training module 340 trains the model using a classification algorithm. Since multiple machine learning pipelines exist in a machine learning application, training module 340 trains multiple models.

[0063] The ranking module 350 ranks multiple trained models. In some embodiments, the ranking module 350 defines a harness associated with a performance metric (e.g., classification accuracy) to evaluate the performance of the trained models. For example, the ranking module 350 applies the trained models to a test set to quantify the accuracy of the trained models. The test set includes data different from the data used to train the models. In some embodiments, the machine learning application 300 splits the labels and feature matrices into training and test sets. The training set is provided to the training module 340 to train the models, and the test set is provided to the ranking module 350 to rank the models.

[0064] Common metrics used in accuracy measurement include: Precision = TP / (TP + FP) and Recall = TP / (TP + FN), where precision is the number of correctly predicted (TP) outcomes out of the total predicted count (TP or true positives + FP or false positives), and recall is the number of correctly predicted (TP) outcomes out of the actual total number of occurrences (TP + FN or false negatives). F-score (F-score = 2) PR / (P + R) unifies precision and recall into a single metric.

[0065] The results of testing the trained models against a test setup estimate how well the trained models perform prediction tasks against performance metrics. The ranking module 350 can determine a ranking score for each trained model, indicating the measured performance and / or accuracy of the trained model. The ranking module 350 selects one of the trained models based on the ranking, for example, the training model with the best performance.

[0066] The ranking module 350 then deploys the selected trained model so that it can be used for prediction based on new data. In some embodiments, the ranking module 350 transfers the artifact to a database in a computer system, such as a server of an organization in an industry associated with an industry problem. The artifact is the output created by the machine learning process and includes, for example, the selected trained model, other trained models, model checkpoints, features, labels, etc. The computer system also provides the selected trained model to other computer systems, whereby the selected trained model is used for prediction based on new data.

[0067] Database 360 ​​stores data associated with machine learning application 300, such as data received, used, and generated by machine learning application 300. For example, Database 360 ​​stores datasets, standard features, feature matrices, transformers, annotation times, training sets, test sets, machine learning pipelines, decisions made in each step of the machine learning pipeline, algorithms, hyperparameters, trained models, ranking scores of trained models, etc.

[0068] Figures 4A-4C This illustrates, according to one embodiment, training a model based on a dataset using an industry-specific machine learning application 300. Figure 4A In this process, dataset 410 is input into annotation module 320, and annotation module 320 outputs annotation schedule 420. Annotation schedule 420 includes annotations, each of which is associated with a deadline.

[0069] exist Figure 4B In this process, feature generation data 430 is input to feature engineering module 330, which outputs feature matrix 440. Feature generation data 430 includes some or all of the data in the dataset associated with timestamps prior to the deadline. Feature matrix 440 includes multiple features, feature values, and feature importance factors. Some of the multiple features are standard features included in machine learning application 300, provided to the user by machine learning application 300, and mapped by the user to variables in the dataset.

[0070] exist Figure 4CIn this process, feature matrix 440, labeled values ​​425 from labeling schedule 420, and machine learning pipeline 450 are input into training module 340. Machine learning pipeline 450 includes interpolator 453, scaler 455, and classifier 457. Machine learning pipeline 450 is one of several machine learning pipelines in machine learning application 300. Several machine learning pipelines are specific to industry problems. Training module 340 trains model 460 using machine learning pipeline 450: training module 340 uses interpolator 453 to detect missing values ​​and provide new values ​​for missing values; training module 340 uses scaler 455 to scale the range of feature values; and training module 340 uses classifier 457 to perform supervised machine learning.

[0071] The training module 340 also generates trained models using each of the other machine learning pipelines in the machine learning application 300. In some embodiments, those trained models are ranked based on their predictive performance, and the trained model determined to have the best performance is deployed and used for predictions based on new data.

[0072] Figure 5 This is a flowchart illustrating a method 500 for generating a model for predicting industry problems according to one embodiment. In some embodiments, the method is performed by a machine learning application 160, although in other embodiments some or all of the operations in the method may be performed by other entities. In some embodiments, the operations in the flowchart are performed in a different order and include different and / or additional steps.

[0073] Machine learning application 160 receives a dataset of 510 units used to generate a model. The dataset can be received from a client device associated with the user or from a data source, such as... Figure 1 One of the 120 data sources. The 160 machine learning applications are selected from multiple machine learning applications based on industry problems. Each of the multiple machine learning applications corresponds to a different industry problem and includes standard features specific to that industry problem and a machine learning pipeline specific to that industry problem. Example industry problems include application monitoring, service level agreement violation detection, user action prediction, etc.

[0074] In some embodiments, standard features have been previously generated and / or selected by the machine learning server 110 based on the industry problem. Standard features may be features that have proven important in the historical training of the model used to solve the industry problem, common variables in the dataset used to train the model for solving the industry problem, features that are logically related to the industry problem, or some combination thereof.

[0075] In some embodiments, the machine learning pipeline has been previously generated by the machine learning server 110 based on the domain of the industry problem. For example, the machine learning server 110 identifies the domain of the industry problem based on a description of the industry problem. This domain is associated with a business type. The machine learning server 110 selects an objective function from a plurality of objective functions based on the identified domain. Each of the plurality of objective functions is specific to the corresponding domain and is used to select the best machine learning pipeline for prediction in the corresponding domain. The machine learning server 110 then applies the objective function to a pool of machine learning pipelines to select machine learning pipelines for the appropriate machine learning applications from the pool of machine learning pipelines.

[0076] Machine learning application 160 provides 520 standard features for display to a client device associated with a user. In some embodiments, machine learning application 160 provides 520 standard features in a user interface. The user interface allows a user associated with a client device to map variables in a dataset to selected standard features in the machine learning application. In some embodiments, the user interface allows a user to map one variable in a dataset to one standard feature, multiple variables in a dataset to one standard feature, and / or one variable in a dataset to multiple standard features.

[0077] In response to the provision of standard features, machine learning application 160 receives mappings from 530 datasets on a client device to standard features selected within the machine learning application. For example, machine learning server 110 receives a mapping from the user's variable "username" to the standard feature "ID". After receiving the mapping, machine learning server 110 can convert the values ​​of the variables into new values ​​to serve as values ​​for the standard features. For example, the variable "username" includes multiple text strings representing the user's name, and the machine learning server converts these text strings into numerical values ​​to serve as the values ​​for the standard feature "ID".

[0078] Machine learning application 160 applies machine learning pipelines 540 to a dataset to train multiple models, at least based on a mapping. Each machine learning pipeline specifies steps in the training process. In some embodiments, the training process includes data imputation, feature scaling, and classification.

[0079] In some embodiments, the machine learning application 160 generates multiple features, including one or more standard features mapped to one or more variables in the dataset from a selected machine learning application, and one or more other features. The machine learning application 160 extracts one or more other features from variables in the dataset that are not mapped to any standard feature. To extract such features, the machine learning application 160 may identify variables in the dataset that are not mapped to any of the standard features in the standard feature pool, select primitives from the primitive pool based on the identified variables, and apply the primitives to those variables.

[0080] Machine learning application 160 ranks multiple trained models 550. In some embodiments, machine learning application 160 ranks trained models 570 by defining a test device associated with a performance metric (e.g., classification accuracy) and ranking the trained models based on their performance. The performance of each trained model can be measured by feeding a test set into the trained models and comparing the output of the trained models with known predictions associated with the test set.

[0081] Machine learning applications 160 to select 560 generated models from multiple trained models based on ranking. The selected trained model will be used to make predictions on new data.

[0082] Figure 6 This illustrates the use of, according to one embodiment, as Figure 1 A high-level block diagram of the functional view of a typical computer system 600, such as a machine learning server 110.

[0083] The illustrated computer system includes at least one processor 602 coupled to a chipset 604. The processor 602 may include multiple processor cores on the same die. The chipset 604 includes a memory controller center 620 and an input / output (I / O) controller center 622. Memory 606 and a graphics adapter 612 are coupled to the memory controller center 620, and a display 618 is coupled to the graphics adapter 612. Storage device 608, keyboard 610, pointing device 614, and network adapter 616 may be coupled to the I / O controller center 622. In some other embodiments, the computer system 600 may have additional, fewer, or different components, and these components may be coupled in different ways. For example, embodiments of the computer system 600 may lack a display and / or keyboard. Additionally, in some embodiments, the computer system 600 may be instantiated as a rack-mounted blade server or a cloud server instance.

[0084] Memory 606 stores instructions and data used by processor 602. In some embodiments, memory 606 is random access memory. Storage device 608 is a non-transitory computer-readable storage medium. Storage device 608 may be an HDD, SSD, or other type of non-transitory computer-readable storage medium. Data processed and analyzed by machine learning server 110 may be stored in memory 606 and / or storage device 608.

[0085] Pointing device 614 may be a mouse, trackball, or other type of pointing device, and is used in conjunction with keyboard 610 to input data into computer system 600. Graphics adapter 612 displays images and other information on display 618. In some embodiments, display 618 includes touchscreen functionality for receiving user input and selections. Network adapter 616 couples computer system 600 to network 140.

[0086] Computer system 600 is adapted to execute computer modules for providing the functions described herein. As used herein, the term "module" refers to computer program instructions and other logic for providing specified functions. Modules can be implemented in hardware, firmware, and / or software. A module may include one or more processes, and / or be provided only by a portion of those processes. Modules are typically stored on storage device 608, loaded into memory 606, and executed by processor 602.

[0087] Specific naming of components, capitalization of terms, attributes, data structures, or any other programming or structural aspects are not mandatory or important, and the mechanisms for implementing the described embodiments may have different names, formats, or protocols. Furthermore, as described, the system may be implemented via a combination of hardware and software, or entirely with hardware components. Moreover, the specific functional divisions among the various system components described herein are merely exemplary and not mandatory; a function performed by a single system component may alternatively be performed by multiple components, and a function performed by multiple components may alternatively be performed by a single component.

[0088] Some of the descriptions above present characteristics in terms of the algorithms and symbolic representations of information operations. These algorithmic descriptions and representations are the means by which those skilled in the art of data processing most effectively communicate the substance of their work to others skilled in the art. Although these operations are described functionally or logically, they should be understood as being implemented by computer programs. Furthermore, it has been shown that, without loss of generality, it is sometimes convenient to arrange these operations as modules or to refer to them by functional names.

[0089] Unless it is explicitly stated otherwise from the foregoing discussion, it should be appreciated that throughout the description, the use of terms such as “processing” or “operation” or “computation” or “determining” or “displaying” refers to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities within the computer system’s memory or registers or other such information storage, transmission or display devices.

[0090] Some embodiments described herein include processing steps and instructions described in algorithmic form. It should be noted that the processing steps and instructions of the embodiments may be embodied in software, firmware, or hardware, and when embodied in software, may be downloaded to reside on and operate from different platforms used by a real-time network operating system.

[0091] Finally, it should be noted that the language used in this specification has been chosen primarily for readability and guidance purposes, and may not have been chosen to describe or limit the subject matter of the invention. Therefore, the disclosure of embodiments is intended to be illustrative rather than restrictive.

Claims

1. A computer-implemented method for generating a model for predicting industry-specific problems, comprising: The dataset used to generate the model is received through a machine learning application, which is selected from multiple machine learning applications based on the industry problem. Each of the multiple machine learning applications corresponds to a different industry problem and includes standard features specific to the corresponding industry problem and a machine learning pipeline specific to the corresponding industry problem. The standard features in the machine learning application are provided for display in a user interface on a client device associated with the user, the user interface allowing the user to map variables in the dataset to standard features in the machine learning application; In response to providing the standard features, the client device receives a mapping from variables in the dataset to standard features in a selected machine learning application; Applying the machine learning pipeline of the selected machine learning application to the dataset to train multiple models based at least on the mapping, wherein applying the machine learning pipeline of the selected machine learning application to the dataset to train the multiple models includes: Generate multiple features including one or more standard features of one or more variables in the selected machine learning application that are mapped to the dataset, and one or more other features, wherein the one or more other features are extracted from one or more other variables in the dataset that are not mapped to any of the standard features of the selected machine learning application; The multiple features that generate include the one or more standard features of the one or more variables mapped to the dataset and the one or more other features include: Identify variables in the dataset that are not mapped to any of the standard features in the selected machine learning application; Primitives are selected from a primitive pool based on the identified variables, the primitives including functions for transforming the variables into features; and The primitive is applied to the variable to synthesize one of the other features among the one or more other features; Ranking multiple trained models; and The generated model is selected from the plurality of trained models based on the ranking.

2. The computer-implemented method according to claim 1 further includes: Identify the semantic representation of the dataset; as well as The primitive is selected from the primitive pool based on the semantic representation of the dataset having at least one attribute that matches the primitive.

3. The computer-implemented method according to claim 1 further includes: A representation vector is generated for the dataset, and the representation vector encodes the data describing the dataset; as well as The primitives are selected from the primitive pool by inputting the representation vector into a machine learning model, wherein the machine learning model is trained to output one or more primitives given an input vector.

4. The computer-implemented method according to claim 3, wherein, The data description of the dataset encoded in the representation vector is at least one of the following: the number of tables in the dataset, the number of columns in each table in the dataset, the average number of columns in one or more tables in the dataset, or the average number of rows in one or more tables in the dataset.

5. The computer-implemented method according to claim 1, further comprising: The annotation function is applied to the dataset to generate annotation times, each annotation time including an annotation and a deadline associated with the annotation.

6. The computer-implemented method according to claim 5, wherein, The annotation function is included in the selected machine learning application and is specific to the industry problem corresponding to the selected machine learning application.

7. The computer-implemented method according to claim 5, wherein, The annotation function includes customizable parameters and also includes: Receive values ​​of the customizable parameters from the client device, the values ​​being predictions specific to the industry problem; and The annotation function is customized based on the received values.

8. The computer-implemented method according to claim 3 or claim 4, wherein, Generating the representation vector for the dataset includes applying a hash function to the dataset or applying a Rabin fingerprint algorithm to the dataset.

9. The computer-implemented method according to any one of claims 1 to 8, comprising: Generate a feature matrix, the feature matrix including at least one synthetic feature and at least one standard feature of the plurality of standard features which are mapped to at least one of the plurality of variables via input.

10. The computer-implemented method according to claim 9, wherein, Generating the feature matrix includes: An absolute ranking score is determined for each of the at least one synthetic feature and at least one standard feature of the plurality of standard features that are mapped to at least one of the plurality of variables via the input; Based on the corresponding absolute ranking scores, a subset of features is selected from the at least one synthetic feature and at least one standard feature from the plurality of standard features mapped to at least one variable of the plurality of variables via the input; and Generate the feature matrix to include a subset of the features.

11. The computer-implemented method according to claim 10, wherein, The determination of the absolute ranking score for each of the at least one synthetic feature and at least one of the at least one standard feature of the plurality of variables mapped to at least one of the plurality of variables via the input is performed using the Goodman-Kruskal Tau metric.

12. The computer-implemented method according to claim 10, further comprising: An importance factor is determined for each of the at least one synthetic feature and at least one of the at least one standard feature that is mapped to at least one of the plurality of variables via the input, wherein the selection of a subset of the features is further performed based on the corresponding importance factor.

13. The computer-implemented method according to claim 12, wherein, The determination of the importance factor for each of the at least one synthetic feature and at least one of the at least one standard feature of the plurality of variables mapped to at least one of the plurality of variables via the input is performed using a transformer.

14. A non-transitory computer-readable storage medium for storing executable computer program instructions, said instructions being executable to perform operations for generating a model for making predictions for an industry problem, said operations including: The dataset used to generate the model is received through a machine learning application, which is selected from multiple machine learning applications based on the industry problem. Each of the multiple machine learning applications corresponds to a different industry problem and includes standard features specific to the corresponding industry problem and a machine learning pipeline specific to the corresponding industry problem. The standard features in the machine learning application are provided for display in a user interface on a client device associated with the user, the user interface allowing the user to map variables in the dataset to standard features in the machine learning application; In response to providing the standard features, the client device receives a mapping from variables in the dataset to standard features in a selected machine learning application; Applying the machine learning pipeline of the selected machine learning application to the dataset to train multiple models at least based on the mapping, wherein applying the machine learning pipeline of the selected machine learning application to the dataset to train the multiple models at least based on the mapping includes: Generate multiple features including one or more standard features of one or more variables in the selected machine learning application that are mapped to the dataset, and one or more other features, wherein the one or more other features are extracted from one or more other variables in the dataset that are not mapped to any of the standard features of the selected machine learning application; The multiple features that generate include the one or more standard features of the one or more variables mapped to the dataset and the one or more other features include: Identify variables in the dataset that are not mapped to any of the standard features in the selected machine learning application; Primitives are selected from a primitive pool based on the identified variables, the primitives including functions for transforming the variables into features; and The primitive is applied to the variable to synthesize one of the other features among the one or more other features; Ranking multiple trained models; and The generated model is selected from the plurality of trained models based on the ranking.

15. The non-transitory computer-readable storage medium of claim 14, further comprising: Generate a feature matrix, the feature matrix including at least one synthetic feature and at least one standard feature of the plurality of standard features which are mapped to at least one of the plurality of variables via input.

16. The non-transitory computer-readable storage medium according to claim 15, wherein, Generating the feature matrix includes: An absolute ranking score is determined for each of the at least one synthetic feature and at least one standard feature of the plurality of standard features that are mapped to at least one of the plurality of variables via the input; Based on the corresponding absolute ranking scores, a subset of features is selected from the at least one synthetic feature and at least one standard feature from the plurality of standard features mapped to at least one variable of the plurality of variables via the input; and Generate the feature matrix to include a subset of the features.

17. The non-transitory computer-readable storage medium according to claim 16, wherein, The determination of the absolute ranking score for each of the at least one synthetic feature and at least one of the at least one standard feature of the plurality of variables mapped to at least one of the plurality of variables via the input is performed using the Goodman-Kruskal Tau metric.

18. The non-transitory computer-readable storage medium of claim 16, further comprising: For each of the selected features, an importance factor is determined.

19. The non-transitory computer-readable storage medium according to claim 18, wherein, The determination of the importance factor for each of the selected features is performed using a transformer.

20. The non-transitory computer-readable storage medium of claim 15, wherein the operation further comprises: Identify the semantic representation of the dataset; as well as The primitive is selected from the primitive pool based on the semantic representation of the dataset having at least one attribute that matches the primitive.

21. The non-transitory computer-readable storage medium according to any one of claims 14 to 20, wherein the operation further comprises: A representation vector is generated for the dataset, and the representation vector encodes the data describing the dataset; as well as The primitives are selected from the primitive pool by inputting the representation vector into a machine learning model, wherein the machine learning model is trained to output one or more primitives given an input vector.

22. The non-transitory computer-readable storage medium according to claim 21, wherein, The data description of the dataset encoded in the representation vector is at least one of the following: the number of tables in the dataset, the number of columns in each table in the dataset, the average number of columns in one or more tables in the dataset, or the average number of rows in one or more tables in the dataset.

23. The non-transitory computer-readable storage medium according to claim 21, wherein, Generating the representation vector for the dataset includes applying a hash function to the dataset or applying a Rabin fingerprint algorithm to the dataset.

24. A system comprising: A computer processor is used to execute computer program instructions; as well as A non-transitory computer-readable storage device storing computer program instructions executable by the computer processor to perform operations for generating models for predicting industry problems, the operations including: The dataset used to generate the model is received through a machine learning application, which is selected from multiple machine learning applications based on the industry problem. Each of the multiple machine learning applications corresponds to a different industry problem and includes standard features specific to the corresponding industry problem and a machine learning pipeline specific to the corresponding industry problem. The standard features in the machine learning application are provided for display in a user interface on a client device associated with the user, the user interface allowing the user to map variables in the dataset to standard features in the machine learning application; In response to providing the standard features, the client device receives a mapping from variables in the dataset to standard features in a selected machine learning application; Applying the machine learning pipeline of the selected machine learning application to the dataset to train multiple models at least based on the mapping, wherein applying the machine learning pipeline of the selected machine learning application to the dataset to train the multiple models at least based on the mapping includes: Generate multiple features including one or more standard features of one or more variables in the selected machine learning application that are mapped to the dataset, and one or more other features, wherein the one or more other features are extracted from one or more other variables in the dataset that are not mapped to any of the standard features of the selected machine learning application; The multiple features that generate include the one or more standard features of the one or more variables mapped to the dataset and the one or more other features include: Identify variables in the dataset that are not mapped to any of the standard features in the selected machine learning application; Primitives are selected from a primitive pool based on the identified variables, the primitives including functions for transforming the variables into features; and The primitive is applied to the variable to synthesize one of the other features among the one or more other features; Ranking multiple trained models; and The generated model is selected from the plurality of trained models based on the ranking.

25. The system according to claim 24, wherein, The operation also includes: Identify the semantic representation of the dataset; and The primitive is selected from the primitive pool based on the semantic representation of the dataset having at least one attribute that matches the primitive.

26. The system according to claim 24, wherein, The operation also includes: Generate representation vectors for the dataset, the representation vectors encoding the data describing the dataset; and The primitives are selected from the primitive pool by inputting the representation vector into a machine learning model, wherein the machine learning model is trained to output one or more primitives given an input vector.

27. The system according to claim 26, wherein, Generating the representation vector for the dataset includes applying a hash function to the dataset or applying a Rabin fingerprint algorithm to the dataset.

28. The system according to claim 24, wherein, The operation also includes: Generate a feature matrix, the feature matrix including at least one synthetic feature and at least one standard feature of the plurality of standard features which are mapped to at least one of the plurality of variables via input.

29. The system according to claim 28, wherein, Generating the feature matrix, which includes the at least one synthetic feature and at least one standard feature from the plurality of standard features mapped to at least one variable from the plurality of variables via the input, comprises: An absolute ranking score is determined for each of the at least one synthetic feature and at least one standard feature of the plurality of standard features that are mapped to at least one of the plurality of variables via the input; Based on the corresponding absolute ranking scores, a subset of features is selected from the at least one synthetic feature and at least one standard feature from the plurality of standard features mapped to at least one variable of the plurality of variables via the input; and Generate the feature matrix to include a subset of the features.

30. The system according to claim 29, wherein, The determination of the absolute ranking score for each of the at least one synthetic feature and at least one of the at least one standard feature of the plurality of variables mapped to at least one of the plurality of variables via the input is performed using the Goodman-Kruskal Tau metric.

31. The system according to claim 29, further comprising: An importance factor is determined for each of the at least one synthetic feature and at least one of the at least one standard feature that is mapped to at least one of the plurality of variables via the input, wherein the selection of a subset of the features is further performed based on the corresponding importance factor.

Images