Machine learning feature recommendation

Abstract

To provide methods, systems and storage media for recommending eligible machine learning features.SOLUTION: A method comprises: receiving a command to initiate creation of a machine learning model via a user interface; receiving specification of a target field included in one or more database tables storing multiple sets of machine learning training data; automatically identifying, within the one or more database tables, one or more eligible machine learning features for predicting the target field; automatically evaluating a level of impact of each of the eligible machine learning features on prediction accuracy of the machine learning model; creating a set of recommended machine learning features among the one or more eligible machine learning features; ranking the set of recommended machine learning features on the basis of the level of impact of each feature on the prediction accuracy of the machine learning model; and receiving input for identifying one or more selected machine learning features.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] The use of automatic classification using machine learning can significantly reduce manual effort and errors compared to manual classification. One method of performing automatic classification includes using machine learning to predict categories for input data. For example, using machine learning, incoming tasks, incidents, and cases can be automatically categorized and routed to assigned parties. Generally, automatic classification using machine learning requires training data that includes past experience. Once trained, the machine learning model can be applied to new data to infer classification results. For example, newly reported incidents can be automatically categorized, assigned, and routed to personnel. However, creating an accurate machine learning model can be a difficult and time-consuming task that requires a significant investment and typically requires subject matter expertise. For example, selecting input features that result in an accurate model typically requires a deep understanding of the dataset and how the features affect the predicted outcome. [Brief description of the drawings]

[0002] Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.

[0003] [Figure 1] FIG. 1 is a block diagram illustrating an example of a network environment for creating and utilizing machine learning models.

[0004] [Diagram 2] 1 is a flow chart illustrating an embodiment of a process for generating a machine learning solution.

[0005] [Diagram 3] 1 is a flow chart illustrating one embodiment of a process for automatically identifying recommended features for a machine learning model.

[0006] [Figure 4]1 is a flow chart illustrating one embodiment of a process for automatically identifying recommended features for a machine learning model.

[0007] [Diagram 5] 1 is a flow chart illustrating one embodiment of an evaluation process for automatically identifying recommended features for a machine learning model.

[0008] [Figure 6] 1 is a flow chart illustrating an embodiment of a process for creating an offline model for determining a performance metric for a feature. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present invention can be implemented in many aspects, including processes, apparatus, systems, compositions, computer program products embodied on a computer-readable storage medium, and / or processors configured to execute instructions stored on and / or provided by a memory coupled to the processor. In this specification, these embodiments, or any other form the present invention may take, may be referred to as techniques. In general, the order of steps of a disclosed process may be changed within the scope of the present invention. Unless otherwise stated, a component such as a processor or memory described as being configured to perform a task may be implemented as a general-purpose component temporarily configured to perform that task at a given time, or as a specific component manufactured to perform that task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0010] A detailed description of one or more embodiments of the present invention is provided below, along with the accompanying drawings illustrating the principles of the present invention. Although the present invention is described in relation to such embodiments, the present invention is not limited to any embodiment. The scope of the present invention is limited only by the claims, and the present invention encompasses many alternatives, modifications, and equivalents. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. These details are provided for illustrative purposes, and the present invention may be practiced according to the claims without some or all of these specific details. For purposes of clarity, technical materials known in the art related to the present invention have not been described in detail so as not to unnecessarily obscure the present invention.

[0011] Techniques for selecting machine learning features are disclosed. When building a machine learning model, feature selection can have a significant impact on the accuracy and usefulness of the model. However, properly selecting features that improve the accuracy of the model without subject matter expertise and a deep understanding of the machine learning problem can be a challenge. With the disclosed techniques, machine learning features that result in significant improvements in the predictive accuracy of the machine learning model can be automatically recommended and selected. Furthermore, little or no subject matter expertise is required. For example, a user with minimal understanding of the input dataset can successfully generate a machine learning model that can accurately predict classification results. In some embodiments, a user can utilize the machine learning platform through a software service, such as a software-as-a-service web application. The user provides an input dataset to the machine learning platform, for example, identifying one or more database tables. The provided dataset includes a plurality of qualified features. The qualified features can include features that are useful in accurately predicting machine learning results and features that are useless or have a small impact on accurately predicting machine learning results. Accurately identifying useful features can result in highly accurate models and improve resource usage and performance. For example, training a model with useless features can be a significant resource drain that can be avoided by accurately identifying and ignoring useless features. In various embodiments, a user specifies a desired target field for prediction, and a machine learning platform using the disclosed technology can generate a set of recommended machine learning features from the provided input dataset for use in building a machine learning model. In some embodiments, the recommended machine learning features are determined by applying a series of ratings to the eligible features to filter useless features and identify useful features. After the set of recommended features is determined, it can be presented to the user. For example, in some embodiments, the features are ranked in order of improvement to the prediction outcome.In some embodiments, the machine learning model is trained with features selected by a user based on the recommended features, for example, the model may be automatically trained with recommended features that are automatically identified and ranked in order of improvement to the prediction outcome.

[0012] In some embodiments, a designation of a desired target field for machine learning prediction and one or more tables for storing machine learning training data are received. For example, a customer of a software-as-a-service platform specifies one or more customer database tables. The tables may include data from past experience such as categorized incoming tasks, incidents, and cases. For example, the classification may include categorizing the type of task, incident, or case and assigning an appropriate person to resolve the issue. In some embodiments, the machine learning data is stored in other suitable data structures other than a database. In various embodiments, the desired target field is a classification result, which may be a column in one of the received tables. Because the data in the received database table is not necessarily prepared as training data, the data may include both useful and useless fields for predicting a classification result. In some embodiments, eligible machine learning features are identified in one or more tables for building a machine learning model to perform predictions on the desired target field. For example, from the database data, fields are identified as possible or eligible features for training a machine learning model. In some embodiments, the eligible features are based on columns of the table. The qualified machine learning features are evaluated using a pipeline of different evaluations to sequentially filter out one or more of the qualified machine learning features and identify a set of recommended machine learning features among the qualified machine learning features. By sequentially filtering out features from the qualified features, features with less impact on the model prediction accuracy are selected. The remaining features are the recommended features that have predictive value. Each step of the filtering pipeline identifies additional features that are not useful (and features that could be useful). For example, in some embodiments, one filtering step removes features for which the feature data is unnecessary or out of range.Features that are sparsely registered in the respective database tables or features where all of the values ​​of the feature are identical (e.g., constant) may be filtered out. In some embodiments, non-nominal columns are filtered out. In some embodiments, the filtering step calculates an impact score for each eligible feature. Features with an impact score below a certain threshold may be removed from the recommendations. In some embodiments, a performance metric is evaluated for each eligible feature. For example, the increase in the area under the precision-recall curve (AUPRC) of the model may be evaluated for a particular feature. In some embodiments, a model is trained offline to convert the impact score into a performance metric by evaluating feature selections on a large cross-section of machine learning problems. The model may then be applied to a particular customer's machine learning problem to determine a performance metric that may be used to rank the eligible features. Once identified, a set of recommended machine learning features is provided for use in building a machine learning model. For example, a customer may select a machine learning model from the recommended features and request that the machine learning model be trained with the provided data and the selected features. The model may then be integrated into the customer's workflow to predict the desired target field. For example, with little or no subject matter expertise in both datasets and machine learning, features can be automatically recommended (and selected) for a machine learning model that can be used to infer a target field.

[0013] FIG. 1 is a block diagram illustrating an example of a network environment for creating and utilizing machine learning models. In the illustrated example, clients 101, 103, and 105 access services on server 121 via network 111. The services include prediction services that utilize machine learning. For example, the services may include both the ability to generate machine learning models using recommended features and services for applying the generated models to predict outcomes, such as classification outcomes. Network 111 may be a public or private network. In some embodiments, network 111 is a public network, such as the Internet. In various embodiments, clients 101, 103, and 105 are network clients, such as web browsers, for accessing services provided by server 121. In some embodiments, server 121 provides services including web applications for utilizing a machine learning platform. Server 121 may provide one or more servers, including a server for identifying recommended features for training machine learning models. Server 121 may utilize database 123 for providing certain services and / or for storing data associated with users. For example, database 123 may be a configuration management database (CMDB) used by server 121 to provide customer service and store customer data. In some embodiments, database 123 stores customer data related to tasks, incidents, cases, and the like. Database 123 may also be used to store information related to feature selection for training machine learning models. In some embodiments, database 123 may store customer configuration information related to managed assets, such as associated hardware and / or software configurations.

[0014] In some embodiments, each of clients 101, 103, and 105 may access server 121 to create a customer machine learning model. For example, clients 101, 103, and 105 may represent one or more different customers each desiring to create a machine learning model that can be applied to predict outcomes. In some embodiments, server 121 provides clients, such as clients 101, 103, and 105, with interactive tools to select and / or confirm feature selections for training the machine learning model. For example, a customer of a software-as-a-service platform, via a client, such as clients 101, 103, and 105, provides relevant training data, such as customer data, to server 121 as training data. The provided customer data may be data stored in one or more tables of database 123. Along with the provided training data, the customer selects a desired target field, such as one of the table columns of the provided table. Using the provided data and the desired target field, server 121 recommends a set of features that predict the desired target field with a high degree of accuracy. A customer can select a subset of recommended features from which to train a machine learning model. In some embodiments, the model is trained using the provided customer data. In some embodiments, as part of the feature selection process, the customer is provided with a performance metric for each recommended feature. The performance metric provides the customer with a quantified value for how much a particular feature improves the predictive accuracy of the model. In some embodiments, the recommended features are ranked based on their impact on predictive accuracy.

[0015] In some embodiments, the trained machine learning model is incorporated into an application for inferring a desired target field. For example, the application may receive incoming reports of support incident events and predict categories for the incidents and / or assign the reported incident events to personnel. The support incident application may be hosted by server 121 and accessed by clients, such as clients 101, 103, and 105. In some embodiments, each of clients 101, 103, and 105 may be a network client running on one of many different computing devices, including laptops, desktops, mobile devices, tablets, kiosks, smart TVs, and the like.

[0016] Although single instances of some components are shown to simplify the diagram, there may be additional instances of any component shown in FIG. 1. For example, server 121 may include one or more servers. Some of the servers of server 121 may be web application servers, training servers, and / or interference servers. As shown in FIG. 1, the servers are simplified as a single server 121. Similarly, database 123 may not be directly connected to server 121, may be multiple databases, and / or may be replicated or distributed across multiple components. For example, database 123 may include one or more different servers for each customer. As another example, clients 101, 103, and 105 are just a few examples of possible clients for server 121. Fewer or more clients may be connected to server 121. In some embodiments, components not shown in FIG. 1 may also be present.

[0017] FIG. 2 is a flow chart illustrating one embodiment of a process for creating a machine learning solution. For example, using the process of FIG. 2, a user may request a machine learning solution to a problem. The user may identify a desired target field for prediction and provide a reference to data that may be used as training data. The provided data is analyzed and input features are recommended for training a machine learning model. The recommended features may be provided to the user and a machine learning model may be trained based on the features selected by the user. The trained model is incorporated into the machine learning solution to predict the user's desired target field. In some embodiments, the machine learning platform for creating the machine learning solution is hosted as a software-as-a-service web application. In some embodiments, a user requests a solution via a client, such as clients 101, 103, and / or 105 of FIG. 1. In some embodiments, the machine learning platform including the created machine learning solution is hosted on server 121 of FIG. 1.

[0018] At 201, a machine learning solution is requested. For example, a customer may want to use the machine learning solution to automatically predict assignees for incoming support incident incident reports. In some embodiments, a user requests the machine learning solution via a web application. When requesting a solution, the user may specify target fields the user would like predicted and provide related training data. In some embodiments, the training data provided is historical customer data. The customer data may be stored in a customer database. In some embodiments, the user provides one or more database tables as training data. The database tables may also include the desired target fields. In some embodiments, the user specifies multiple target fields. If predictions for multiple fields are desired, the user may specify multiple fields together and / or request multiple different machine learning solutions. In some embodiments, the user also specifies other properties of the machine learning solution, such as the processing language, stop words, filters for the provided data, and the name and description of the desired model, among others.

[0019] At 203, recommended input features are determined. For example, a set of eligible machine learning features based on the requested machine learning solution is determined. From the eligible features, a set of recommended features is identified. In some embodiments, the recommended features are identified by evaluating the eligible machine learning features using different evaluation pipelines. At each stage of the pipeline, one or more of the eligible machine learning features may be sequentially filtered out. At the end of the pipeline, a set of recommended features is identified. In some embodiments, identifying the recommended features includes determining one or more metrics associated with the features, such as an impact score or a performance metric. For example, an offline trained model may be applied to each feature to determine a performance metric that quantifies how much the feature increases the area under the precision-recall curve (AUPRC) of a model trained with the feature. In some embodiments, an appropriate threshold may be utilized for each metric to determine whether the feature is recommended for use during training.

[0020] In some embodiments, the qualified machine learning features are based on input data provided by a user. For example, in some embodiments, a user provides one or more database tables or other suitable data structures as training data. If a database table is provided, the qualified machine learning features may be based on columns of the table. In some embodiments, the data type of each column is determined, and columns having a nominal data type are identified as qualified features. In some embodiments, data from a particular column may be eliminated if the column data is unlikely to be useful for prediction. For example, columns may be removed based on how sparsely registered the data is, the occurrence of stop words, the relative distribution of different values ​​for the column, etc.

[0021] At 205, features are selected based on the recommended input features. For example, using an interactive user interface, a set of recommended machine learning features for use in building the machine learning model is presented to the user. In some embodiments, the exemplary user interface is implemented as a web application or web service. The user can select from the displayed recommended features to determine a set of features to use to train the machine learning model. In some embodiments, the recommended input features determined at 203 are automatically selected as default features for training. No user input may be required to select the recommended input features. In some embodiments, the recommended input features may be presented in a ranked order based on how each affects the predictive accuracy of the model. For example, the most relevant input features are ranked first. In various embodiments, the recommended features are displayed with an impact score and / or a performance metric. For example, the impact score may measure how much impact the feature has on the model accuracy. The performance metric may quantify how much the model improves if the feature is used for training. For example, in some embodiments, the displayed performance metric is based on the increase in the area under the precision-recall curve (AUPRC) of the machine learning model when the feature is used. Other performance metrics may be used as appropriate. By ranking and quantifying different features, a user with little or no subject matter expertise can easily select appropriate input features to train a highly accurate model.

[0022] At 207, a machine learning model is trained using the selected features. For example, using the features selected at 205, a training dataset is prepared and used to train the machine learning model. The model predicts the desired target field specified at 201. In some embodiments, the training data is based on the customer data received at 201. Data that is not useful for training is removed from the customer data, such as data from table columns that correspond to features not selected at 205. For example, data corresponding to columns associated with features identified as having little or no impact on the accuracy of the prediction is excluded from the dataset used to train the machine learning model.

[0023] At 209, the machine learning solution is hosted. For example, an application server and machine learning platform hosts a service for applying the trained machine learning model to input data. For example, a web service applies the trained model to automatically categorize incoming incident reports. Categorization may include identifying the type of incident and the assignee. Once classified, the hosted solution can assign and route the incident to the predicted assignee. In some embodiments, the hosted application is a customer machine learning solution for a customer of the software-as-a-service platform. In some embodiments, the solution is hosted on server 121 of FIG. 1.

[0024] FIG. 3 is a flow chart illustrating one embodiment of a process for automatically identifying recommended features for a machine learning model. Using the process of FIG. 3, a user can automate the creation of a machine learning model by utilizing recommended features identified from potential training data. The user specifies a desired target field and provides potential training data. The machine learning platform identifies recommended fields for creating a machine learning model from the provided data to predict the desired target field. In some embodiments, the process of FIG. 3 is performed at 201 of FIG. 2. In some embodiments, the process of FIG. 3 is performed on a machine learning platform at server 121 of FIG. 1.

[0025] At 301, model creation is initiated. For example, a customer initiates the creation of a machine learning model via a web service application. In some embodiments, the customer initiates model creation by accessing a model creation webpage via a software-as-a-service platform for creating automated workflows. The service may be part of a larger machine learning platform that allows users to incorporate trained models to predict outcomes. In some embodiments, the predicted outcomes may be used to automate workflow processes, such as forwarding an incident report to an assigned party once the appropriate party is automatically predicted using the trained model.

[0026] At 303, training data is identified. For example, a user designates data as possible training data. In some embodiments, the user indicates one or more database tables from a customer database or other suitable data structure that store the possible training data. The data may be historical customer data. For example, the historical customer data may include incoming incident reports and their assigned personnel stored in one or more database tables. In some embodiments, the identified training data may include a large number of possible input features and may not be properly prepared as high quality training data. For example, a particular column of data may be sparsely populated or contain only the same constant values. As another example, the data type of a column may be improperly configured. For example, nominal or numeric data values ​​may be stored as text in the identified database table. In various embodiments, the identified training data needs to be cleaned before it can be effectively used as training data. For example, data from one or more columns that have little or no impact on the model prediction accuracy is removed.

[0027] At 305, a desired target field is selected. For example, a user specifies a desired target field for machine learning prediction. In some embodiments, the user selects a column field from the data identified at 303. For example, the user may select a categorical type for incident reports to indicate the user's desire to create a machine learning model to predict the categorical type of incoming incident reports. In some embodiments, the user may select from possible input features of the training data provided at 303. In some embodiments, the user selects multiple desired target fields to be predicted together.

[0028] At 307, model configuration is complete. For example, the user may provide additional configuration options such as a name and description for the model. In some embodiments, the user may specify optional stop words. For example, the stop words may be provided to prepare the training data. In some embodiments, the stop words are removed from the provided data. In some embodiments, the user may specify a processing language and / or additional filters for the provided data. For example, stop words for the specified language may be added by default or suggested. With respect to the specified additional filters, condition filters may be applied to create a representative data set from the training data identified at 303. In some embodiments, rows of the provided table may be removed from the training data by applying one or more specified condition filters. For example, the table may include a "Status" column, the possible values ​​of which may be "New", "In Progress", "Pending", and "Resolved". A condition may be specified to utilize as training data only rows in which the "Status" field has the value "Resolved". As another example, a condition may be specified to utilize as training data only rows created after a specified date or time frame.

[0029] FIG. 4 is a flow chart illustrating one embodiment of a process for automatically identifying recommended features for a machine learning model. For example, using the feature selection pipeline of FIG. 4, eligible features of a dataset can be evaluated in real time to determine how much each possible feature will impact the machine learning model for predicting the desired target field. In various embodiments, a set of recommended features can be determined and selected for training the machine learning model. The recommended features are selected based on their accuracy in predicting the desired target field. For example, useless features are not recommended. In some embodiments, the process of FIG. 4 is performed at 203 of FIG. 2. In some embodiments, the process of FIG. 4 is performed on a machine learning platform at server 121 of FIG. 1.

[0030] At 401, data is retrieved from a database table. For example, possible training data sets stored in one or more identified database tables are identified by a user, and relevant data is retrieved. In some embodiments, a condition filter is applied to the relevant data before (or after) the data is retrieved. For example, based on the condition filter, only certain rows of the database table may be retrieved. As another example, stop words are removed from the retrieved data. In some embodiments, the data is retrieved from the identified tables to a machine learning training server.

[0031] At 403, column data types are identified. For example, the data type of each column of data is identified. In some embodiments, the column data types configured in the database table are not specific enough to be used to evaluate the associated features. For example, nominal values ​​may be stored as text or binary large object (BLOB) values ​​in the database table. As another example, numeric or date types may also be stored as text (or string) data types. In various embodiments, at 403, the column data types are identified automatically without user intervention.

[0032] In some embodiments, the data type is identified by first scanning all the different values ​​of a column and analyzing the scanned results. Properties of the column may be utilized to determine the valid data type of the column's values. For example, text data may be identified at least in part by the number of spaces and the amount of text length variation in the column field. As another example, if there is little or no variation in the actual values ​​stored in the column field, the column data type may be determined to be a nominal data type. For example, a column having five discrete values ​​but values ​​stored as string values ​​may be identified as a nominal type. In some embodiments, the distribution of the value types is used as a factor in identifying the data type. For example, if a high percentage of the values ​​in a column are numeric, the column may be classified as a numeric data type.

[0033] At 405, preprocessing is performed on the data columns. In some embodiments, a set of preprocessing rules is applied to remove useless columns. For example, columns with sparsely registered fields are removed. In some embodiments, a threshold is utilized to determine if a column is sparsely registered and a candidate for removal. For example, in some embodiments, a threshold of 20% is used. Columns with less than 20% of the data registered are useless columns and may be removed. As another example, columns where all values ​​are constants are removed. In some embodiments, columns where one value dominates over other values ​​are removed, for example, if the dominant value appears in more than 80% (or other threshold amount) of the records. Columns where every value is unique or an ID may also be removed. In some embodiments, non-nominal columns are removed. For example, columns with binary data or text strings may be removed. In various embodiments, the preprocessing step removes only a subset of all eligible features from consideration as recommended input features.

[0034] At 407, the qualified machine learning features are evaluated. For example, the qualified machine learning features are evaluated for their impact on training an accurate machine learning model. In some embodiments, the qualified machine learning features are evaluated using an evaluation pipeline to sequentially filter out features by their usefulness in predicting the desired target value. For example, in some embodiments, the first evaluation step can determine an impact score, such as a relief score, to identify the distinction the column brings to the classification model. Columns with a relief score below a threshold may be removed from the recommendations. As another example, in some embodiments, the second evaluation step can determine an impact score, such as an information gain or weighted information gain, for the column. With the selected features and the desired target field, the impact score can be determined by comparing the improvement of the feature by using the change in information entropy when considering the features. Columns with an information gain or weighted information gain score below a threshold may be removed from the recommendations. In some embodiments, a third evaluation set can determine a performance metric for each feature. For example, a model is created offline to convert an impact score, such as an information gain or weighted information gain score, into a performance metric, such as one based on the increase to the area under the precision-recall curve (AUPRC) for the model. In various embodiments, the trained model is applied to the impact scores to determine an AUPRC-based performance metric for each remaining eligible feature. Using the determined performance metric, columns whose performance metric is below a threshold may be removed from recommendation. Although three evaluation steps are described above, fewer or additional steps may be utilized as appropriate based on the desired outcome for the set of features to be recommended. For example, one or more different evaluation techniques may be applied in addition to or to replace the described evaluation steps to further reduce the number of eligible features.

[0035] In various embodiments, a set of recommended machine learning features for building a machine learning model is identified by applying sequential evaluation steps. In some embodiments, sequential evaluation steps are necessary to determine which features result in accurate models. Any one evaluation step alone may be insufficient and inaccurately identify bad features for training for recommendation. For example, a feature may have a high relief score but a low weighted information gain score. A low weighted information gain score indicates that the feature should not be used for training. In some embodiments, a key or similar identifier column is a bad feature for training because it has little predictive value. The column may have a high impact score when evaluated under one of the evaluation steps, but is filtered from being recommended by a subsequent evaluation step.

[0036] At 409, recommended features are provided. For example, the remaining features are recommended as input features. In some embodiments, the set of recommended features is provided to the user via a graphical user interface of a web application. The recommended features may be provided with quantified metrics regarding how much impact each of the features has on the model accuracy. In some embodiments, the features are provided in ranked order, allowing the user to select the most impactful features to train the machine learning model.

[0037] In some embodiments, useless features are also provided along with the recommended features. For example, the user is provided with a set of features that are identified as useless or as having little impact on the model accuracy. This information can help the user gain a better understanding of the machine learning problem and solution.

[0038] FIG. 5 is a flow chart illustrating one embodiment of an evaluation process for automatically identifying recommended features for a machine learning model. In some embodiments, the evaluation process is a multi-stage process for sequentially filtering out features from the qualified machine learning features to identify a set of recommended machine learning features. The process may be performed in real time, utilizing data provided as possible training data from which qualified machine learning features are identified. Although described with particular evaluation steps with respect to FIG. 5, alternative embodiments of the evaluation process may utilize fewer or more evaluation steps and incorporate different evaluation techniques. In some embodiments, the process of FIG. 5 is performed at 203 of FIG. 2 and / or at 407 of FIG. 4. In some embodiments, the process of FIG. 5 is performed on a machine learning platform at server 121 of FIG. 1.

[0039] At 501, features are evaluated using the determined relief scores. In various embodiments, an impact score using a relief-based technique is determined at 501 and used to filter one or more eligible machine learning features to identify a set of machine learning features to be recommended. For example, an impact score based on the relief score for each feature is determined. Columns with a relief score below a threshold may be removed from recommendations. In some embodiments, the relief score corresponds to the impact the column has in differentiating different classification results. In various embodiments, for each feature, multiple adjacent rows are selected. The rows are selected based on having similar values ​​(or mathematically close or proximate values) except for the value for the column currently being evaluated. For example, for a table with three columns A, B, and C, column A is evaluated by selecting rows that have similar values ​​for corresponding columns B and C (i.e., values ​​for column B are similar for all selected rows and values ​​for column C are similar for all selected rows). This impact score utilizes the selected rows to determine how much column A impacts the desired target field. In an example, the target field may correspond to one of columns B or C. Using the selected adjacent rows, an impact or relief score is calculated for each eligible feature. The scores may be normalized and compared to a threshold. Features whose relief scores are below the threshold may be identified as useless columns and removed from further consideration as recommended input features. Features whose relief scores meet the threshold are further evaluated for consideration as recommended input features at 503. In some embodiments, the eligible features are ranked by the determined relief scores, and a feature may be removed from consideration as a recommended input feature if the feature is not ranked high enough. For example, in some embodiments, a maximum number of features based on the ranking (such as the top 10 or top 10% of eligible features) are retained for further evaluation at 503.

[0040] At 503, the features are evaluated using weighted information scores. In various embodiments, an impact score using information gain techniques is determined at 503 and used to filter one or more eligible machine learning features to identify a set of machine learning features to be recommended. For example, an impact score based on the weighted information gain score for each feature is determined. Columns with a weighted information gain score below a threshold may be removed from the recommendations. In some embodiments, the weighted information gain score of a feature corresponds to the change in information entropy when the value of the feature is known. The weighted information gain score is an information gain metric, weighted by a target distribution of different known values ​​for the feature. In some embodiments, the weights are proportional to the frequency of a given target value. In some embodiments, a non-weighted information score may be used as an alternative impact score.

[0041] In various embodiments, the eligible features are ranked by the determined weighted information gain scores, and a feature may be removed from consideration as a recommended input feature if it is not ranked high enough. For example, in some embodiments, the largest number of features based on the ranking (such as the top 10 or top 10% of the eligible features) are retained for further evaluation at 505.

[0042] At 505, a performance metric is determined for the features. In various embodiments, a performance metric is determined for each of the remaining eligible features using the feature's corresponding impact score determined at 503. The performance metric is used to filter one or more of the eligible machine learning features to identify a set of recommended machine learning features. For example, the weighted information gain score (or, for some embodiments, the unweighted information gain score) is converted to a performance metric, for example, by applying a model created offline. In some embodiments, the model is a regression model and / or a trained machine learning model to predict the increase in area under the precision-recall curve (AUPRC) as a function of the weighted information gain score. In various embodiments, the offline model is applied to the impact score from step 503 to infer a performance metric, such as an AUPRC-based performance metric for the model when utilizing the feature being evaluated. The AUPRC-based performance metric determined for each of the remaining eligible features may be used to rank the remaining features and filter out features that do not meet or fall within a certain threshold. In some embodiments, the eligible features are ranked by the determined AUPRC-based performance metric, and a feature may be removed from consideration as a recommended input feature if the feature does not rank high enough. For example, in some embodiments, a maximum number of features based on the ranking (such as the top 10 or top 10% of the eligible features) are retained for post-processing at 507.

[0043] In some embodiments, accurate determination of a performance metric, such as an AUPRC-based performance metric, can be time-consuming and resource-intensive. By utilizing an offline prepared model (such as a transformation model) to determine a performance metric from the weighted information gain scores, the performance metric can be determined in real time. The time- and resource-intensive task is shifted from the process of FIG. 5, particularly from step 505, to the creation of a transformation model, which can be pre-computed and applied to multiple machine learning problems. For example, after the transformation model is created, the model can be applied across multiple machine learning problems to multiple different customers and datasets.

[0044] At 507, post-processing is performed on the qualified features. For example, the remaining qualified features are processed for consideration as recommended machine learning features. In some embodiments, the post-processing performed at 507 includes a final filtering of the remaining qualified features. The post-processing step may be utilized to determine a final ranking of the remaining qualified features based on predicted model performance. In some embodiments, the final ranking is based on the performance metric determined at 505. For example, the feature with the highest expected improvement is ranked first based on its performance metric. In various embodiments, features that do not meet the final threshold or are outside the final threshold range or ordered ranking may be removed from the recommendation. In some embodiments, none of the remaining qualified features meet the final threshold for recommendation. For example, even the top-ranked features do not significantly improve the prediction accuracy over the naive model. In this scenario, none of the remaining qualified features may be recommended. In various embodiments, the remaining qualified features after the final filtering are a set of recommended machine learning features, each including a performance metric and an associated ranking. In some embodiments, a set of non-recommended features is also created, e.g., any features that are determined not to significantly improve the model predictive accuracy based on the evaluation process are identified as useless.

[0045] FIG. 6 is a flow chart illustrating an embodiment of a process for creating an offline model for determining a performance metric for a feature. Using the process of FIG. 6, an offline model is created for converting the impact scores of the features into a performance metric. For example, a weighted information gain score (or, for some embodiments, an unweighted information gain score) is used to predict the increase in the area under the precision-recall curve (AUPRC) performance metric. The performance metric may be utilized to evaluate the expected improvement the feature has in improving the accuracy of the model prediction. In various embodiments, the model is created as part of an offline process and applied during a real-time process for feature recommendation. In some embodiments, the offline model created is a machine learning model. In some embodiments, the offline model created using the process of FIG. 6 is utilized at 203 of FIG. 2, at 407 of FIG. 4, and / or at 505 of FIG. 5. In some embodiments, the model is created on a machine learning platform at server 121 of FIG. 1.

[0046] At 601, a dataset is received. For example, multiple datasets are received for building an offline model. In some embodiments, hundreds of datasets are utilized to build an accurate offline model. The received dataset may be a customer dataset stored in one or more database tables.

[0047] At 603, relevant features of the dataset are identified. For example, columns of the received dataset are processed for relevant features and features corresponding to non-relevant columns of the dataset are removed. In some embodiments, the data is pre-processed to identify column data types and non-nominal columns are filtered out to identify relevant features. In various embodiments, only the relevant features are utilized to train the offline model.

[0048] At 605, an impact score is determined for the identified features of the datasets. For example, an impact score is determined for each of the identified features. In some embodiments, the impact score is a weighted information gain score. In some embodiments, a non-weighted information gain score is used as an alternative impact score. In determining the impact score, a pair of the identified features may be selected, one as an input and the other as a target. An impact score may be calculated using the selected pair to calculate a weighted information gain score. A weighted information gain score may be determined for each of the identified features of each dataset. In some embodiments, the impact score is determined using the techniques described with respect to step 503 of FIG. 5.

[0049] At 607, a comparison model is constructed for each identified feature. For example, a machine learning model is trained with each identified feature and the corresponding model is created as a baseline model. In some embodiments, the baseline model is a naive model. For example, the baseline model may be a naive probability-based classifier. In some embodiments, the baseline model may predict outcomes by always predicting the most likely outcome by randomly selecting outcomes or by using other suitable naive classification techniques. Both the trained model and the baseline model are comparison models for the identified features. The trained model is a machine learning model that utilizes the identified features for prediction, and the baseline model represents a model where the features are not utilized for prediction.

[0050] At 609, a performance metric is determined using the comparison model. By comparing the prediction results and accuracy of the two comparison models for each identified feature, a performance metric can be determined for that feature. For example, for each identified feature, the area under the precision-recall curve (AUPRC) can be evaluated for the trained model and the baseline model. In some embodiments, the difference between the two AUPRC results is the performance metric of the feature. For example, the performance metric of the feature can be expressed as the increase in AUPRC between the comparison models. For each identified feature, the performance metric is associated with an impact score. For example, the increase in AUPRC is associated with a weighted information gain score.

[0051] At 611, a regression model is constructed to predict a performance metric. Using the impact score and performance metric pairs determined at 605 and 609, respectively, a regression model is created to predict a performance metric from the impact score. For example, a regression model is created to predict an increase in the area under the precision-recall curve (AUPRC) of a feature as a function of the feature's weighted information gain score. In some embodiments, the regression model is a machine learning model trained using the impact score and performance metric pairs determined at 605 and 609 as training data. In various embodiments, the trained model may be applied in real time to predict a performance metric of a feature after the impact score is determined. For example, the trained model may be applied at step 505 of FIG. 5 to determine a performance metric of a feature to evaluate an expected improvement in model quality associated with the feature.

[0052] Although the above embodiments have been described in considerable detail for purposes of clarity of understanding, the present invention is not limited to the details provided. There are many alternative ways of implementing the present invention. The disclosed embodiments are illustrative and not restrictive.

Claims

1. receiving a command via a user interface to initiate generation of a machine learning model; receiving, via the user interface, a designation of a target field included in one or more database tables storing sets of machine learning training data for prediction of the machine learning model; automatically identifying, from the one or more database tables, one or more qualified machine learning features for predicting the target field; automatically assessing an impact level of each of the eligible machine learning features on the predictive accuracy of the machine learning model; generating, via the user interface, a set of recommended machine learning features from the one or more eligible machine learning features, and ranking the set of recommended machine learning features based on their respective levels of impact on predictive accuracy of the machine learning model; receiving an input via the user interface identifying one or more selected machine learning features; The method according to claim 1,

2. building the machine learning model using the one or more selected machine learning features to predict the target field. The method of claim 1.

3. and pre-processing the sets of machine learning training data. The method of claim 1.

4. The method further comprises removing one or more sets of machine learning training data that are less than a threshold from the plurality of sets of machine learning training data. The method according to claim 3.

5. the one or more database tables include one or more incident reports; the target field includes a category type of a particular incident report among the one or more incident reports. The method of claim 1.

6. the target field includes a first target and a second target; The method of claim 1.

7. The step of evaluating an impact level of each of the eligible machine learning features on the predictive accuracy of the machine learning model includes: assessing a first impact level on the predictive accuracy of the machine learning model based on the first target; assessing a second impact level on the predictive accuracy of the machine learning model based on the second target; and Including, The method according to claim 6.

8. receiving additional input via the user interface specifying a model name, a model description, one or more stop words, a processing language, or a combination thereof; The method of claim 1.

9. a particular qualified machine learning feature among the one or more qualified machine learning features corresponds to a column of the one or more database tables; The method of claim 1.

10. evaluating an impact level of each of the eligible machine learning features on the predictive accuracy of the machine learning model includes calculating a confidence score. The method of claim 1.

11. the confidence score corresponds to an impact level of a particular column of the one or more database tables in the machine learning model predicting the target field. The method of claim 10.

12. 1. A system comprising a processor and a memory coupled to the processor, the memory configured to provide instructions to the processor that, when executed, cause the processor to: receiving a command via a user interface to initiate generation of a machine learning model; receiving, via the user interface, a designation of a target field included in one or more database tables storing sets of machine learning training data for prediction of the machine learning model; automatically identifying, from the one or more database tables, one or more qualified machine learning features for predicting the target field; automatically assessing an impact level of each of the eligible machine learning features on the predictive accuracy of the machine learning model; generating, via the user interface, a set of recommended machine learning features from the one or more eligible machine learning features, and ranking the set of recommended machine learning features based on their respective levels of impact on predictive accuracy of the machine learning model; receiving an input via the user interface identifying one or more selected machine learning features; A system that executes the above.

13. and further causing the processor to construct the machine learning model using the one or more selected machine learning features to predict the target field. The system of claim 12.

14. causing the processor to further perform the step of pre-processing the sets of machine learning training data. The system of claim 12.

15. the one or more database tables include one or more incident reports; the target field includes a category type of a particular incident report among the one or more incident reports. The system of claim 12.

16. the target field includes a first target and a second target; The system of claim 12.

17. A non-transitory computer-readable storage medium that, when executed by a processor, causes the processor to: receiving a command via a user interface to initiate generation of a machine learning model; receiving, via the user interface, a designation of a target field included in one or more database tables storing sets of machine learning training data for prediction of the machine learning model; automatically identifying, from the one or more database tables, one or more qualified machine learning features for predicting the target field; automatically assessing an impact level of each of the eligible machine learning features on the predictive accuracy of the machine learning model; generating, via the user interface, a set of recommended machine learning features from the one or more eligible machine learning features, and ranking the set of recommended machine learning features based on their respective levels of impact on predictive accuracy of the machine learning model; receiving an input via the user interface identifying one or more selected machine learning features; A computer-readable storage medium for causing a computer to execute the method.

18. the target field includes a first target and a second target; 20. The computer-readable storage medium of claim 17.

19. The step of evaluating an impact level of each of the eligible machine learning features on the predictive accuracy of the machine learning model includes: assessing a first impact level on the predictive accuracy of the machine learning model based on the first target; assessing a second impact level on the predictive accuracy of the machine learning model based on the second target; and Including, 20. The computer-readable storage medium of claim 18.

20. further causing the processor to receive, via the user interface, additional input specifying a model name, a model description, one or more stop words, a processing language, or a combination thereof.

20. The computer-readable storage medium of claim 17.

Images