Machine learning feature recommendation

Abstract

A pre-trained model is generated that is trained to predict a measure of expected model performance based at least in part on a feature correlation score associated with a text field data type. A specification of a target field of interest for machine learning prediction and stored input content of one or more text fields is received. A corresponding feature correlation score is computed for each of the one or more text fields of stored input content. Based on the corresponding computed feature correlation score, a pre-trained model is used to predict a corresponding measure of expected model performance for each of the one or more text fields of stored input content. The predicted measure of expected model performance is provided for use in feature selection among the one or more text fields of stored input content for generating a machine learning model to predict the target field of interest.

Description

[0001] Cross-references to other applications

[0002] This application is a continuation-in-part of pending U.S. Patent Application No. 16 / 931,906, filed July 17, 2020, entitled “MACHINE LEARNING FEATURE RECOMMENDATION,” which is incorporated herein by reference for all purposes. Background Technology

[0003] The use of automated classification using machine learning can significantly reduce human work and errors compared to manual classification. One approach to performing automated classification involves using machine learning to predict the category of input data. For example, machine learning can be used to automatically categorize incoming tasks, events, and cases and route them to the responsible party. Typically, automated classification using machine learning requires training data based on past experience. Once trained, the machine learning model can be applied to new data to infer classification results. For example, newly reported events can be automatically classified, assigned, and routed to the responsible party. However, creating accurate machine learning models is a significant investment and can be a difficult and time-consuming task, typically requiring subject-matter expertise. For instance, selecting input features that result in an accurate model typically requires a deep understanding of the dataset and how those features affect the predictions. Attached Figure Description

[0004] Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

[0005] Figure 1 This is a block diagram illustrating an example of a network environment used to create and utilize machine learning models.

[0006] Figure 2 This is a flowchart illustrating an embodiment of a process used to create a machine learning solution.

[0007] Figure 3 This is a flowchart illustrating an embodiment of the processing for automatically identifying recommended features used in machine learning models.

[0008] Figure 4 This is a flowchart illustrating an embodiment of the processing for automatically identifying recommended features used in machine learning models.

[0009] Figure 5 This is a flowchart illustrating an embodiment of an evaluation process for automatically identifying recommended features for machine learning models.

[0010] Figure 6This is a flowchart illustrating an embodiment of a process for creating an offline model used to determine performance metrics for features.

[0011] Figure 7 This is a flowchart illustrating an example of processing for automatically identifying and evaluating text fields as potential features for machine learning models.

[0012] Figure 8 This is a flowchart illustrating an embodiment of a process for evaluating the eligibility of a text field as a feature for a machine learning model in order to predict a desired target field.

[0013] Figure 9 This is a flowchart illustrating an embodiment of a process for preparing input text field data to determine the impact on the score.

[0014] Figure 10 This is a flowchart illustrating an embodiment of a process for determining performance metrics for text field features. Detailed Implementation

[0015] This invention can be embodied in a variety of ways, including as a method; apparatus; system; component of an object; a computer program product embodied on a computer-readable storage medium; and / or a processor, such as a processor configured to execute instructions stored on and / or provided by memory coupled to the processor. In this specification, these implementations or any other forms that the invention may take are referred to as techniques. Generally, within the scope of this invention, the order of steps of the disclosed processes can be changed. Unless otherwise stated, components such as processors or memory described as being configured to perform tasks can be implemented as general-purpose components temporarily configured to perform tasks at a given time or manufactured as specific components to perform tasks. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data such as computer program instructions.

[0016] The principles of the present invention, illustrated below, are as follows: Figure 1 This description provides a detailed account of one or more embodiments of the invention. The invention is described in relation to such embodiments, but is not limited to any particular embodiment. The scope of the invention is limited only by the claims, and the invention encompasses many substitutions, modifications, and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for illustrative purposes, and the invention may be practiced without some or all of these specific details. For clarity, technical materials known in the art related to the invention have not been described in detail, so as not to unnecessarily obscure the invention.

[0017] Techniques for selecting machine learning features are disclosed. Feature selection can significantly impact the accuracy and usability of a machine learning model when it is being built. However, appropriately selecting features to improve model accuracy can be challenging without subject matter expertise and a deep understanding of the machine learning problem. Using the disclosed techniques, machine learning features can be automatically recommended and selected, resulting in a significant improvement in the predictive accuracy of the machine learning model. Furthermore, subject matter expertise is required almost entirely or not at all. For example, a user with minimal understanding of the input dataset can successfully generate a machine learning model that can accurately predict classification results. In some embodiments, users can utilize the machine learning platform via software services such as software-as-a-service web applications.

[0018] In various embodiments, a user provides an input dataset to a machine learning platform, such as identifying one or more database tables. The provided dataset includes multiple qualified features. Qualified features can include features that are useful in accurately predicting machine learning outcomes and features that are useless or have a small impact on accurately predicting machine learning outcomes. Accurately identifying useful features can result in a highly accurate model and improve resource usage and performance. For example, training a model with useless features can be a significant resource drain, which can be avoided by accurately identifying and ignoring useless features. In various embodiments, the user specifies a desired target field for prediction, and the machine learning platform using the disclosed techniques can generate a set of recommended machine learning features from the provided input dataset for use in building the machine learning model. In some embodiments, recommended machine learning features are determined by applying a series of evaluations to qualified features to filter out useless features and identify useful features. Once a set of recommended features is determined, it can be presented to the user. For example, in some embodiments, the features are ordered in order of improvement to the prediction outcome. In some embodiments, the machine learning model is trained using features selected by the user based on the recommended features. For example, the model can be automatically trained using recommended features that are automatically identified and ordered by improving the prediction outcome.

[0019] In some embodiments, one or more tables are received that specify and store machine learning training data for a desired target field used for machine learning prediction. For example, a client of a software-as-a-service platform specifies one or more client database tables. The tables may include data from past experience, such as incoming tasks, events, and cases that have been categorized. For example, categorization may include classifying the type of task, event, or case, and assigning the appropriate party to be responsible for resolving the problem. In some embodiments, the machine learning data is stored in a separate, suitable data structure from the database. In various embodiments, the desired target field is a classification result, which may be a column in one of the received tables. Since the received database table data may not necessarily be prepared as training data, the data may include both useful and useless fields for predicting classification results. In some embodiments, qualified machine learning features for building a machine learning model to perform predictions against the desired target field are identified within one or more tables. For example, fields are identified from database data as potential or qualified features for training the machine learning model. In some embodiments, qualified features are based on columns of the tables. Qualified machine learning features are evaluated using different evaluation pipelines to successively filter out one or more qualified machine learning features to identify a recommended set of machine learning features from the qualified machine learning features. By sequentially filtering features from qualified features, features with minimal impact on model prediction accuracy are eliminated. The remaining features are recommended features with predicted values. Each step of the filtering pipeline identifies additional features that are not helpful (as well as potentially helpful features). For example, in some embodiments, a filtering step removes features where the feature data is unnecessary or out of range. Features sparsely distributed in their respective database tables or features where all values ​​of the feature are the same (e.g., constants) can be filtered out. In some embodiments, non-nominal columns are filtered out. In some embodiments, the filtering step calculates an impact score for each qualified feature. Features with impact scores below a certain threshold can be removed from the recommendations. In some embodiments, a performance metric is evaluated for each qualified feature. For example, with respect to a particular feature, the increase in the area under the precision-recall curve (AUPRC) can be evaluated. In some embodiments, the model is trained offline to convert the impact score into a performance metric by evaluating feature selection across a large cross-section for the machine learning problem. The model can then be applied to a specific client's machine learning problem to determine the performance metrics that can be used to rank qualified features. Once identified, a set of recommended machine learning features is provided for use in building the machine learning model. For example, a client can choose from recommended features and request to train a machine learning model using the provided data and the selected features. The model can then be integrated into the client's workflow to predict desired target fields.For example, in both datasets and machine learning, features can be automatically recommended (and selected) for machine learning models that can be used to infer target fields, even with little or no subject-specific expertise.

[0020] In some embodiments, qualifying features include data as text input data. For example, text input data can be text input of variable and / or arbitrary length, such as user input collected from input text fields, email subjects or bodies, chat conversations, etc. In various embodiments, one or more columns in potentially other identified table data may include text input as a potential feature for predicting a desired target field. For example, the user specifies the desired target field and a database table. Input text fields included in the table are evaluated as qualifying features to determine a performance metric for how well each input text field predicts the desired target field. In some embodiments, the fields provided by the user are sorted and the text input fields are included in the sorted qualifying fields. Like other qualifying features, the text input fields are evaluated to determine an impact score for the feature. In some embodiments, the impact score may be calculated as a relief score. For example, in some embodiments, the relief score is a weighted and normalized relief score. Multiple weighted and normalized relief scores may be calculated for the same qualifying feature, and an average impact score may be used.

[0021] In some embodiments, the determined influence scores are used to predict performance metrics. Performance metric predictions can be determined by applying a machine learning model trained offline. For example, using filtered selection scores and text field density scores, a machine learning model can predict performance metrics for text input fields. In some embodiments, performance metrics are based on the expected increase in the area under the model at the precision-recall curve (AUPRC). The applied model converts influence scores into performance metrics by evaluating feature selection across a large cross-section for the machine learning problem. This training for the model can be performed offline before evaluating qualified features. By leveraging an offline-trained model, performance metrics for qualified features can be quickly determined using the determined influence scores of the features. In various embodiments, while at least one input to the trained model is the influence score of a text input field, additional inputs such as the text field density of the field may be appropriate to further improve the accuracy of performance metric predictions. In various embodiments, the predicted performance metrics can be used to rank and recommend qualified features on a user-supplied dataset.

[0022] In some embodiments, a pre-trained model is generated to predict a metric of expected model performance based at least in part on feature relevance scores associated with the text field data type. For example, the model can be trained offline by evaluating feature selection across a large cross-section for a machine learning problem. Specifically, the model is trained to predict performance scores or metrics for features with text field data types. Using feature relevance scores such as influence scores, the model can predict the expected model performance for qualified features. For example, performance can be provided by the expected improvement of a feature over the area under the precision-recall curve (AUPRC). In some embodiments, one or more text fields of input content are received, specifying a desired target field for machine learning prediction. For example, a user specifies a desired target field, such as a field from a customer database table. The user also specifies additional fields, such as one or more text fields from the same database table or other database tables. Additional fields are qualified features that may be useful for predicting results for the desired target field. Qualified features can be specified by the user for evaluation to determine which qualified feature should be recommended for predicting the desired target field. In some embodiments, a corresponding feature relevance score is calculated for each of the one or more text fields storing the input content. For example, an influence score is calculated for each qualified text field feature. The influence score can be a filtered selection score, such as a normalized, weighted, or averaged filtered selection score. In some embodiments, based on the corresponding computed feature relevance score, a pre-trained model is used to predict a corresponding metric of the expected model performance for each of one or more text fields used to store the input content. For example, using a pre-trained model, the computed influence / relevance score is used to infer the expected model performance for each of one or more text field features. In some embodiments, the expected model performance is a performance metric, such as the expected improvement in the area under the precision-recall curve (AUPRC). The metric of the predicted expected model performance is provided for use in feature selection among one or more text fields storing the input content to generate a machine learning model to predict a desired target field. For example, the predicted performance metric can be used to recommend which text field features should be used to create a machine learning model to predict a desired target field. In some embodiments, text field features are sorted by performance metric, and only features that meet the performance threshold are recommended. The user can select from recommended text field features from other qualified and sorted non-text field features to generate a machine learning model to predict a desired target field.

[0023] Figure 1This is a block diagram illustrating an example of a network environment used to create and utilize machine learning models. In the example shown, clients 101, 103, and 105 access services on server 121 via network 111. The services include predictive services utilizing machine learning. For example, the services may include both the ability to generate machine learning models using recommended features and services for applying the generated models to predict outcomes such as classification results. Network 111 may be a public or private network. In some embodiments, network 111 is a public network such as the Internet. In various embodiments, clients 101, 103, and 105 are network clients, such as web browsers for accessing services provided by server 121. In some embodiments, server 121 provides services including web applications for utilizing a machine learning platform. Server 121 may be one or more servers, including servers for identifying recommended features for training machine learning models. Server 121 may utilize database 123 to provide certain services and / or for storing data associated with users. For example, database 123 may be a configuration management database (CMDB) used by server 121 for providing customer service and storing customer data. In some embodiments, database 123 stores customer data related to customer tasks, events, and cases. Database 123 may also be used to store information related to feature selection for training machine learning models. In some embodiments, database 123 may store customer configuration information related to managed assets, such as associated hardware and / or software configurations.

[0024] In some embodiments, each of clients 101, 103, and 105 can access server 121 to create a custom machine learning model. For example, clients 101, 103, and 105 may represent one or more different clients, each wanting to create a machine learning model that can be applied to predict outcomes. In some embodiments, server 121 provides clients such as clients 101, 103, and 105 with interactive tools for selecting and / or confirming feature selections used to train the machine learning model. For example, a client of a software-as-a-service platform provides relevant training data, such as client data, to server 121 via clients such as clients 101, 103, and 105. The provided client data may be data stored in one or more tables in database 123. Along with the provided training data, the client selects a desired target field, such as one of the columns of the provided table. Using the provided data and the desired target field, server 121 recommends a set of features to predict the desired target field with high accuracy. The client can select a subset of the recommended features and train the machine learning model from that subset. In some embodiments, the provided client data is used to train the model. In some embodiments, as part of the feature selection process, a performance metric for each recommended feature is provided to the client. The performance metric provides the client with a quantitative value relating to the degree to which a particular feature improves the model's prediction accuracy. In some embodiments, the recommended features are ranked based on their impact on prediction accuracy.

[0025] In some embodiments, a trained machine learning model is incorporated into the application to infer desired target fields. For example, the application may receive incoming reports supporting event conditions and predict the category for the event and / or assign the reported event conditions to the responsible party. The event-supporting application may be hosted by server 121 and accessed by clients such as clients 101, 103, and 105. In some embodiments, each of clients 101, 103, and 105 may be a web client running on one of many different computing devices, including laptops, desktop computers, mobile devices, tablets, information stations, smart TVs, etc.

[0026] Although some individual instances of components have been shown to simplify the diagram, it is possible that... Figure 1 Additional instances of any components shown. For example, server 121 may include one or more servers. Some servers in server 121 may be web application servers, training servers, and / or interference servers. Figure 1As shown, the server is simplified to a single server 121. Similarly, database 123 may not be directly connected to server 121, may be more than one database, and / or may be replicated or distributed across multiple components. For example, database 123 may include one or more different servers for each client. As another example, clients 101, 103, and 105 are just a few examples of potential clients for server 121. Fewer or more clients may connect to server 121. In some embodiments, there may also be Figure 1 Components not shown in the diagram.

[0027] Figure 2 This is a flowchart illustrating an embodiment of a process used to create a machine learning solution. For example, using... Figure 2 In this process, users can request machine learning solutions to a problem. Users can identify desired target fields for prediction and provide references to data that can be used as training data. The provided data is analyzed, and input features are recommended for training the machine learning model. Recommended features are provided to the user, and the machine learning model can be trained based on the user-selected features. The trained model is then incorporated into the machine learning solution to predict the user's desired target field. In some embodiments, the machine learning platform used to create the machine learning solution is hosted as a software-as-a-service web application. In some embodiments, users access the solution via, for example, through... Figure 1 Client request solutions such as client 101, 103, and / or 105. In some embodiments, the machine learning platform including the created machine learning solution is hosted on... Figure 1 On server 121.

[0028] At position 201, a machine learning solution is requested. For example, a customer might want to use a machine learning solution to automatically predict the responsible party for an incoming support event situation report. In some embodiments, the user requests a machine learning solution via a web application. When requesting a solution, the user can specify the target field they want to predict and provide relevant training data. In some embodiments, the provided training data is historical customer data. Customer data can be stored in a customer database. In some embodiments, the user provides one or more database tables as training data. The database tables may also include the desired target field. In some embodiments, the user specifies multiple target fields. In cases where predictions are desired for multiple fields, the user can specify multiple fields together and / or request multiple different machine learning solutions. In some embodiments, the user also specifies other properties of the machine learning solution, among other things, such as the processing language, stop words, filters used for the provided data, and the desired model name and description.

[0029] At point 203, recommended input features are determined. For example, a set of qualified machine learning features is determined based on the requested machine learning solution. A set of recommended features is identified from the qualified features. In some embodiments, recommended features are identified by evaluating the qualified machine learning features using a pipeline of different evaluations. At each stage of the pipeline, one or more of the qualified machine learning features may be filtered out sequentially. At the end of the pipeline, a set of recommended features is identified. In some embodiments, the identification of recommended features includes determining one or more metrics associated with the feature, such as impact score or performance metrics. For example, an offline-trained model may be applied to each feature to determine a performance metric that quantifies how much the feature will increase the area under the precision-recall curve (AUPRC) of the model trained using that feature. In some embodiments, an appropriate threshold may be applied for each metric to determine whether the feature is recommended for use in training.

[0030] In some embodiments, qualified machine learning features are based on input data provided by the user. For example, in some embodiments, the user provides one or more database tables or other suitable data structures as training data. When database tables are provided, qualified machine learning features may be based on the columns of the table. In some embodiments, the data type of each column is determined, and columns with a nominal data type are identified as qualified features. In some embodiments, data from certain columns may be excluded if the column data is unlikely to aid in prediction. For example, columns may be removed based on how sparsely populated the data is, the presence of stop words, the relative distribution of different values ​​for the column, etc.

[0031] At 205, features are selected based on recommended input features. For example, using an interactive user interface, a set of recommended machine learning features for use in building a machine learning model is presented to the user. In some embodiments, the example user interface is implemented as a web application or web service. The user can select from the displayed recommended features to determine a set of features for training the machine learning model. In some embodiments, the recommended input features determined at 203 are automatically selected as the default features for training. User input may not be required for selecting recommended input features. In some embodiments, recommended input features may be presented in sorted order based on how each recommended input feature affects the model's prediction accuracy. For example, the most relevant input features are sorted first. In various embodiments, recommended features are displayed along with influence scores and / or performance metrics. For example, influence scores can measure how much a feature affects model accuracy. Performance metrics can quantify how much the model will improve when the feature is used for training. For example, in some embodiments, the displayed performance metrics are based on the increase in the area under the precision-recall curve (AUPRC) of the machine learning model when the feature is used. Other performance metrics may be used appropriately. By sorting and quantifying different features, users with little or no subject-specific expertise can easily select appropriate input features to train a highly accurate model.

[0032] At 207, the selected features are used to train the machine learning model. For example, using the features selected at 205, a training dataset is prepared and used to train the machine learning model. The model predicts the desired target field specified at 201. In some embodiments, the training data is based on customer data received at 201. The customer data may be stripped of data that is useless for training, such as data from table columns corresponding to features not selected at 205. For example, data corresponding to columns associated with features identified as having little or no impact on the accuracy of predictions may be excluded from the dataset used to train the machine learning model.

[0033] At 209, a hosted machine learning solution. For example, an application server and machine learning platform host services for applying trained machine learning models to input data. For instance, a web service applies a trained model to automatically categorize incoming event reports. Categorization may include identifying the type of event and the responsible party. Once categorized, the hosted solution can assign and route events to the predicted responsible party. In some embodiments, the hosted application is a custom machine learning solution for a client of a Software as a Service platform. In some embodiments, the solution is hosted on... Figure 1 On server 121.

[0034] Figure 3This is a flowchart illustrating an embodiment of a process for automatically identifying recommendation features used in machine learning models. Figure 3 In this process, users can automate the creation of machine learning models by leveraging recommended features identified from potential training data. The user specifies the desired target field and provides potential training data. The machine learning platform identifies recommended fields from the provided data to create a machine learning model that predicts the desired target field. In some embodiments, Figure 3 The processing in Figure 2 Executed at position 201. In some embodiments, Figure 3 The processing in Figure 1 It is executed on the machine learning platform at server 121.

[0035] At 301, model creation is initiated. For example, a customer initiates the creation of a machine learning model via a web service application. In some embodiments, a customer initiates model creation by accessing a model creation webpage via a software-as-a-service platform used to create automated workflows. The service may be part of a larger machine learning platform that allows users to combine trained models to predict outcomes. In some embodiments, the predictions may be used to automate workflow processing, such as routing event reports to the assigned party once the appropriate party has been automatically predicted using the trained model.

[0036] At step 303, training data is identified. For example, the user specifies data as potential training data. In some embodiments, the user points to one or more database tables from a customer database or another suitable data structure storing potential training data. The data may be historical customer data. For example, historical customer data may include incoming event reports stored in one or more database tables and the parties responsible for their assignment. In some embodiments, the identified training data includes a large number of potential input features and may not have been properly prepared as high-quality training data. For example, some data columns may be sparsely populated or contain only the same constant values. As another example, the data type of the columns may be incorrectly configured. For example, nominal or numeric data values ​​may be stored as text in the identified database table. In various embodiments, preparation of the identified training data is required before it can be effectively used as training data. For example, data from one or more columns that has little or no impact on the model's prediction accuracy may be removed.

[0037] At 305, select the desired target field. For example, the user specifies the desired target field for machine learning prediction. In some embodiments, the user selects a column field from the data identified at 303. For example, the user can select a category type for event reporting to express the user's expectation of creating a machine learning model to predict the category type of incoming event reports. In some embodiments, the user can select from potential input features of the training data provided at 303. In some embodiments, the user selects multiple desired target fields that are predicted together.

[0038] Model configuration is completed at 307. For example, the user can provide additional configuration options such as model name and description. In some embodiments, the user can specify optional stop words. For example, stop words can be supplied to prepare training data. In some embodiments, stop words are removed from the provided data. In some embodiments, the user can specify the processing language and / or additional filters for the provided data. For example, stop words for a specified language can be added by default or suggested. Regarding the specified additional filters, conditional filters can be applied to create the represented dataset from the training data identified at 303. In some embodiments, rows of a table provided can be removed from the training data by applying one or more specified conditional filters. For example, the table may contain a "Status" column with possible values ​​"New," "In Progress," "Hold," and "Resolved." Conditions can be specified to use only rows where the "Status" field has the value "Resolved" as training data. As another example, conditions can be specified to use only rows created after a specified date or time frame as training data.

[0039] Figure 4 This is a flowchart illustrating an embodiment of a process for automatically identifying recommendation features used in machine learning models. For example, using... Figure 4 The feature selection pipeline can evaluate eligible features of a dataset in real time to determine how each potential feature will affect the machine learning model used to predict the desired target field. In various embodiments, a set of recommended features is determined, and the machine learning model can be trained from this set of recommended features. The recommended features are selected based on their accuracy in predicting the desired target field. For example, useless features are not recommended. In some embodiments, Figure 4 The processing in Figure 2 Executed at position 203. In some embodiments, Figure 4 The processing in Figure 1 It is executed on the machine learning platform at server 121.

[0040] At step 401, data is retrieved from a database table. For example, a user identifies a potential training dataset stored in one or more identified database tables and retrieves the associated data. In some embodiments, conditional filters are applied to the associated data before (or after) data retrieval. For example, based on conditional filters, only certain rows of the database tables can be retrieved. As another example, stop words are removed from the retrieved data. In some embodiments, data is retrieved from the identified tables to a machine learning training server.

[0041] At step 403, the column data type is identified. For example, the data type of each column is identified. In some embodiments, the column data type configured in the database table is not specific enough to be used to evaluate the characteristics of the association. For example, nominal values ​​may be stored in the database table as text or binary large object (BLOB) values. As another example, number or date types may also be stored as text (or string) data types. In various embodiments, at step 403, the column data type is automatically identified without user intervention.

[0042] In some embodiments, the data type is identified by first scanning all distinct values ​​through the column and analyzing the scan results. The properties of the column can be used to determine the valid data type of the column values. For example, text data can be identified at least in part by the number of spaces and the amount of variation in the text length within the column field. As another example, a column data type can be identified as a nominal data type if there is little or no variation in the actual values ​​stored in the column field. For example, a column with five discrete values ​​but stored as string values ​​can be identified as a nominal type. In some embodiments, the distribution of value types is used as a factor in identifying the data type. For example, if a high percentage of the values ​​in a column are numbers, then the column can be classified as a numeric data type.

[0043] At 405, preprocessing is performed on the data columns. In some embodiments, a set of preprocessing rules is applied to remove useless columns. For example, columns with sparsely populated fields are removed. In some embodiments, a threshold is used to determine whether a column is sparsely populated and is a candidate for removal. For example, in some embodiments, a 20% threshold is used. Columns with less than 20% of the data populated are unnecessary columns and can be removed. As another example, columns where all values ​​are constants are removed. In some embodiments, columns where one value dominates other values ​​are removed, for example, the dominant value appears in more than 80% (or another threshold amount) of the records. Columns where each value is unique or an ID can also be removed. In some embodiments, non-nominal columns are removed. For example, columns with binary data or text strings can be removed. In various embodiments, the preprocessing step eliminates only a subset of all eligible features from consideration as recommended input features.

[0044] At 407, qualified machine learning features are evaluated. For example, qualified machine learning features are evaluated for their impact on training an accurate machine learning model. In some embodiments, an evaluation pipeline is used to evaluate qualified machine learning features, filtering features sequentially according to their usefulness in predicting the desired target value. For example, in some embodiments, the first evaluation step may determine an impact score, such as a filter selection score, to identify the difference a column makes to the classification model. Columns with filter selection scores below a threshold can be removed from the recommendations. As another example, in some embodiments, the second evaluation step may determine an impact score, such as information gain or weighted information gain for a column. Using the selected features and the desired target field, the impact score can be determined by comparing improvements in the features using changes in information entropy when considering the features. Columns with information gain or weighted information gain scores below a threshold can be removed from the recommendations. In some embodiments, a third evaluation setting may determine a performance metric for each feature. For example, the model is created offline to convert the impact scores (such as information gain or weighted information gain scores) into performance metrics (such as a performance metric based on the increase in the area under the precision-recall curve (AUPRC) used for the model). In various embodiments, the trained model is applied to influence the score to determine an AUPRC-based performance metric for each remaining eligible feature. Using the determined performance metric, columns with performance metrics below a threshold can be removed from the recommendations. While three evaluation steps have been described above, fewer or additional steps can be appropriately utilized based on the desired results for a set of recommended features. For example, one or more different evaluation techniques can be applied in addition to or instead of the described evaluation steps to further reduce the number of eligible features.

[0045] In various embodiments, a set of recommended machine learning features for building a machine learning model is identified by applying successive evaluation steps. In some embodiments, successive evaluation steps are necessary to determine which features contribute to an accurate model. Any single evaluation step may be insufficient and may incorrectly identify features that are poor for training purposes for recommendation. For example, a feature may have a high filtering selection score but a low weighted information gain score. A low weighted information gain score indicates that the feature should not be used for training. In some embodiments, keywords or similar identifier columns are poor features for training because they have few predictive values. A column may have a high influence score when evaluated in one of the evaluation steps, but will be filtered out from the recommendations made by the successive evaluation steps.

[0046] At position 409, recommended features are provided. For example, the remaining features are recommended as input features. In some embodiments, a set of recommended features is provided to the user via a graphical user interface of a web application. The recommended features may be provided with a quantitative measure of how much influence each feature has on the model's accuracy. In some embodiments, the features are provided in sorted order, allowing the user to select the most influential features for training the machine learning model.

[0047] In some embodiments, useless features are also provided along with recommended features. For example, a user is provided with a set of features identified as useless or having a minor impact on model accuracy. This information can help the user gain a better understanding of machine learning problems and solutions.

[0048] Figure 5 This is a flowchart illustrating an embodiment of an evaluation process for automatically identifying recommended features for a machine learning model. In some embodiments, the evaluation process is a multi-step process for sequentially filtering features from qualified machine learning features to identify a set of recommended machine learning features. The process utilizes data provided as potential training data from which qualified machine learning features are identified and can be executed in real time. Although regarding... Figure 5 Specific evaluation steps have been described, but alternative embodiments of the evaluation process may utilize fewer or more evaluation steps and may combine different evaluation techniques. In some embodiments, Figure 5 The processing is in Figure 2 203 locations and / or at Figure 4 Executed at position 407. In some embodiments, Figure 5 The processing is in Figure 1 It was executed on the machine learning platform at server 121.

[0049] At 501, a defined filter selection score is used to evaluate the feature. In various embodiments, an influence score based on a filter selection technique is determined at 501, and the influence score is used to filter one or more eligible machine learning features to identify a set of recommended machine learning features. For example, an influence score based on the filter selection score for each feature is determined. Columns with filter selection scores below a threshold can be removed from the recommendations. In some embodiments, the filter selection score corresponds to the influence a column has in distinguishing different classification outcomes. In various embodiments, multiple neighboring rows are selected for each feature. Rows are selected based on having similar values ​​(or values ​​that are mathematically close or adjacent) in addition to the value used for the currently evaluated column. For example, for a table with three columns A, B, and C, column A is evaluated by selecting rows with similar values ​​for the corresponding columns B and C (i.e., the values ​​for column B are similar for all selected rows, and the values ​​for column C are similar for all selected rows). This influence score is used to determine how much influence column A has on the desired target field. In the example, the target field may correspond to one of columns B or C. Using the selected neighboring rows, an influence score or filtering selection score is calculated for each eligible feature. The scores can be normalized and compared to a threshold. Features with filtering selection scores falling below the threshold are identified as useless columns and can be excluded from further consideration as recommended input features. Features with filtering selection scores meeting the threshold are further evaluated at 503 for consideration as recommended input features. In some embodiments, eligible features are sorted according to their determined filtering selection scores, and if a feature is not ranked high enough, it can be removed from consideration as recommended input features. For example, in some embodiments, only the largest number of features based on the ranking (such as the top ten eligible features or the top 10% of eligible features) are retained for further evaluation at 503.

[0050] At 503, a weighted information gain score is used to evaluate the feature. In various embodiments, an impact score using information gain techniques is determined at 503, and this impact score is used to filter one or more qualified machine learning features to identify a set of recommended machine learning features. For example, an impact score based on the weighted information gain score for each feature is determined. Columns with weighted information gain scores below a threshold can be removed from the recommendations. In some embodiments, the weighted information gain score of a feature corresponds to a change in information entropy when the value of the feature is known. The weighted information gain score is an information gain measure that is weighted by a target distribution of different known values ​​for the feature. In some embodiments, the weighting is proportional to the frequency of a given target value. In some embodiments, an unweighted information score can be used as a replacement impact score.

[0051] In various embodiments, qualified features are sorted according to a determined weighted information gain score, and if a feature is not ranked high enough, it can be removed from consideration as a recommended input feature. For example, in some embodiments, only the largest number of features based on the ranking (such as the top ten qualified features or the top 10% of qualified features) are retained for further evaluation at 505.

[0052] At 505, a performance metric is determined for the feature. In various embodiments, the corresponding influence score of the feature determined at 503 is used to determine a performance metric for each of the remaining eligible features. The performance metric is used to filter one or more eligible machine learning features to identify a set of recommended machine learning features. For example, a weighted information gain score (or, in some embodiments, an unweighted information gain score) is converted into a performance metric by applying a model that has been created offline. In some embodiments, the model is a regression model and / or a trained machine learning model for predicting an increase in the area under the precision-recall curve (AUPRC) as a function of the weighted information gain score. In various embodiments, the offline model is applied to the influence score from step 503 to infer a performance metric, such as an AUPRC-based performance metric, for the model when utilizing the evaluated features. The AUPRC-based performance metric determined for each remaining eligible feature can be used to rank the remaining features and filter out those features that do not meet a specific threshold or fall within a specific threshold range. In some embodiments, qualified features are sorted according to a determined AUPRC-based performance metric, and a feature may be removed from consideration as a recommended input feature if it is not ranked high enough. For example, in some embodiments, only the largest number of features based on the ranking (such as the top ten qualified features or the top 10% of qualified features) are retained for post-processing at 507.

[0053] In some embodiments, the accurate determination of performance metrics, such as those based on AUPRC, can be time-consuming and resource-intensive. Performance metrics can be determined in real-time by leveraging offline-prepared models (such as transformation models) to determine performance metrics from weighted information gain scores. Time- and resource-intensive tasks are thus transformed from... Figure 5 The processing, and particularly the transition from step 505, to the creation of the transformation model, can be pre-computed and applied to multiple machine learning problems. For example, once the transformation model is created, it can be applied across multiple machine learning problems and for multiple different clients and datasets.

[0054] At 507, post-processing is performed on the qualified features. For example, the remaining qualified features are processed for consideration as recommended machine learning features. In some embodiments, the post-processing performed at 507 includes a final filtering of the remaining qualified features. The post-processing step can be used to determine the final ranking of the remaining qualified features based on the predicted model performance. In some embodiments, the final ranking is based on the performance metric determined at 505. For example, the feature with the highest expected improvement is ranked first based on its performance metric. In various embodiments, features that do not meet the final threshold or fall within the final threshold range or outside the ordered ranking can be removed from the recommendations. In some embodiments, none of the remaining qualified features meet the final threshold for recommendation. For example, even the top-ranked feature does not significantly improve prediction accuracy on the naive model. In this case, the remaining qualified features may not be recommended. In various embodiments, the remaining qualified features after the final filtering are a set of recommended machine learning features, and each includes a performance metric and an associated ranking. In some embodiments, a set of non-recommended features is also created. For example, any feature determined based on the evaluation process to not significantly improve model prediction accuracy is identified as useless.

[0055] Figure 6 This is a flowchart illustrating an embodiment of a process for creating an offline model used to determine performance metrics for features. Using Figure 6 The process involves creating an offline model to convert feature impact scores into performance metrics. For example, weighted information gain scores (or, in some embodiments, unweighted information gain scores) are used to increase the area under the precision-recall curve (AUPRC) performance metric. Performance metrics can be used to evaluate the expected improvement a feature has in improving the accuracy of the model's predictions. In various embodiments, the model is created as part of the offline process and applied during real-time processing for feature recommendation. In some embodiments, the created offline model is a machine learning model. In some embodiments, in Figure 2 203 places Figure 4 407 locations and / or Figure 5 505 locations used Figure 6 The offline model is created through processing. In some embodiments, in Figure 1 The model was created on the machine learning platform at server 121.

[0056] At position 601, a dataset is received. For example, multiple datasets are received for building an offline model. In some embodiments, hundreds of datasets are used to build an accurate offline model. The received datasets may be customer datasets stored in one or more database tables.

[0057] At 603, relevant features of the dataset are identified. For example, columns of the received dataset are processed for relevant features, and features corresponding to irrelevant columns of the dataset are removed. In some embodiments, data is preprocessed to identify column data types, and non-nominal columns are filtered out to identify relevant features. In various embodiments, only relevant features are used to train the offline model. In some embodiments, text field input columns are identified within the received dataset. For example, a database table may include one or more text field input fields containing text input of variable or arbitrary length. Fields are identified as potential qualified features for predicting desired target fields and are evaluated as text field input features rather than nominal types.

[0058] At 605, an influence score is determined for the identified features in the dataset. For example, an influence score is determined for each identified feature. In some embodiments, the influence score is a weighted information gain score. In some embodiments, an unweighted information gain score is used as a replacement influence score. When determining the influence score, a pair of identified features can be selected, one as input and the other as target. The selected pair can be used to compute the influence score to calculate the weighted information gain score. A weighted information gain score can be determined for each identified feature in each dataset. In some embodiments, an influence score is used for each feature in the dataset. Figure 5 The technique described in step 503 is used to determine the influence score. In some embodiments, the influence score is an average weighted score. For example, it can be used to determine the influence score based on the following information. Figures 7 to 10 The processing described here is a technique for determining the impact of text field input features on the score.

[0059] At 607, a comparative model is built for each identified feature. For example, a machine learning model is trained using each identified feature, and a corresponding model is created as a baseline model. In some embodiments, the baseline model is a naive model. For example, the baseline model could be a naive probability-based classifier. In some embodiments, the baseline model can predict the outcome by always predicting the most likely outcome, by randomly selecting the outcome, or by using another appropriate naive classification technique. The trained model and the baseline model together form a comparative model used for the identified features. The trained model is the machine learning model that uses the identified features for prediction, and the baseline model represents the model in which the features are not used for prediction.

[0060] At point 609, a performance metric is determined using a comparative model. A performance metric can be determined for each identified feature by comparing the predictions and accuracy of the two comparative models. For example, for each identified feature, the area under the precision-recall curve (AUPRC) can be evaluated against the training model and the baseline model. In some embodiments, the difference between the two AUPRC results is a performance metric for the feature. For example, a feature's performance metric can be expressed as the increase in AUPRC between the comparative models. For each identified feature, the performance metric is associated with the influence score. For example, an increase in AUPRC is associated with the weighted information gain score.

[0061] At position 611, a regression model is built to predict the performance metric. Using the influence score and performance metric pair determined at positions 605 and 609, respectively, a regression model is created to predict the performance metric from the influence score. For example, a regression model is created to predict the increase in the area under the precision-recall curve (AUPRC) of a feature as a function of its weighted information gain score. In some embodiments, the regression model is a machine learning model trained using the influence score and performance metric pair determined at positions 605 and 609 as training data. In various embodiments, once the influence score is determined, the trained model can be applied in real time to predict the performance metric of the feature. For example, the trained model can be... Figure 5 Step 505 is applied to determine the performance metrics of the features in order to evaluate the expected improvements in the quality of the model associated with the features.

[0062] Figure 7 This is a flowchart illustrating an embodiment of a process for automatically identifying and evaluating text fields as potential features for machine learning models. For example, using... Figure 7 In the processing of text fields, if the text field is used as an input feature for predicting the desired target field, then the text field can be evaluated to determine the expected model performance. In some embodiments, Figure 7 The processing can be done by Figure 3 The processing is initiated. For example, using... Figure 3 This processing allows users to automate the creation of machine learning models by leveraging recommended text field features identified from potential training data to predict desired target fields. Figure 7 The identified text fields are processed and evaluated for use as features in recommendations. The text fields are evaluated as variable and / or arbitrary length text fields, rather than being converted to a nominal type and evaluated as a nominal type. Similarly, in some embodiments, Figure 4 Feature selection pipeline depends Figure 7The processing is used to evaluate in real time how potential text field features will affect the machine learning model used to predict the desired target field. In some embodiments, in Figure 3 Use at step 303 Figure 7 The text fields evaluated in the processing are identified as potential training data. In some embodiments, Figure 7 The various steps of the processing are by Figure 4 The processing is executed. For example, in some embodiments, in Figure 4 Execute step 701 at point 401. Figure 4 Execute step 703 at point 403. Figure 4 Perform step 705 at position 405 and / or 407, and / or at position 405 and / or 407. Figure 4 Step 707 is executed at point 409. In some embodiments, at... Figure 1 Server 121 and / or Figure 2 Executed on machine learning platform at 203 Figure 7 The processing is used to at least partially determine the recommended input features.

[0063] At 701, a text field column is received as input data. For example, a text field column of a database table or dataset is identified by the user as potential training data. Once identified, the text field column is received as input data that can be evaluated. In some embodiments, the text field column includes entries corresponding to text of variable or arbitrary length.

[0064] At point 703, the column data type used for the received text field column is identified as text field data. For example, the entries in the received text field column are evaluated to determine that the column data type is text field data. This evaluation step may be necessary to determine that the data type of the received text field column is actually text data and not another type such as a nominal type compatible with text data. For example, in some cases, data stored in a text field column is stored as text data, but an other data type such as nominal, integer, number, or other appropriate data type could describe the data more accurately and / or efficiently. At point 703, the column data type used for the received text field column is confirmed as text field data.

[0065] At 705, the readability of the text field as a feature is evaluated. For example, a column of text fields is evaluated as a readability feature for predicting a desired target field. In some embodiments, the text field is first evaluated to determine a feature relevance score, such as an influence score in predicting a desired target field. Example influence scores can be calculated as weighted and normalized filtered selection scores. In some embodiments, the filtered selection score is a ReliefFscore, a statistical measure indicating feature relevance based on how well the feature values ​​distinguish the target among similar instances. The Euclidean norm / Flobenius norm of the ReliefFscore can be calculated based on the text feature dimension and normalized using the distribution of the target features to derive a weighted and normalized filtered selection score. Using the calculated feature relevance scores, a performance metric can be determined. For example, a corresponding metric of the expected model performance can be predicted by applying a pre-trained model to the calculated influence scores. In some embodiments, other metrics of the text data, such as text field density, are also evaluated and utilized in the prediction. In some embodiments, the performance metric corresponds to the readability of the text field as a feature for predicting a desired target field. For example, the higher the prediction performance metric, the more qualified the text field is and / or the more highly recommended it is as a feature for predicting the desired target field.

[0066] At 707, recommendations are provided for the evaluated text field. For example, recommendations are made regarding the text field received at 701 using the determined pass / fail assessment. In some embodiments, the recommendations include ranking the evaluated text field among other latent features. As a helpful guide for users to choose among different latent features, the recommendations may include the expected improvement in model performance when relying on the evaluated text field as an input feature. In some embodiments, the text field may be recommended only if the determined performance metric exceeds a minimum performance threshold. In various embodiments, users can leverage the provided recommendations to select features for automatically creating machine learning models to predict desired target fields.

[0067] Figure 8 This is a flowchart illustrating an embodiment of a process for evaluating the eligibility of a text field as a feature for a machine learning model in order to predict a desired target field. In some embodiments, Figure 8 The processing and evaluation of text field data provided as potential training data can be performed in real time. In some embodiments, Figure 8 The processing is in Figure 2 203 places Figure 4 405 and / or 407, and / or Figure 7 This is performed at point 705. In some embodiments, when evaluating a text field, Figure 8 The various steps of the processing are by Figure 5 The processing is executed. For example, in some embodiments, in Figure 5 Execute step 803 at point 501. Figure 5 Execute step 805 at point 503, and / or at point 503. Figure 5 Step 807 is performed at positions 505 and / or 507. In some embodiments, at... Figure 1 Executed on the machine learning platform at server 121 Figure 8 The processing. In some embodiments, Figure 8 The processing portion is also used to train the offline performance metric prediction model. For example, in some embodiments, in Figure 6 Step 605 utilizes the influence scores and other relevant metrics determined at steps 801, 803, and / or 805 to train the offline performance metric prediction model. Then, at step 807, the pre-trained model is used to determine the corresponding performance metric for the text field.

[0068] At 801, the input text field data is processed. For example, processing and / or preprocessing of the text field data may be performed to prepare intermediate data required to calculate the impact score. Processing may include determining statistical measures of the text data and preparing multiple evaluation samples from the text data. In some embodiments, processing includes determining a term frequency-inverse document frequency (TF-IDF) metric for the provided text data and / or performing a projection of the text data to reduce the number of dimensions. Other appropriate processing may be performed, such as determining the text field density. In various embodiments, the input text field data may correspond to entries in a text field column of a specified database table or dataset.

[0069] At 803, a weighted filter selection score is calculated. For example, using the intermediate data prepared at 801, a weighted filter selection score is calculated for the text field. In some embodiments, the weighted filter selection score is a normalized filter selection score. Each calculated weighted filter selection score may correspond to a stratified sample set of the input data. By calculating weighted filter selection scores over multiple samples of the input data, the data can be appropriately sampled with minimal resource requirements compared to calculating weighted filter selection scores over the entire input text field data. For example, in some cases, three stratified samples are prepared at 801, and three weighted filter selection scores are calculated at 803, one corresponding to each prepared sample.

[0070] At 805, the average weighted filter selection score is determined. For example, the average weighted filter selection score is calculated using the weighted filter selection score computed from 803. The average weighted filter selection score can be a normalized filter selection score and can correspond to the influence score used for the text field. In some embodiments, the magnitude of the influence score corresponds to how much influence the text field has on predicting the desired target field. While the influence score expresses the relevance of a feature in predicting the desired target field, it may not quantify improvements in model performance if the text field is used as an input feature for a machine learning model. In some embodiments, the determined average weighted filter selection score and any other appropriate text field metric (such as the text field density computed at 801) are used to train an offline performance metric prediction model.

[0071] At point 807, a performance metric for the text field is determined. For example, the performance metric can be predicted using the determined average weighted filtered selection score and any additional text field metrics, such as text field density. In some embodiments, this is achieved by applying a pre-trained model (such as using...). Figure 6 The performance metric is inferred from the pre-trained model (processing the offline-trained model). By utilizing the pre-trained model, a metric for the expected model performance can be determined in real time. During the training of the performance metric prediction model, significant computationally and resource-intensive operations are instead performed offline. In various embodiments, the determined performance metric may correspond to an increase in the area under the precision-recall curve (AUPRC) for the text field features. This increase may correspond to the difference between a trained model using similar text fields as features for prediction and a baseline model utilizing appropriate naive classification techniques (such as always predicting the most likely outcome). The determined performance metric provides an indication of the expected performance increase for a trained model utilizing text field features compared to a machine learning model that does not utilize text field features. In some embodiments, the performance metric is used to determine recommendations for text fields as potential or qualified features for predicting a desired target field.

[0072] Figure 9 This is a flowchart illustrating an embodiment of a process for preparing input text field data to determine the impact on the score. In some embodiments, Figure 9 The processing in Figure 4 405 and / or Figure 8 Executed at 801, and prior to the calculation of the score or feature relevance used to determine the impact of the text field on model performance. In some embodiments, Figure 9 The processing in Figure 1 It is executed on the machine learning platform at server 121. In some embodiments, Figure 9The processing portion is also used to train the offline performance metric prediction model. For example, in some embodiments, Figure 9 The processing is performed together with additional steps to... Figure 6 Step 605 determines the impact score for the text field.

[0073] At 901, an information metric is evaluated for the text input data. For example, an information metric, such as a statistical metric, is determined for the text input data. The information metric is calculated in real time and may include metrics such as term frequency-inverse document frequency (TF-IDF) metrics. As another example, an information metric such as text field density may be calculated for the text input data. In some embodiments, the information metric may be determined using a sample of the text input data or by evaluating the entire dataset of text input data. In various embodiments, the text input data may correspond to entries in a text field column in a specified database table or dataset.

[0074] At position 903, a random projection is performed on the input data being evaluated. For example, for a large dataset with a high number of dimensionalities, a random projection is performed to reduce the number of dimensionalities. In some embodiments, the number of dimensionalities can be reduced to a more efficient number, such as 100 dimensions.

[0075] At 905, an input sample dataset is created. For example, one or more samples of text input data are created for evaluation. In some embodiments, the text input data is too large to efficiently compute a single impact score on the entire dataset. Instead, multiple sample datasets are created. Each can be scored for its impact, and the sample impact scores are then averaged. In various embodiments, stratified sampling is applied to create multiple sample datasets. The created datasets may include a sufficient sample of the text input data. For example, in some embodiments, the created datasets cover approximately 10% of the text input data.

[0076] Figure 10 This is a flowchart illustrating an embodiment of a process for determining performance metrics for text field features. In some embodiments, Figure 10 The processing in Figure 5 505 locations Figure 7 705 and / or Figure 8 Executed at point 807. In some embodiments, by Figure 10 The processing utilizes impact scores and additional information metrics. Figure 8 and / or Figure 9 The processing and calculation are performed. In some embodiments, Figure 10 The processing is in Figure 1 It was executed on the machine learning platform at server 121.

[0077] At position 1001, an influence score for the text field is received. For example, an influence score for the text field, such as an average weighted filtered selection score, is received. The influence score can be a measure of the relevance of features in a desired target field when using the text field as a model feature. In some embodiments, the received influence score is calculated in real time and can be calculated over one or more sample sets of input text data for the text field. In various embodiments, the text field and its input text data can correspond to entries in a text field column in a specified database table or dataset.

[0078] At 1003, additional metrics for the text field are received. For example, additional metrics such as text field density are received and prepared as input features. In some embodiments, using additional metrics as input features for predictive performance metrics improves prediction results compared to relying solely on calculated impact scores. In various embodiments, the additional metrics may be computed in real time and may be computed on one or more sample sets of the input text data for the text field or on the entire text field dataset.

[0079] At point 1005, a prediction model is applied to determine a performance metric for the text field. For example, the performance metric prediction model is trained offline and applied at point 1005 to predict the expected model performance. In various embodiments, the input features for the prediction model include an influence score received at point 1001 and one or more information metrics received at point 1003. These received input features can be computed in real time along with the inferred performance metric. Conversely, the generation of the prediction model can be resource- and computationally expensive, and can be achieved, for example, by using... Figure 6 The processing benefits from offline training. In some embodiments, when comparing two comparative models, the performance metric for prediction corresponds to an increase in the area under the precision-recall curve (AUPRC) for the text field features. For example, the metric could correspond to the performance difference between a model trained using similar text fields as features for prediction and a baseline model utilizing appropriate naive classification techniques (such as always predicting the most likely outcome). The performance metric for prediction provides an indication of the expected performance increase for a model trained using text field features compared to a machine learning model that does not utilize text field features. In some embodiments, the performance metric is used to determine recommendations for text fields as potential or qualified features for predicting a desired target field.

[0080] While the foregoing embodiments have been described in some detail for clarity of understanding, the invention is not limited to the details provided. Many alternative methods of implementing the invention exist. The disclosed embodiments are illustrative rather than restrictive.

Claims

1. A method comprising: Generate a pre-trained model, which is trained to predict a metric of expected model performance based at least in part on feature relevance scores associated with the text field data type. Receives one or more text fields storing the input content and a specified target field for machine learning prediction; By generating one or more sample datasets for each of the one or more text fields storing the input content, a corresponding feature relevance score is calculated for each of the one or more text fields storing the input content, wherein the one or more generated sample datasets for each of the one or more text fields storing the input content are hierarchical samples; Based on the corresponding calculated feature relevance scores, a pre-trained model is used to predict the corresponding metric of the expected model performance for each of the one or more text fields storing the input content; as well as The predicted performance metrics of the model are provided for use in feature selection among the one or more text fields storing the input content, in order to generate a machine learning model to predict the desired target field.

2. The method of claim 1, wherein calculating a corresponding feature relevance score for each of the one or more text fields storing input content includes determining a statistical metric for each of the one or more text fields.

3. The method of claim 2, wherein the statistical metric is at least partially based on the term frequency-inverse document frequency (TF-IDF) metric.

4. The method of claim 1, further comprising determining a relevance score for each of the one or more generated sample datasets.

5. The method of claim 1, wherein calculating the corresponding feature relevance score for each of the one or more text fields comprises averaging the relevance scores of one or more samples for each of the one or more text fields.

6. The method of claim 1, wherein using a pre-trained model to predict a corresponding metric of expected model performance for each of the one or more text fields storing input content comprises applying the pre-trained model to one or more information metrics for each of the one or more text fields.

7. The method of claim 6, wherein the one or more information metrics include a text field density metric.

8. The method of claim 1, wherein the calculated feature relevance score for each of the one or more text fields storing the input content is a weighted and normalized filter selection score.

9. The method of claim 1, wherein the corresponding metric for the expected model performance of each of the one or more text fields storing input content is based on the increase in the area under the precision-recall curve associated with the machine learning model compared to the baseline model, to predict the desired target field.

10. The method of claim 1, further comprising sorting the one or more text fields storing the input content based on a metric of the predicted expected model performance for use in feature selection for generating a machine learning model to predict a desired target field.

11. The method of claim 1, wherein the one or more text fields storing input content include text collected from input text fields, email subjects, email bodies, or chat conversations.

12. A system comprising: One or more processors; as well as A memory coupled to the one or more processors, wherein the memory is configured to provide instructions to the one or more processors, the instructions causing the one or more processors to: Generate a pre-trained model, which is trained to predict a metric of expected model performance based at least in part on feature relevance scores associated with the text field data type. Receives one or more text fields storing the input content and a specified target field for machine learning prediction; The corresponding feature relevance score is calculated for each of the one or more text fields storing the input content by causing the one or more processors to determine a statistical metric for each of the one or more text fields, and wherein the statistical metric is at least partially based on the term frequency-inverse document frequency (TF-IDF) metric. Based on the corresponding calculated feature relevance scores, a pre-trained model is used to predict the corresponding metric of the expected model performance for each of the one or more text fields storing the input content; as well as The predicted performance metrics of the model are provided for use in feature selection among the one or more text fields storing the input content, in order to generate a machine learning model to predict the desired target field.

13. The system of claim 12, wherein the memory is further configured to provide instructions to the one or more processors, the instructions causing the one or more processors to: Generate one or more sample datasets for each of the one or more text fields that store the input content; For each sample dataset in one or more generated sample datasets, determine the relevance score of the sampled data; and For each of the one or more text fields, the relevance scores of one or more determined samples are averaged.

14. The system of claim 12, wherein a corresponding metric that causes the one or more processors to use a pre-trained model to predict the expected model performance for each of the one or more text fields storing input content includes one or more information metrics that cause the one or more processors to apply the pre-trained model to each of the one or more text fields, and wherein the one or more information metrics include a text field density metric.

15. The system of claim 12, wherein the calculated feature relevance score for each of the one or more text fields storing input content is a weighted and normalized filter selection score.

16. The system of claim 12, wherein the corresponding metric for the expected model performance of each of the one or more text fields storing input content is based on the increase in the area under the precision-recall curve associated with the machine learning model compared to the baseline model, to predict the desired target field.

17. A computer program product embodied in a non-transitory computer-readable medium and comprising computer instructions for: Generate a pre-trained model, which is trained to predict a metric of expected model performance based at least in part on feature relevance scores associated with the text field data type. Receives one or more text fields storing the input content and a specified target field for machine learning prediction; By determining a statistical metric for each of the one or more text fields storing the input content, a corresponding feature relevance score is calculated for each of the one or more text fields, and wherein the statistical metric is at least partially based on the term frequency-inverse document frequency (TF-IDF) metric. Based on the corresponding calculated feature relevance scores, a pre-trained model is used to predict the corresponding metric of the expected model performance for each of the one or more text fields storing the input content; as well as The predicted performance metrics of the model are provided for use in feature selection among the one or more text fields storing the input content, in order to generate a machine learning model to predict the desired target field.

Images