Systems and methods for generating datasets from heterogeneous sources for machine learning

Abstract

A computer system is provided that is programmed to select a set of features from a large number of features. The features of the set are selected based on meta-gradient information returned from a machine learning process that has been performed on an earlier selected set of features. The process can iterate until the selected set of features converges or otherwise meets or exceeds a given threshold.

Description

[0001] Cross Reference To One or More Related Applications

[0002] This application claims priority to U.S. Provisional Application No. 62 / 650,521 filed March 30, 2018, and U.S. Patent Application No. 16 / 368,804 filed March 28, 2019, the entire contents of which are hereby incorporated by reference.

[0003] TECHNICAL SUMMARY

[0004] The technology described herein relates to generating data from heterogeneous data. More particularly, the technology described herein relates to using a machine learning feedback loop to automatically generate a dataset from heterogeneous sources.

[0005] INTRODUCTION

[0006] Modern society generates a large variety of data. Data can vary from weather data (e.g., temperature) to traffic data (e.g., amount of cars on a given road), to service call data (e.g., number of calls received), to financial data (e.g., amount of transactions), to power production / consumption data (e.g., kilo-watt-hours), to many other forms of data. Finding associations or connections between different types of data and a target problem (e.g., predicting a hurricane) can be valuable in modern society.

[0007] However, due to the combinatorial explosion, finding associations or appropriate datasets among a large variety of datasets can be an incredibly difficult problem. For example, when there are hundreds or thousands of different datasets (each containing a number of different features or columns), the number of combinations can potentially exceed the number of molecules in the universe. Thus, evaluating each combination can become computationally infeasible.

[0008] Accordingly, it will be appreciated that there exists a continuing need for new and improved technologies, systems, and processes in the art of technology and other technologies. For example, technologies for generating, designing, or determining datasets from a variety of heterogeneous sources. SUMMARY

[0009] A computer system is provided that is programmed to select a set of features from a large number of features. The set of features is selected based on meta-gradient information returned from a machine learning process performed on an earlier selected set of features. The process can iterate until a selected set of features is found that meets or exceeds a given threshold.

[0010] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the DETAILED DESCRIPTION. This Summary is neither intended nor should it be construed to identify key or essential features of the claimed subject matter, nor to limit the scope of the claimed subject matter. Rather, this Summary is intended to present concepts of the subject matter described in this document in a generalized form so as to provide a broad BRIEF DESCRIPTION OF DRAWINGS

[0011] These and other features and advantages will be better understood by reference to the following detailed description when considered in conjunction with the accompanying drawings.

[0012] Figure 1 is an example system architecture diagram in accordance with certain example embodiments;

[0013] Figure 2 illustrates an example feature engineering module of the system shown in Figure 1

[0014] Figure 3 illustrates an example data set scanner module of the system shown in Figure 1

[0015] Figure 4 illustrates an example policy designer module of the system shown in Figure 1

[0016] Figure 5 shows an example computing device that can be used in some embodiments to implement features described herein. DETAILED DESCRIPTION

[0017] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular nodes, functional entities, techniques, protocols, etc. in order to provide an understanding of the described technology. It will be apparent to one skilled in the art in light of this description that implementations can be practiced without certain specific details. In other instances, detailed descriptions of well-known methods, devices, techniques, etc. are omitted in order to not obscure the description with material deemed to be unnecessary for understanding the technology.

[0018] The sections used in this DETAILED DESCRIPTION are in place merely to provide the reader with a general understanding of the subject matter of each section; as will be seen from the following, descriptions of many features cross over multiple sections, and headings should not be construed as affecting the meaning of the descriptions included in any section.

[0019] In many places in this document, including but not limited to Figure 1 ​​​In the description of the software modules and the actions performed by the software modules, the software modules and the actions performed by the software modules are described in terms of performing actions or storing data. This is done for ease of description; it is understood that, whenever a software module is described to perform an action, it is the underlying hardware elements (such as processor and memory devices) that perform the action according to instructions included in the software module. Additional details regarding this are provided below Figure 5 in the description of the software modules and the actions performed by the software modules.

[0020] SUMMARY

[0021] Certain example embodiments relate to automatically designing or generating new data sets using feedback loops and constraint theory from heterogeneous data sources. In certain examples, data is collected from a plurality of different data sources, and then subsequently processed to generate an engineered design data set (or engineered design features). The original features and engineered design features are stored in a hierarchical graph that is used to select features that will be included in a set of features for further analysis or processing. Machine learning is used to process the selected features of the set of features to generate meta-gradients for changing or updating the hierarchical graph of features (and the data sets to which those features belong). A new set of features is then selected, and the process continues to iterate until the selected set of features produces a model and output data that matches at least one target signal. The target signal can be, for example, a production quota, a next day prediction for a stock market, a next quarter earnings for a company, a target Sharpe ratio, a weather prediction, etc. The technical improvements provided by the technology herein can be applied in different fields, from healthcare, to media, to education, to finance, to securities, to transportation, and many other industries and fields that have different problems but have large (or many different) data sets that can be analyzed using the technology herein.

[0022] The methods discussed herein allow for processing hundreds or thousands (or even more) of different features (and computationally infeasible number of combinations of those features).

[0023] Figure 1 of the description

[0024] Figure 1is an example computer system architecture diagram in accordance with certain example embodiments. Computer system 100 receives and processes data sets A, B, C through N. These data sets can be from the same (e.g., data source A) or different data sources (e.g., data sources A, B, C, N...). Computer system 100 is configured to handle any number of data sets from any number of data sources. The data sets can be different types of data and can be in any form. For example, different types of data include temperature data gathered from one or more temperature sensors, electronic exchange data for one or more securities, service call data such as the total number of calls a company receives in a 24 hour period, and many other types of data. Essentially, any type of data can be included in the multiple different data sets supplied to system 100 by the data sources. In practice, the technology herein is designed to work with hundreds or thousands of different data sets and their corresponding sources.

[0025] In certain instances, the data sources can include internal data sources (e.g., data sources operated by the same organization that is operating computer system 100). The data sources can include data cable service providers (e.g., a data “cable” service analogous to the way in which Reuters is a news service). In certain instances, the data sources can be subscribed to by system 100. The data sources and the data formats for the data sets supplied by those data sources can be heterogeneous or homogenous in nature, and as such any type of data format can be acceptable.

[0026] In certain examples, the data from the various data sources can be stored in a data warehouse or data lake (not shown) that can then be queried and operated on by computer system 100. For example, the data sources can supply data sets A, B, C, N to such a data warehouse, and system 100 can access the data warehouse to process the data sets stored therein.

[0027] ETL (extract, transform, load) module 102 is part of computer system 100. In certain examples, the ETL module can also be its own dedicated computer system in communication with computer system 100. ETL module 102 is responsible for: 1) extracting data from the data sources (or data warehouses in which the data is stored); 2) transforming the data from its stored (or raw format) into a format more suitable for analysis by training system 104; and 3) loading the data into another database or other storage for further processing by training system 104 and the modules therein.

[0028] In certain example embodiments, training system 104 can be its own computer system separate from ETL module 102. Training system 104 can be implemented in a cloud-based computer environment and can be implemented across one or more physical computer nodes (e.g., as shown inFigure 5 The different components or modules of the training system 104 can be implemented on virtual machines, which can be implemented on corresponding physical computer hardware, in some examples.

[0029] The training system 104 includes three separate modules that operate to generate a data set 112 that can be applied to a model (see Figure 4 ) to achieve a target signal 114. The three modules include a feature engineering module 106 (described in Figure 2 ), a data set scanner module 108 (described in Figure 3 ), and a strategy designer module 110 (described in Figure 4 ).

[0030] The feature engineering module 106 operates on the data passed through the ETL module 102. The module 106 generates new columns of data based on the obtained source data. As one non-limiting example, the feature engineering module 106 can take the average of two (or more) segments of data to generate a third segment of data. For example, it can average high temperature values and low temperature values from an original data set to create a third value (the average). As explained below, other types of functions, transformations, etc. can be used to engineer the feature data. The engineered data set (which can include the original data set) is passed onto the data set scanner module 108. In some examples, the functionality provided by the feature engineering module 106 can be skipped; for example, if no engineering of features is needed for the processing.

[0031] The data set scanner module 108 is part of a feedback loop used by the training system 104 to generate the desired output (the data set 112 and the model for that data set) for the target signal 114. In some examples, the data set scanner module 108 uses a heuristic hierarchical graph search on the data (which is organized by data sets and features of individual data sets) to select a subset of features (also referred to herein as columns) to test. The selected set of features (multiple features) is then passed to the strategy designer module 110, another part of the feedback loop employed by the training system 104.

[0032] The policy designer module 110 uses machine learning to develop a policy for the selected feature set data received from the dataset sweeper module 108. This policy, along with meta-gradient information (also referred to as meta- gradient or meta-gradient data) and a model generated for the selected feature set, loops back to the dataset sweeper 108, which iterates on what features to select next (taking into account the meta-gradient information). This forms a feedback loop between the dataset sweeper 108 and the policy designer module 110. The returned meta-gradient information can indicate how effective or good each feature in the selected feature set is with respect to the target, and / or how effective the collection of features is with respect to the target signal.

[0033] Figure 2 Description of the Drawings - Feature Engineering Design:

[0034] Figure 2 FIGURE 1 illustrates an example system 100 in accordance with certain example embodiments. The system 100 includes an ETL module 102, a feature engineering module 106, a dataset sweeper module 108, and a policy designer module 110. t . And each dataset can include one or more features X0-X n (also referred to as columns), each feature having individual values (rows). The input data 202 can be output from the ETL module 102 and / or a database or other storage system that has been populated with data from the ETL module 102 as shown in FIGURE 2. Figure 1

[0035] The feature engineering module 106 receives the input data 202 and engineers additional features based on the input data. The engineering of additional features can use transformations, domain-specific encodings, categorical encodings, or other processes that take some subset of the input data to engineer additional features from that input data. One example of how a feature can be engineered can be to take the average of two features to engineer a third feature (e.g., take the average of X0and X1to create a data value under the engineered feature X n+1 The nature of the transformations, encodings, or other processes performed by the feature engineering module 106 can be based on the type of data being handled. For example, the feature engineering can be selected based on data that is weather data versus car performance data, or data from an electronic trading system.

[0036] In any case, the result of the processes performed by the feature engineering module 106 results in output data 204. The output data 204 includes the original input data 202 (e.g., D1-D t ​features that feature engineering module 106 has engineered or otherwise added. Engineered features can be generated into one or more additional datasets D v In certain examples, engineered features from an original dataset can be included into the same engineered dataset.

[0037] In certain examples, features can be engineered from multiple different datasets. For example, a feature from a first dataset and a feature from a second dataset can be used to engineer a third feature (which can be included in a first, second, or third new or existing dataset). For example, an engineered feature can be a calculated ratio between the average temperature in New York City and the average trading volume in the New York Stock Exchange. In certain examples, all engineered features can belong to their own dataset (e.g., a dataset of engineered features).

[0038] Features and datasets can be arranged in a hierarchical graph 206 (a type of data structure) in which each dataset (e.g., D1 - D v ) is a child node of a root node, and each feature (e.g., X2 - X9) is a child node of a node of a respective dataset. Thus, the output from feature engineering module 106 can be graph 206. In certain examples, each engineered dataset is included as a new node (e.g., D v In certain examples, all engineered features can be included as part of the same engineered feature dataset. In certain examples, engineered features are organized into engineered feature datasets based on the original data. Thus, engineered features based on dataset D1 can include those engineered features based on features from D1, and other features based on other datasets can be included in separate engineered feature datasets.

[0039] It will be appreciated that feature engineering module 106 can be configured to operate automatically with any type of input. It will also be appreciated that, in certain example embodiments, the hierarchical relationship between features and engineered features (e.g., that they share the same grandparent node) can help address combinatorial explosion in selecting combinations of features (as discussed further herein).

[0040] Figure 3 Description - Dataset Scanner

[0041] Figure 3 illustrates a feature engineering module 106 according to certain example embodiments Figure 1An example dataset scanner module 108 of the system 100 is illustrated in FIG. 6. The dataset scanner module 108 is configured to look for a set of features (e.g., a plurality of features) that perform "well" (e.g., they exceed a predetermined threshold) in obtaining a target signal (e.g., satisfying a given function or model).

[0042] The dataset scanner module receives features, a hierarchy of those features (e.g., the tree 206), and a dataset and / or a feature performance metric (e.g., meta-gradient information or how well a given feature or dataset performs with respect to a given target or metric). The dataset scanner module 108 then uses this data, as described herein.

[0043] In certain examples, the dataset scanner module 108 will operate with at least one penalty function, at least one constraint function, and / or at least one objective function. In certain examples, each overall use of the system 100 to determine a set of features until convergence will use or rely on a single objective function. Such an objective function can remain throughout the process of determining a set of features, a model, and a strategy for that objective function. In certain examples, different objective functions can be used for different processes that are used to find features for the same target signal (examples of target signals can be buy / sell indicators for stocks, or weather predictions for cities). The penalty functions and constraint functions can remain constant across a given process of determining a set of features. In certain examples, these functions can be switchable, such that a user or administrator can turn them on and off as desired (e.g., they can be based on user input). For example, a plurality of different objective functions can be stored within the system 100, and can be usable depending on the nature of the target or the type of features being sought. In certain examples, functions can be turned on (or turned off) based on how features or sets of features have been processed.

[0044] Constraint functions indicate situations that will not occur when selecting features. For example, one constraint can be that each selected feature must come from a different dataset, or if 4 features are selected, they must come from at least 3 different datasets. The constraints act to filter out sets of features that are not allowed.

[0045] Penalty functions can be used to penalize a set of situations related to feature selection. In other words, something can be allowed to happen, but it is more "costly" to include and consider that thing (e.g., a feature). The penalty function can operate such that the more the penalty function is broken, the worse the result will be scored. For example, the penalty function can be that each feature comes from the same dataset. If features from the same dataset are selected, then a 1% penalty is applied to each feature from the same dataset (e.g., if 5 features from the same dataset are selected, then a 5% penalty is applied).

[0046] The objective function is the goal to be achieved for a feature set. The objective function can be thought of as defining the value of a permissible found solution (e.g., each feature set found by the strategy designer). The objective function relates to the goal or objective signal being sought (e.g., the objective used during training of a model for a given feature set). An example of an objective signal in the stock market example can be the total return on investment. The objective function can take a weighted sum of the returns of the positions (and potentially strategies) and subtract from the weighted sum, for example, transaction costs (e.g., the cost of executing each trade) or other costs, values, etc... to achieve a calculation that is "closer" to the actual returns that can be generated. In other words, in some cases, the objective function can be used to further act on or use the signals from the model and / or strategy output.

[0047] In some examples, multiple functions of each function can be included (e.g., 4 penalty functions, 2 constraint functions, and 3 objective functions - however in most cases, one objective function will be used per process). These functions are used in conjunction with the information returned from the strategy designer 110 (e.g., meta-gradient information) to select features from the graph search.

[0048] The dataset scanner module 108 includes two sub-modules, one that determines what features to select (the hierarchical heuristic graph search module 302), and one that iteratively improves on the found feature sets (the optimizer module 304). Thus, these two modules work in conjunction to select features, which are then passed to the strategy designer 110 for testing.

[0049] The hierarchical heuristic graph search module 302 starts with the full set of features provided by the feature engineering module 106 and the dataset (e.g., the first iteration of the process). This can be, for example, the graph 206. The module 302 is responsible for the initial selection of features to test. In some examples, the feature selection for a feature set can occur by selecting features based on which dataset the feature belongs to rather than picking features at random. This technique allows for a reduction in the search space for selecting features, thereby increasing the efficiency of the training process.

[0050] Consider an example where datasets A, B, and C include corresponding features. The module 302 can initially pick features from the parent dataset. For example, AAC is initially selected, such that one feature from dataset A is selected, another feature from dataset A is selected, and one feature from dataset C is selected. In some examples, the selection of the initial dataset group (e.g., AAC) can be random. In some examples, the initial selection can be based on a weighted probability of each dataset and / or features within those datasets (e.g., AAC was selected because dataset A was weighted as 2, C was weighted as 1, and B was weighted as 0.25). Thus, the initial randomized selection can be influenced by the weights.

[0051] Once the parent dataset is selected, individual features within those datasets can be selected. In some examples, this can be random. Thus, for example, if dataset A includes features 1-6, module 302 can select feature 1 and feature 4. As with the selection of the dataset, the selection of the features can be influenced by the weights assigned to the features within the dataset. In some examples, the selection of the features is performed without replacement. Thus, a feature will not be selected twice, as the total set of features from which to draw is reduced after the selection of that feature. In some examples, constraints can be applied to control how the features are selected. For example, a constraint can prevent more than n features from being selected from dataset A.

[0052] Hierarchical heuristic graph search module 302 can also include heuristics that control how frequently a given dataset is sampled. In some examples, a relevance value or other score for a dataset can be stored in relation to the target signal that is being sought. For example, if the target signal is to predict the next hurricane, a dataset with South Atlantic temperature values can be weighted more heavily than a dataset with customer service call data. Heuristics can also be applied to individual features based on how those features map to the target signal.

[0053] The likelihood that a dataset (or feature) is more or less relevant to a given target can be determined by testing only that dataset (or feature) against the given target. In some example embodiments, each dataset and / or feature can include a relevance score that is used by the heuristics to determine how frequently a given dataset or feature will be chosen. The relevance score (e.g., the weighted probability discussed above) can indicate how relevant a given dataset or feature is to the target (e.g., how relevant it is to the likelihood of a hurricane occurring).

[0054] Once hierarchical heuristic graph search module 302 has selected an initial set of features, that set of features can be passed to optimizer module 304. In some examples, the optimizer is skipped for the initial set of features and passed directly to policy designer 110.

[0055] Optimizer module 304 is configured to take the initial selection of features and optimize those features (and their corresponding datasets) to more closely approach the target function. In other words, if the initial selection of features is drawn from datasets AAC, optimizer 304 can (after multiple iterations) determine that datasets “R,” “E,” “W” provide the best results (along with the features within those individual datasets). This is accomplished by passing the selected features to policy designer 110 and subsequently optimizing which features to select from graph 206 based on the returned meta-gradient information and / or other data returned from policy designer 110.

[0056] As an example, assume that A1A3C4 (e.g., the first and third features in dataset A and the fourth feature in dataset C) is the set of features processed by the policy designer 110. The metagradient information returned from the policy designer 110 can include information that A1 performed well (e.g., represented by a value of 2), A3 performed poorly (a value of 0.1), C4 performed okay (represented by a value of 1), and the combination of A1C4 performed very well (represented by a value of 4). This metagradient information can be used to modify the selected features for the next iteration (other modifications can be possible based on the particular use case) by dropping A3 and keeping A1 and C4. The graph search in 302 and / or the optimizer 304 can then be used to select a replacement for A3 (the dropped feature) to be included in the selected feature set, which is then passed back to the policy designer 110 for processing.

[0057] In certain examples, a new selected feature set is selected without replacement (e.g., for this next iteration, A3, A1, and C4 are removed from the set of possible selections— A1 and C4 have already been selected, and A3 has already been dropped). In certain examples, A3 is not just dropped from the set of possible features for the next iteration (e.g., it can be removed from the set of possible features entirely for all future iterations of the objective). Thus, in certain examples, the total number of features from which to select can gradually decrease as more and more iterations are performed (and more and more features are “dropped”). In certain examples, a feature (e.g., A3 in the above example) can be removed as a possible selection for only one iteration (or some other predetermined number of iterations). In certain examples, the number of iterations for which a feature is removed is based on the metagradient for that feature (e.g., the worse the metagradient, the more iterations for which the feature is removed as a selection).

[0058] In certain examples, the metagradient information 410 is used to determine what features (or groups of features— i.e., any subset of the features in the selected feature set) within the selected feature set (e.g., the set processed by the policy designer 110) should be kept and what features should be dropped. In other words, how should the feature set be changed and / or how should the features within the feature set be replaced and / or kept. In certain examples, the type of metagradient information can depend on the type of optimization procedure used.

[0059] In certain examples, the dataset scanner module 304 can include different types of optimizers. The optimizers can be based on one or more of the following optimization algorithms: gradient descent, Newton-Raphson, Nelder-Mead, ant colony, greedy search, etc. The optimizer used can depend on the particular problem or objective. For example, if the model being developed is a decision tree, the optimizer can determine the level of usage of particular features in the decision tree. In other words, if feature "A" is used 5 times, and feature "B" is used 1 time, the optimizer can favor including feature "A" in the feature set being analyzed (e.g., the optimizer will pick it more frequently).

[0060] The following is an illustrative example of how the optimizer can work. First, an initial grouping of the dataset is determined. For example, AAB - or one feature from dataset A, another feature from dataset A, and a third feature from dataset B. Next, a selection of features from those datasets can be selected - for example, Al, A4, and B6 (e.g., feature 1 of A, feature 4 of A, and feature 6 of B). This group of features is then passed to the strategy designer 110, which returns meta-gradient information, model data, and a generated strategy for the tested feature set. As noted herein, the meta-gradient information can represent how well each feature within the tested feature set is doing, and how well the collective set of features is doing. The meta-gradient information can represent the sensitivity of the objective function to the features within the tested feature set, and the fitness of the collective set of features. In certain examples, each instance of meta-gradient information returned from the strategy designer can represent a score of how likely that feature set is to be used again (e.g., how well it is performing).

[0061] In certain examples, multiple different instances of meta-gradient information can be combined to infer which features within those selected feature sets are "more" relevant or better for achieving a given objective. For example, assume that features A, B, and C are in a first feature set, and features A, B, and D are in a second feature set. The returned meta-gradient information for ABC can be a value of 2 (where a higher number indicates that the combination is performing "better"), and the meta-gradient information for ABD can be 1. Thus, the individual meta-gradient information for A and B can be 3 (2+1), C can be 2, and D can be 1. The dataset scanner module can then adjust the transition probabilities of selecting A, B, C, and D according to the returned meta-gradient information (e.g., as part of a perturbation function). This can cause A and B to be selected (or more likely to be retained in the next feature set to be tested) more frequently than previously (relatively), while D is selected less frequently than A and B.

[0062] The optimizer module 304 uses the returned policy and the returned meta-gradient information to update the heuristics and transition probabilities in the graph search used to store the dataset and features. In certain examples, constraint functions and / or penalty functions are also applied. Thus, the original graph and / or the heuristics associated with the graph can be updated and / or modified based on the returned policy and meta-gradient information.

[0063] Figure 3 The convergence block 306 in the selected feature set 306 tests whether convergence criteria are met for the selected feature set. If there is convergence, the final feature set 310 and / or the dataset based on the feature set is output. If there is no convergence, a new selected feature is passed to the policy designer 110.

[0064] In certain examples, convergence can be achieved by reaching or exceeding a predefined level of performance (return on investment, accuracy, etc...). In certain examples, convergence can be met by passing a predefined number of iterations without producing a new best performing model. In certain examples, convergence can be met when a consecutive number of iterations have a performance improvement below a defined threshold. For example, if the threshold is 0.01, and the "best model" is 1, and a series of models are developed with values of 1, 1.0002, 1.001, 0.9999, then the models can be considered to have converged. In certain examples, convergence can be achieved when the best model is not improved after a given number of iterations.

[0065] Description of the policy designer module Figure 4 :

[0066] Figure 4 FIG. 1 illustrates an example system in accordance with certain example embodiments. Figure 1 An example policy designer module 110 of the system shown in FIG. 1.

[0067] The list or set of features selected by the dataset scanner 108 is passed to the policy designer module 110. Upon receipt of the list of features, the policy designer module (or another module) will retrieve the data in the columns associated with those features and generate a new dataset. During this process, NaN (not a number) data is handled (e.g., set to zero or removed from the dataset), and the customized dataset is passed to the policy module 110.

[0068] In particular, the customized dataset is passed into an expectation learner 404. Expectation learning is part of the machine learning module 402, and the expectation learner 404 learns the objective. The data is also passed to a policy learner 406, which also takes in the expectations and errors from the expectation learner 404. The policy learner module 406 takes the expectations about the objective and converts them into actionable policies 412 (which are scaled by the scaling transformer module 408). In certain instances, the policies can be referred to as policies. The policies reflect the strategy that would be applied to the selected set of features to achieve (or at least seek to achieve) the objective signal. For example, if the objective signal is investment returns, then the derived policies for the selected set of features would be the strategies that would seek to achieve that objective for those features.

[0069] One of the potential outputs from the machine learning module 402 is a model 414. The model 414 can be used on future datasets of the selected set of features to determine the policies that should be implemented based on the input data. Thus, if a particular model is developed using the techniques herein with a feature set list A. Then that same model can be used again with the same feature set list A with updated data (e.g., including new data from the past 6 months) in 6 months to develop a policy based on or using the originally created model.

[0070] The model 414 can include or involve the parameters of the model, the model parameterization, the generated model, and / or the policy 412 (sometimes referred to as a policy). Thus, in certain examples, the model 414 and the policy 412 generated from the policy designer module 110 can be the same.

[0071] The following is an example where the model 414 and the policy 412 can be the same (or nearly the same). The model 414 is generated to provide a signal for buying and selling stocks (e.g., every day). A policy 412 can be generated that uses the buy / sell signal to execute the buy and sell of a given stock (or group of stocks, etc...). In such an example, the model and the policy are functionally equivalent (or at least very similar). However, another policy 412 can be generated that uses the same buy / sell signal from the same model, which further acts on the buy / sell signal. For example, the policy can use the signal such that a buy is executed only when there are two consecutive days of model output buy signals. In such an example, the policy 412 further acts on the signal output from the model.

[0072] As another example, consider a model that predicts whether it will rain on a given day (e.g., the output from the model is simply a yes / no for a rain prediction). A policy that utilizes the output of the model (e.g., it acts on the output) can be generated that decides whether a person should take an umbrella.

[0073] The scaling converter 408 takes the output signal from the machine learning module 402 and generates a human understandable element form (or other machine readable form). In certain instances, the scaling converter 408 acts as a translator between the machine learning module 402 and whatever is consuming the output from the machine learning module 402. If the consumer is a human, this can result in, for example, red / green light output, or a plus or minus output.

[0074] The meta-gradient calculator 400 takes the resulting model 414 developed by the machine learning module 402 and processes the model to determine important things about the model developed by the machine learning module 402. This can use, for example, LIME (Local Interpretable Model- Agnostic Interpretation), deep Taylor decomposition / expansion, Gini score, etc. In other words, the model developed by the machine learning module 402 is processed to obtain meta-gradient information 410 about the model (or features within the model). It will be appreciated that different techniques for obtaining the meta-gradient information 410 can be used in accordance with the techniques described herein.

[0075] Finally, the strategy designer 110 returns the meta-gradient information 410 to the data set scanner 108, to the model 414 that converted the input signal to the output signal, and / or to the strategy (e.g., which has been scaled by the scaling converter module 408). This information can be used to further evaluate and perturb the graph search performed by the data set scanner module 108. In certain examples, this strategy can be passed to a user to determine how to proceed with this particular developed strategy.

[0076] Example Pseudocode

[0077] The following is example pseudo code for the data set scanner 108:

[0078] Requirements : F: Set of all feature sets

[0079] Requirements : H_p : Probability of selecting a feature in the hierarchy of all features

[0080] Requirements : : Number of features to find

[0081] 1: f = Select n features from F with H_p

[0082] 2: While not done (e.g., not converged) do :

[0083] 3: Strategy Designer Strategy, Model, Meta-Gradient

[0084] 4: if J (Strategy) ≥ max( J ( Historical Strategy ))

[0085] 5: Strategy, Model , f -> Best Strategy, Best Model, Best f

[0086] 6: Disturbance_f ( Meta-Gradient, H_p ) → f

[0087] 7: Return ( Best Strategy, Best Model, Best f )

[0088] For the pseudocode example of the dataset scanner 108, the set of all features (or all features) is provided as " F The probability of selecting a feature for a given set of selected features is provided as "". H_p As explained in this paper, this value can be static (e.g., the relevance value of a given feature) and can be further modified using meta-gradient data generated during processing performed by the policy designer. In some examples, the number of features to be sought (e.g., which will be included in a given feature set) can be provided as " In some examples, this could be the maximum number of features, or a strict value where all selected feature sets have that number of features. Next, a selected feature set is generated from the set of all feature sets. The features within this selected set can be based on… H_p Choose based on probability.

[0089] Once the first feature set is selected, the process continues until convergence. During this process, each selected feature set and loss function is provided to the policy designer 110, which returns the policy, model, and meta-gradient data. If the objective achieved by the policy (e.g., "J(policy)") is better than all previously achieved objectives for a previously developed policy, the new optimal policy, along with its corresponding model and the features used to develop the model and policy, is stored. A new feature set is selected (or features within the feature set are replaced / changed) based on a perturbation of all possible feature sets (or a set of all possible features). This can take into account the returned meta-gradient information and / or the initial selection probabilities. Once convergence is achieved, the optimal policy, optimal model, and optimal selected feature set are returned as outputs.

[0090] The following is sample pseudocode for Strategy Designer 110:

[0091] Requirements: f: Set of features

[0092] Requirements :: : Loss Function

[0093] 1: Randomly initialize theta

[0094] 2: Train theta on Compute objective

[0095] 3: Compute meta-gradient of theta with respect to J

[0096] 4: Return

[0097] 5: theta ( , Meta-Gradient , Figure 5 )

[0098] For the pseudocode example of the strategy designer 110, the input can include the selected feature set ( f ) and loss function ( The initialization of θ may include the initialization of the model's parameters, model parameterization, and the model and / or policy (sometimes called the strategy). The model (e.g., θ) is then trained on a loss function in a manner that minimizes the loss function relative to the given model and / or policy. This may include generating the model, the model's parameters, model parameterization, and / or policy. Once the model is trained, it is then trained relative to the trained model and / or policy (e.g., it is for a given feature set). f Provide value) Calculate target ( J This is sometimes referred to as the objective function. Meta-gradient information is then computed against the model and / or policy with respect to the computed objective. Such meta-gradients are per-feature and / or per-feature set.

[0099] Figure 5 of the description

[0100] Figure 5is a block diagram of an example computing device 500 (which can also be referred to as, e.g., a “computing device,” a “computer system,” or a “computing system”) according to some embodiments. In some embodiments, the computing device 500 includes one or more of the following: one or more processors 502; one or more memory devices 504; one or more network interface devices 506; one or more display interfaces 508; and one or more user input adapters 510. Additionally, in some embodiments, the computing device 500 is connected to or includes a display device 512. As will be explained below, these elements (e.g., the processors 502, the memory devices 504, the network interface devices 506, the display interfaces 508, the user input adapters 510, the display device 512) are hardware devices (e.g., electronic circuits or combinations of circuits) configured to perform various different functions for the computing device 500.

[0101] In some embodiments, each or any of the processors 502 is or includes, e.g., a single-core or multi-core processor, a microprocessor (which can be referred to as a central processing unit or CPU), a digital signal processor (DSP), a microprocessor associated with a DSP core, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, or a system on a chip (SOC) (e.g., an integrated circuit that includes a CPU and other hardware components such as memory, networking interfaces, etc.). And / or, in some embodiments, each or any of the processors 502 uses an instruction set architecture such as x86 or Advanced RISC Machines (ARM).

[0102] In some embodiments, each or any of the memory devices 504 is or includes random access memory (RAM) (e.g., dynamic RAM (DRAM) or static RAM (SRAM)), flash memory (based on, e.g., NAND or NOR technology), a hard disk, a magneto-optical medium, an optical medium, cache memory, registers (e.g., that hold instructions), or other types of devices that store volatile or non-volatile storage of execution data and / or instructions (e.g., software executed on or by the processors 502). The memory devices 504 are examples of non-transitory computer-readable storage media.

[0103] In some embodiments, each or any of the network interface devices 506 includes one or more circuits (such as a baseband processor and / or a wired or wireless transceiver) and implements Layer 1, Layer 2, and / or higher for one or more wired communication technologies (such as Ethernet (IEEE 802.3)) and / or wireless communication technologies (such as Bluetooth, WiFi (IEEE 802.11), GSM, CDMA2000, UMTS, LTE, LTE-A Advanced, and / or other short-range, medium-range, and / or long-range wireless communication technologies). The transceiver may include circuitry for both the transmitter and receiver. The transmitter and receiver may share a common housing and may share some or all of the circuitry within the housing to perform transmission and reception. In some embodiments, the transmitter and receiver of the transceiver may not share any common circuitry and / or may be located in the same or separate housings.

[0104] In some embodiments, each or any of the display interfaces 508 is or includes one or more circuits that receive data from the processor 502, generate (e.g., via a discrete GPU, an integrated GPU, a CPU performing graphics processing, etc.) corresponding image data based on the received data, and / or output the generated image data (e.g., via a High Definition Multimedia Interface (HDMI), DisplayPort Interface, Video Graphics Array (VGA) Interface, Digital Video Interface (DVI), etc.) to a display device 512 that displays the image data. Alternatively or additionally, in some embodiments, each or any of the display interfaces 508 is or includes, for example, a video card, a video adapter, or a graphics processing unit (GPU).

[0105] In some embodiments, each or any of the user input adapters 510 is or includes one or more circuits that receive and process input from one or more user input devices. Figure 5 User input data (not shown in the diagram), wherein the user input device is included in, attached to, or otherwise communicates with the computing device 500, and the circuitry outputs data to the processor 502 based on the received input data. Alternatively or additionally, in some embodiments, each or any of the user input adapters 510 is or includes, for example, a PS / 2 interface, a USB interface, a touchscreen controller, etc.; and / or the user input adapter 510 facilitates input data from the user input device (…). Figure 5 Input (not shown in the image) is provided by a user input device such as, for example, a keyboard, mouse, touchpad, touchscreen, etc.

[0106] In some embodiments, display device 512 can be a liquid crystal display (LCD) display, a light emitting diode (LED) display, or other type of display device. In embodiments in which display device 512 is a component of computing device 500 (e.g., the computing device and display device are included in a unified housing), display device 512 can be a touch screen display or a non-touch screen display. In embodiments in which display device 512 is connected to computing device 500 (e.g., external to computing device 500 and in communication with computing device 500 via a wired and / or via wireless communication techniques), display device 512 is, for example, an external monitor, a projector, a television, a display screen, etc.

[0107] In various embodiments, computing device 500 includes one, or two, or three, four or more of each or any of the above-mentioned elements (e.g., processor 502, memory device 504, network interface device 506, display interface 508, and user input adapter 510). Alternatively or additionally, in some embodiments, computing device 500 includes one or more of: a processing system including processor 502; a memory or storage system including memory device 504; and a network interface system including network interface device 506.

[0108] In various embodiments, computing device 500 can be arranged in many different ways. As just one example, computing device 500 can be arranged such that processor 502 includes: a multi-core (or single core) processor; a first network interface device (which implements, for example, WiFi, Bluetooth, NFC, etc.); a second network interface device that implements one or more cellular communication technologies (e.g., 3G, 4G LTE, CDMA, etc.); a memory or storage device (e.g., RAM, flash memory, or a hard disk). The processor, first network interface device, second network interface device, and memory device can be integrated as part of the same SOC (e.g., one integrated circuit chip). As another example, computing device 500 can be arranged such that: processor 502 includes two, three, four, five or more multi-core processors; network interface device 506 includes a first network interface device that implements Ethernet and a second network interface device that implements WiFi and / or Bluetooth; and memory device 504 includes RAM and flash memory or a hard disk.

[0109] As noted previously, whenever an action is described in this document as being performed by a software module or software process, the action is actually performed by underlying hardware elements in accordance with instructions included in the software module. Consistent with the foregoing, in various embodiments, the use of Figure 5The example of computing device 500 implements computer system 100, ETL module 102, training system 104, feature engineering design module 106, dataset scanner 108, policy designer 110—each of which, for clarity, will be referred to individually as a “component”—or any combination thereof. In such an embodiment, the following applies to each component: (a) Figure 5 The elements of the computing device 500 shown herein (i.e., one or more processors 502, one or more memory devices 504, one or more network interface devices 506, one or more display interfaces 508, and one or more user input adapters 510, or suitable combinations or subsets thereof) are configured, adapted to, and / or programmed to implement each or any combination of the actions, activities, or features described herein as being performed by the components and / or any software modules described herein as being included within the components; (b) alternatively or additionally, with regard to the presence of one or more software modules described herein within the components, in some embodiments such software modules (and any data described herein as being disposed of and / or used by the software modules) are stored in memory device 504 (e.g., in various embodiments, in volatile memory devices such as RAM or instruction registers, and / or in non-volatile memory devices such as flash memory or hard disks), and all actions described herein as being performed by the software modules are performed by processor 502 in combination as appropriate with other elements in and / or connected to the computing device 500 (i.e., network interface devices 506, display interfaces 508, user input adapters 510, and...). (c) Alternatively or additionally, in some embodiments, such data is stored in memory device 504 (e.g., in some embodiments, in a volatile memory device such as RAM, and / or in a non-volatile memory device such as flash memory or hard disk) and / or by processor 502 in combination with other elements in and / or connected to computing device 500 (i.e., network interface device 506, display interface 508, user input adapter 510) as appropriate. (d) Alternatively or additionally, in some embodiments, memory device 502 stores instructions that, when executed by processor 502, cause processor 502 to, as appropriate, combine with other elements in and / or connected to computing device 500 (i.e., memory device 504, network interface device 506, display interface 508, user input adapter 510, and / or display device 512) to perform each or any combination of actions described herein as performed by components and / or any software modules described herein as included within components.

[0110] Consistent with the techniques described herein, as one example, in embodiments in which an instance of the computing device 500 is used to implement the training system 104, the memory device 504 can load program instructions for the functionality of the feature engineering module 106, the data set scanner 108, and the policy designer module 110. Data for all features to be processed by the feature engineering module 106 can be loaded from the memory device. The loaded features can be processed according to the program instructions of the feature engineering module 106 to generate engineered features, which are then stored to the memory device 504. A process (which can operate within a virtual machine implementing the modules described herein) can then execute the data set scanner and / or the policy designer as described herein.

[0111] Figure 5 The hardware configuration shown in FIG. 1 and described above is provided as an example, and the subject matter described herein can be utilized in conjunction with a variety of different hardware architectures and elements. For example, in many of the figures in this document, individual functional / action blocks are shown; in various embodiments, the functionality of those blocks can be implemented using (a) individual hardware circuits, (b) using application specific integrated circuits (ASICs) that are specifically configured to perform the described functionality / actions, (c) using one or more digital signal processors (DSPs) that are specifically configured to perform the described functionality / actions, (d) using the hardware configurations described above with reference to FIG. 1, (e) via other hardware arrangements, architectures, and configurations, and / or via combinations of the techniques described in (a) through (e). Technical advantages of the described subject matter

[0112] Selected terms

[0113] In certain example embodiments, the techniques herein allow for improved techniques in selecting features from a large number of possible features (e.g., hundreds or thousands). Certain examples use meta-gradient information generated from models developed using machine learning. The meta-gradient information is then used to select new features to test against additional sets of features. This type of approach allows the process to handle problems with an extremely large number of possible combinations. For example, the techniques herein can be used to handle more than 10 17 combinations - which would otherwise be computationally infeasible to process. The techniques herein employ a smarter approach to selecting or generating features to test (via the meta-gradient information) than random selection, which can be a more conventional approach. This smarter approach can improve convergence and / or the number of search iterations needed to reach convergence. This allows for more efficient (e.g., superior to random) use of computational resources (CPU, memory, etc...) in finding a solution.

[0114] ​The technical features described herein can thus improve the speed at which seemingly unrelated combinations of data can be analyzed and processed to determine previously unknown correlations between different data sets and features in those data sets.

[0115] In certain examples, these techniques improve the models developed while avoiding limitations associated with data processing inequalities (e.g., which cannot be improved upon by engineering features alone). The improved models are generated by finding higher quality data arrangements rather than simply adding more features to the traditional approach of developing models. Using fewer (and smarter) features to develop models allows for faster learning and run times.

[0116] The technical features herein can also allow for very diverse sets of data to be analyzed together. For example, the number of mobile phones sold in June can be one data set, the amount of cars in a mall can be another data set, and the weather can be a third data set. Features from these three data sets can be analyzed together using the techniques described herein to reveal important insights. This is made possible by the meta-gradient based feedback loop that allows for fast exploration and discovery of useful combinations of features from a large number of heterogeneous data sets. The meta-gradient information can provide information about data sets, features, or combinations of features that allows for more targeted and useful selection of features from across heterogeneous data sets.

[0117] The techniques herein also allow for high levels of customization through the use of objectives, constraints, and penalties.

[0118] Additional applications of the described subject matter

[0119] Whenever a given item is described herein as being present "in some embodiments," "in various embodiments," "in certain embodiments," "in some example embodiments," "in some example implementations," "in an example embodiment," or otherwise, it should be interpreted that such given item is present in at least one embodiment, though not necessarily in all embodiments. In accordance with the foregoing, whenever a claim refers to "a," "an" or "the" item, such claim should be interpreted to mean that such given item is present in at least one embodiment, though not necessarily in all embodiments. The terms "comprises," "comprising," "includes," "including," "has," "having," "contains," "containing," or any other similar term are intended to be interpreted as including the stated item, but not excluding other items. The term "comprises" is intended to be interpreted as including the stated item, but not excluding other items. The term "consisting of is intended to be interpreted as including only the stated item, and excluding other items. The term "consisting essentially of is intended to be interpreted as including the stated item, and excluding other items, except for those items that do not materially affect the essential characteristics and functions of the item. The term "and / or" is intended to be interpreted as including any and all combinations of one or more of the associated listed items. The terms "a," "an," and "the" are intended to be interpreted to mean "at least one" or "one or more." The term "example" is used to provide examples of the subject matter discussed, not to imply or warrant that the subject matter is or has been made by, or is in any way associated with, the assignee hereof. The terms "include," "including," and "includes" should be interpreted as including the stated item, but not excluding other items. The term "optional" is intended to mean that the item is not required, but can be included if desired.

[0120] As used herein, the term "non-transitory computer-readable storage medium" includes a register, a cache memory, a ROM, a semi-conductor memory device such as a D-RAM, S-RAM, or other RAM, a magnetic medium such as a flash memory, a hard disk, a magneto-optical medium, an optical medium such as a CD-ROM, a DVD, or a Blu-ray disc, or other types of devices for non-transitory electronic data storage. The term "non-transitory computer-readable storage medium" does not include a transitory propagating electromagnetic signal.

[0121] Figures 1-4

[0122] Although process steps, algorithms, etc. are described herein without limitation as including reference to ​Processes can be described or claimed in terms of a specific order of actions or steps. Unless otherwise specified, such processes can be practiced with the order of actions or steps described or claimed, in any order that is possible. In other words, any sequence or order of steps that can be described or claimed in this document that is not expressly set forth in a claim is not intended to be limiting. Moreover, steps can be added or omitted, and other steps can be substituted, in any order or combination, without departing from the scope of the present disclosure. Further, although some steps can be described or implied as being performed sequentially, in some embodiments, these steps can be performed in parallel (or concurrently) with each other. In addition, the depiction of a process by its depiction in a drawing does not imply that the illustrated process is exclusive of other variations and modifications, nor does it mean that the illustrated process is essential, nor does it imply that a particular step is necessary. Further, the depicted process is not exclusive, and the steps recited need not be performed in the order illustrated, or all explained.

[0123] While various embodiments have been illustrated and described, the claims are not limited to any particular embodiment or example. No aspect, step, feature, or element of any above-described embodiment should be construed as being an essential element or necessary for the practice of the invention. All structural and functional equivalents to elements described above as well as those known to those of ordinary skill in the art are expressly incorporated herein by reference and intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether these embodiments are ultimately claimed.

Claims

1. A computer system comprising: a non-transitory storage medium configured to store a plurality of data sets, each data set comprising a plurality of features; a processing system comprising at least one hardware processor, the processing system configured to: (a) select a feature set comprising a plurality of features from the plurality of data sets; (b) train a model using data values of the plurality of features of the feature set, wherein each of the plurality of features has a probability value selected for the selected feature set; (c) compute meta-gradient data for each feature in the feature set based on the trained model, wherein the meta-gradient data represents a sensitivity of an objective function to the features within the feature set and a fit of the collective set of overall features; (d) adjust the probability value of at least one of the plurality of features based on the computed meta-gradient data; (e) make a change by taking into account the computed meta-gradient data to select which of the features of the plurality of data sets are included in the feature set; and repeat at least (b)-(e) until a convergence criterion is met for the feature set.

2. The computer system of claim 1, wherein the plurality of data sets and the plurality of features are stored in a hierarchical graph data structure.

3. The computer system of claim 2, wherein the hierarchical graph data structure comprises a root node having a plurality of first child nodes, each first child node corresponding to one of the plurality of data sets, wherein each of the plurality of first child nodes has a plurality of second child nodes, each second child node corresponding to one of the plurality of features of the corresponding data set, wherein the change to select which features are included is performed using the hierarchical graph data structure.

4. The computer system of claim 1, wherein the change of features included in the feature set is further based on applying at least one of a penalty function or at least one constraint function.

5. The computer system of claim 1, wherein the change of features included in the feature set is further based on a selection of a data set from among the plurality of data sets, and then a selection of features from within the selected data set for the selected feature set.

6. The computer system of claim 5, wherein each data set of the plurality of data sets is assigned a selection probability, and the selection of the data set is further based on the selection probability.

7. The computer system of claim 5, wherein each feature of the plurality of features within each data set of the plurality of data sets is assigned a selection probability, and the change of which features are selected to be included in the feature set is further based on the selection probability.

8. The computer system of claim 1, wherein the processing system is further configured to: prior to (a), compute a relevance value for each of the plurality of data sets, the relevance value indicating a relevance of the corresponding data set to a target signal.

9. The computer system of claim 8, wherein the selection of the feature set in (a) is further based on the relevance value associated with each data set of the plurality of data sets.

10. The computer system of claim 1, wherein the model is trained with a machine learning process comprising an expected learner and a policy learner. ​ 11. A method performed on a computer system comprising at least one hardware processor, the method comprising: storing a plurality of data sets to a data storage device coupled to the computer system, each data set comprising a plurality of features; selecting a feature set from among the plurality of features of the plurality of data sets; and performing a process of (a)-(d) until a convergence criterion (e) is met for the feature set, the process comprising: (a) training a model using data values of those features included in the feature set, (b) computing meta-gradient data for the feature set with respect to the trained model, wherein the meta-gradient data represents a sensitivity of an objective function to features within the feature set and a fit of the collective set of overall feature sets, (c) adjusting a selection probability of at least one of the plurality of features based on the computed meta-gradient data, (d) replacing at least one feature in the feature set by taking into account the selection probability adjusted according to the meta-gradient data, and (e) determining whether the convergence criterion is met for the feature set in (d).

12. The method of claim 11, further comprising: storing a selection probability value for each of the plurality of features, wherein the application of the computed meta-gradient data adjusts the selection probability value of at least one feature.

13. The method of claim 11, wherein the plurality of data sets and the plurality of features are stored in a hierarchical graph data structure, the hierarchical graph data structure comprising a root node having a plurality of first child nodes, each first child node corresponding to one of the plurality of data sets, wherein each of the plurality of first child nodes has a plurality of second child nodes, each second child node corresponding to one of the plurality of features of the corresponding data set, wherein the hierarchical graph data structure includes at least one value for a selection probability, wherein the at least one value is adjusted based on the computed meta-gradient data.

14. The method of claim 11, wherein the replacement of features in the feature set is further based on application of at least one of a penalty function or at least one constraint function.

15. The method of claim 11, wherein, The replacement of features in the feature set is further based on a selection of a data set from among the plurality of data sets and then a selection of features from within the selected data set for the selected feature set.

16. The method of claim 11, wherein the replacement of at least one feature is performed without replacement.

17. The method of claim 11, wherein the model is trained using a machine learning process comprising an expected learner and a policy learner.

18. A non-transitory computer-readable storage medium storing computer- executable instructions for use by a computer system comprising at least one hardware processor, the computer system coupled to a storage device storing a plurality of data sets, each data set comprising a plurality of features, the computer-executable instructions comprising instructions that cause the computer system to: (a) select a feature set comprising a plurality of features from the plurality of data sets; (b) train a model using data values of the plurality of features of the feature set, wherein each of the plurality of features has a probability value selected for the selected feature set; (c) computing meta-gradient data based on the trained model, wherein the meta-gradient data represents a sensitivity of the objective function to features within the feature set and a fitness of the overall feature set body; (d) adjusting a probability value of at least one feature of the plurality of features based on the computed meta-gradient data; (e) changing, by taking into account the computed meta-gradient data, which of the features of the plurality of data sets are included in the feature set; and repeating at least (b)-(e) until a convergence criterion is satisfied for the feature set.

19. The non-transitory computer-readable storage medium of claim 18, wherein the changing of the features included in the feature set is further based on applying at least one of a penalty function or at least one constraint function.

Images