Integrated Future Engineering

JP2024502027A5Active Publication Date: 2024-11-27ALTERYX INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023539955
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2020-12-30
Filing Date
2021-12-16
Publication Date
2024-11-27
Estimated Expiration
2041-12-16

AI Technical Summary

Technical Problem

Current feature engineering tools face challenges in efficiently combining data from multiple sources due to architectural limitations, leading to time-consuming processes and difficulties in user interaction, which affects the accuracy and efficiency of machine learning models.

Method used

An integrated feature engineering system that generates primitives from multiple data entities, applies time parameters, and synthesizes features using entity sets, allowing for automated and efficient feature creation without requiring additional data during processing, and trains models based on these features.

Benefits of technology

The system enhances data processing efficiency, reduces user complexity, and improves feature engineering outcomes by enabling rapid and accurate feature creation, even with limited data, ultimately leading to more effective machine learning model predictions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

The feature engineering application receives a plurality of datasets from different data sources to train a model for making predictions based on new data, generates primitives based on the datasets, and the primitives are applied to variables of the datasets to synthesize features. Additionally, the feature engineering application also receives a time parameter specifying a time value for generating time-based features, and after the primitives are generated and the time parameter is received, aggregates a plurality of data entities based on primary variables of the plurality of data entities, generates an entity set based on the aggregation, and then synthesizes features including time-based features based on the entity set, at least some of the primitives, and the time parameter.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The described aspects relate generally to processing data streams, and in particular to integrated feature engineering in which automated feature engineering is integrated with entity set creation to perform machine learning on the stream's data. [Background technology]

[0002] Feature Engineering is the process of identifying and extracting predictive features in complex data typically analyzed by companies and other enterprises. Features are important for the accuracy of predictions made by machine learning models. Therefore, feature engineering is often the determining factor for the success of data analytics projects. Feature engineering is often a time-consuming process and typically requires a significant amount of data to achieve good prediction accuracy. Often, the data used to create features is from different sources and data combination is required before feature engineering. However, there are architectural challenges in moving data between tools for combining data and tools for creating features, making the process of creating features even more time-consuming. In addition, the architectural challenges make it more difficult for data analytics engineers to interact with the feature creation process. Therefore, current feature engineering tools cannot efficiently serve the data processing needs of enterprises. Summary of the Invention

[0003] The above and other problems are addressed by a method, a computer-implemented system, and a computer-readable memory. An aspect of the method includes receiving a plurality of data entities from different data sources. The plurality of data entities are used to train a model that makes predictions based on new data. The method further includes generating primitives based on the plurality of data entities. Each of the primitives is configured to be applied to variables of the plurality of data entities to synthesize features. The method further includes receiving a time parameter from a client device associated with a user. The time parameter specifies a time value and is to be used to synthesize features from the plurality of data entities. The method further includes generating an entity set by aggregating the plurality of data entities after the primitives are generated and the time parameter is received. The method further includes synthesizing a plurality of features based on the entity set, the primitives, and the time parameter. The method further includes training a model based on the plurality of features.

[0004] An aspect of the computer-implemented system includes a computer processor for executing computer program instructions. Additionally, the system includes a non-transitory computer readable memory storing computer program instructions executable by the computer processor to perform operations. The operations include receiving a plurality of data entities from different data sources. The plurality of data entities are used to train a model to make predictions based on new data. The operations further include generating primitives based on the plurality of data entities. Each of the primitives is configured to be applied to variables of the plurality of data entities to synthesize features. The operations further include receiving a time parameter from a client device associated with a user. The time parameter specifies a time value and is to be used to synthesize features from the plurality of data entities. The operations further include generating an entity set by aggregating the plurality of data entities after the primitives are generated and the time parameter is received. The operations further include synthesizing a plurality of features based on the entity set, the primitives, and the time parameter. Additionally, the operations also include training a model based on the plurality of features.

[0005] An embodiment of the non-transitory computer readable memory stores executable computer program instructions. The instructions are executable to perform operations. The operations include receiving a plurality of data entities from different data sources. The plurality of data entities are used to train a model to make predictions based on new data. The operations further include generating primitives based on the plurality of data entities. Each of the primitives is configured to be applied to variables of the plurality of data entities to synthesize features. The operations further include receiving a time parameter from a client device associated with a user. The time parameter specifies a time value and is to be used to synthesize features from the plurality of data entities. The operations further include generating an entity set by aggregating the plurality of data entities after the primitives are generated and the time parameter is received. The operations further include synthesizing a plurality of features based on the entity set, the primitives, and the time parameter. The operations further include training a model based on the plurality of features.

[0006] The drawings depict various embodiments for illustrative purposes only. Those skilled in the art will readily appreciate from the discussion that follows that alternative embodiments of the structures and methods illustrated herein may be utilized without departing from the principles of the embodiments described herein. In the various drawings, the same reference numbers and names indicate the same elements. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram illustrating a machine learning environment including a machine learning server according to one embodiment. [Diagram 2] FIG. 1 is a block diagram illustrating a feature engineering application according to one aspect. [Diagram 3] FIG. 1 illustrates a user interface that allows user input for primitive and time parameters according to one embodiment. [Figure 4] FIG. 1 illustrates a user interface for entity set creation according to one aspect. [Diagram 5] 1 is a flow chart illustrating a method for synthesizing features by using data entities received from different data sources according to one aspect. [Figure 6] FIG. 2 is a high-level block diagram illustrating a functional diagram of an exemplary computer system for use as the machine learning server of FIG. 1 according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] 1 is a block diagram illustrating a machine learning environment 100 including a machine learning server 110 according to one embodiment. The environment 100 further includes multiple data sources 120 connected to the machine learning server 110 via a network 130. Although the illustrated environment 100 includes only one machine learning server 110 coupled to multiple data sources 120, embodiments can have multiple machine learning servers and a single data source.

[0009] The data sources 120 provide electronic data to the data analysis system 110. The data sources 120 may be storage devices, such as hard disk drives (HDDs) or solid state drives (SSDs), a computer that manages and provides access to multiple storage devices, a storage area network (SAN), a database, or a cloud storage system. Additionally, the data sources 120 may be computer systems capable of retrieving data from different sources. Different data sources may be associated with different users, different organizations, or different departments within the same organization. The data sources 120 may be remote from the machine learning server 110 and provide data over the network 130. Additionally, some or all of the data sources 120 may be directly coupled to the data analysis system and provide data without passing the data over the network 130.

[0010] The data provided by the data sources 120 may be organized into data records (e.g., rows). Each data record includes one or more values. For example, a data record provided by the data sources 120 may include a series of comma-separated values. The data describes information related to an enterprise using the data analysis system 110. For example, the data from the data sources 120 may describe computer-based interactions with content accessible at a website and / or application (e.g., click tracking data). The enterprise may be in one or more of a variety of industries, such as, for example, computer technology and manufacturing.

[0011] The machine learning server 110 is a computer-based system utilized to build machine learning models and provide machine learning models that can be used to make predictions based on data. Exemplary predictions include application monitoring, network traffic data flow monitoring, user action predictions, and the like. Data is collected, aggregated, or otherwise accessed from multiple data sources 120 via a network 130. The machine learning server 110 can implement scalable software tools and hardware resources employed in accessing, preparing, blending, and analyzing data from a wide variety of data sources 120. The machine learning server 110 can be a computing device used to implement machine learning functions, including feature engineering and modeling techniques described herein.

[0012] The machine learning server 110 can be configured to support one or more software applications, illustrated in FIG. 1 as feature engineering application 150 and training application 160. The feature engineering application 150 performs integrated feature engineering, in which automated feature engineering is integrated with the creation of entity sets. The integrated feature engineering process begins by generating feature engineering algorithms and parameters from individual data entities, then proceeds to create entity sets by combining the individual data entities, and further proceeds to extract predictive variables, or features, from the data in the entity sets. Each feature is a variable potentially related to a prediction (called a target prediction) that a corresponding machine learning model will be used to make.

[0013] The feature engineering application 150 generates primitives based on individual data entities. In one aspect, the feature engineering application 150 selects primitives from a pool of primitives based on individual data entities. The pool of primitives is maintained by the feature engineering application 150. The primitives define individual calculations that can be applied to raw data of a dataset to create one or more new features with associated values. The selected primitives can be applied and stacked across different kinds of data to constrain input and output data types to create new calculations. The feature engineering application 150 allows a user (e.g., a data analysis engineer, etc.) to provide time values ​​that can later be used by the feature engineering application 150 to create time-based features. Time-based features are features extracted from data associated with a particular time or time period. Time-based features may be used to train a model to make time-based predictions, such as predictions for a particular time or time period.

[0014] After generating the primitives and receiving the time parameters from the user, the feature engineering application 150 combines the individual data entities to generate an entity set. In some aspects, the feature engineering application 150 generates the entity set based on variables in the individual data entities. For example, the feature engineering application 150 identifies two individual data entities that have a common variable, determines a parent-child relationship between them, and generates an intermediate data entity based on the parent-child relationship. The feature engineering application 150 combines the intermediate data entities to generate the entity set.

[0015] After the entity set is generated, the feature engineering application 150 synthesizes features by applying primitives and time parameters to the data of the entity set. It then evaluates the features to determine their importance through an iterative process that applies different parts of the data to the features at each iteration. The feature engineering application 150 removes some of the features at each iteration to obtain a subset of features that are more useful for prediction than the removed features. For each feature of the subset, the feature engineering application 150 determines an importance coefficient, for example using a random forest. The importance coefficient indicates how important / relevant the feature is to the target prediction. The features of the subset and their importance coefficients can be sent to a training application 160 that builds a machine learning model.

[0016] Compared to conventional feature engineering tools, the architecture of feature engineering application 150 facilitates more efficient data processing and provides a less complicated experience for users who can provide input to the feature engineering process. Feature engineering application 150 integrates automated feature engineering and entity set creation in a manner that allows data input (either from data source 120 or user) for both automated feature engineering and entity set creation at the beginning of the integrated process, avoiding requesting data in the middle of the process. In this way, feature engineering application 150 can operate efficiently since it does not need to wait for responses from other entities during operation. Furthermore, all user input (including editing primitives and providing time parameters) at the beginning of the integrated feature engineering process also provides a better experience for users. The user does not need to monitor the rest of the process, thus overcoming the challenges faced by conventional feature engineering tools.

[0017] Another advantage of the feature engineering application 150 is that the use of primitives makes the feature engineering process more efficient than traditional feature engineering processes in which features are extracted from raw data. Furthermore, the feature engineering application 150 can evaluate the primitives based on the evaluation and importance coefficients of the feature(s) generated from the primitives. It is possible to generate metadata describing the evaluation of the primitives and use the metadata to determine whether to select the primitives for different data or different prediction problems. Traditional feature engineering processes can generate a huge number of features (e.g., millions, etc.) without providing guidance or solutions to engineer faster and better features. Yet another advantage of the feature engineering application 150 is that it does not require a large amount of data to evaluate the features. Instead, it applies an iterative approach to evaluate the features, using a different portion of the data in each iteration.

[0018] The training application 160 trains a machine learning model with the features and feature importance coefficients received from the feature engineering application 150. Different machine learning techniques, such as linear support vector machines (linear SVMs), boosting other algorithms (e.g., AdaBoost), neural networks, logistic regression, naive Bayes, memory-based learning, random forests, bagged trees, decision trees, boosted trees, or boosted stumps, may be used in different embodiments. The generated machine learning model makes target predictions when applied to features extracted from a new dataset (e.g., a dataset from the same or a different data source 120).

[0019] In some embodiments, the training application 160 validates predictions before deploying the trained model to a new dataset. For example, the training application 160 applies the trained model to a validation dataset to quantify the accuracy of the model. Common metrics applied to accuracy measures include Precision=TP / (TP+FP) and Recall=TP / (TP+FN), where Precision is the number of outcomes (TP or true positives) that the model correctly predicts out of the total number that the model predicted (TP+FP or false positives), and Recall is the number of outcomes (TP) that the model correctly predicts out of the total number that actually occurred (TP+FN or false negatives). The F-score (F-score=2*PR / (P+R)) unifies precision and recall into a single measure. In one embodiment, the training application 160 iteratively retrains the machine learning model until the occurrence of a stopping condition, such as, for example, an indication of an accuracy measure that the machine learning model is sufficiently accurate, or the number of training rounds performed.

[0020] Network 130 represents a communication path between machine learning server 110 and data source 120. In one aspect, network 130 is the Internet and uses standard communication technologies and / or protocols. Thus, network 130 may include links using technologies such as, for example, Ethernet, 802.11, worldwide interoperability for microwave access (WiMAX), 3G, Long Term Evolution (LTE), digital subscriber line (DSL), asynchronous transfer mode (ATM), InfiniBand, PCI Express Advanced Switching, etc. Similarly, networking protocols used in network 130 may include Multiprotocol Label Switching (MPLS), transmission control protocol / Internet protocol (TCP / IP), User Datagram Protocol (UDP), hypertext transport protocol (HTTP), simple mail transfer protocol (SMTP), file transfer protocol (FTP), etc.

[0021] Data exchanged over network 130 may be represented using technologies and / or formats including hypertext markup language (HTML), extensible markup language (XML), etc. Additionally, all or portions of the links may be encrypted using conventional encryption techniques, such as, for example, secure sockets layer (SSL), transport layer security (TLS), virtual private network (VPN), Internet Protocol security (IPsec), etc. In another aspect, the entities may use custom and / or proprietary data communication technologies in place of or in addition to those described above.

[0022] 2 is a block diagram illustrating a feature engineering application 200 according to one embodiment. The feature engineering application 200 receives multiple data entities from different data sources and synthesizes features from the data entities. The feature engineering application 200 is an embodiment of the feature engineering application 150 of FIG. 1. The feature engineering application 200 includes a primitive generation module 210, a time parameter module 220, an entity feature module 230, a feature composition module 240, and a database 250. Those skilled in the art will recognize that other embodiments may have different and / or other components than those described herein and that functionality may be distributed among the components in a different manner.

[0023] The primitive generation module 210 generates a list of primitives based on the received data entities and allows a user to edit the list. In some aspects, the primitive generation module 210 generates the list of primitives by selecting primitives from a pool of primitives maintained by the feature engineering application 200 based on the data entities. In some aspects, the primitive generation module 210 obtains information indicating one or more relationships between the data entities before generating the list of primitives. The primitive generation module 210 selects primitives for the list from the pool of primitives based on the data entities as well as on the information indicating the relationships between the data entities. The information may indicate one or more parent-child relationships between the data entities. The primitive generation module 210 may obtain the information by receiving information from a user or by determining one or more parent-child relationships, such as, for example, using the methods described below in connection with the entity feature module 230.

[0024] The pool of primitives includes a large number of primitives, e.g., hundreds or thousands of primitives. Each primitive includes an algorithm that, when applied to data, performs a computation on the data and generates a feature having an associated value. A primitive is associated with one or more attributes. A primitive's attributes can be a description of the primitive (e.g., a natural language description that specifies the computation performed by the primitive when applied to data), an input type (i.e., the type of input data), a return type (i.e., the type of output data), metadata for the primitive that indicates how useful the primitive was in a previous feature engineering process, or other attributes.

[0025] In some aspects, the pool of primitives includes a number of different types of primitives. One type of primitive is an aggregation primitive. When applied to a dataset, an aggregation primitive identifies related data in the dataset, makes decisions on the related data, and creates values ​​that summarize and / or aggregate the decisions. For example, the aggregation primitive "count" identifies values ​​in related rows of the dataset, determines whether each of the values ​​is a non-NULL value, and returns (outputs) the number of non-NULL values ​​in the rows of the dataset. Another type of primitive is a transformation primitive. When applied to a dataset, a transformation primitive creates a new variable from one or more existing variables of the dataset. For example, the transformation primitive "weekend" evaluates a timestamp of the dataset and returns a binary value (e.g., true or false) indicating whether the date indicated by the timestamp is a weekend. Another exemplary transformation primitive evaluates a timestamp and returns a count indicating the number of days until a specified date (e.g., the number of days until a particular holiday).

[0026] In some aspects, the primitive generation module 210 selects primitives based on the received data entities using a skim view approach, a summary view approach, or both approaches. In the skim view approach, the primitive generation module 210 identifies one or more semantic representations of each data entity. The semantic representation of a data entity describes characteristics of the data entity and may be obtained without performing calculations on the data of the data entity. Examples of semantic representations of a data set include the presence of one or more specific variables (e.g., column names) of the data entity, the number of columns, the number of rows, the input type of the data entity, other attributes of the data entity, and combinations thereof. To select primitives using the skim view approach, the primitive generation module 210 determines whether the identified semantic representation of the data entity matches the attributes of a primitive in the pool. If there is a match, the primitive generation module 210 selects the primitive.

[0027] The skim view approach is a rules-based analysis. The determination of whether an identified semantic representation of a data entity matches an attribute of a primitive is based on rules maintained by the feature engineering application 200. The rules specify which semantic representation of a data entity matches which attribute of a primitive, for example, based on a match between keywords in the semantic representation of the data entity and keywords of the attribute of the primitive. As an example, the semantic representation of a data entity is a column named "Birth Date", and the primitive generation module 210 selects a primitive with an input type of "Birth Date" that matches the semantic representation of the dataset. In another example, the semantic representation of a dataset is a column named "Timestamp", and the primitive generation module 210 selects a primitive with an attribute indicating that the primitive is appropriate for use with data that indicates a timestamp.

[0028] In the summary view approach, the primitive generation module 210 generates representative vectors from the data entities. The representative vectors encode data describing the data entities, such as data indicating the number of tables, the number of columns per table, the average number of each column, and the average number of each row in the data entity. The representative vectors thus act as fingerprints of the dataset. A fingerprint is a compact representation of a dataset and may be generated by applying one or more fingerprint functions to the dataset, such as, for example, a hash function, Rabin's fingerprint algorithm, or other types of fingerprint functions.

[0029] The primitive generation module 210 selects primitives for the data entities based on the representative vectors. For example, the primitive generation module 210 inputs the representative vectors of the data entities into a machine learning model. The machine learning model outputs primitives for the dataset. The machine learning model is trained, for example by the primitive selection module 210, to select primitives for the dataset based on the representative vectors. The machine learning model may be trained based on training data including a plurality of representative vectors for a plurality of training data entities and a set of primitives for each of the plurality of training data entities. The set of primitives for each of the plurality of training data entities was used to generate features determined to be useful for making predictions based on the corresponding training dataset. In some aspects, the machine learning model is trained in succession. For example, the primitive generation module 210 may further train the machine learning model based on the representative vectors of the data entities and at least some of the selected primitives.

[0030] In some aspects, the primitive generation module 210 also generates primitives based on input from a user (e.g., a data analysis engineer). The primitive generation module 210 provides the user with a list of primitives selected from a pool of primitives based on the data entity for display in a user interface. The user interface allows the user to edit the primitives, for example, add other primitives to the list of primitives, create new primitives, remove primitives, perform other types of actions, or combine. The primitive generation module 210 updates the list of primitives based on the user's edits. Thus, the primitives generated by the primitive generation module 210 incorporate the user's input. More information about user information is described below in connection with FIG. 4.

[0031] The time parameter module 220 determines one or more cutoff times based on one or more time values. The cutoff time is the time at which to make a prediction. Data associated with a timestamp before the cutoff time can be used to extract features for the label. However, data associated with a timestamp after the cutoff time should not be used to extract features for the label. The cutoff time may be specific to a subset of primitives generated by the primitive generation module 210 or may be global to all primitives generated. In some aspects, the time parameter module 220 receives a time value from a user. The time value may be a timestamp or a time period. The time parameter module 220 allows a user to provide different time values ​​for different primitives or the same time value for multiple primitives.

[0032] In some aspects, the primitive generation module 210 and the time parameter module 220, separately or together, provide a graphical user interface (GUI) that allows a user to edit primitives and enter time values. An example GUI provides tools that allow a user to view, search, and edit primitives generated by the primitive generation module 210, and to provide values ​​for one or more of the primitive's time parameters. In some aspects, the GUI allows the user to select the number of data entities to use to synthesize the feature. The GUI provides the user with options to control whether all or a portion of the data entities received by the feature engineering application 200 are used for the feature synthesis. More details about the GUI are described in connection with FIG.

[0033] The entity feature module 230 synthesizes features based on the data entities received from the feature engineering application 200, the primitives generated from the primitive generation module 210, and the cutoff time determined by the time parameter module 220. After the primitives are generated and the time parameters are received, the entity feature module 230 creates an entity set from the data entities received by the feature engineering application 200. In some aspects, the entity feature module 230 determines variables of the data entities and identifies two or more data entities that share a common variable as a subset. The entity feature module 230 may identify a subset of multiple data entities from the data entities received from the feature engineering application 200.

[0034] For each subset, the entity feature module 230 determines a parent-child relationship of the subset. It identifies a data entity of the subset as a parent entity and each of one or more other data entities of the subset as a child entity. In some aspects, the entity feature module 230 determines the parent-child relationship based on a primary variable of the data entity. A primary variable of a data entity is a variable having a value that uniquely identifies an entity (e.g., a user, an action, etc.). Examples of primary variables are variables associated with user identity information, action identity information, object identity information, or a combination thereof. The entity feature module 230 identifies the primary variables in each data entity and determines a hierarchy among the primary variables, for example, by performing a rule-based analysis. The entity feature module 230 maintains rules that specify which variables are higher in the hierarchy than other variables. For example, a rule may specify that a "user ID" variable has a higher position in the hierarchy than a "user action ID" variable.

[0035] In some embodiments, the entity feature module 230 determines the primary variable of the data entity based on user input. For example, the entity feature module 230 detects variables of the data entity, provides the detected variables for display to the user, and provides an opportunity for the user to identify which variable is the primary variable. As another example, the entity feature module 230 determines the primary variable and provides an opportunity for the user to confirm or disapprove the decision. The entity feature module 230 combines the subset of data entities based on parent-child relationships to generate an intermediate data entity. The entity feature module 230 may support a GUI to facilitate user input. An example GUI is described below in connection with FIG. 5. In some embodiments, the entity feature module 230 determines the primary variable of the data entity without user input. For example, the entity feature module 230 compares a variable of the data entity with a variable of another data entity of a similar data type. Examples of data types include a numeric data type, a categorical data type, a time series data type, a text data type, etc. The entity feature module 230 determines a matching score for each variable of the data entity, where the matching score indicates the probability that the variable matches a variable of another data entity. The entity feature module 230 selects the variable of the data entity with the highest matching score as the primary variable of the data entity. Additionally, the entity feature module 230 combines the intermediate data entities to generate an entity set that incorporates all data entities received by the feature engineering application 200.

[0036] In some aspects, the entity feature module 230 determines whether to normalize the data entity by, for example, determining whether there are multiple primary variables in the data entity. In one aspect, the entity feature module 230 determines whether there are duplicate values ​​of different variables in the data entity. For example, if a "Colorado" value of a location variable always corresponds to a "Mountain" value of a region variable, then the "Colorado" value and the "Mountain" value are duplicate values. In response to the decision to normalize the data entity, the entity feature module 230 normalizes the data entity by splitting the data entity into two new data entities. Each of the two new data entities includes a subset of the variables of the data entity. The process just described is referred to as normalization. In some aspects of the normalization process, the entity feature module 230 identifies a first primary variable and a second primary variable from the variables of a given entity. Examples of primary variables of a data entity include, for example, user identity information, action identity information, object identity information, or a combination thereof. The entity feature module 230 categorizes the variables of the given entity into a first variable group and a second variable group. The first variable group includes a first primary variable and one or more other variables of the given entity related to the first primary variable. The second variable group includes a second primary variable and one or more other variables of the given entity related to the second primary variable. The entity feature module 230 generates one of two new entities according to the variables of the first group and the values ​​of the variables of the first group, and generates the other of the two new entities according to the variables of the second group and the values ​​of the variables of the first group. The entity feature module 230 then identifies a subset of data entities from the two new data entities and the received data entity excluding the normalized data entity.

[0037] The entity feature module 230 applies the primitives and the cutoff time to the entity set to synthesize a group of features and an importance factor for each feature in the group. The entity feature module 230 extracts data from the entity set based on the cutoff time and applies the primitives to the extracted data to synthesize a plurality of features. In some aspects, the entity feature module 230 applies each of the selected primitives to at least a portion of the extracted data to synthesize one or more features. For example, the entity feature module 230 applies a "weekend" primitive to a column named "timestamp" of the entity set to synthesize a feature indicating whether a date falls on a weekend. The entity feature module 230 can synthesize a large number of features, e.g., hundreds or millions of features, for the entity set.

[0038] The entity feature module 230 evaluates the features and removes some of the features based on the evaluation to obtain a feature group. In some embodiments, the entity feature module 230 evaluates the features through an iterative process. In each round of iteration, the entity feature module 230 applies the features that were not removed by the previous iteration (also called "remaining features") to a different portion of the extracted data and determines a usefulness score for each of the features. The entity feature module 230 removes some of the features with the lowest usefulness scores from the remaining features.

[0039] In some aspects, the entity feature module 230 uses a random forest to determine a utility score for a feature. The utility score for a feature indicates how useful the feature is for predictions made based on the entity set. In some aspects, the entity feature module 230 repeatedly applies different portions of the entity set to the feature to evaluate the utility of the feature. For example, in a first iteration, the entity feature module 230 applies a predetermined percentage (e.g., 25%) of the entity set to the feature to construct a first random forest. The first random forest includes several decision trees. Each decision tree includes multiple nodes. Every node corresponds to a feature and includes a condition that describes how to traverse the tree through the node based on the value of the feature (e.g., take one branch if the date is on a weekend, take another branch otherwise). The feature for each node is determined based on information gain or Gini impurity reduction. The feature that maximizes the information gain or Gini impurity reduction is selected as the split feature. The evaluation module 320 determines the individual utility scores of the features based on either the information gain or the reduction of Gini impurity by the features across the decision trees. The individual utility scores of the features are specific to one decision tree. After determining the individual utility scores of the features for each of the random forest decision trees, the evaluation module 320 determines a first utility score of the feature by combining the individual utility scores of the features. In one example, the first utility score of the feature is the average of the individual utility scores of the features. The entity feature module 230 removes 20% of the features with the lowest first utility scores, leaving 80% of the features remaining. These features are referred to as the first remaining features.

[0040] In the second iteration, the entity feature module 230 applies the first remaining features to a different portion of the entity set. The different portion of the entity set can be 25% of the entity set different from the portion of the entity set used in the first iteration, or can be 50% of the entity set that includes the portion of the entity set used in the first iteration. The evaluation module 320 builds a second random forest using the different portion of the entity set and determines a second utility score for each of the remaining features using the second random forest. The entity feature module 230 removes 20% of the first remaining features and the remainder of the first remaining features (i.e., 80% of the first remaining features form the second remaining features).

[0041] Similarly, in each next iteration, the entity feature module 230 applies the remaining features from the previous round to a different portion of the entity set, determines the utility scores of the remaining features from the previous round, and removes some of the remaining features to obtain a smaller group of features. The entity feature module 230 can continue the iterative process until it determines that a condition is met. The condition can be that a number of features remain below a threshold, that the lowest utility score of the remaining features is above a threshold, that the entire entity set has been applied to the features, that a threshold number of rounds have been completed in the iteration, other conditions, or a combination thereof. The remaining features from the last round, i.e., the features not removed by the entity feature module 230, are selected to train the machine learning model.

[0042] After the iteration is finished and the feature group is obtained, the entity feature module 230 determines an importance coefficient for each feature in the group. The importance coefficient of a feature indicates how important the feature is to predicting the target variable. In some embodiments, the entity feature module 230 determines the importance coefficient using a random forest, e.g., one constructed based on at least a portion of the entity set. In some embodiments, the entity feature module 230 constructs a random forest based on the selected features and the entity set. The entity feature module 230 determines an individual ranking score for the selected features based on each decision tree of the random forest, and obtains an average of the individual ranking scores as the ranking score for the selected features. The entity feature module 230 determines an importance coefficient for the selected features based on the ranking scores. For example, the entity feature module 230 ranks the selected features based on the ranking scores, and determines that the importance score of the highest ranked selected feature is 1. The entity feature module 230 then determines the ratio of the ranking score of each of the remaining selected features to the ranking score of the highest ranked selected feature as the importance coefficient for the corresponding selected feature.

[0043] In some aspects, the entity feature module 230 adjusts the importance score of the feature by inputting the feature and a different portion of the entity set into a machine learning model. The machine learning model outputs a second importance score for the feature. The entity feature module 230 compares the importance factor to the second importance score to determine whether to adjust the importance factor. For example, the entity feature module 230 can change the importance factor to an average of the importance factor and the second importance factor.

[0044] 3 illustrates a user interface 300 that allows user input for primitives and time parameters according to one embodiment. The user interface 300 is provided by the primitive generation module 210 and the time parameters module 220 of the feature engineering application 200. The user interface 300 allows a user to view and interact with primitives that will be used to synthesize features. It also allows a user to input values ​​for time parameters. The user interface 300 includes a search bar 310, a display portion 320, a training data portion 330, and a time parameters portion 340. Other embodiments of the user interface 300 have more, fewer, or different components.

[0045] The search bar 310 allows the user to type a search term to find a primitive they want to view or otherwise interact with. The display 320 presents a list of primitives. It presents all primitives selected by the primitive generation module 210 or those that match the search term entered by the user. In some aspects, the primitives are listed in order, for example, based on the importance of the primitive to the prediction, the relevance of the primitive to the user's search term, or other factors. For each primitive in the list, the display presents the name, category (aggregation or transformation), and description of the primitive. The name, category, and description of the primitive explain the function and algorithm of the primitive and help the user understand the primitive. Each primitive in the list is associated with a checkbox that the user may click to select the primitive. Although not shown in FIG. 3, the user interface 300 allows the user to remove selected primitives, add new primitives, change the order of primitives, or perform other types of interactions with the primitives. The training data portion 330 allows the user to select the maximum number of data entities that will be used to create a feature. What has just been described provides the user with the opportunity to control the amount of training data that will be used to create a feature. The user interface allows the user to specify a tradeoff between the processing time to generate a feature and the quality of the feature.

[0046] The time parameters section 340 provides options for the user to input a time value, e.g., by specifying a time, by defining a time window, or both. Additionally, the time parameters section 340 also provides an option for the user to select whether the time value is to be used as a common time or a case-specific time. A common time applies to all primitives used to create a feature, whereas a case-specific time applies to a subset of primitives.

[0047] FIG. 4 illustrates a user interface 400 for entity set creation according to one embodiment. The user interface is provided by the entity feature module 230 of the feature engineering application 200. The user interface 400 displays to a user the data entities 410, 420, 430 received by the feature engineering application 200. In addition, the user interface 400 also displays information of the data entities, such as, for each data entity, the data entity name, the number of columns, and the primary key. The data entity name helps the user identify the data entity. Since the columns of a data entity correspond to variables, the number of columns corresponds to the number of variables of the data entity, and the primary column corresponds to the primary variable. The user interface 400 also displays the parent-child relationship between the two data entities 410 and 420, and the common columns shared by the two data entities. The data entity 410 is shown as the parent data entity, and the data entity 420 is shown as the child data entity. The user interface 400 provides a drop-down icon 440 (individually referred to as drop-down icon 440) for each primary and common column displayed in the user interface 400 to allow the user to change the columns. For example, in response to receiving a selection of a drop-down icon, the user interface 400 provides the user with a list of candidate columns and allows the user to select a different column. In some aspects, the entity feature module 230 detects columns of the data entities and selects candidate columns from the detected columns. Additionally, the user interface 400 also allows the user to add a new relationship. Although the user interface 400 shows three data entities and one parent-child relationship, there may be more data entities and more parent-child relationships. In some aspects, a data entity is a parent in one parent but a child in a different subset.

[0048] 5 is a flowchart illustrating a method 500 of synthesizing features by using data entities received from different data sources according to one embodiment. In some embodiments, the method is performed by the machine learning server 110, while in other embodiments, some or all of the operations in the method may be performed by other entities. In some embodiments, the operations in the flowchart are performed in a different order and / or include different and / or additional steps.

[0049] The machine learning server 110 receives 510 multiple data entities from different data sources. The different data sources may be associated with different users, different organizations, or different departments within the same organization. The multiple data entities are used to train a model that makes predictions based on new data. A data entity is a set of data that includes one or more variables. Exemplary predictions include application monitoring, network traffic data flow monitoring, user action predictions, etc.

[0050] The machine learning server 110 generates 520 primitives based on the multiple data entities. Each of the primitives is configured to be applied to variables of the multiple datasets to synthesize features. In some aspects, the machine learning server 110 selects primitives from a pool of primitives based on the multiple data entities. The machine learning server 110 may provide the selected primitives for display to the user and enable the user to edit the selected primitives, e.g., add, remove, reorder, etc. of the primitives. The machine learning server 110 generates the primitives based on the selections and the user edits.

[0051] The machine learning server 110 receives 530 time values ​​from a client device associated with the user. The time values ​​are used to synthesize one or more time-based features from the multiple data entities. In some aspects, the machine learning server 110 determines one or more cutoff times based on the time values ​​and extracts data from the entity set based on the one or more cutoff times. The extracted data will be used by the machine learning server 100 to synthesize one or more time-based features from the extracted data.

[0052] After the primitives are generated and the time values ​​are received, the machine learning server 110 generates an entity set 540 by aggregating the multiple data entities. The machine learning server 110 identifies subsets of data entities from the multiple data sets. Each subset includes two or more data entities that share a common variable. The machine learning server 110 then generates an intermediate data entity by aggregating the data entities of each subset. In some aspects, the machine learning server 110 determines a primary variable of each data entity of the subset, and identifies the data entity of the subset as a parent entity based on the primary variable of the data entities of the subset, and identifies each of one or more other data entities included in the parent as a child entity. The machine learning server 110 aggregates the data entities of the subset based on the parent-child relationship. The machine learning server 110 generates an entity set by aggregating the intermediate data entities.

[0053] In some embodiments, the machine learning server 110 generates two new data entities from a given data entity of the plurality of data entities based on variables of the given data entity. Each of the two new data entities includes a subset of variables of the given data entity. For example, the machine learning server 110 identifies a first primary variable and a second primary variable from the variables of the given entity. Then, the machine learning server 110 categorizes the variables of the given entity into a first variable group and a second variable group. The first variable group includes the first primary variable and one or more other variables of the given entity related to the first primary variable. The second variable group includes the second primary variable and one or more other variables of the given entity related to the second t primary variables. The machine learning server 110 generates one of the two new entities with the variables of the first group and the values ​​of the variables of the first group, and generates the other of the two new entities with the variables of the second group and the values ​​of the variables of the first group. The machine learning server 110 identifies a subset of data entities from the two new data entities and the plurality of data entities excluding the giving data entity.

[0054] The machine learning server 110 synthesizes 550 a plurality of features by applying the primitives and time values ​​to the entity set. In some embodiments, the machine learning server 110 applies the primitives to the entity set to generate a pool of features. The machine learning server 110 then iteratively evaluates the pool of features to remove some features from the pool of features to obtain a plurality of features. At each iteration, the machine learning server 110 evaluates the usefulness of at least some of the plurality of features by applying different portions of the entity set to the evaluated features, and removes some of the evaluated features based on the usefulness of the evaluated features to generate a plurality of features.

[0055] The plurality of features includes one or more time-based features. In some embodiments, the machine learning server 110 determines one or more cutoff times based on the time values. The machine learning server 110 then extracts data from the entity set based on the one or more cutoff times and generates one or more time-based features from the extracted data.

[0056] The machine learning server 110 trains 550 a model based on the plurality of features. The machine learning server 110 includes using different machine learning techniques in different embodiments. Exemplary machine learning techniques include, for example, linear support vector machines (linear SVMs), boosting of other algorithms (e.g., AdaBoost), neural networks, logistic regression, naive Bayes, memory-based learning, random forests, bagged trees, decision trees, boosted trees, boosted stumps, etc. The trained model is then used to make predictions in terms of new datasets.

[0057] FIG. 6 is a high-level block diagram illustrating a functional diagram of an exemplary computer system 600 for use as the machine learning server 110 of FIG. 1 according to an embodiment.

[0058] The illustrated computer system includes at least one processor 602 coupled to a chipset 604. The processor 602 can include multiple processor cores on the same die. The chipset 604 includes a memory controller hub 620 and an input / output (I / O) controller hub 622. A memory 606 and a graphics adapter 612 are coupled to the memory controller hub 620, and a display 618 is coupled to the graphics adapter 612. A storage device 608, a keyboard 610, a pointing device 614, and a network adapter 616 may be coupled to the I / O controller hub 622. In some other aspects, the computer system 600 may have additional, fewer, or different components, and the components may be coupled differently. For example, aspects of the computer system 600 may be devoid of a display and / or a keyboard. Additionally, the computer system 600 may be instantiated as a rack-mounted blade server or as a cloud server instance in some aspects.

[0059] The memory 606 holds instructions and data used by the processor 602. In some aspects, the memory 606 is a random access memory. The storage device 608 is a non-transitory computer-readable recording medium. The storage device 608 can be a HDD, SSD, or other type of non-transitory computer-readable recording medium. Data processed and analyzed by the machine learning server 110 can be stored in the memory 606 and / or the storage device 608.

[0060] Pointing device 614 may be a mouse, trackball, or other type of pointing device and is used in combination with keyboard 610 to input data into computer system 600. Graphics adapter 612 displays images and other information on display 618. In some aspects, display 618 includes touch screen capabilities for receiving user input and selections. Network adapter 616 connects computer system 600 to network 160.

[0061] The computer system 600 is adapted to execute computer modules for providing the functionality described herein. The term "module" as used herein refers to computer program instructions and other logic for providing a particular functionality. A module may be implemented in hardware, firmware, and / or software. A module may include one or more processes and / or may be provided by only a portion of a process. A module is typically stored in the storage device 608, loaded into the memory 606, and executed by the processor 602.

[0062] The particular naming of components, capitalization of terms, attributes, data structures, or other programming or structural aspects are not required or important, and mechanisms for implementing the described aspects may have different names, formats, or protocols. Further, the system may be implemented through a combination of hardware and software as described, or entirely in hardware elements. Furthermore, the particular division of functionality among various system components described herein is merely exemplary and not required, and functions performed by a single system component may instead be performed by multiple components, and functions performed by multiple components may instead be performed by a single component.

[0063] Some portions of the description above are presented in terms of algorithms and symbolic representations of operations of information. These algorithmic descriptions and representations are used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. It will be understood that the operations described herein, while described functionally or logically, are implemented by computer programs. Furthermore, it has been shown to be convenient at times, without loss of generality, to refer to the described arrangements of operations as modules or by functional names.

[0064] As is apparent from the above discussion, unless specifically stated otherwise, throughout this specification, discussions using terms such as "processing" or "calculating" or "calculating" or "determining" or "displaying" are understood to refer to the acts and processes of a computer system or similar electronic computing device that manipulates and transforms data that is represented as physical (electronic) quantities in the computer system's memories or registers or other such information storage, transmission or display devices.

[0065] Certain aspects described herein include process steps and instructions written in the form of an algorithm. It should be noted that the process steps and instructions of the aspects may be embodied in software, firmware or hardware, and when embodied in software, may be downloaded to reside on and be operated from different platforms for use by a real-time network operating system. Finally, as a general rule, the language used in the specification has been selected for ease of reading and educational purposes, and may not have been selected to define or delimit the boundaries of the subject matter of the invention. Thus, the disclosure of the present embodiments is intended to be illustrative, and not limiting.

Claims

1. receiving a plurality of data entities from different data sources; generating primitives based on the plurality of data entities, each of the primitives including an algorithm that, when applied to one or more of the plurality of data entities, synthesizes a feature having an associated value; generating an entity set by aggregating the plurality of data entities; generating a pool of features by applying the primitives to the set of entities; updating the pool of features over a number of iterations, each of the number of iterations comprising: determining a utility score for each feature in the pool of features using a portion of the entity set that is different from the portion of the entity set used in a different iteration; and Removing at least one feature from the pool of features based on the usefulness score. and outputting the updated pool of features as a plurality of features in response to a stopping condition. combining the plurality of features by training a machine learning model using the features to generate an output based on new data; and 23. A method comprising:

2. Generating the entity set by aggregating the plurality of data entities includes: identifying subsets of data entities from the plurality of data entities, each subset of data entities including two or more data entities that share a common variable; generating an intermediate data entity by aggregating the data entities in each subset of the data entities; generating the entity set by aggregating the intermediate data entities; 2. The method of claim 1, comprising:

3. Generating the intermediate data entity by aggregating the data entities in each of the subsets of data entities includes: determining a primary variable for each data entity in said subset of data entities; identifying a data entity in the subset of data entities as a parent entity and each of one or more other data entities in the subset of data entities as child entities based on each of the primary variables determined for each data entity in the subset of data entities; 3. The method of claim 2, comprising:

4. Identifying the subset of data entities from the plurality of data entities comprises: generating two new data entities from a given data entity of the plurality of data entities based on variables defining the given data entity, each of the two new data entities including a subset of the variables defining the given data entity; identifying the subset of data entities from the plurality of data entities excluding the given data entity and the two new data entities; 3. The method of claim 2, comprising:

5. Generating the two new data entities from the given data entity of the plurality of data entities based on variables that define the given data entity, identifying a first primary variable and a second primary variable from the variables defining the given entity; categorizing the variables defining the given entity into a first group of variables and a second group of variables, the first group of variables including the first primary variable and one or more other variables defining the given entity related to the first primary variable, and the second group of variables including the second primary variable and one or more other variables defining the given entity related to the second primary variable; generating one of the two new entities according to the first group of variables and respective values ​​for the first group of variables; generating another of the two new entities with the second group of variables and respective values ​​for the second group of variables; 5. The method of claim 4, comprising:

6. Composing the plurality of features by applying the primitives to the set of entities is based on a time value; and determining one or more cutoff times based on the time values; extracting data from the entity set based on the one or more cutoff times; synthesizing the plurality of features from the extracted data; and The method of claim 1 further comprising:

7. a computer processor for executing computer program instructions; receiving a plurality of data entities from different data sources; generating primitives based on the plurality of data entities, each of the primitives including an algorithm that, when applied to one or more of the plurality of data entities, synthesizes a feature having an associated value; generating an entity set by aggregating the plurality of data entities; generating a pool of features by applying the primitives to the set of entities; updating the pool of features over a number of iterations, each of the number of iterations comprising: determining a utility score for each feature in the pool of features using a portion of the entity set that is different from the portion of the entity set used in a different iteration; and Removing at least one feature from the pool of features based on the usefulness score. and outputting the updated pool of features as a plurality of features in response to a stopping condition. combining the plurality of features by training a machine learning model using the features to generate an output based on new data; and a non-transitory computer readable memory storing computer program instructions executable by said computer processor to perform operations including: A system comprising:

8. Generating the entity set by aggregating the plurality of data entities includes: identifying subsets of data entities from the plurality of data entities, each subset of data entities including two or more data entities that share a common variable; generating an intermediate data entity by aggregating the data entities in each subset of the data entities; generating the entity set by aggregating the intermediate data entities; 8. The system of claim 7, further comprising:

9. Generating the intermediate data entity by aggregating the data entities in each of the subsets of data entities comprises: determining a primary variable for each data entity in said subset of data entities; identifying a data entity in the subset of data entities as a parent entity and each of one or more other data entities in the subset of data entities as child entities based on each of the primary variables determined for each data entity in the subset of data entities; 9. The system of claim 8, further comprising:

10. Identifying the subset of data entities from the plurality of data entities comprises: generating two new data entities from a given data entity of the plurality of data entities based on variables defining the given data entity, each of the two new data entities including a subset of the variables defining the given data entity; identifying the subset of data entities from the plurality of data entities excluding the given data entity and the two new data entities; 9. The system of claim 8, further comprising:

11. Generating the two new data entities from the given data entity of the plurality of data entities based on variables that define the given data entity, identifying a first primary variable and a second primary variable from the variables defining the given entity; categorizing the variables defining the given entity into a first group of variables and a second group of variables, the first group of variables including the first primary variable and one or more other variables defining the given entity related to the first primary variable, and the second group of variables including the second primary variable and one or more other variables defining the given entity related to the second primary variable; generating one of the two new entities according to the first group of variables and respective values ​​for the first group of variables; generating another of the two new entities with the second group of variables and respective values ​​for the second group of variables; The system of claim 10, further comprising:

12. Composing the plurality of features by applying the primitives to the set of entities is based on a time value; and determining one or more cutoff times based on the time values; extracting data from the entity set based on the one or more cutoff times; synthesizing the plurality of features from the extracted data; and The system of claim 7 further comprising:

13. 1. A non-transitory computer readable memory storing executable computer program instructions for processing data blocks in a data analysis system, the computer program instructions comprising: receiving a plurality of data entities from different data sources; generating primitives based on the plurality of data entities, each of the primitives including an algorithm that, when applied to one or more of the plurality of data entities, synthesizes a feature having an associated value; generating an entity set by aggregating the plurality of data entities; generating a pool of features by applying the primitives to the set of entities; updating the pool of features over a number of iterations, each of the number of iterations comprising: determining a utility score for each feature in the pool of features using a portion of the entity set that is different from the portion of the entity set used in a different iteration; and Removing at least one feature from the pool of features based on the usefulness score. and outputting the updated pool of features as a plurality of features in response to a stopping condition; combining the plurality of features by training a machine learning model using the features to generate an output based on new data; and 16. A non-transitory computer readable memory executable to perform operations including:

14. Generating the entity set by aggregating the plurality of data entities includes: identifying subsets of data entities from the plurality of data entities, each subset of data entities including two or more data entities that share a common variable; generating an intermediate data entity by aggregating the data entities in each subset of the data entities; generating the entity set by aggregating the intermediate data entities; 14. The non-transitory computer readable memory of claim 13, comprising:

15. The method of generating the intermediate data entity by aggregating the data entities in each of the subsets of data entities, comprising: determining a primary variable for each data entity in said subset of data entities; identifying a data entity in the subset of data entities as a parent entity and each of one or more other data entities in the subset of data entities as child entities based on each of the primary variables determined for each data entity in the subset of data entities; 15. The non-transitory computer readable memory of claim 14, comprising:

16. Identifying the subset of data entities from the plurality of data entities comprises: generating two new data entities from a given data entity of the plurality of data entities based on variables defining the given data entity, each of the two new data entities including a subset of the variables defining the given data entity; identifying the subset of data entities from the plurality of data entities excluding the given data entity and the two new data entities; 15. The non-transitory computer readable memory of claim 14, comprising:

17. Composing the plurality of features by applying the primitives to the set of entities is based on a time value; and determining one or more cutoff times based on the time values; extracting data from the entity set based on the one or more cutoff times; synthesizing the plurality of features from the extracted data; and 14. The non-transitory computer readable memory of claim 13, further comprising: During each of the plurality of iterations, determining the utility score for each feature in the pool of features comprises: constructing a random forest comprising decision trees having nodes representing different features in the pool of features; and applying the portion of the entity set to the random forest that is different from the portion of the entity set used in the different iteration sets.

14. The non-transitory computer readable memory of claim 13, wherein the non-transitory computer readable memory is implemented by: During each of the plurality of iterations, determining the utility score for each feature in the pool of features comprises: constructing a random forest comprising decision trees having nodes representing different features in the pool of features; and applying the portion of the entity set to the random forest that is different from the portion of the entity set used in the different iteration sets.

2. The method of claim 1, wherein the method is performed by: During each of the plurality of iterations, determining the utility score for each feature in the pool of features comprises: constructing a random forest comprising decision trees having nodes representing different features in the pool of features; and applying the portion of the entity set to the random forest that is different from the portion of the entity set used in the different iteration sets. The system according to claim 7, characterized in that the system is performed by: