Metadata-Driven Feature Stores for Machine Learning Systems

Abstract

The method includes identifying one or more transformations to be applied to generate one or more features or feature sets. The method also includes generating metadata identifying the one or more features or feature sets and the one or more transformations. The method further includes determining the one or more features or feature sets for specified data using the metadata and storing the one or more determined features or feature sets in a feature store. Additionally, the method includes outputting at least some of the one or more determined features or feature sets, or data associated with at least some of the one or more determined features or feature sets, from the feature store to at least one machine learning model.

Description

[Technical field]

[0001] The present disclosure is directed generally to machine learning systems and, more specifically, to a metadata-driven feature store for machine learning systems. [Background technology]

[0002] Machine learning systems are developed and deployed to perform numerous functions in computing and other systems. Many machine learning systems are designed using a process called feature engineering, which involves identifying features to be used with a machine learning model. Features refer to data that are input into a trained machine learning model so that the trained machine learning model can use the features to perform inference. Proper identification of features to be used with a machine learning model is often a critical or important aspect in the design of systems that use machine learning. Summary of the Invention

[0003] The present disclosure relates to a metadata-driven feature store for machine learning systems.

[0004] In a first embodiment, the method includes identifying one or more transformations to be applied to generate one or more features or feature sets. The method also includes generating metadata identifying the one or more features or feature sets and the one or more transformations. The method further includes determining the one or more features or feature sets for specified data using the metadata, and storing the one or more determined features or feature sets in a feature store. Additionally, the method includes outputting at least some of the one or more determined features or feature sets, or data associated with at least some of the one or more determined features or feature sets, from the feature store to at least one machine learning model.

[0005] In a second embodiment, the apparatus includes at least one processing device configured to identify one or more transformations to be applied to generate one or more features or feature sets. The at least one processing device is also configured to generate metadata identifying the one or more features or feature sets and the one or more transformations. The at least one processing device is further configured to determine the one or more features or feature sets for specified data using the metadata and store the one or more determined features or feature sets in a feature store. Additionally, the at least one processing device is configured to output at least some of the one or more determined features or feature sets, or data associated with at least some of the one or more determined features or feature sets, from the feature store to at least one machine learning model.

[0006] In a third embodiment, a non-transitory computer-readable storage medium has computer-readable program code stored thereon that, when executed, causes one or more processors to identify one or more transformations to be applied to generate one or more features or feature sets. The medium also has computer-readable program code stored thereon that, when executed, causes the one or more processors to generate metadata identifying the one or more features or feature sets and the one or more transformations. The medium further has computer-readable program code stored thereon that, when executed, causes the one or more processors to use the metadata to determine the one or more features or feature sets for specified data and to store the one or more determined features or feature sets in a feature store. Additionally, the medium has computer-readable program code stored thereon that, when executed, causes the one or more processors to output at least some of the one or more determined features or feature sets, or data associated with at least some of the one or more determined features or feature sets, from the feature store to at least one machine learning model.

[0007] Other technical features may become readily apparent to those skilled in the art from the following figures, description, and claims. [Brief description of the drawings]

[0008] For a more complete understanding of the present disclosure, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which: [Figure 1] FIG. 1 illustrates an exemplary system that supports a metadata-driven feature store according to this disclosure. [Diagram 2] FIG. 2 illustrates an example device that supports a metadata-driven feature store according to this disclosure. [Diagram 3] FIG. 3 illustrates one exemplary architecture for supporting a metadata-driven feature store according to this disclosure. [Figure 4A] FIG. 4A illustrates one exemplary method for using a metadata-driven feature store according to this disclosure. [Figure 4B] FIG. 4B illustrates one exemplary method for using a metadata-driven feature store according to this disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] 1-4B, described below, and the various embodiments used to illustrate the principles of the present disclosure, are for illustrative purposes only and should not be construed as limiting the scope of the present disclosure in any way. Those skilled in the art will understand that the principles of the present disclosure may be implemented in any type of suitably arranged device or system.

[0010] The following definitions are provided for terms and phrases used in this disclosure: A "feature" refers to a single logical input or sequence of inputs provided to a machine learning model. A feature can have an associated subject type, and each instance of a feature can be associated with a subject identifier (ID) and a value. A "subject type" refers to a category of person, place, or thing to which machine learning is to be applied. A "subject identifier" or "subject ID" refers to a unique identifier for a particular person, place, or thing to which machine learning is to be applied. A "time-series feature" refers to a feature that may change over time for a particular subject ID. A time-series feature can have a timestamp associated with each instance of the time-series feature. An "interval" refers to any difference in time between subsequent timestamps for a time-series feature, which may be common in a variety of situations (such as when the time-series feature represents a physical value that is sampled at regular intervals or otherwise). Sampled or discrete time series features may be aggregated or resampled at regular intervals or other times.

[0011] A "data frame" refers to a tabular representation of data with rows and columns. Data frames are often used in data science as an in-memory representation, although data frames can be mapped to other types of data structures, such as relational database tables and record-oriented file representations (such as CSV, Parquet, ORC, ORC2, or HDF5 files). Data frames and database tables may be used interchangeably without loss of generality. A "feature set" refers to a collection of features associated with the same machine learning target type, where the feature set indicates how to combine multiple features to form a single data frame for input to a machine learning model. In addition to a list of features, a feature set may include a time interval, a pointer to a specific feature to be used in identifying a primary timestamp for the feature set, and one or more resampling functions that may be used to convert different time series features to a common interval. Feature sets may be created explicitly by a user or implicitly, such as through actions performed on the underlying data. A “point-in-time join” refers to the act of joining features into a feature set such that, for each associated timestamp, all feature values ​​represent the most recent values ​​for those features at or before the timestamp.

[0012] As mentioned above, many machine learning systems are designed using a process called feature engineering, which involves identifying features to be used with a machine learning model. Proper identification of features to be used with a machine learning model is often a critical or important aspect in the design of a system that uses machine learning. Often, determining the features to be input to a machine learning model involves identifying data transformations to be applied to available source data. Exemplary types of data transformations that may be performed using available source data include functions such as value mapping, value scaling, and data aggregation.

[0013] A feature may be said to be "materialized" when transformed values ​​for that feature are stored directly (rather than evaluating the feature transformations on the source data each time the data is used). A "feature store" refers to a centralized or other repository of materialized feature data. Feature stores are typically used to support capabilities such as (i) feature sharing and discoverability across different teams of people, (ii) reuse of named features in both training and prediction (inference) contexts, and (iii) providing a point-in-time view of multiple features (such as the latest version of each feature at a particular point in time).

[0014] A feature store typically includes two separate data storages: offline storage and online storage. The offline storage is generally used to store data related to training of machine learning models, and the offline storage typically represents a file-based storage or data warehouse. The online storage is generally used to store data related to inferences performed using trained machine learning models (often referred to as "production data"), and the online storage is typically implemented as an in-memory key-value store.

[0015] Typical feature stores are implemented as standalone components and are not metadata-driven. As a result, they cannot provide much in the way of automation, and many required functions need to be performed manually by end users. For example, end users may need to define the transition from experiment to production, recalculation of updated features, resampling of time series features, and aggregation of composite features. End users are often also responsible for monitoring end-to-end data lineage and providing access control enforcement outside of the feature store. In many cases, end users provide basic information about feature names and types, and features are manually designed using Structured Query Language (SQL) code. Also, direct computation of features for in-memory inference is generally not possible with feature stores.

[0016] As one particular example of the type of process that can be followed using a typical feature store, data can be ingested from a source external to the feature store and stored in the offline storage. This is typically performed using custom code written by the developer (where the code is not managed by the feature store). The developer can also create registry entries that correspond to columns of tables in the offline storage. Historical features for machine learning model training can be retrieved in batch mode, which is typically performed using SQL code written based on feature tables in the offline storage. Point-in-time joins can be used to join features from multiple tables in the offline storage. Recent data from the offline storage can be materialized into the online storage, such as by using a SQL query with a point-in-time join. Time series features, or externally provided time series features, can be used to define a point-in-time, which can also be stored in the online storage. For inference, recently requested features can be retrieved from the key-value store (online storage) via an application programming interface (API) and provided to the machine learning model for use during inference.

[0017] Unfortunately, this type of approach can suffer from several drawbacks. For example, feature stores may not allow data transformations to be performed within the feature store, or may only allow minor data transformations to be performed within the feature store. As a result, features often need to be transformed before being ingested into the feature store, and most or all of the ingestion logic needs to be hand-built outside of the feature store. The ingestion logic also needs to be designed to work at scale, and a separate path for experimentation (without rewriting everything) typically does not exist. Also, data experimentation and machine learning model design are typically performed using sample data stored in flat files (such as comma-separated values, or "CSV" format), and there is no path to reuse this work in a production feature store that can retrieve that data from other types of sources (such as relational and time-series databases). Furthermore, automatically recalculating features based on changes in upstream data is generally not possible because transformations are performed outside of the feature store. Furthermore, in many cases, different features may represent aggregations calculated at different intervals, and combining these features using point-in-time joins may result in inaccurate results. Existing feature stores cannot perform the resampling required to give correct results due to lack of metadata. In addition, as a standalone component, a feature store typically cannot track where individual data records originated and where those data records are used, which means that end-to-end data lineage needs to be implemented outside the feature store. Finally, the online storage of a feature store is typically populated from the most recent features in the offline storage.Online storage can provide low latency for inference, but this means that data can become stale due to the time it takes to ingest new data into the offline storage and then propagate the new data to the online storage.

[0018] This disclosure provides various embodiments of a metadata-driven feature store for use with machine learning model-based systems. As described in more detail below, metadata (such as information defining the transformations necessary to create one or more features or feature sets from source data) can be created via automated or manual processes, such as via direct APIs calls. The metadata drives the process for efficiently computing and storing features or feature sets from both experimental data (representing data used to design, train, and evaluate one or more machine learning models) and production data (representing data used by one or more trained machine learning models for large-scale batch training and during inference). The metadata is stored for use by the metadata-driven feature store, which can be tightly integrated with upstream and downstream processes of the machine learning workflow.

[0019] As a specific example of functionality that may be supported by a metadata-driven feature store, the metadata-driven feature store may be used to capture and use metadata associated with features or feature sets that have been defined and determined using experimental data. For example, the system may enable one or more users to define individual features or feature sets that are determined using experimental data, and data transformations used to generate the features or feature sets may be defined. Metadata identifying the features or feature sets and transformations may be captured and used to calculate values ​​of the features or feature sets based on the experimental data, and the calculated features or feature sets may be embodied in the feature store in an appropriate format. Features or feature sets stored in the feature store may be easily evaluated by one or more users and used to perform training of one or more machine learning models.

[0020] As production data becomes available, additional transformations between the production data and the features or feature sets defined using the experimental data can be identified. Additional metadata identifying the features or feature sets and the additional transformations can be captured and used to calculate the values ​​of the features or feature sets based on the production data, and the calculated features or feature sets can again be embodied in the feature store in an appropriate format. Among other things, this approach allows production data to be captured and features or feature sets determined and stored in the feature store, while applying any required changes to the production data, such as changes required due to production data representing aggregates or other information calculated at different time intervals. In some cases, the feature store can optionally be populated directly from the incoming production data. In addition, new features or feature sets can be easily defined for the production data as needed.

[0021] In this way, the metadata-driven feature store can be used to enable transformations of data ingested into the feature store, and these transformations can be built into the feature store itself, scaled as needed, and tested as needed. The same type of data used for inference can also be used during data experimentation and machine learning model design, which can help facilitate easier and more accurate data experimentation and model design. Furthermore, features or feature sets can be recalculated as needed, and changes or corrections can be made to support point-in-time join operations for data related to different time intervals. Furthermore, the metadata-driven feature store can be used to track where individual data records originated from and where they are used, thereby supporting end-to-end data lineage within the feature store. In addition, the metadata-driven feature store can be used to store production data more efficiently and in a timely manner, since the production data does not need to be transported first through offline storage. Ultimately, a metadata-driven feature store can provide a robust and scalable solution that enables users to move quickly from experimentation to production and avoid writing a lot of custom code associated with populating and updating the feature store.

[0022] FIG. 1 illustrates one exemplary system 100 supporting a metadata-driven feature store according to the present disclosure. For example, the illustrated system 100 may be used to support a metadata-driven feature store used to store metadata associated with features or feature sets processed by one or more machine learning models, as described in more detail below. As illustrated in FIG. 1, the system 100 includes user devices 102a-102d, one or more networks 104, one or more application servers 106, and one or more database servers 108 associated with one or more databases 110 and / or one or more file servers 111. Each of the user devices 102a-102d communicates over the network 104, such as via a wired or wireless connection. Each of the user devices 102a-102d represents any suitable device or system used by at least one user to provide or receive information, such as a desktop computer, a laptop computer, a smartphone, and a tablet computer. However, any other or additional types of user devices may be used in the system 100.

[0023] Network 104 facilitates communication between various components of system 100. For example, network 104 may communicate Internet Protocol (IP) packets, Frame Relay frames, Asynchronous Transfer Mode (ATM) cells, or other suitable information between network addresses. Network 104 may include one or more local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), all or a portion of a global network such as the Internet, or any other communication system in one or more locations. In some cases, network 104 may include at least one network within an enterprise or other organization that uses one or more machine learning models to perform one or more functions.

[0024] The application server 106 is coupled to the network 104 and is coupled to or otherwise in communication with a database server 108 and / or a file server 111. The application server 106, the database server 108, the database 110, and / or the file server 111 support the use of at least one metadata-driven feature store. For example, the application server 106 can execute one or more applications 112, and the applications 112 can be configured to define and use one or more metadata-driven feature stores. In some cases, the data for each metadata-driven feature store can be physically stored in the database 110 and accessed via the database server 108 and / or physically stored in the file server 111. Note that the database server 108 or the file server 111 may also be used within the application server 106 to store information, in which case the application server 106 itself stores information used to support one or more metadata-driven feature stores.

[0025] The database server 108 and / or file server 111 operate to store in the database 110 and facilitate retrieval of various information used, generated, or collected by the application server 106 and the user devices 102a-102d. For example, the database server 108 and / or file server 111 may store data associated with one or more metadata-driven feature stores, such as feature stores that include features or feature sets used by one or more machine learning models, and metadata associated with those features or feature sets.

[0026] Note that in the system 100 of FIG. 1, it is envisioned that the metadata-driven feature store may be provided by the system 100 itself using the application server 106, the database server 108, the database 110, and / or the file server 111. However, the metadata-driven feature store may be implemented in any other suitable manner. For example, the system 100 may be coupled to at least one external network 114, such as the Internet. This may enable at least one metadata-driven feature store for an organization associated with the system 100 to be created by a remote server 116, a cloud-based environment, or any other suitable device external or remote to the system 100. Data associated with the at least one metadata-driven feature store may also be stored by an external database server 118, in an external database 120, in an external file server, or in any other suitable manner. Implemented in this manner, multiple metadata-driven feature stores may be created and managed using server 116 and may have data stored in database 120 and / or external file servers on behalf of multiple systems 100, such as different systems 100 associated with different enterprises. In general, the present disclosure is not limited to any particular implementation or environment of one or more metadata-driven feature stores.

[0027] Although FIG. 1 illustrates one example of a system 100 supporting a metadata-driven feature store, various modifications may be made to FIG. 1. For example, the system 100 may include any number of user devices 102a-102d, networks 104, 114, an application server 106, a remote server 116, database servers 108, 118, databases 110, 120, and a file server 111. Also, these components may be located in any suitable locations and may be distributed over a wide area. In addition, although FIG. 1 illustrates one example operating environment in which one or more metadata-driven feature stores may be used, this functionality may be used in any other suitable system.

[0028] 2 illustrates one exemplary device 200 supporting a metadata-driven feature store according to the present disclosure. One or more instances of device 200 may be used to, for example, at least partially implement the functionality of application server 106, database server 108, and / or file server 111 of FIG. 1. However, the functionality of application server 106, database server 108, and / or file server 111 may be implemented in any other suitable manner. In some embodiments, device 200 illustrated in FIG. 2 may form at least a portion of user devices 102a-102d, although each of these components may be implemented in any other suitable manner.

[0029] 2, device 200 represents a computing device or system including at least one processing unit 202, at least one storage unit 204, at least one communication unit 206, and at least one input / output (I / O) unit 208. Processing unit 202 is capable of executing instructions that may be loaded into memory 210. Processing unit 202 includes any suitable number and type of processors or other processing devices in any suitable arrangement. Exemplary types of processing units 202 include one or more microprocessors, microcontrollers, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or discrete circuits.

[0030] Memory 210 and persistent storage 212 are examples of storage devices 204 and represent any structure capable of storing and facilitating retrieval of information (such as data, program code, and / or other suitable information, either temporary or persistent). Memory 210 may represent random access memory or any other suitable volatile or non-volatile storage device. Persistent storage 212 may include one or more components or devices that support long-term storage of data, such as read-only memory, a hard drive, a flash memory, or an optical disk.

[0031] The communications unit 206 supports communication with other systems or devices. For example, the communications unit 206 may include a network interface card or a wireless transceiver that facilitates communication over a wired or wireless network, such as the networks 104, 114. The communications unit 206 may support communication over any suitable physical or wireless communications link.

[0032] I / O unit 208 allows for the input and output of data. For example, I / O unit 208 may provide a connection for user input via a keyboard, mouse, keypad, touch screen, or other suitable input device. I / O unit 208 may also send output to a display, printer, or other suitable output device. Note, however, that I / O unit 208 may be omitted if device 200 does not require local I / O, such as if device 200 represents a server or other device that can be accessed remotely.

[0033] Although Figure 2 illustrates one example of a device 200 supporting a metadata-driven feature store, various modifications may be made to Figure 2. For example, computing and communication devices and systems come in a wide variety of configurations, and Figure 2 is not intended to limit the disclosure to any particular computing or communication device or system.

[0034] Figure 3 illustrates one exemplary architecture 300 supporting a metadata-driven feature store according to the present disclosure. For ease of explanation, the architecture 300 illustrated in Figure 3 is described as being implemented on or provided by the application server 106 and database 110 and / or file server 111 in the system 100 illustrated in Figure 1, where the application server 106 may be implemented using one or more instances of the device 200 illustrated in Figure 2. However, the architecture 300 illustrated in Figure 3 may be implemented on or provided by any other suitable device and implemented in any other suitable system.

[0035] As shown in FIG. 3, architecture 300 includes data store 302, which represents a storage device (such as database 110 and / or file server 111) that can be used to store and retrieve various features or feature sets used by one or more machine learning models or other devices or systems. The types of features or feature sets stored in data store 302 can vary widely based on the particular application for the machine learning models. Data store 302 may be implemented in any suitable manner and may use any suitable technology to store and retrieve information. As a particular example, data store 302 may be implemented using a file store, a key-value store, an object storage service, or an SQL database. In general, this disclosure is not limited to any particular type of database, file server, or other data storage technology. In some cases, data store 302 may be implemented to have low latency to enable rapid throughput of data.

[0036] Using architecture 300, a data scientist or other personnel can access experimental data 304, which generally represents data that can be used to design, train, and evaluate one or more machine learning models to perform at least one desired function. The experimental data 304 can be stored in any suitable location, such as in database 110 and / or file server 111. The experimental data 304 can also have any suitable form, such as where the experimental data 304 is defined using one or more data frames. The data scientist or other personnel can define one or more features or feature sets to be generated using at least a portion of the experimental data 304 and identify one or more transformations to be applied to the experimental data 304 to generate the features or feature sets. Any suitable transformations used to generate the features or feature sets can be identified here, such as mapping the experimental data 304 to feature values, scaling the experimental data 304 to feature values, or aggregating the experimental data 304 to create feature values. This transformation may be explicitly specified or may be implicitly identified, such as by tracking personnel actions taken on the experimental data 304 using an interactive API (such as the Pandas API, or the SQL API).

[0037] The identified features or feature sets and the identified transformations are used to define or generate feature / feature set metadata 306. The metadata 306 generally defines one or more features or feature sets to be generated and one or more transformations to be performed to generate the one or more features or feature sets from the source data. For example, the metadata 306 may include names and data types for the defined features or feature sets. The metadata 306 may also include user-supplied metadata, such as a user-supplied description of each of the one or more features or feature sets. The metadata 306 may further include information defining a transformation to be performed to generate feature values ​​for the features or feature sets using the experimental data 304. This transformation may be represented in the metadata 306 in any suitable manner, such as in a manner that makes it easy to determine the source rows and columns of the experimental data 304 whose values ​​may affect the value of each feature. If the feature is a time series feature, the desired aggregation interval (if any) may also be stored as part of the metadata 306 for that feature.

[0038] Upon request, or in response to any other suitable trigger, features or feature sets may be determined using the experimental data 304 (e.g., according to the metadata 306) and instantiated into the data store 302. Generally, each instantiation operation involves storing the features or feature sets identified using the experimental data 304 as stored features 308 in the data store 302. The stored features 308 may include features stored in any suitable manner, such as in a file or a relational database table. For example, in some embodiments, a file containing the stored features 308 may be partitioned (e.g., via ranges or hashes) according to one or more criteria, such as one, some, or all of the following criteria: object type, object ID, feature name, time range of the feature value, and the feature value itself. In other embodiments, the data store 302 represents an SQL database, and each feature or feature set may be stored in a table.

[0039] Once the stored features 308 are available in the data store 302, various features and feature sets may be evaluated by personnel, such as by querying the data store 302 via an API or other mechanism and receiving selected features 308 from the data store 302. The selected features 308 identified for evaluation (or a portion of the data associated with the selected features 308) may be exported, such as to one or more machine learning models represented as clients 310 of the data store 302. The selected features 308 identified for evaluation may also be exported in any suitable form, such as one or more data frames. For example, each evaluation may include querying the data store 302 for stored features 308 that meet one or more specified criteria and obtaining a data frame containing those stored features 308. Each data frame may be suitable for input to a machine learning model or client 310. As a particular example, the data store 302 may be queried by providing a request that identifies a list of feature names or feature set names, any filter conditions for subject IDs, and an optional timestamp range. Stored features 308 that satisfy this requirement (or a portion of the data associated with stored features 308 that satisfy this requirement) may be identified, merged, and output as a Data Frame. Note that when evaluating a Feature Set, a desired time interval is provided in a query of the Data Store 302, and individual stored features 308 that satisfy the query may be resampled if they do not match the current interval. A point-in-time join may also be performed across the features in the Feature Set to generate a query result (e.g., a Data Frame of the Feature Set). Note also that if the Data Store 302 is an SQL database, an SQL query with appropriate evaluation conditions may be used to retrieve the selected features 308.

[0040] This approach allows the experimental data 304 to be used to identify various features 308 that can be stored, analyzed, tested, and adjusted as necessary or desired. Additionally, this approach may allow the features 308 to be defined and generated using the same type of data that may be acquired during inference. It is meant that the experimental data 304 need not be a flat file, but can represent data from one or more relational or time series databases (although the experimental data 304 may still be flat). Additionally, as can be seen, the data store 302 may allow changes or corrections to be made to the information to support operations such as point-in-time join operations on data associated with different time intervals.

[0041] At some point, the production data 312 may become available for use. The production data 312 may then be stored in at least one production store 314. The production data 312 may be stored in any suitable location, such as in the database 110 and / or the file server 111 implementing the production store 314. The production data 312 may represent data used for large-scale training of machine learning models and / or provided to one or more trained machine learning models (such as one or more clients 310) for use during inference. It is possible (and rather common) for the production data 312 to differ in at least one respect from the experimental data 304, such as in terms of format or content. As a result, one or more additional transformations may be defined based on the production data 312 to generate the same feature or feature set previously defined for the experimental data 304. Note that the one or more additional transformations may be created manually (e.g., by a data scientist or other personnel) or inferred, such as based on operations invoked by personnel using the production data 312. However one or more additional transformations are identified, the features or feature sets to be generated and the additional transformations used to generate those features or feature sets using production data 312 may be used to define or generate additional feature / feature set metadata 306. Again, additional metadata 306 may include the name and data type of the defined feature or feature set, any user-supplied metadata, as well as information defining the transformations to be performed to generate feature values ​​for the feature or feature set using production data 312. The additional transformations may be represented in additional metadata 306 in any suitable manner, such as in a manner that is easy to determine the source rows and columns of production data 312 whose values ​​may affect the value of each feature.

[0042] After some of the production data 312 is loaded into the production store 314, features or feature sets can be determined using the production data 312 (e.g., according to the additional metadata 306) and embodied in the data store 302. This allows features or feature sets based on the production data 312 to be stored in the data store 302 as additional stored features 308. One or more clients 310 (e.g., one or more trained machine learning models) can use the same API calls or other queries previously used during evaluation of the stored features 308 on the experimental data 304 to obtain the selected stored features 308 for the production data 312 (e.g., in the form of a data frame) and process those features 308. In some cases, an optional feature cache 316 can be used to store a subset of the stored features 308 in the data store for faster access by one or more clients 310.

[0043] As new production data 312 arrives, the production store 314 may be updated with the new data. A change propagation function 318 may be used herein to determine whether any of the new production data 312 (and associated time ranges) need to be updated, such as when the intervals of the new production data 312 do not match. In some cases, the change propagation function 318 may determine whether the new production data 312 needs to be adjusted based on the particular incoming production data 312 received and the dependency mappings in the metadata 306. If changes are required, the change propagation function 318 may resample at least a portion of the new production data 312 to the appropriate time. The change propagation function 318 may use any suitable technique to identify and resample data herein. In some embodiments, for example, the change propagation function 318 may be implemented using the ANALYTICS CONTAINER ENGINE (ACE) from C3.AI, Inc.

[0044] In some cases, it is also possible to perform one or more in-memory transformations 320 on at least a portion of the production data 312 to generate additional features or feature sets directly from the production data 312. These features or feature sets may similarly be embodied for storage as features 308 in the data store 302 or for storage in a feature cache 316 (for low latency processing). Additional metadata 306 may be generated and used to implement the processing performed by the in-memory transformations 320. If necessary, a change propagation function 318 may be used to modify at least a portion of the production data 312 being processed by the one or more in-memory transformations 320 to take into account different time intervals.

[0045] Additionally, at some point after the production data 312 has been received and processed, it may be necessary or desirable to define and use one or more additional features or feature sets based on the production data 312. This may include modified versions of previously defined features or feature sets or new features or feature sets. In these cases, one or more additional features or feature sets and their associated transformations may be defined. Additional metadata 306 related to these additional features or feature sets and associated transformations may then be generated. The one or more additional features or feature sets may be determined using the production data 312 (such as according to the additional metadata 306). The additional features 308 may then be placed in the data store 302 and embodied into the data store 302 so that they may be provided to one or more machine learning models or clients 310.

[0046] In this manner, the metadata 306 can be used to define transformations that are applied to generate and update features 308 that are stored in the data store 302. Users can also use the metadata 306 to infer the lineage of information passing through the data store 302, and at least some of the metadata 306 defined can actually identify the particular client 310 for which a particular feature 308 is being determined. In addition, users can easily transition between experimental and production modes of operation, meaning that they can easily switch between designing features for use by machine learning models and computing those features for analysis by machine learning models.

[0047] There are various other functions that may be implemented within or using the feature store architecture 300 as shown in FIG. 3. For example, the architecture 300 may be configured to capture snapshots of features 308 in the data store 302. A snapshot represents the state of a feature or set of features at a particular point in time. In some cases, it may be desirable to capture feature snapshots when training a machine learning model, since the snapshots are then used to train other models, ensuring that the models are trained on the exact same data. When using machine learning models in certain industries (such as finance), it may be legally required to retain snapshots for audit purposes. The metadata-driven design of the feature store makes it easy to capture and leverage feature snapshots.

[0048] One exemplary process for capturing a feature or feature set snapshot may be performed as follows: When a feature or feature set is queried from the feature store 302 by a client 310, the client 310 may provide a query, such as one with a list of object IDs (or a filter expression to generate that list) and a time range. The process described above may be used to identify a subset of feature data stored in the data store 302 that satisfies the query. The stored feature data that satisfies the query may be copied to another location in the data store 302. Various techniques for copying the stored features 308 may be used. In some cases, the entire data file set may be copied as is to the new location. Metadata generated as described below may then be used to filter data outside the scope of the snapshot (although this may include extra data that was not part of the original query, such as in the case where the files include a longer time range than requested). In other cases, the stored feature data that satisfies the query may be merged and rewritten as a new copy in an optimized format, such as by simply including the data that satisfies the query. In still other cases, where previous snapshots have any identical files, only one copy of the common files may need to be kept, and each snapshot can maintain references to these files. A combination of these approaches may also be used. However the copy is generated, a metadata entry for the snapshot can be added to the metadata 306 for the associated feature or feature set, and this metadata entry can provide information about the snapshot (such as the original filter criteria for the snapshot, the snapshot timestamp, and bidirectional links between the snapshot and the metadata for the feature or feature set included in the snapshot).The export command allows data from a snapshot to be copied to an external location (such as an object storage bucket), and the external copy can be archived or used independently of the feature store 302. Future feature requests from the client 310 can specify this snapshot, and data can be returned from the snapshot instead of the feature store 302 (by using the snapshot metadata).

[0049] As another example feature that may be implemented, business or other requirements may dictate that certain data in the production store 314 cannot be disclosed to unauthorized users. To satisfy these requirements, an access control layer (such as one based on a role-based access control model) may be implemented in the production store 314. The same requirements may generally be applied to the transformed data (features 308) retrieved from the feature store 308 by the client 310. In existing systems, users may need to manually configure the access control system for the feature store and manually re-implement all existing access rules for the feature data. With the feature store shown in FIG. 3, the transformations identified by the metadata 306 may be used to create mappings from rows and columns in the production store 314 to the features 308 in the feature store 302 and vice versa. This may be similar to the mappings created to determine change propagation. Using these mappings, the access control settings from the production store 314 may be used to infer access control settings for the features 308 in the feature store 302. One example of this may occur as follows:

[0050] Given a feature 308 (and optionally a list of targets), the mapping can be used to determine the rows and columns in the production store 314 that will be used to create the feature 308. Access settings for the mapped rows and columns in the production store 314 can be identified, and this can be referred to as the source access control set. The access control settings can be found to be as restrictive as or more restrictive than all the settings in the access control set. For example, a partial ordering of all the access rights can be created, where the lowest access rights can represent no access and the highest access rights can represent access to everyone. A highest access level can be selected that is less than or equal to all the access settings in the source access control set. The selected access level can then be applied to the feature 308 (and possibly its target list). Optionally, a user can manually override the access control settings calculated for the feature 308, which can be useful if the calculated settings prove to be too restrictive or not necessary (such as if a given feature 308 has been anonymized with respect to the original source data).

[0051] Note that the functionality shown in or described with respect to FIG. 3 may be implemented in an electronic device, such as a computing device, in any suitable manner. For example, in some embodiments, at least some of the functionality shown in or described with respect to FIG. 3 may be implemented or supported using one or more software applications or other software instructions executed by one or more processing units of application server 106, database server 108, file server 111, device 200, or other devices. In other embodiments, at least some of the functionality shown in or described with respect to FIG. 3 may be implemented or supported using dedicated hardware components. In general, the functionality shown in or described with respect to FIG. 3 may be performed using any suitable hardware or any suitable combination of hardware instructions and software / firmware instructions.

[0052] Also note that the various functions described above with respect to FIG. 3 may be implemented in a number of ways. For example, identification of the transformations to be applied to the experimental data 304 or production data 312 may occur in any suitable manner. And the metadata 306 defining the transformations may be expressed in any suitable manner. As a particular example, U.S. patent application Ser. No. 17 / 698,934, entitled “Intelligent Data Processing System with Multi-Interface Frontend and Backend” (attorney docket CAII01-00011), which is incorporated herein by reference in its entirety, describes how data transformations may be identified and used to generate a directed acyclic graph that represents such data transformations. The same or similar mechanisms may be used here to track how transformations should be applied to the experimental data 304 or production data 312 and record information defining those transformations. However, any other suitable approach may be used here. Additionally, note that the metadata 306 may be stored in any suitable storage location, such as the production store 314, the data store 302, the database 110, or the file server 111.

[0053] Finally, note that in Figure 3 it is assumed that experimental data 304 is used to define features or feature sets and their associated transformations, and that production data 312 is then obtained and used to generate those features or feature sets (possibly using different transformations). However, the use of experimental data 304 is not required. As a result, the same operations and processes as described above with respect to experimental data 304 may in fact be performed using production data 312, which means that features or feature sets and their associated transformations may be defined using production data 312.

[0054] Although Figure 3 illustrates one example of an architecture 300 supporting a metadata-driven feature store, various modifications may be made to Figure 3. For example, functions and components may be added, omitted, combined, further subdivided, duplicated, or arranged in any other suitable configuration in architecture 300 according to particular needs. As a particular example, architecture 300 may include any suitable number of data stores and be used by any suitable number of clients.

[0055] Figures 4A and 4B illustrate one exemplary method 400 for using a metadata-driven feature store according to the present disclosure. For ease of explanation, the method 400 illustrated in Figures 4A and 4B is described as being performed using the application server 106 and the database 110 and / or file server 111 in the system 100 shown in Figure 1, where the application server 106 may be implemented using one or more instances of the device 200 shown in Figure 2. However, the method 400 illustrated in Figures 4A and 4B may be performed using any other suitable device and in any other suitable system.

[0056] As shown in Figures 4A and 4B, one or more features or feature sets to be generated using the experimental data are identified in step 402, and one or more transformations to be performed to generate the one or more features or feature sets using the experimental data are identified in step 404. This may include, for example, the processing unit 202 of the application server 106 enabling a user to identify the experimental data 304 to be used and the one or more features or feature sets to be generated using the experimental data 304. This may also include the processing unit 202 of the application server 106 enabling a user to identify or invoke one or more transformations to be applied to the experimental data 304 to generate the features or feature sets. In step 406, metadata associated with the one or more features or feature sets and the one or more transformations are generated. This may include, for example, the processing unit 202 of the application server 106 generating metadata 306 identifying the features / feature sets and the transformations. In step 408, one or more transformations are performed to generate one or more features or feature sets using the experimental data, and the one or more features or feature sets are embodied into the feature store. This may include, for example, the processing unit 202 of the application server 106 performing a transformation using the experimental data 304 defined by the metadata 306 to generate the feature or feature set. Note that this assumes that the feature or feature set was not previously generated if a transformation was defined, which may be the case in some cases. This may also include the processing unit 202 of the application server 106 storing the feature / feature set in the data store 302 as stored features 308.

[0057] At step 410, at least one query is received for one or more features or feature sets to be used for the evaluation. This may include, for example, the processing unit 202 of the application server 106 receiving a query from the client 310 for features 308 stored in the data store 302 that match one or more specified criteria. At step 412, one or more selected features or feature sets are provided for use during the evaluation. This may include, for example, the processing unit 202 of the application server 106 retrieving one or more of the stored features 308 from the data store 302 based on one or more criteria in the query and providing the retrieved features / feature sets in a data frame. The retrieved information may be used in any suitable manner during the evaluation. If desired, various of steps 402-412 may be repeated based on the results of the evaluation or for any other suitable reasons, such as to change the features or feature sets. Among other things, this may enable a user to review the features or Feature sets generated using the experimental data 304 and adjust the features or Feature sets as necessary or desired (such as by modifying or deleting existing features / Feature sets or adding new features / Feature sets).

[0058] At some point, production data may be obtained for use during inference, such as when production data 312 is obtained from any suitable source and optionally stored in at least one production store 314. In step 414, one or more transformations are identified to be performed using the production data to generate one or more features or feature sets. This may include, for example, the processing unit 202 of the application server 106 enabling a user to identify or invoke one or more transformations to be applied to the production data 312 to generate the same features or feature sets previously defined for the experimental data 304. In step 416, metadata associated with the one or more features or feature sets and the one or more transformations is generated. This may include, for example, the processing unit 202 of the application server 106 generating additional metadata 306 identifying the features / feature sets and transformations for the production data 312. If necessary, in step 418, one or more modifications may be implemented to the production data. This may include, for example, the processing unit 202 of the application server 106 performing a change propagation function 318 to resample one or more portions of the production data 312 associated with a different time interval than one or more other portions of the production data 312. In step 420, one or more transformations are performed using the production data or the modified production data to generate one or more features or feature sets, and the one or more features or feature sets are embodied in the feature store. This may include, for example, the processing unit 202 of the application server 106 performing a transformation using the production data 304 as defined by the additional metadata 306 to generate the features or feature sets. The one or more transformations may be performed here using the production data 312 in the production store 314, using one or more in-memory transformations 320, or in any other suitable manner. This may also include the processing unit 202 of the application server 106 storing the features / feature sets in the data store 302 as additional stored features 308.

[0059] At step 422, at least one query is received for one or more features or feature sets to be used for inference. This may include, for example, the processing unit 202 of the application server 106 receiving a query from the client 310 for stored features 308 in the data store 302 that match one or more specified criteria previously defined using features 308 of the experimental data 304. For inference, the stored features 308 sought here represent features produced using the production data 312. At step 424, one or more selected features or feature sets are provided for use during inference. This may include, for example, the processing unit 202 of the application server 106 retrieving one or more of the stored features 308 from the data store 302 and providing the retrieved features / feature sets in a data frame. The retrieved information may be used in any suitable manner during inference.

[0060] Optionally, there may be times when one or more features or feature sets used with the production data 312 need to be updated or deleted, or when one or more features or feature sets need to be added for use with the production data 312. If so, in step 426, one or more features or feature sets used with the production data may be updated and additional metadata may be generated for use. This may include, for example, the processing unit 202 of the application server 106 enabling the user to modify the features or feature sets used with the production data 312 or to create new features or feature sets for use with the production data 312. This may also include the processing unit 202 of the application server 106 enabling the user to identify one or more transformations used to generate the updated features or feature sets. This may further include the processing unit 202 of the application server 106 generating additional metadata 306 associated with the updated features or feature sets and the associated transformations.

[0061] 4A and 4B illustrate one example of a method 400 for using a metadata-driven feature store, various modifications may be made to FIG. 4A and 4B. For example, although shown as a series of steps, various steps in FIG. 4A and 4B may overlap, be performed in parallel, be performed in a different order, or be performed any number of times. Also, as discussed above, various steps described as being performed using experimental data 304 may instead be performed using production data 312.

[0062] Below, we describe an example embodiment of the present disclosure that implements a metadata-driven feature store for a machine learning system, although other embodiments may be used in accordance with the teachings of the present disclosure.

[0063] In a first embodiment, the method includes identifying one or more transformations to be applied to generate one or more features or feature sets. The method also includes generating metadata identifying the one or more features or feature sets and the one or more transformations. The method further includes using the metadata to determine the one or more features or feature sets for specified data and storing the one or more determined features or feature sets in a feature store. Additionally, the method includes outputting at least some of the one or more determined features or feature sets, or data associated with at least some of the one or more determined features or feature sets, from the feature store to at least one machine learning model.

[0064] In a second embodiment, the apparatus includes at least one processing device configured to identify one or more transformations to be applied to generate one or more features or feature sets. The at least one processing device is also configured to generate metadata identifying the one or more features or feature sets and the one or more transformations. The at least one processing device is further configured to determine the one or more features or feature sets for specified data using the metadata and store the one or more determined features or feature sets in a feature store. Additionally, the at least one processing device is configured to output at least some of the one or more determined features or feature sets, or data associated with at least some of the one or more determined features or feature sets, from the feature store to at least one machine learning model.

[0065] In a third embodiment, a non-transitory computer-readable storage medium has computer-readable program code stored thereon that, when executed, causes one or more processors to identify one or more transformations to be applied to generate one or more features or feature sets. The medium also has computer-readable program code stored thereon that, when executed, causes the one or more processors to generate metadata identifying the one or more features or feature sets and the one or more transformations. The medium further has computer-readable program code stored thereon that, when executed, causes the one or more processors to use the metadata to determine the one or more features or feature sets for specified data and to store the one or more determined features or feature sets in a feature store. Additionally, the medium has computer-readable program code stored thereon that, when executed, causes the one or more processors to output at least some of the one or more determined features or feature sets, or data associated with at least some of the one or more determined features or feature sets, from the feature store to at least one machine learning model.

[0066] Any one of the following features, or any suitable combination, may be used with the first, second, or third embodiment. The one or more transformations applied may represent one or more transformations applied to the experimental data to generate one or more features or feature sets of the experimental data. At least some of the one or more determined features or feature sets, or data associated with at least some of the one or more determined features or feature sets, may be output for evaluation. The one or more transformations may include one or more first transformations, and the metadata may include the first metadata. One or more second transformations applied to generate the one or more features or feature sets for the production data may be identified, second metadata identifying the one or more features or feature sets and the one or more second transformations may be generated, and the second metadata may be used to determine the one or more features or feature sets for the production data. The one or more second determined features or feature sets may be stored in a feature store. And at least some of the one or more second determined features or feature sets, or data associated with at least some of the one or more second determined features or feature sets, may be output for inference. A query for the determined features or feature sets stored in the feature store may be received, and any determined features or feature sets that match one or more criteria specified in the query, or data associated with any determined features or feature sets that match one or more criteria specified in the query, may be output. A determination may be made whether portions of the specified data are associated with different time intervals, and at least one portion of the specified data may be resampled to a time interval associated with at least one other portion of the specified data. The metadata may be used to determine one or more features or feature sets of the specified data by performing one or more in-memory transforms, and the one or more determined features or feature sets may be stored in a feature cache of the feature store.One or more additional transformations to be applied to generate the one or more additional features or feature sets may be identified, and additional metadata identifying the one or more additional features or feature sets and the one or more additional transformations may be generated, and the additional metadata may be used to determine the one or more additional features or feature sets for the additional specified data. The one or more determined additional features or feature sets may be stored in a feature store, and data related to at least some of the one or more determined additional features or feature sets or at least some of the one or more determined additional features or feature sets may be output from the feature store. A snapshot of the specified ones of the one or more features or feature sets in the feature store identified as satisfying the query may be generated, the snapshot may be stored in the feature store, and a metadata entry may be generated identifying the snapshot in the metadata associated with the specified feature or feature set. An access control set associated with a specified portion of data used to generate a specified one of one or more features or feature sets may be identified, and access control settings for the specified feature or feature set in the feature store may be identified (the access control settings may be as restrictive as all of the settings in the access control set, or may be more restrictive).

[0067] In some embodiments, various functions described in this patent document are implemented or supported by a computer program formed from computer readable program code and embodied in a computer readable medium. The phrase "computer readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer readable medium" includes any type of medium that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), hard disk drive (HDD), compact disk (CD), digital video disk (DVD), or any other type of memory. "Non-transitory" computer readable medium excludes wired, wireless, optical, or other communication links that carry transitory electrical or other signals. Non-transitory computer readable medium includes media on which data can be stored permanently and media on which data can be stored and later overwritten, such as rewritable optical disks or erasable storage devices.

[0068] It may be advantageous to explain definitions of certain words and phrases used throughout this specification. The terms "application" and "program" refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, associated data, or portions thereof that are adapted for implementation in suitable computer code (including source code, object code, or executable code). The term "communicate" and its derivatives encompass both direct and indirect communication. The terms "include" and "comprise" and their derivatives mean inclusive without limitation. The term "or" is inclusive and / or. The phrase "associated with" and its derivatives may mean include, contained in, interconnected with, contain, included in, connected to, coupled with, in communication with, cooperate with, interleaved, juxtaposed, adjacent to, connected to, having, having a characteristic of, relating to, or the like. The phrases "at least one of" and "one or more of," when used in conjunction with a list of items, mean that different combinations of one or more of the listed items may be used and that only one item in the list may be required. For example, "at least one of: A, B, and C" includes any of the following combinations: A, B, C, A and B, A and C, B and C, and A and B and C.

[0069] The statements in this disclosure should not be read as suggesting that any particular element, step, or function is an essential or critical element that must be included in the scope of the claims. The scope of the patented subject matter is defined solely by the allowed claims. Moreover, none of the claims invoke 35 U.S.C. § 112(f) with respect to any of the appended claims or claim elements, unless the precise words "means for" or "step for" are expressly used in a particular claim followed by a participle phrase that identifies the function. The use of terms such as (but not limited to) "mechanism," "module," "device," "unit," "component," "element," "member," "apparatus," "machine," "system," "processor," or "controller" in the claims are understood and intended to refer to structures known to those skilled in the art, as further modified or enhanced by the features of the claims themselves, and are not intended to invoke 35 U.S.C. § 112(f).

[0070] While this disclosure has described certain embodiments and generally associated methods, modifications and permutations of these embodiments and methods will be apparent to those skilled in the art. Thus, the above description of exemplary embodiments does not define or constrain the disclosure. Other modifications, substitutions, and alterations are also possible without departing from the spirit and scope of the disclosure, as defined by the following claims.

Claims

1. 1. A method comprising: identifying one or more transformations to be applied to generate one or more features or feature sets; generating metadata identifying the one or more features or feature sets and the one or more transformations; using the metadata; determining said one or more features or feature sets for the specified data; storing the one or more determined features or feature sets in a feature store; outputting at least some of the one or more determined features or feature sets, or data associated with at least some of the one or more determined features or feature sets, from the feature store to at least one machine learning model; A method comprising:

2. the one or more transformations applied represent one or more transformations applied to the experimental data to generate the one or more features or feature sets for the experimental data; and At least some of the one or more determined characteristics or feature sets, or data associated with at least some of the one or more determined characteristics or feature sets, are output for evaluation. The method of claim 1.

3. The one or more transformations include one or more first transformations. The metadata includes first metadata; and The method further comprises: identifying one or more second transformations to be applied to the production data to generate the one or more features or feature sets; generating second metadata identifying the one or more features or feature sets and the one or more second transformations; using the second metadata; determining the one or more features or feature sets for the production data; storing one or more second determined features or feature sets in the feature store; outputting at least some of the one or more second determined features or feature sets or data associated with at least some of the one or more second determined features or feature sets for inference; The method of claim 2 , comprising:

4. The method further comprises: receiving a query for the determined features or feature sets stored in the feature store; The step of outputting at least some of the one or more determined features or feature sets, or data associated with at least some of the one or more determined features or feature sets, may further comprise: outputting any determined features or feature sets that match one or more criteria specified in the query, or data associated with any determined features or feature sets that match the one or more criteria specified in the query; The method of claim 1 , comprising:

5. The method further comprises: determining whether the specified portions of data are associated with different time intervals; resampling at least one portion of the designated data to a time interval associated with at least one other portion of the designated data; The method of claim 1 , comprising:

6. using the metadata to determine the one or more features or feature sets for the specified data includes performing one or more in-memory transformations; and storing the one or more determined features or feature sets comprises storing the one or more determined features or feature sets in a feature cache of the feature store. The method of claim 1.

7. The method further comprises: identifying one or more additional transformations to be applied to generate one or more additional features or feature sets; generating additional metadata identifying the one or more additional features or feature sets and the one or more additional transformations; using said additional metadata; determining said one or more additional features or feature sets for additional specified data; storing the one or more determined additional features or feature sets in the feature store; outputting at least some of the one or more additional determined features or feature sets, or data associated with at least some of the one or more additional determined features or feature sets, from the feature store; The method of claim 1 , comprising:

8. The method further comprises: generating a snapshot of a designated one of the one or more features or feature sets in the feature store identified as satisfying a query; storing the snapshot in the feature store; generating a metadata entry identifying the snapshot in the metadata associated with the specified feature or feature set; The method of claim 1 , comprising:

9. The method further comprises: identifying an access control set associated with the specified portion of data used to generate a specified one of the one or more features or feature sets; identifying an access control setting for the specified feature or feature set in the feature store; the access control settings being at least as restrictive as any setting in the access control set; The method of claim 1 , comprising:

10. 1. An apparatus comprising at least one processing unit, The at least one processing device comprises: Identifying one or more transformations to be applied to generate one or more features or feature sets; generating metadata identifying the one or more features or feature sets and the one or more transformations; Using the metadata to determine the one or more features or feature sets for the specified data; storing the one or more determined features or feature sets in a feature store; and outputting at least some of the one or more determined features or feature sets, or data associated with at least some of the one or more determined features or feature sets, from the feature store to at least one machine learning model; The apparatus is configured to:

11. the one or more transformations applied represent one or more transformations applied to the experimental data to generate the one or more features or feature sets for the experimental data; and The at least one processing device comprises: At least some of the one or more determined characteristics or feature sets, or data associated with at least some of the one or more determined characteristics or feature sets, are output for evaluation. The apparatus of claim 10 , configured to:

12. The one or more transformations include one or more first transformations. The metadata includes first metadata; and The at least one processing device further comprises: identifying one or more second transformations to be applied to the production data to generate the one or more features or feature sets; generating second metadata identifying the one or more features or feature sets and the one or more second transformations; determining the one or more features or feature sets for the production data using the second metadata; storing the one or more second determined features or feature sets in the feature store; and outputting at least some of the one or more second determined features or feature sets or data associated with at least some of the one or more second determined features or feature sets for inference; The apparatus of claim 11 , configured as follows:

13. The at least one processing device is further configured to receive a query for the determined features or feature sets stored in the feature store; and To output at least some of the one or more determined features or feature sets, or data associated with at least some of the one or more determined features or feature sets, the at least one processing device may further include: outputting any determined features or feature sets that match one or more criteria specified in the query, or data associated with any determined features or feature sets that match one or more criteria specified in the query; The apparatus of claim 10 , configured to:

14. The at least one processing device further comprises: determining whether the specified portions of data are associated with different time intervals; and resampling at least one portion of the designated data to a time interval associated with at least one other portion of the designated data; The apparatus of claim 10 , configured to:

15. the at least one processing device is configured to perform one or more in-memory transformations to use the metadata to determine the one or more features or feature sets for the specified data; and the at least one processing unit is configured to store the one or more determined features or feature sets in a feature cache of the feature store.

11. The apparatus of claim 10.

16. The at least one processing device further comprises: Identifying one or more additional transformations to be applied to generate one or more additional features or feature sets; generating additional metadata identifying the one or more additional features or feature sets and the one or more additional transformations; using the additional metadata to determine the one or more additional features or feature sets for the additional specified data; storing the one or more determined additional features or feature sets in the feature store; and outputting from the feature store at least some of the one or more additional determined features or feature sets, or data associated with at least some of the one or more additional determined features or feature sets; The apparatus of claim 10 , configured to:

17. A non-transitory computer readable storage medium having computer readable program code stored thereon, The computer readable program code, when executed, causes one or more processors to: Identifying one or more transformations to be applied to generate one or more features or feature sets; generating metadata identifying the one or more features or feature sets and the one or more transformations; Using the metadata to determine the one or more features or feature sets for the specified data; storing the one or more determined features or feature sets in a feature store; and outputting at least some of the one or more determined features or feature sets, or data associated with at least some of the one or more determined features or feature sets, from the feature store to at least one machine learning model; A non-transitory computer-readable storage medium.

18. the one or more transformations applied represent one or more transformations applied to the experimental data to generate the one or more features or feature sets for the experimental data; and The computer readable program code, which when executed causes the one or more processors to output at least some of the one or more determined characteristics or feature sets or data associated with at least some of the one or more determined characteristics or feature sets, comprises: computer readable program code that, when executed, causes the one or more processors to output at least some of the one or more determined characteristics or feature sets, or data associated with at least some of the one or more determined characteristics or feature sets, for evaluation; 20. The non-transitory computer-readable storage medium of claim 17, comprising:

19. The one or more transformations include one or more first transformations. The metadata includes first metadata; and The medium further has stored thereon computer readable program code that, when executed, causes the one or more processors to: identifying one or more second transformations to be applied to the production data to generate the one or more features or feature sets; generating second metadata identifying the one or more features or feature sets and the one or more second transformations; determining the one or more features or feature sets for the production data using the second metadata; storing the one or more second determined features or feature sets in the feature store; and outputting at least some of the one or more second determined features or feature sets or data associated with at least some of the one or more second determined features or feature sets for inference; To make it so, 20. The non-transitory computer-readable storage medium of claim 18.

20. The medium, when executed, causes the one or more processors to receive a query for a determined feature or feature set stored in the feature store; and The computer readable program code, which when executed causes the one or more processors to output at least some of the one or more determined characteristics or feature sets or data associated with at least some of the one or more determined characteristics or feature sets, comprises: When executed, outputting to the one or more processors any determined features or feature sets that match one or more criteria specified in the query, or data associated with any determined features or feature sets that match the one or more criteria specified in the query. To make it so, 20. The non-transitory computer-readable storage medium of claim 17.

21. The medium further has stored thereon computer readable program code that, when executed, causes the one or more processors to: determining whether the specified portions of data are associated with different time intervals; and resampling at least one portion of the designated data to a time interval associated with at least one other portion of the designated data; To make it so, 20. The non-transitory computer-readable storage medium of claim 17.

22. The computer readable program code that, when executed, causes the one or more processors to use the metadata to determine the one or more features or feature sets for the specified data, comprises: When executed, causes the one or more processors to perform one or more in-memory transformations; and The computer readable program code that, when executed, causes the one or more processors to store the one or more determined characteristics or feature sets, when executed, causing the one or more processors to store the one or more determined features or feature sets in a feature cache of the feature store.

20. The non-transitory computer-readable storage medium of claim 17.

23. The medium further has stored thereon computer readable program code that, when executed, causes the one or more processors to: Identifying one or more additional transformations to be applied to generate one or more additional features or feature sets; generating additional metadata identifying the one or more additional features or feature sets and the one or more additional transformations; using the additional metadata to determine the one or more additional features or feature sets for the additional specified data; storing the one or more determined additional features or feature sets in the feature store; and outputting from the feature store at least some of the one or more additional determined features or feature sets, or data associated with at least some of the one or more additional determined features or feature sets; To make it so, 20. The non-transitory computer-readable storage medium of claim 17.

Images