Learning Interpretable Tabular Data Using Sequential Sparse Attention
TabNet addresses the interpretability and feature selection challenges of neural networks on tabular data by using sequential sparse attention, enhancing model performance and trust in decision-making through local and global feature importance.
Patent Information
- Application Number
- KR1020227003420
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-08-02
- Filing Date
- 2020-08-02
- Publication Date
- 2026-11-06
- Estimated Expiration
- 2040-08-02
AI Technical Summary
Existing machine learning models, particularly neural networks, struggle with interpretability and efficient feature selection for tabular data, leading to overparameterization and a lack of trust in decision-making processes.
TabNet, a neural network designed for tabular data, uses sequential sparse attention to select features at each decision step, providing interpretability through local and global feature importance, and enables flexible feature selection and prediction of missing values using unsupervised pre-training.
TabNet outperforms traditional models in classification and regression tasks by offering interpretable and efficient feature selection, enhancing trust in decision-making processes and improving model performance on tabular data.
Smart Images

Figure 112022011177623-PCT00014_ABST
Abstract
Description
Technology Field
[0001] The present disclosure relates to interpretable tabular data learning using sequential sparse attention. Background Technology
[0002] Machine learning models receive input and generate an output, such as a predicted output, based on the received input. Some machine learning models are parametric models that generate an output based on the received input and the values of the model's parameters.
[0003] Some machine learning models are deep models that use multiple layers to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes one or more hidden layers and an output layer that generate an output by applying a non-linear transformation to the received input, respectively.
[0004] One aspect of the present disclosure provides a method for interpreting tabular data. The method comprises receiving a set of features in a deep tabular data learning network (TabNet) running on data processing hardware. For each of a plurality of sequential processing steps, the method further comprises the step of the data processing hardware selecting a subset of relevant features from a set of features using a sparse mask of TabNet; the step of the data processing hardware processing the subset of relevant features using a feature transformer of TabNet to generate a decision step output and information for the next processing step in the plurality of sequential processing steps; and the step of the data processing hardware providing the information to the next processing step. The method further comprises the step of the data processing hardware determining a final decision output by aggregating the decision step outputs generated for the plurality of sequential processing steps.
[0005] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the method also includes, for each of a plurality of processing steps, a step in which data processing hardware determines an aggregate of the number of times each feature of a set of features is processed in each preceding processing step of a plurality of sequential processing steps using an attentive transformer of TabNet. In these implementations, the attentive transformer may include a fully connected layer (FC) and batch normalization (BN).
[0006] In some examples, the step of providing information to the next processing step includes providing information to an attributive transformer of TabNet that determines an aggregation of the number of times each feature of a set of features is processed in each preceding processing step of a plurality of sequential processing steps based on the provided information. The set of features may include tabular data. Optionally, the feature transformer of TabNet may include a plurality of neural network layers including a fully-connected layer (FC), batch normalization (BN), and generalized linear unit (GLU) nonlinearity, respectively. Here, a first portion of the plurality of neural network layers may be shared across each of the plurality of sequential processing steps, and a second portion of the remainder of the plurality of neural network layers may depend on the corresponding processing step of the plurality of sequential processing steps.
[0007] The decision step output generated by processing a subset of related features can pass through the rectified linear unit (ReLU) of TabNet. Additionally, the subset of features may correspond to interdependent feature columns of tabular data. In some implementations, the method also includes the step of receiving tabular data containing one or more masked features in data processing hardware; and the step of the data processing hardware predicting one or more masked features using unsupervised pre-training.
[0008] Another aspect of the present disclosure provides a system for interpreting tabular data. The system includes data processing hardware and memory hardware that communicates with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including receiving a set of features from a deep tabular data learning network (TabNet) executed on the data processing hardware. For each of a plurality of sequential processing steps, the operations also include selecting a subset of relevant features from a set of features using a sparse mask of TabNet; processing the subset of relevant features using a feature transformer of TabNet to generate a decision step output and information for the next processing step in the plurality of sequential processing steps; and providing the information to the next processing step. The operations also include determining a final decision output by aggregating the decision step outputs generated for the plurality of sequential processing steps.
[0009] This aspect may include one or more of the following optional features. In some implementations, operations also include, for each of the multiple processing steps, using the attentive transformer of TabNet to determine the aggregation of the number of times each feature of the set of features was processed in each preceding processing step of the multiple sequential processing steps. In these implementations, the attentive transformer may include a fully connected layer (FC) and batch normalization (BN).
[0010] In some examples, providing information to the next processing step involves providing information to an attributive transformer of TabNet that determines an aggregation of the number of times each feature of a set of features has been processed in each preceding processing step of a plurality of sequential processing steps based on the provided information. The set of features may include tabular data. Optionally, the feature transformer of TabNet may include a plurality of neural network layers, each including a fully-connected layer (FC), batch normalization (BN), and generalized linear unit (GLU) nonlinearity. Here, a first portion of the plurality of neural network layers may be shared across each of the plurality of sequential processing steps, and a second portion of the remainder of the plurality of neural network layers may depend on the corresponding processing step of the plurality of sequential processing steps.
[0011] The decision step output generated by processing a subset of relevant features can pass through TabNet's ReLU (rectified linear unit). Additionally, the subset of features may correspond to interdependent feature columns in the tabular data. In some implementations, operations also include receiving tabular data containing one or more masked features and predicting one or more masked features using unsupervised pre-training.
[0012] Details of one or more embodiments of the present disclosure are described in the description below and in the accompanying drawings. Other aspects, features, and advantages will be apparent from the description and drawings and from the claims. Brief explanation of the drawing
[0013] Figure 1 is a schematic diagram of an exemplary data management environment.
[0014] Figures 2a–2g are schematic diagrams of exemplary models for interpreting table data.
[0015] Figure 3 is a flowchart of an exemplary operation arrangement for interpreting table data.
[0016] FIG. 4 is a schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein.
[0017] Similar reference symbols in various drawings indicate similar elements. Specific details for implementing the invention
[0018] Neural networks have generally been successful in processing data types such as images, text, and audio. However, one type of data that appears to be lacking, despite being one of the most common data types in computing, is tabular data. Tabular data refers to data in a table format. Often, instead of using neural networks, there is a tendency to use decision trees for tabular data due to their ability to be interpretable, expressively efficient, and train quickly. For example, the interpretability of decision trees allows one to understand how a decision tree arrives at a specific result, rather than simply receiving the result without context. Decision trees are interpretable because decisions can be traced through their nodes. In the realm of tabular data, there may be trust issues arising from concerns regarding decisions that led to a specific result. For instance, how did a decision tree predict what the employee's bonus compensation should be? Or did a loan processing tool predict that an applicant qualified for a loan of a certain size? In these scenarios, someone might want to examine the context of the decision.
[0019] Generally, neural networks are based on stacked layers that can lead to overparameterization. Conventional parameterized neural networks tend not to find optimal solutions for tabular decision manifolds. However, when properly adapted to tabular data, neural networks can provide similar benefits to those expected based on their performance on other data types. For example, neural networks are particularly useful for large datasets and can use backpropagation on their inputs to guide efficient learning toward a goal. Through neural networks designed for tabular data, neural networks can also efficiently encode multiple data types (e.g., images) alongside tabular data, minimize the need for feature engineering—a core aspect of current tree-based tabular data learning methods—enable learning from streaming data, which has proven difficult for current decision tree models (e.g., decision trees make great efforts to dynamically adapt split points), and enable representation learning for domain adaptation, generative modeling, and semi-supervised learning.
[0020] To address some of the existing shortcomings of neural networks on tabular data while retaining the general benefits of neural networks, TabNet is a neural network designed to learn in a decision tree manner. In other words, TabNet aims to provide interpretability and sparse feature selection. TabNet takes raw tabular data as input without arbitrary feature preprocessing and is trained using gradient descent-based optimization to learn flexible features and enable flexible integration into end-to-end learning. Furthermore, TabNet uses sequential attention to select features to interpret at each decision step; this makes them interpretable and allows for focus on prominent features. Feature selection can vary for each input (i.e., instant-wise) in a single deep learning architecture with end-to-end learning. Through this approach, TabNet has been proven to outperform or perform as well as other tabular learning models (e.g., decision tree models) on various datasets for classification and regression problems. Regarding interpretability, TabNet provides two types of interpretability. It allows for local interpretability to visualize the importance of input features and how they are combined, while also possessing global interpretability to quantify the contribution of each input feature relative to the trained model. Furthermore, for tabular data, TabNet can predict missing features using techniques such as unsupervised pre-training.
[0021] Feature selection generally refers to the process of selecting a subset of features from a larger feature pool based on how useful each feature is for a given prediction. When features are selected based on the entire training dataset, this type of feature selection is referred to as a global feature selection method. In contrast, instance-wise feature selection methods refer to selecting features individually for each input. While some methods train models to maximize mutual information between selected features on a given response variable, TabNet uses soft feature selection with controllable sparsity in end-to-end learning, enabling a single model to jointly perform feature selection and output mapping. This soft feature selection is then embedded with controllable sparsity using sequential attention. Furthermore, unlike some models that attempt to map tabular data to different data types, TabNet can apply sequential attention to supervised or self-supervised learning on tabular data. TabNet is designed to have tree-like functionality in which linear combinations of features are selected, along with a feature selection process to represent decision boundaries. TabNet learns sparse instance-specific selection using training datasets, constructs a sequential multi-stage architecture where each decision step contributes to a portion of the decision based on selected features, performs non-linear processing of the selected features, and mimics an ensemble through higher dimensions and more steps. Through this approach, TabNet combines the benefits of tree-based decision models for tabular data with the benefits of neural networks.
[0022] FIG. 1 illustrates an example of a data management environment (100). A user device (110) associated with a user (10) generates user data (12) during the execution of its computing resources (112) (e.g., data processing hardware (114) and / or memory hardware (116)). For example, the user (10) uses one or more applications (e.g., real-time applications) running on the data processing hardware (114) of the user device (110) to generate user data (12). In some examples, the user device (110) is a local device (e.g., associated with the location of the user (10)) that uses its own computing resources (112) and has the ability to communicate with one or more remote systems (130) (e.g., via a network (120)). Additionally or alternatively, the user device (110) operates an application for the user (10) by utilizing its access to remote resources (e.g., remote computing resources (132)). User data (12) generated through the use of the user device (110) is initially stored locally (e.g., in a data storage (118) of memory hardware (116)), and then can be communicated to a remote system (130) or transmitted to a remote system (130) via a network (120) upon generation. For example, the user device (110) communicates user data to a storage system (140) using the remote system (130).
[0023] In some examples, the user (10) utilizes computing resources (132) of a remote system (130) (e.g., a cloud computing environment) for the storage and / or management of user data (12). In these examples, the remote system (130) may receive user data (12) as generated by various user applications. Here, user data (12) may refer to a data stream (e.g., a continuous or generally continuous feed of data reaching the remote system (130) for storage and / or further processing) or a distinct set of user data (12) transmitted to the remote system (130) at the coordination of the user (10). Much like the user device (110), the remote system (130) includes computing resources (132) such as remote data processing hardware (134) (e.g., servers and / or CPUs) and memory hardware (136) (e.g., disks, databases, or other forms of data storage).
[0024] In some configurations, remote computing resources (132) are resources utilized by various systems associated with and / or communicating with the remote system (130). FIG. 1 illustrates a remote system (130) communicating with a data storage system (140) and a manager (150), but the remote system (130) may also communicate with other data-related systems (e.g., a query system, and / or an analysis system). The remote system (130) having computing resources (132) may be configured to host one or more functions of the manager (150) and / or the data processing model (200). In some implementations, the remote system (130) is a distributed system in which computing resources (132) are distributed across one or more locations accessible via a network (120).
[0025] In some examples, the storage system (140) is configured to operate a data warehouse (142) (e.g., a data repository and / or multiple databases) as a means of storing data for a user (10) (or multiple users). Generally speaking, the data warehouse (142) may be designed to store data from one or more sources and to analyze, report, and / or integrate data from those sources. The data warehouse (142) enables users (e.g., organizational users) to have a central storage repository and a storage data access point. By including user data (12) in a central repository such as the data warehouse (142), the data warehouse (142) can simplify data retrieval for functions such as data analysis and / or data reporting (e.g., by an analysis system). Additionally, data warehouses (142) may be configured to store a significant amount of data so that a user (10) (e.g., an organizational user) can store a large amount of historical data to understand data trends. Since data warehouses (142) may be the primary or sole data storage repository for user data (12), the storage system (140) may receive a large amount of data from user devices (110) associated with the user (10). Additionally or alternatively, as a storage system (140), the storage system (140) and / or storage warehouse (142) may be configured for data security (e.g., data redundancy) for multiple users (e.g., multiple employees of an organization) and / or simultaneous multi-user access from a single data source. In some configurations, the data warehouse (142) is persistent and / or non-volatile so that data is not overwritten or deleted by new incoming data.
[0026] Generally speaking, data storage systems (140) receive user data (12) in a table format referred to as table data, in which user data (12) populates the rows and columns of the table. Using table data, user data (12) within the table may have rows and columns corresponding to schemas or headings associated with user data (12). For example, user data (12) may refer to commercial transactions made by user (10). In this example, user data (12) may include columns for seller, buyer, transaction price, transaction quantity, and other transaction data collected by user (10) in relation to their transactions. Here, each row may have a heading or schema such as a transaction number or identifier and / or a time entry associated with the transaction. Because the storage system (140) can receive user data (12) in a specific format (e.g., transaction table format), the storage system (140) is configured to store user data (12) so that elements of the format associated with the user data (12) (e.g., relationships, headings, or other schemas) (e.g., providing additional context or definition for the user data (12)) are accessible to other systems communicating with the data storage system (140) (e.g., data retrieval systems such as query systems or data analysis systems).
[0027] Referring further to FIG. 1, the data management environment (100) also includes a manager (150). The manager (150) is generally configured to coordinate (e.g., optimize) data storage. The manager (150) can perform such optimization by using a data processing model (200) (also referred to as TabNet (200)) to execute and / or coordinate actions related to a system communicating with a data storage system (140). Here, TabNet (200) is configured as a machine learning model capable of learning aspects of tabular data received from the manager (150) while performing data storage. Based on this learning, TabNet (200) can assist various management functions. For example, TabNet (200) is configured to identify missing or incomplete data (referred to as missing features) in user data (12) and to predict or generate (or suggest) data entries for the missing data. Similarly, instead of predicting entries for incomplete data, the user (10) may request that the TabNet (200) predict that the entry should be based on other data stored for the user (10) (via the manager (150)). For example, the user (10) is a company that wants to know how much to pay a new employee based on the employee's previous experience and skills. Here, the TabNet (200) will interpret employee financial user data (12) stored in the storage system (140), including, for example, salary, position, department, years of experience, etc., and will output a prediction for the new employee's salary based on the user data (12) it interprets. In some examples, the manager (150) uses the TabNet (200) to interpret the interdependencies of the user data (12) in order to understand how the user data (12) can be stored while still preserving the relationships between the columns of the user data (12).In other words, TabNet (200) can identify split points that indicate locations to split user data (12) for storage purposes.
[0028] In some implementations such as FIG. 1, the manager (150) is configured to receive user data (12) from the user (10) and to enable storage operations in the storage system (140). For example, the manager (150) may collect the user data (12) upon receipt and transform the user data (12) into a storage-optimized format. Here, collection refers to acquiring and / or retrieving the user data (12) into the storage system (140) (e.g., a data warehouse (142)) so that the system(s) (e.g., by query systems and / or analysis systems) can use the collected user data. During collection, the manager (150) may use TabNet (200) to verify the format of user data (12) (e.g., to comply with acceptable formats for the storage system (140)) and / or route the user data (12) to a data warehouse (142) (e.g., specific data storage locations within the data warehouse (142) designated for the user data (12) of the user (10).
[0029] Referring to FIGS. 2a–2g, TabNet (200) generally includes an encoder (202) and a decoder (204). TabNet (200) receives a set of features (206, 206a–n) from the encoder (202), processes a subset (216) of the features (206a–n) at each processing decision step (S) into a decision step output (208), and processes the decision step outputs (208) (i.e., a plurality of processed feature representations) into a final decision output (208 FIt is configured to aggregate into ). Table data refers to table-format data that generally has multiple features (206a-n), more generally referred to as table entries. Generally speaking, the table format includes a series of columns and multiple row entries for each column. Each row entry for a specific column is typically associated with a column heading. In other words, a column with a heading of "Salary" (e.g., a categorical heading) will have row entries of numeric values representing salaries. Because table entries can be essentially numeric or categorical, similarly, feature (206) may refer to a numeric feature (206) or a categorical feature (206). For example, FIG. 2a illustrates a plurality of features (206, 206a-n) corresponding to a table of user data (12), where the headings “Age”, “Marital Status”, “Degree”, “Job Role”, Department (“Department”), “Current Salary”, “Additional Compensation”, “Previous Salary”, “Additional Leave”, and “Employment Date” are categorical features (206), and some of the row entry data corresponding to “Age”, “Current Salary”, previous year additional compensation (“Previous Year Additional Compensation”), previous salary (“Previous Salary”), “Additional Leave”, and “Employment Date” are numeric features (206). When a feature (206) is a numeric feature (206), TabNet (200) can be configured to use this numeric feature (206) in its raw form. When the feature (206) is a categorical feature (206), the TabNet (200) can be configured to map the categorical features (206) to trainable embeddings. Here, the TabNet (200) typically does not perform global normalization on the feature(s) (206), but rather performs batch normalization (BN) during processing (e.g., in the feature transformer (222) and / or the attentive transformer (212).Each determination step (S) can receive features (206) of the same dimension size.
[0030] Here, the encoder (202) can perform classification or regression. The encoder (202) generally includes a feature selector (210) and a feature processor (220) at each decision step (S). The encoder (202) performs multi-stage processing (e.g., illustrated in FIG. 2c) with N decision steps (S). In a plurality of decision steps (S), the current step (S i The input for ) is the previous step (S i-1 It is processing information from ). For example, in FIG. 2a, the feature selector (210) initially (e.g., previous step (S i-1 The feature processor (220) describes processing information (regarding a subset (216) of features (206) from ), and (e.g., the current stage (S i It generates processing information regarding a subset (216) of features (206) for ). Then, each determination step (S i-n The judgment outputs (208) of ) are the final judgment output (208 F It is used to generate ). For example, FIG. 2a is a final judgment output (208 F Examples of determination outputs (208a-c) from three steps (S) that form ) are provided.
[0031] Feature selection is generally, response variable (final judgment output (208 FIt refers to a process for selecting a reduced number of explanatory variables (e.g., features (206)) to explain )). Feature selection helps prevent overfitting problems because the feature selection process ensures that the variables (e.g., features (206)) that TabNet (200) decides to process are actually related to the desired decision output. By using feature selection, TabNet (200) also gains interpretability due to the fact that the model is generally easier to interpret when it can be seen that the model uses meaningful variables and removes or ignores redundant or meaningless variables. For example, if TabNet (200) is configured to predict year-end bonus rewards, the feature (206) (or variable) corresponding to an employee's marital status is less important or potentially meaningless when compared to features (206) such as current salary and job role.
[0032] In some examples, the feature selector (210) receives a plurality of features (206) and selects prominent features (206) from the plurality of features (206) corresponding to a desired judgment output to form a subset (216) of related features (206R). For example, in the example of FIG. 2a, when the desired judgment output is a prediction of year-end bonus rewards, the feature selector (210) identifies the related features (206R) among the plurality of features (206), job-related features (206) as a first subset (216, 216a) of features (206), and accounting-related features (206) as a second subset (216, 216b) of features (206).
[0033] In some implementations, to perform feature selection, the feature selector (210) includes an attentional transformer (212) and a feature mask (214). For example, the feature mask (214) is a learnable mask for soft selection of prominent features (206). In some examples, the feature mask (214) uses a sparse selection (referred to as a sparse mask) to select the most prominent features (206) (e.g., illustrated as relevant features (206R, 206Ra-g) in FIG. 2a) so that the learning capacity of the decision step (S) is not wasted on irrelevant features (206). This allows the TabNet (200) to be parameter-efficient because each feature (206) is meaningfully selected. For example, the first and second subsets (216a-b) of “job-related” and “accounting-related” can significantly affect the value of an employee’s year-end bonus compensation when compared with other features (206) such as “age”, “marital status”, or “date of employment”.
[0034] In some configurations, the attentive transformer (212) is a preceding stage (S i-1A mask (214) is obtained based on processed features (222) from (e.g., from a feature processor (220)). To obtain the sparse mask (214), the attentive transformer (212) may use sparsemax normalization (e.g., as illustrated in FIG. 2c). Here, sparsemax refers to an activation function similar to a traditional activation function such as softmax, but differs in that sparsemax can output sparse probabilities (i.e., a small number of non-zero probabilities and most terms having a probability of zero). For example, sparsemax generates probabilities for features (206), but instead of generating non-zero probabilities for all input features (206) (like softmax), sparsemax generates non-zero probabilities only for the relevant features (206R) of a subset (216). The remaining or irrelevant features (206) are generally features (206) for which the sparsemax function is unlikely to be set to zero. Therefore, sparsemax normalization encourages Euclidean projection onto stochastic simplexes. Sparsemax normalization allows for efficient feature selection performance while aiming for sparse feature selection for most datasets. For example, the following expression can represent the mask function from the attentive transformer (212): Here is the prior scale term, and is a trainable function, and is the previous step (S i-1 It refers to the processed feature (222) from ). P[i] can be expressed as follows: Here, y is a relaxation parameter that forces the feature (206) to be used in only one decision step (S) when y = 1. Thus, as y increases, there is more flexibility in using the feature (206) in multiple decision steps (S). This allows attention to sparsely masked features to be tunable. In some implementations, as illustrated by Equation (1), the attentive transformer (212) obtains a trainable mask by using a trainable function. For example, the attentive transformer (212) receives the features (206) and processes the features (206) through a single layer (L, L1) mapping that is modulated with previous scale information to understand how much attention was previously focused on specific features (206) in previous decision steps (S). In other words, the previous scale is all previous steps (S i-1 For ) (e.g., based on aggregation), each feature (206) is at the current determination stage (S i ) determines how much was previously used by the encoder (202) as an associated feature (206R) for feature processing. In some examples, such as FIG. 2c, the single layer (L1) structure of the attentive transformer (212) includes a fully connected layer (FC) that performs batch normalization (BN). In some examples, the sparsity of the selected feature (206) for a subset (216) of associated features (206R) can be controlled using sparsity regularization in the form of entropy. For example, the following loss formula can be used to control sparsity: Here is a small number for numerical stability, B is the batch size, and D is the dimension size of the features (206). Then, the sparsity realization can be combined with the total loss of TabNet (200).
[0035] The feature processor (220) is configured to process the filtered features (206) of a subset (216) of features (206) into processed features (222) to form a decision stage output (208). Once the feature processor (220) processes the subset (216) of features (206) into processed features (222), information regarding the decision stage (S), referred to as the decision stage output (208) and decision stage information (226), is supplied to a subsequent decision stage (Si+1). For example, FIGS. 2b–2d illustrate the splitting after the feature transformer (224) of the feature processor (220). As illustrated in FIG. 2d, the splitting is performed at the current stage (S i The feature (222) processed from )(e.g., illustrated as the first step (S1)) is passed to the subsequent step (S i+1 ) is supplied to a feature selector (210) of the second stage (S2), for example. For example, the split supplies the processed features (222) from the first stage (S1) to the attentive transformer (212) of the feature selector (210). The attentive transformer (212) may use these processed features (222) in its previous scale functionality (e.g., illustrated in FIG. 2c) to determine which features (206) should be selected as a subset (216) of the relevant features (206R) in the second stage (S2).
[0036] In some examples, to ensure parameter-efficient and robust learning at high capacity, the feature processor (220) includes a feature transformer (224). The feature transformer (224) comprises a plurality of processing layers (L, L 1-n It may include ). In some implementations, the feature transformer (224) may include all determination steps (S i-N Some layers (L) shared across ) (e.g., the first and second layers (L, L) in FIG. 2c) 1-2As shown in ) as well as some decision step dependent layers (L) (e.g., the third and fourth layers (L, L) in FIG. 2c) 3-4 It consists of (illustrated as ). In the shared layers and decision stage dependent layers (L), the feature transformer (224) can perform concatenation between the layers (L). For example, FIG. 2c shows two shared layers (L, L 1-2 ) exemplifies, wherein each sharing (L) includes BN (batch normalization) and GLU (gated linear unit) nonlinearity after the FC (fully-connected layer). Here, the feature transformer (224) is each sharing layer (L 1-2 Connect ) together, and connect the decision stage dependency layers (L 3-4 ) supplies to. In this example, the feature transformer (224) supplies two decision step dependency layers (L 3-4 It includes ), where, similar to the shared layers (L), each decision-step dependent layer (L) includes batch normalization (BN) and gated linear unit (GLU) nonlinearities after the fully-connected layer (FC). Each decision-step dependent layer (L 3-4 ) can not only be contiguous together but also shared layers (L 1-2 It can be connected to ). Using these shared decision step layers (L) and decision step dependency layers (L), the feature transformer (224) processes a subset (216) of features (206) (e.g., each associated feature (206R) of the subset (216)) into processed features (222).
[0037] In some configurations, to determine the decision embedding (242) for each decision step (S), the feature processor (220) includes a rectified linear unit (RELU) (240). The RELU is a type of activation function that generates a linear relationship between 0 for all negative values and all positive values. Here, the RELU (240) functions well for sparse learning because it is sparsely activated (i.e., activated only on positive values). In these configurations, the RELU (240) receives the decision step output (208) for the processed feature(s) (222) and generates an embedding (e.g., the decision step embedding (242)) to represent the decision step output (208). For example, the decision embedding (242) It can be composed of, where is the total number of decision steps. In some examples, the RELU (240) is combined (e.g., concatenated) with the decision step embedding (242) from subsequent and / or previous decision steps (S) to form the total decision embedding (242) for all steps (S) communicated to the decoder (204) to decode the decision embedding(s) (242) into a recognizable feature (206), and the current step (S) to be combined (e.g., concatenated). i Outputs the determination step embedding (242) from ).
[0038] In some implementations, the RELU (240) also transmits decision step information (226) to the aggregator (250). Here, the aggregator (250) is configured to receive decision step information (226) from each decision step (S). In the decision step information (226), the aggregator (250) may concatenate this information together. In some examples, the aggregator (250) also receives information regarding feature selection (210) (e.g., as a feedback line from the mask (214) shown in FIG. 2c and FIG. 2d). Here, the aggregation of the decision step information (226) and / or feature selection information is such that the TabNet (200) makes a final decision (208 F It can be used for additional interpretability of the functionality of TabNet (200) so as to provide some context regarding ) to the user (10) or manager (150). In other words, TabNet (200) can notify the user (10) or manager (150) that the relevant features (206R) for predicting year-end bonus rewards are job-related (e.g., relevant features (206Ra-c)) and accounting-related (e.g., relevant features (206Rd-g)).
[0039] The decoder (204) of the TabNet (200) is configured to receive an encoded representation of a decision output (208) and to decode the encoded representation into an interpretable decision output (208). For example, the decoder (204) enables the encoding representation to be converted from a computer-readable encoded representation, which is efficient for encoder functions, into a human-readable decoded representation (i.e., reconstructed features) of the decision output (208). Referring to FIGS. 2b, 2c, and 2e, the decoder (204) receives a decision embedding (242) as an encoded representation of a decision output from the encoder (202) and uses a feature transformer (224) to convert the decision embedding (242) into a decision output (208). For example, the decoder (204) includes fully-connected (FC) layers after one or more feature transformers (224) to decode an encoded representation (e.g., a decision embedding (242)). In some examples, the encoder (202) is such that the decoder (204) converts each decision step output (208) into a final decision output (208 F The decision embedding (242) at each step (S) is communicated to the decoder (204) so as to be configured to be combined into ). For example, the decoder (204) has a final decision (208 F Linear mapping can be applied to obtain ). As an example of multiple steps, FIG. 2e shows two judgment step outputs (208) from the first step (S1) and the second step (S2) as a final judgment output (208 F An encoder (202) that connects to ) is illustrated.
[0040] FIG. 2g illustrates a practical example of the functionality of TabNet (200). Here, a user (10) provides user data (12) to TabNet (200) (e.g., through a manager (150)). The user data (12) is in the form of a table having features (206) corresponding to age, capital gains, education, occupation, gender, and relationships. Here, the user (10) uses TabNet (200) to predict whether each individual corresponding to each row of the data will be predicted to have an annual income of more than $50,000. Although not illustrated, TabNet (200) will first be trained on a set of training examples containing actual income information or actual data labels that indicate some aspects of income based on a set of features (206) prior to inference (i.e., real-time application). Once the training process trains the TabNet (200), the TabNet (200) can receive user data (12) illustrated in FIG. 2g having multiple features (206), and a prediction of whether each individual will have an income exceeding 50,000 based on what the TabNet (200) has learned from the user data (12) (e.g., the final judgment output (208) of the decoder (204) F Can generate ))
[0041] Generally speaking, TabNet (200) performs supervised learning for a specific task. However, in some examples, performing unsupervised representation learning prior to supervised learning improves the functionality of TabNet (200) for supervised learning a specific task. For example, TabNet (200) can be used for the task of predicting missing features (206, 206M) in tabular data. This can be advantageous because actual tabular datasets often have interdependent feature columns that TabNet (200) can learn to understand in order to predict missing features (206M) based on other related features (206) in the same column or related interdependent columns. For example, FIG. 2g illustrates user data (12) as tabular entries having a number of missing features (206, 206M, 206Ma-n). Here, for missing entries regarding education level, TabNet (200) can predict the missing feature (206M) using related features (206), such as occupation. Similarly, the missing feature (206M) regarding gender can be predicted from other gender entries by combining related entries. To perform unsupervised representation learning, TabNet (200) uses its own supervised goal with a pretrainer (260). For example, In the case of a binary mask, the encoder (202) of TabNet (200) is Inputs, and the decoder (204) reconstructs the features It outputs. Here, the encoder (202) has the previous scale term By initializing it as such, TabNet (200) can highlight known features (206), and S can be multiplied in the decoder's fully-connected layers to simply output unknown or missing features (206M). In unsupervised representation learning, normalization with a population standard deviation of the actual data can be important for potentially cutting off very different values for the features (206).
[0042] FIG. 3 is a flowchart of an exemplary operation arrangement for a method (300) for interpreting tabular data. In operation (302), the method (300) receives a set of features (206, 206a-n) from a deep tabular data learning network (TabNet) (200) running on data processing hardware (132). In operation (304), the method (300) performs a plurality of sequential processing steps (S, S i-n Sub-operations (304a-c) are performed for each of the following. In operation (304a), the method (300) uses the sparse mask (214) of the TabNet (200) to select a subset (216, 216a-n) of relevant features from a set of features (206, 206a-n). In operations (304b), the method (300) uses the feature transformer (224) of the TabNet (200) to perform a plurality of sequential processing steps (S, S i-n ) at the determination step output (208) and the next processing step (S, S i+1 A subset (216, 216a-n) of related features is processed to generate information (226) about ). In operation (304c), the method (300) processes the information (226) in the next processing step (S, S i+1 Provides to ). In operation (306), the method (300) comprises a plurality of sequential processing steps (S, S i-n A final judgment output (208, 208a-n) is obtained by aggregating the judgment step outputs (208, 208a-n) generated for ) FDetermines ).
[0043] FIG. 4 is a schematic diagram of an exemplary computing device (400) that may be used to implement the systems (e.g., Manager (150) and / or TabNet (200)) and methods (e.g., Method (300)) described in this document. The computing device (400) is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components, their connections and relationships, and their functions depicted herein are merely exemplary and are not intended to limit the implementations of the inventions described and / or claimed herein.
[0044] The computing device (400) includes a processor (410), memory (420), a storage device (430), a high-speed interface / controller (440) connected to the memory (420) and high-speed expansion ports (450), and a low-speed interface / controller (460) connected to the low-speed bus (470) and the storage device (430). Each of the components (410, 420, 430, 440, 450, and 460) is interconnected using various buses and may be mounted on a common motherboard or suitably in other ways. The processor (410) may process instructions for execution within the computing device (400), including instructions stored in the memory (420) or on the storage device (430), to display graphic information for a graphical user interface (GUI) on an external input / output device such as a display (480) coupled to the high-speed interface (440). In other implementations, multiple processors and / or multiple buses may be appropriately used with multiple memories and memory types. Additionally, multiple computing devices (400) may be connected, and each device provides parts of the necessary operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).
[0045] Memory (420) stores information non-transiently within the computing device (400). Memory (420) may be a computer-readable medium, volatile memory unit(s), or non-volatile memory unit(s). Non-transient memory (420) may be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) temporarily or permanently for use by the computing device (400). Examples of non-volatile memory include, but are not limited to, flash memory and ROM (read-only memory) / PROM (programmable read-only memory) / EPROM (erasable programmable read-only memory) / EEPROM (electronically erasable programmable read-only memory) (e.g., commonly used in firmware such as boot programs). Examples of volatile memory include, but are not limited to, RAM (random access memory), DRAM (dynamic random access memory), SRAM (static random access memory), and PCM (phase change memory), as well as disks or tapes.
[0046] The storage device (430) may provide mass storage for the computing device (400). In some implementations, the storage device (430) is a computer-readable medium. In various different implementations, the storage device (430) may be an array of devices including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or devices in a storage area network or other configurations. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that perform one or more methods such as those described above when executed. The information carrier is a computer or machine-readable medium, such as memory (420), the storage device (430), or memory on the processor (410).
[0047] The high-speed controller (440) manages bandwidth-intensive operations for the computing device (400), while the low-speed controller (460) manages lower bandwidth-intensive operations. Such job assignments are merely exemplary. In some implementations, the high-speed controller (440) is coupled to memory (420), to a display (480) (e.g., via a graphics processor or accelerator), and to high-speed expansion ports (450), which can accommodate various expansion cards (not shown). In some implementations, the low-speed controller (460) is coupled to a storage device (430) and a low-speed expansion port (490). A low-speed expansion port (490) that may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device, such as a switch or a router, such as through a network adapter.
[0048] The computing device (400) may be implemented in a number of different forms, as illustrated in the drawing. For example, it may be implemented as a standard server (400a), or as a laptop computer (400b) in a group of such servers (400a), or as part of a rack server system (400c).
[0049] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical networks, integrated networks, specially designed application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include one or more computer programs executable and / or interpretable on a programmable system comprising a storage system, at least one input device and at least one output device, and at least one programmable processor which may be coupled to receive data and instructions from the storage system and at least one input device and at least one output device and transmit data and instructions to them.
[0050] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in high-level procedures and / or object-oriented programming languages and / or in assembly language / machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transient computer-readable medium, device and / or device (e.g., magnetic disks, optical disks, memory, PLD (Programmable Logic Device)) used to provide machine instructions and / or data to a programmable processor comprising a machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.
[0051] The processes and logic flows described herein may be executed by one or more programmable processors that execute one or more computer programs to perform functions by operating on input data to generate outputs. The processes and logic flows may also be executed by special-purpose logic networks, such as field programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose microprocessors and any one or more processors of any type of digital computer. Generally, the processor will receive instructions and data from read-only memory or random access memory, or both. Essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, the computer will also include one or more mass storage devices for storing data, such as magnetic, magneto-optical disks or optical disks, or will be operatively coupled to receive data from them, transfer data to them, or perform both. However, the computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and all forms of non-volatile memory, media, and memory devices, including CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or included in a special-purpose logic network.
[0052] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device for displaying information to a user, such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor or a touchscreen, and optionally, a keyboard and pointing device, such as a mouse or a trackball, that enables the user to provide input to the computer. Other types of devices may also be used to provide interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form including acoustic, voice, or tactile input. Additionally, the computer may interact with the user by transmitting documents to a device used by the user and receiving documents from a device used by the user, for example, by transmitting web pages to a web browser on the user's client device in response to requests received from a web browser.
[0053] Many embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other embodiments are within the scope of the following claims.
Claims
Claim 1 As a method, receiving a set of features (206, 206a-n) from a deep tabular data learning network (TabNet) (200) running on data processing hardware (132); a plurality of sequential processing steps (S, S i-n For each of the following: the data processing hardware (132) uses the sparse mask (214) of TabNet (200) to produce a final decision output (208, 208) from the set of features (206, 206a-n). F A step of selecting a subset of features (216, 216a-n) related to ); wherein the data processing hardware (132) uses the feature transformer (224) of the TabNet (200) to select the final decision output (208, 208 F By processing a subset (216, 216a-n) of features related to ), the plurality of sequential processing steps (S, S i-n A determination step output (208) for prediction based on a set of the above features (206, 206a-n) and a next processing step (S, S i+1 A step of generating information (226) about ); and the data processing hardware (132) passes the information (226) to the next processing step (S, S i+1 A step of providing to ); and the data processing hardware (132) wherein the plurality of sequential processing steps (S, S i-n By aggregating the decision step outputs (208, 208a-n) generated for ), the final decision output (208, 208) as a final prediction based on the set of features (206, 206a-n) F The method includes a step of determining ), and the plurality of processing steps (S, S i-n For each of the above, the data processing hardware (132) uses the attentive transformer (212) of the TabNet (200) to perform the plurality of sequential processing steps (S, S i-n Each of the preceding processing steps (S, S) of ) i-1 A method further comprising the step of determining an aggregate of the number of times each feature (206) of the set of features (206, 206a-n) is processed. Claim 2 delete Claim 3 In claim 1, the attentive transformer (212) comprises FC (fully-connected layer) and BN (batch normalization), method (300). Claim 4 As a method, receiving a set of features (206, 206a-n) from a deep tabular data learning network (TabNet) (200) running on data processing hardware (132); a plurality of sequential processing steps (S, S i-n For each of the following: the data processing hardware (132) uses the sparse mask (214) of TabNet (200) to produce a final decision output (208, 208) from the set of features (206, 206a-n). F A step of selecting a subset of features (216, 216a-n) related to ); wherein the data processing hardware (132) uses the feature transformer (224) of the TabNet (200) to select the final decision output (208, 208 F By processing a subset (216, 216a-n) of features related to ), the plurality of sequential processing steps (S, S i-n A determination step output (208) for prediction based on a set of the above features (206, 206a-n) and a next processing step (S, S i+1 A step of generating information (226) about ); and the data processing hardware (132) passes the information (226) to the next processing step (S, S i+1 A step of providing to ); and the data processing hardware (132) wherein the plurality of sequential processing steps (S, S i-n By aggregating the decision step outputs (208, 208a-n) generated for ), the final decision output (208, 208) as a final prediction based on the set of features (206, 206a-n) F It includes a step of determining ), and the information (226) is used in the next processing step (S, S i+1 The step of providing to ) is based on the provided information (226) and the plurality of sequential processing steps (S, S i-n Each of the preceding processing steps (S, S) of ) i-1 A method (300) comprising the step of providing the information (226) to an attentive transformer (212) of the TabNet (200) that determines an aggregation of the number of times each feature (206) of the set of features (206, 206a-n) is processed. Claim 5 In claim 1, the set of features (206) comprises tabular data (12), method (300). Claim 6 In claim 1, the feature transformer (224) of the TabNet (200) comprises a plurality of neural network layers (L, L) each including FC (fully-connected layer), BN (batch normalization), and GLU (generalized linear unit) nonlinearity. a-n A method (300) including ). Claim 7 In claim 6, the plurality of neural network layers (L, L 1-n The first part of ) is the plurality of sequential processing steps (S, S i-n ) shared across each of the plurality of neural network layers (L, L 1-n The remaining second part of ) is the plurality of sequential processing steps (S, S i-n Each processing step (S, S) of ) i A method (300) specialized for ). Claim 8 A method (300) wherein, in any one of claims 1 to 7, the decision step output (208) generated by processing a subset (216, 216a-n) of the relevant features passes through the ReLU (rectified linear unit) (240) of the TabNet (200). Claim 9 In any one of claims 1 to 7, the method (300) wherein the subset of features (216, 216a-n) corresponds to interdependent feature columns of table data (12). Claim 10 A method (300) comprising, in any one of claims 1 to 7, receiving table data (12) containing one or more missing features (206, 206M) in the data processing hardware (132); and the data processing hardware (132) predicting the one or more missing features (206, 206M) using unsupervised pre-training by TabNet (200). Claim 11 The apparatus comprises data processing hardware (132); and memory hardware (134) that communicates with the data processing hardware (132) and stores instructions that cause the data processing hardware (132) to perform operations when executed on the data processing hardware (132), wherein the operations include: receiving a set of features (206, 206a-n) in a deep tabular data learning network (TabNet) (200) executed on the data processing hardware (132); and a plurality of sequential processing steps (S, S i-n For each: using the sparse mask (214) of TabNet (200), the final decision output (208, 208) from the set of features (206, 206a-n) F Selecting a subset of features (216, 216a-n) related to ); using the feature transformer (224) of the TabNet (200), the final decision output (208, 208 F A subset of features (216, 216a-n) related to ) is processed to produce a determination step output (208) for prediction based on a set of features (206, 206a-n) and a next processing step (S, S i+1 Generating information (226) regarding ); and the information (226) to the next processing step (S, S i+1 Providing to ); and the plurality of sequential processing steps (S, S i-n By aggregating the judgment step outputs (208, 208a-n) generated for ), the final judgment output (208, 208) as a final prediction based on the set of features (206, 206a-n) F ) including determining, and the operations include the plurality of processing steps (S, S i-n For each of the above, the plurality of sequential processing steps (S, S) are performed using the attentive transformer (212) of the TabNet (200). i-n Each of the preceding processing steps (S, S) of ) i-1 A system (100) further comprising determining an aggregation of the number of times each feature (206) of the set of features (206, 206a-n) is processed. Claim 12 delete Claim 13 In claim 11, the attentive transformer (212) comprises a fully-connected layer (FC) and a batch normalization (BN), in a system (100). Claim 14 The apparatus comprises data processing hardware (132); and memory hardware (134) that communicates with the data processing hardware (132) and stores instructions that cause the data processing hardware (132) to perform operations when executed on the data processing hardware (132), wherein the operations include: receiving a set of features (206, 206a-n) in a deep tabular data learning network (TabNet) (200) executed on the data processing hardware (132); and a plurality of sequential processing steps (S, S i-n For each: using the sparse mask (214) of TabNet (200), the final decision output (208, 208) from the set of features (206, 206a-n) F Selecting a subset of features (216, 216a-n) related to ); using the feature transformer (224) of the TabNet (200), the final decision output (208, 208 F A subset of features (216, 216a-n) related to ) is processed to produce a determination step output (208) for prediction based on a set of features (206, 206a-n) and a next processing step (S, S i+1 Generating information (226) regarding ); and the information (226) to the next processing step (S, S i+1 Providing to ); and the plurality of sequential processing steps (S, S i-n By aggregating the judgment step outputs (208, 208a-n) generated for ), the final judgment output (208, 208) as a final prediction based on the set of features (206, 206a-n) F It includes determining ) and the information (226) is used in the next processing step (S, S i+1 Providing to ) is based on the provided information (226) to the plurality of sequential processing steps (S, S i-n Each of the preceding processing steps (S, S) of ) i-1 A system (100) comprising providing the information (226) to an attentive transformer (212) of the TabNet (200) that determines an aggregation of the number of times each feature (206) of the set of features (206, 206a-n) is processed. Claim 15 In claim 11, the set of features (206) comprises table data (12), in a system (100). Claim 16 In claim 11, the feature transformer (224) of the TabNet (200) comprises a plurality of neural network layers (L, L) each including FC (fully-connected layer), BN (batch normalization), and GLU (generalized linear unit) nonlinearity. a-n A system (100) including ). Claim 17 In claim 16, the plurality of neural network layers (L, L 1-n The first part of ) is the plurality of sequential processing steps (S, S i-n ) shared across each of the plurality of neural network layers (L, L 1-n The remaining second part of ) is the plurality of sequential processing steps (S, S i-n Each processing step (S, S) of ) i A system (100) specialized for ). Claim 18 A system (100) wherein, in any one of claims 11 to 17, the decision step output (208) generated by processing a subset (216, 216a-n) of the relevant features passes through the ReLU (rectified linear unit) (240) of the TabNet (200). Claim 19 In any one of claims 11 to 17, the system (100) wherein the subset of features (216, 216a-n) corresponds to interdependent feature columns of table data (12). Claim 20 A system (100) according to any one of claims 11 to 17, wherein the operations further comprise receiving table data (12) containing one or more missing features (206, 206M); and predicting said one or more missing features (206, 206M) using unsupervised pre-training by TabNet (200).
