Explainable tabular data learning using sequence-sparse attention

Abstract

A method (300) of interpreting tabular data includes receiving a set of features (206) at a deep tabular data learning network (TabNet) (200). For each of a plurality of sequential processing steps (S, S i‑N ), the method further includes using a mask (214) of the TabNet to select a subset (216) of relevant features (206, 206R) of the set of features; processing the subset of relevant features using a feature transformer (224) of the TabNet to generate a decision step output (208) and information (226) for a next processing step (S, S i+1 ) of the plurality of sequential processing steps; and providing the information to the next processing step. The method further includes determining a final decision output (208, 208 F ) by aggregating the decision step outputs generated for the plurality of sequential processing steps.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to interpretable tabular data learning using sequential sparse attention. BACKGROUND

[0002] Machine learning models receive inputs and produce outputs, such as predicted outputs, based on the received inputs. Some machine learning models are parametric models and produce outputs based on the received inputs and values of the model parameters.

[0003] Some machine learning models are deep models that employ multiple layers of models to produce outputs for received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to received inputs to produce outputs. SUMMARY

[0004] One aspect of the present disclosure provides a method of interpreting tabular data. The method includes receiving, at a deep tabular data learning network (TabNet) executing on data processing hardware, a set of features. For each of a plurality of sequential processing steps, the method further includes: selecting, by the data processing hardware, a subset of relevant features of the set of features using a mask of the TabNet; processing, by the data processing hardware, the subset of relevant features using a feature transformer of the TabNet to generate a decision step output and information for a next processing step of the plurality of sequential processing steps; and providing, by the data processing hardware, the information to the next processing step; and the method further includes determining, by the data processing hardware, a final decision output by aggregating the decision step outputs generated for the plurality of sequential processing steps.

[0005] Implementations of the present disclosure can include one or more of the following optional features. In some implementations, the method further includes, for each of the plurality of processing steps, determining, by the data processing hardware, a total number of times each feature in the set of features has been processed in each preceding processing step of the plurality of sequential processing steps using an attention transformer of the TabNet. In these implementations, the attention transformer can include a fully connected layer and a batch normalization.

[0006] In some examples, providing the information to the next processing step includes providing the information to an attention transformer of the TabNet, the attention transformer determining, based on the provided information, a total number of times each feature in the set of features has been processed in each preceding processing step of the plurality of sequential processing steps. The set of features can include tabular data. Optionally, the feature transformer of the TabNet includes a plurality of neural network layers, each of the plurality of neural network layers including a fully connected layer, batch normalization, and a generalized linear unit (GLU) nonlinearity. Here, a first portion of the plurality of neural network layers can be shared across each of the plurality of sequential processing steps, and a remaining second portion of the plurality of neural network layers can be dependent on a respective processing step of the plurality of sequential processing steps.

[0007] The decision step output generated by processing the subset of relevant features can be passed through a rectified linear unit (ReLU) of the TabNet. Additionally, the subset of features can correspond to interdependent feature columns of the tabular data. In some implementations, the method further includes: receiving, at the data processing hardware, the tabular data including one or more masked features; and predicting, by the data processing hardware, the one or more masked features using unsupervised pre-training.

[0008] Another aspect of the present disclosure provides a system for interpreting tabular data. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including receiving, at a deep tabular data learning network (TabNet) executing on the data processing hardware, a set of features. For each of a plurality of sequential processing steps, the operations further include: selecting, using a mask of the TabNet, a subset of relevant features of the set of features; processing, using a feature transformer of the TabNet, the subset of relevant features to generate a decision step output and information for a next processing step of the plurality of sequential processing steps; and providing the information to the next processing step; the operations further including determining a final decision output by aggregating the decision step outputs generated for the plurality of sequential processing steps.

[0009] This aspect can include one or more of the following optional features. In some implementations, the operations further include, for each of the plurality of processing steps, determining, using an attention transformer of the TabNet, a total number of times each feature in the set of features has been processed in each preceding processing step of the plurality of sequential processing steps. In these implementations, the attention transformer can include a fully connected layer and batch normalization.

[0010] In some examples, providing information to a next processing step includes providing information to an attention transformer of the TabNet, the attention transformer determining, based on the provided information, a total number of times each of a set of features has been processed in each preceding processing step of a plurality of sequential processing steps. The set of features can include tabular data. Optionally, a feature transformer of the TabNet includes a plurality of neural network layers, each of the plurality of neural network layers including a fully connected layer, batch normalization, and a generalized linear unit (GLU) nonlinearity. Here, a first portion of the plurality of neural network layers can be shared across each of the plurality of sequential processing steps, and a remaining second portion of the plurality of neural network layers can be dependent on a respective processing step of the plurality of sequential processing steps.

[0011] The decision step output produced by processing the subset of related features can be passed through a rectified linear unit (ReLU) of the TabNet. Additionally, the subset of features can correspond to interdependent feature columns of the tabular data. In some implementations, the operations further include receiving the tabular data including one or more masked features and predicting the one or more masked features using unsupervised pre-training.

[0012] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a schematic diagram of an example data management environment.

[0014] Figures 2A-2G is a schematic diagram of an example model for interpreting tabular data.

[0015] Figure 3 is a flowchart of an example arrangement of operations of a method for interpreting tabular data.

[0016] Figure 4 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0017] The same reference numbers in the various drawings indicate the same elements. DETAILED DESCRIPTION

[0018] Neural networks have traditionally been successful at processing data types such as images, text, and audio. There is one type of data, however, that despite being one of the most common types of data in computing, seems to be missing, tabular data. Tabular data refers to data in a table format. Often, tabular data tends to use decision trees over neural networks due to its interpretability, high representation efficiency, and ability to train quickly. For example, the interpretability of decision trees allows someone to understand how a decision tree arrived at a particular result, rather than simply receiving a result without context. Decision trees are interpretable because decisions can be traced via the nodes of the decision tree. In the realm of tabular data, there can be a trust issue embodied in a concern about the decisions that lead to a particular result. For example, how does a decision tree predict that an employee’s bonus pay should be a certain number, or how does a loan processing tool predict that an applicant is eligible for a loan of a particular size? In these situations, someone can want to check the context of that decision.

[0019] Traditionally, neural networks are based on stacked layers that can over-parameterize the neural network. Due to over-parameterization, traditional neural networks tend to not find the optimal solution for tabular decision manifolds. However, if a neural network is adapted sufficiently for tabular data, the neural network can provide similar benefits as people would expect based on its performance for other data types. For example, neural networks are particularly useful for large datasets and are able to apply backpropagation to their inputs to guide effective learning towards a target. With a neural network designed for tabular data, the neural network can also effectively encode multiple data types (e.g., images) in conjunction with tabular data, minimize the need for feature engineering, which is a key aspect of current tree-based tabular data learning methods, enable learning from streaming data (streaming data is currently proven to be difficult for decision tree models (e.g., decision trees struggle to dynamically adapt split points)), and allow representation learning for domain adaptation, generative models, and semi-supervised learning.

[0020] To address some of the pre-existing deficiencies of neural networks for tabular data, while combining the general benefits of neural networks, TabNet is a neural network designed to learn in a decision tree-like fashion. In other words, TabNet aims to provide explainability and sparse feature selection. TabNet takes raw tabular data with no feature pre-processing as input and uses gradient descent-based optimization to train to learn flexible representations and enable flexible integration into end-to-end learning. Additionally, TabNet employs sequential attention to select features for explanation at each decision step; allowing it to be explainable and focus on significant features. In a single deep learning architecture with end-to-end learning, feature selection can be different for each input (i.e., instant-wise). With this approach, TabNet has been shown to outperform or be on par with other tabular learning models (e.g., decision tree models) on various datasets for classification and regression problems. As for explainability, TabNet provides two types of explainability. It allows for local explainability to visualize the importance of input features and how they combine, while also having global explainability to quantify the contribution of each input feature with respect to the trained model. Furthermore, for tabular data, TabNet is able to use techniques such as unsupervised pre-training to predict missing features.

[0021] Feature selection generally refers to the process of selecting a subset of features from a larger pool of features based on how useful each feature is for a given prediction. When features are selected based on the entire training dataset, this type of feature selection is referred to as a global feature selection method. In contrast, instance-wise feature selection methods involve selecting features individually for each input. While some methods train a model to maximize mutual information between selected features in a given response variable, TabNet uses soft feature selection with controllable sparsity in end-to-end learning, which allows a single model to jointly perform feature selection and output mapping. Sequential attention is then used to embed this soft feature selection capability with controllable sparsity. Furthermore, unlike some models that attempt to map tabular data to different data types, TabNet is able to apply sequential attention to supervised or self-supervised learning of tabular data. TabNet is designed to have a tree-like functionality, where a linear combination of features is selected with a feature selection process to represent a decision boundary. TabNet uses a training dataset to learn sparse, instance-wise selection, building a sequential, multi-step architecture where each decision step can contribute to a part of the decision based on selected features, perform nonlinear processing of selected features, and mimic an ensemble via higher dimensionality and more steps. With this approach, TabNet combines the benefits of neural networks with those of tree-based decision models for tabular data.

[0022] Figure 1 An example of a data management environment 100 is shown. A user device 110 associated with a user 10 generates user data 12 during execution of its computing resources 112 (e.g., data processing hardware 114 and / or memory hardware 116). For example, the user 10 generates user data 12 using one or more applications (e.g., real-time applications) operating on the data processing hardware 114 of the user device 110. In some examples, the user device 110 is a local device (e.g., associated with a location of the user 10) that uses its own computing resources 112 with the ability to communicate (e.g., via a network 120) with one or more remote systems 130. Additionally or alternatively, the user device 110 utilizes its access to remote resources (e.g., remote computing resources 132) to operate the applications of the user 10. The user data 12 generated by use of the user device 110 can be initially stored locally (e.g., in a data store 118 of the memory hardware 116) and then transferred to the remote system 130 or sent to the remote system 130 over the network 120 at creation. For example, the user device 110 uses the remote system 130 to transfer the user data to a storage system 140.

[0023] In some examples, the user 10 utilizes the computing resources 132 of the remote system 130 (e.g., a cloud computing environment) to store and / or manage the user data 12. In these examples, the remote system 130 can receive the user data 12 as it is being generated by various user applications. Here, the user data 12 can refer to a data stream (e.g., a continuous or generally continuous feed of data arriving at the remote system 130 for storage and / or further processing) or discrete sets of user data 12 sent to the remote system 130 at the coordination of the user 10. Much like the user device 110, the remote system 130 includes computing resources 132 such as remote data processing hardware 134 (e.g., servers and / or CPUs) and memory hardware 136 (e.g., disks, databases, or other forms of data storage).

[0024] In some configurations, the remote computing resources 132 are resources used by various systems associated with and / or in communication with the remote system 130. Although Figure 1 The remote system 130 is described as being in communication with the data storage system 140 and the manager 150, but the remote system 130 can also be in communication with other data-related systems (e.g., query systems and / or analytics systems). The remote system 130 with its computing resources 132 can be configured to host one or more functions of the manager 150 and / or the data processing model 200. In some implementations, the remote system 130 is a distributed system with its computing resources 132 distributed across one or more locations accessible via the network 120.

[0025] In some examples, storage system 140 is configured to operate data warehouse 142 (e.g., data storage and / or multiple databases) as a data storage device for user 10 (or multiple users). Generally, data warehouse 142 stores data from one or more sources and can be designed to analyze, report on, and / or integrate data from its sources. Data warehouse 142 enables users (e.g., organizational users) to have a central storage repository and storage data access points. By including user data 12 in a central repository such as data warehouse 142, data warehouse 142 can streamline data retrieval for functions such as data analysis and / or data reporting (e.g., through an analytics system). Furthermore, data warehouse 142 can be configured to store large amounts of data, allowing user 10 (e.g., organizational users) to store large amounts of historical data to understand data trends. Because data warehouse 142 can be a primary or sole data storage repository for user data 12, storage system 140 can typically receive large amounts of data from user devices 110 associated with user 10. Additionally or alternatively, as storage system 140, storage system 140 and / or storage warehouse 142 may be configured for data security (e.g., data redundancy), for multiple users from a single data source (e.g., multiple employees of an organization), and / or for simultaneous multi-user access. In some configurations, data warehouse 142 is persistent and / or non-volatile, such that by default, data is not overwritten or erased by newly arriving data.

[0026] Generally, data storage system 140 receives user data 12 in a table format called tabular data, where user data 12 fills the rows and columns of the table. Using tabular data, the user data 12 in the table can have rows and columns corresponding to schemas or headers associated with the user data 12. For example, user data 12 may refer to a business transaction performed by user 10. In this example, user data 12 may include columns for seller, buyer, transaction price, transaction quantity, and other transaction data collected by user 10 regarding their transaction. Here, each row may have headers or schemas, such as transaction number or identifier and / or time entries associated with the transaction. Because storage system 140 can receive user data 12 in a specific format (e.g., transaction table format), storage system 140 is configured to store user data 12 such that elements of the format associated with user data 12 (e.g., relationships, headers, or other schemas) (e.g., providing further context or definition to user data 12) can be accessed by other systems communicating with data storage system 140 (e.g., data retrieval systems such as query systems or data analysis systems).

[0027] Further reference Figure 1The data management environment 100 also includes a manager 150. The manager 150 is generally configured to coordinate (e.g., optimize) data storage. The manager 150 can perform this optimization by using a data processing model 200 (also referred to as TabNet 200) to perform and / or coordinate operations related to systems in communication with the data storage system 140. Here, in performing data storage, the TabNet 200 is configured as a machine learning model that can learn aspects of tabular data received at the manager 150. Based on this learning, the TabNet 200 is able to assist in various management functions. For example, the TabNet 200 is configured to identify missing or incomplete data (referred to as missing features) in the user data 12 and predict or generate (or suggest) data entries for the missing data. Similarly, instead of predicting entries for incomplete data, the user 10 can request the TabNet 200 (through the manager 150) to predict what the entries should be based on other data stored for the user 10. For example, the user 10 is a company that wants to know how much it should pay a new employee based on the new employee’s previous experience and skills. Here, the TabNet 200 would interpret the employee financial user data 12 stored in the storage system 140, which includes, for example, salary, position, department, years of experience, etc., and based on the user data 12 it interprets, output a prediction of the new employee’s salary. In some examples, the manager 150 uses the TabNet 200 to interpret interdependencies of the user data 12 in order to understand how the user data 12 can be stored while still maintaining relationships between columns of the user data 12. In other words, the TabNet 200 can identify split points that dictate where to partition the user data 12 for storage purposes.

[0028] In some implementations, for example Figure 1 The manager 150 is configured to receive the user data 12 from the user 10 and facilitate storage operations at the storage system 140. For example, the manager 150 ingests the user data 12 upon receipt and can convert the user data 12 into a storage-optimized format. Here, ingestion refers to obtaining the user data 12 and / or importing the user data 12 into the storage system 140 (e.g., into the data warehouse 142) to allow the system to use the ingested user data (e.g., through querying the system and / or analyzing the system). During ingestion, the manager 150 can use the TabNet 200 to validate the format of the user data 12 (e.g., to conform to an acceptable format of the storage system 140) and / or route the user data 12 to the data warehouse 142 (e.g., to a particular data storage location in the data warehouse 142 designated for the user data 12 of the user 10).

[0029] Referring to Figures 2A-2G, TabNet 200 generally includes an encoder 202 and a decoder 204. TabNet 200 is configured to receive a set of features 206, 206a-n at the encoder 202, process a subset 216 of the features 206a-n at each processing decision step S into a decision step output 208, and aggregate the decision step outputs 208 (i.e., a plurality of processed feature representations) into a final decision output 208 F . Tabular data refers to data in a tabular format generally having a plurality of features 206a-n, more commonly referred to as table entries. Generally, a tabular format includes a series of columns and a plurality of row entries for each column. Each row entry for a particular column generally relates to a table header. In other words, a column having a table header (e.g., a categorical table header) “Salary” would have row entries representing numerical values of salary. Because table entries can be numerical or categorical in nature, features 206 can likewise refer to numerical features 206 or categorical features 206. For example, Figure 2A depicts a plurality of features 206, 206a-n corresponding to a table of user data 12, where the table headers are “Age,” “Marital Status,” “Education,” “Job Title,” “Department” (“Dept.”), “Current Salary,” “Additional Compensation” (“Add’l Comp”), “Previous Salary” (“Prev. Salary”), “Vacation Accrual,” and “Hire Date” are categorical features 206, and some of the row entry data corresponding to “Age,” “Current Salary,” “Previous Year Additional Compensation” (“Prev Add’l Comp”), “Previous Salary” (“Prev. Salary”), “Vacation Accrual,” and “Hire Date” are numerical features 206. When features 206 are numerical features 206, TabNet 200 can be configured to use the numerical features 206 in their raw form. When features 206 are categorical features 206, TabNet 200 can be configured to map the categorical features 206 to trainable embeddings. Here, TabNet 200 generally does not perform global normalization on features 206, but rather performs batch normalization (BN) during processing (e.g., at feature transformers 222 and / or at attention transformers 212). Each decision step S can receive features 206 having the same size.

[0030] Here, the encoder 202 is capable of performing classification or regression. The encoder 202 generally includes a feature selector 210 and a feature processor 220 at each decision step S. The encoder 202 performs a multi-step process having N decision steps S (e.g., as shown in Figure 2C . For the plurality of decision steps S, the input to a current step S i is processing information from a previous step S i-1 . For example, inFigure 2A In particular, feature selector 210 initially considers processed information (e.g., about a subset 216 of features 206 from a previous step S i-1 and feature processor 220 is generating processed information (e.g., about a subset 216 of features 206 for a current step S i The decision outputs 208 for each decision step S i-n are then used to generate a final decision output 208 F For example, Figure 2A The decision outputs 208a-c from the three steps S that form the final decision output 208 F are shown.

[0031] Feature selection generally refers to the process of selecting a reduced number of explanatory variables (e.g., features 206) to describe a response variable (final decision output 208 F Feature selection helps prevent the problem of overfitting because the process of feature selection ensures that the variables (e.g., features 206) that TabNet 200 decides to process are actually relevant to the desired decision output. By using feature selection, TabNet 200 also gains interpretability by the fact that models are generally easier to interpret when one can see that the model is using meaningful variables and removing or ignoring redundant or meaningless variables. For example, if TabNet 200 is configured to predict end-of-year bonus pay, then a feature 206 (or variable) corresponding to the marital status of an employee is less important or potentially meaningless than features 206 such as current salary and job title.

[0032] In some examples, feature selector 210 receives a plurality of features 206 and selects significant features 206 from the plurality of features 206 corresponding to the desired decision output to form a subset 216 of relevant features 206R. For example, in the example of Figure 2A if the desired decision output is a prediction of end-of-year bonus pay, then from the plurality of features 206, feature selector 210 identifies relevant features 206R as job-related features 206 as a first subset 216, 216a of features 206 and accounting-related features 206 as a second subset 216, 216b of features 206.

[0033] In some implementations, to perform feature selection, feature selector 210 includes an attention transformer 212 and a feature mask 214. For example, feature mask 214 is a learnable mask for soft selection of significant features 206. In some examples, feature mask 214 uses sparse selection (referred to as a sparse mask) to select the most significant features 206 (e.g., in Figure 2AThe relevant features 206R, 206Ra-g) are shown as such, such that the decision step S is not wasted on irrelevant features 206. This allows the TabNet 200 to become parameter efficient, as each feature 206 is meaningfully selected. For example, the first and second subsets 216a-b of“profession related” and“accounting related” can meaningfully influence the value of the employee year-end bonus compensation as compared to other features 206 such as“age,”“marital status,” or“date of hire.”

[0034] In some configurations, the attention transformer 212 obtains the mask 214 based on the processed features 222 from the previous step S i-1 (e.g., from the feature processor 220). To obtain the mask 214, the attention transformer 212 can use sparsemax normalization (e.g., as shown in Figure 2C Here, sparsemax refers to an activation function similar to traditional activation functions such as softmax, but differs in that sparsemax is capable of outputting sparse probabilities (i.e., a small number of non-zero probabilities and a majority of terms with zero probability). For example, sparsemax generates probabilities for the features 206, but rather than generating non-zero probabilities for all input features 206 (like softmax), sparsemax generates non-zero probabilities only for the relevant features 206R of the subset 216. The remaining or irrelevant features 206 are features 206 that typically have small probabilities with the sparsemax function set to zero. Thus, sparsemax normalization encourages the Euclidean projection onto the probability simplex. Sparsemax normalization allows for efficient feature selection performance, while having the goal of sparse feature selection for most datasets. For example, the following expression can represent the mask function from the attention transformer 212:

[0035] M[i] = sparsemax(P[i - 1] x h i (a[i - 1])) (1)

[0036] where P[i - 1] is a prior scale term, h i is a trainable function, and a[i - 1] refers to the processed features 222 from the previous step S i-1 P[i] can be represented as follows:

[0037]

[0038] Where y is a relaxation parameter, such that when y = 1, feature 206 is forced to be used only in one decision step S. Therefore, as y increases, there is greater flexibility in using feature 206 in multiple decision steps S. This allows attention to sparsely masked features to be tunable. In some implementations, as shown in equation (1), attention transformer 212 obtains a trainable mask by using a trainable function. For example, attention transformer 212 receives feature 206 and processes it through a single-layer L,L1 mapping modulated with previous scale information to understand the degree to which a particular feature 206 was previously attended to in previous decision steps S. In other words, the previous scale is for all previous steps S. i-1 Determine (e.g., based on aggregation) each feature 206 as the basis for decision step S in the current decision step. i The extent to which the relevant features 206R from previous feature processing have been used by encoder 202. In some examples, such as Figure 2C As shown, note that the single-layer L1 structure of transformer 212 includes a fully connected layer (FC) that performs batch normalization (BN). In some examples, the sparsity of features 206 selected for a subset 216 of relevant features 206R can be controlled using sparsity regularization in the form of entropy. For example, sparsity can be controlled using the following loss equation:

[0039]

[0040] Where ε is the numerical stability value, B is the batch size, and D is the size of feature 206. The sparsity implementation can then be combined with the total loss of TabNet 200.

[0041] Feature processor 220 is configured to process the filtered features 206, a subset 216 of features 206, into processed features 222 to form decision step output 208. Once feature processor 220 has processed the subset 216 of features 206 into processed features 222, decision step output 208 and information about decision step S (referred to as decision step information 226) are fed to subsequent decision steps S. i+1 .For example, Figures 2B-2D Depicts the splitting after feature transformer 224 of feature processor 220. For example... Figure 1 As shown in 2D, the split will remove the processed feature 222 from the current step S. i (For example, as shown in the first step S1) fed into subsequent steps S i+1selector 210 (e.g., as indicated by the second step S2). For example, the split attention transformer 212 feeds the processed features 222 from the first step S1 to the feature selector 210. The attention transformer 212 can use this processed features 222, e.g., in its previous scale function (e.g., as indicated in Figure 2C the second step S2 to determine which features 206 should be selected as the subset 216 of relevant features 206R.

[0042] In some examples, to ensure parameter-efficient and robust learning with high capacity, the feature processor 220 comprises a feature transformer 224. The feature transformer 224 can comprise a plurality of processing layers L, L 1-n In some implementations, the feature transformer 224 is configured with some layers L (e.g., as indicated in Figure 2C the first and second layers L, L 1-2 ), the feature transformer 224 shares these layers L across all decision steps S i-N and some decision step dependent layers L (e.g., as indicated in Figure 2C the third and fourth layers L, L 3-4 ). For the shared layers and the decision step dependent layers L, the feature transformer 224 can perform concatenation between the layers L. For example, Figure 2C two shared layers L 1-2 are shown, where each shared layer L comprises a fully connected layer (FC) followed by batch normalization BN and a gated linear unit (GLU) nonlinearity. Here, the feature transformer 224 concatenates each shared layer L 1-2 together and feeds this concatenation to the decision step dependent layers L 3-4 In this example, the feature transformer 224 comprises two decision step dependent layers L 3-4 where each decision step dependent layer L comprises a fully connected layer (FC) followed by batch normalization BN and a gated linear unit (GLU) nonlinearity, similar to the shared layers L. Each decision step dependent layer L 3-4 can be concatenated together as well as with the shared layers L 1-2 . Using these shared decision step layers L and decision step dependent layers L, the feature transformer 224 processes the subset 216 of features 206 (e.g., each relevant feature 206R of the subset 216) into processed features 222.

[0043] In some configurations, to determine a decision embedding 242 for each decision step S, the feature processor 220 includes a rectified linear unit (RELU) 240. A RELU is an activation function that produces a linear relationship between all positive values and zero for all negative values. Here, the RELU 240 works well for sparse learning because it is sparsely activated (i.e., activated only on positive values). In these configurations, the RELU 240 receives the decision step output 208 for the processed features 222 and produces an embedding (e.g., a decision step embedding 242) to represent the decision step output 208. For example, the decision embedding 242 can be constructed as where N steps is the total number of decision steps. In some instances, the RELU 240 outputs the decision step embedding 242 to be combined (e.g., concatenated) with the decision step embeddings 242 from subsequent and / or previous decision steps S in order to form an overall decision embedding 242 for all steps S that is passed to the decoder 204 to decode the decision embedding 242 into interpretable features 206. i

[0044] In some implementations, the RELU 240 also passes the decision step information 226 to an aggregator 250. Here, the aggregator 250 is configured to receive the decision step information 226 from each decision step S. With the decision step information 226, the aggregator 250 can string this information together. In some examples, the aggregator 250 also receives information about the feature selection 210 (e.g., in Figure 2C and 2D feedback lines from the mask 214) as shown in FIGS. 1-2. Here, the aggregation of the decision step information 226 and / or the feature selection information can be used for further explanation of the functionality of the TabNet 200 such that the TabNet 200 can provide some context about the final decision 208 F to the user 10 or the manager 150. In other words, the TabNet 200 can inform the user 10 or the manager 150 that the relevant features 206R for predicting the end-of-year bonus compensation are career-related (e.g., relevant features 206Ra-c) and accounting-related (e.g., relevant features 206Rd-g).

[0045] The decoder 204 of the TabNet 200 is configured to receive the encoded representation of the decision output 208 and decode the encoded representation into an interpretable decision output 208. For example, the decoder 204 enables the conversion of the encoded representation from a computer-readable encoded representation effective for the encoder functionality to a human-readable decoded representation (i.e., reconstructed features) of the decision output 208. Referring to Figure 2B ​As described above, the encoder 202 receives the user data 12 as input and outputs the decision embeddings 242. The decoder 204 receives the decision embeddings 242 as an encoded representation of the decisions output from the encoder 202 and uses the feature transformer 224 to transform the decision embeddings 242 into the decision output 208. For example, the decoder 204 includes one or more feature transformers 224 followed by a fully connected (FC) layer to decode the encoded representation (e.g., the decision embeddings 242). In some examples, the encoder 202 communicates the decision embeddings 242 to the decoder 204 at each step S such that the decoder 204 is configured to merge each decision step output 208 into the final decision output 208 F . For example, the decoder 204 can apply a linear mapping to obtain the final decision 208 F . As one example of a multi-step process, Figure 2E the encoder 202 concatenates the two decision step outputs 208 from the first step S1 and the second step S2 into the final decision output 208 F .

[0046] Figure 2G A real-world example of the functionality of the TabNet 200 is described. Here, a user 10 provides user data 12 to the TabNet 200 (e.g., via the manager 150). The user data 12 is in a table format with features 206 corresponding to age, capital gains, education, occupation, gender, and relationship. Here, the user 10 uses the TabNet 200 to predict whether each individual corresponding to each row of data will be predicted to have an annual income greater than fifty thousand dollars. Although not shown, the TabNet 200 will first train on a set of training examples with actual income information or ground truth labels indicating certain aspects of income based on the feature set 206 before inference (i.e., real-time application). Once the training process trains the TabNet 200, the TabNet 200 is able to receive user data 12 with multiple features 206 as shown in Figure 2G and generate a prediction of whether each individual’s income will exceed fifty thousand dollars based on what the TabNet 200 learned from the user data (e.g., the final decision output 208 F of the decoder 204).

[0047] In general, TabNet 200 performs supervised learning for a particular task. In some examples, performing unsupervised representation learning prior to supervised learning improves the functionality of TabNet 200 for supervised learning of the particular task. For example, TabNet 200 can be used for a task of predicting missing features 206, 206M in tabular data. This can be advantageous because often real-world tabular datasets have interdependent feature columns that TabNet 200 can learn to predict a missing feature 206M based on other related features 206 in the same column or related interdependent columns. For example, Figure 2G User data 12 is described as table entries with multiple missing features 206, 206M, 206M a-n. Here, for a missing entry about education level, TabNet 200 can use features 206 such as occupation-related features to predict the missing feature 206M. Similarly, a missing feature 206M of gender can be predicted from other gender entries in conjunction with relationship entries. To perform unsupervised representation learning, TabNet 200 uses a self-supervised objective with a pretrainer 260. For example, with a binary mask S e {0, 1} BxD The encoder 202 of TabNet 200 inputs (1-S) f and the decoder 204 outputs the reconstructed features S f. Here, the encoder 202 can initialize the previous scale term as P[0] = (1-S), such that TabNet 200 emphasizes known features 206, and the decoder’s fully connected layer (FC) is multiplied by S to only output unknown or missing features 206M. In unsupervised representation learning, normalizing with the overall standard deviation of ground truth data can be important for potentially very different values of truncated features 206.

[0048] Figure 3 is a flowchart of an example arrangement of operations of a method 300 for explaining tabular data. At operation 302, the method 300 receives a set of features 206, 206a-n at a deep tabular data learning network (TabNet) 200 executing on data processing hardware 132. At operation 304, the method 300 performs sub-operations 304a-c for each of a plurality of sequential processing steps S, S i-n At operation 304a, the method 300 selects a subset of relevant features 216, 216a-n of the set of features 206, 206a-n using a mask 214 of the TabNet 200. At operation 304b, the method 300 processes the subset of relevant features 216, 216a-n using a feature transformer 224 of the TabNet 200 to generate features 218, 218a-n for a next processing step S, S i-n of the plurality of sequential processing steps S, S i+1decision step outputs 208 and information 226. At operation 304c, the method 300 provides the information 226 to the next processing step Si, S i+1 At operation 306, the method 300 generates decision step outputs 208, 208a-n to determine a final decision output 208, 208 i-n At operation 306, the method 300 generates decision step outputs 208, 208a-n to determine a final decision output 208, 208 F .

[0049] Figure 4 is a schematic diagram of an example computing device 400 that can be used to implement the systems (e.g., the manager 150 and / or the TabNet 200) and methods (e.g., the method 300) described in this document. The computing device 400 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the applications described and / or claimed in this document.

[0050] The computing device 400 includes a processor 410 (e.g., data processing hardware), a memory 420, a storage device 430, a high-speed interface / controller 440 connecting the memory 420 and the high-speed expansion ports 450, and a low-speed interface / controller 460 connecting the low-speed bus 470 and the storage device 430. Each of the components 410, 420, 430, 440, 450, and 460 are interconnected using various busses, and can be mounted on a common motherboard or in other manners as appropriate. The processor 410 can process instructions for execution within the computing device 400, including instructions stored in the memory 420 or on the storage device 430 to

[0051] Memory 420 stores information non-transitorily within computing device 400. Memory 420 can be a computer-readable medium, a volatile memory unit(s), or non-volatile memory unit(s). Non-volatile memory 420 can be a physical device that is used to store programs (e.g., sequences of instructions) or data (program state information) for use by computing device 400 on a temporary or permanent basis. Examples of non-volatile memory include, but are not limited to: flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and optical or tape-based storage.

[0052] Storage device 430 can provide mass storage for computing device 400. In some embodiments, storage device 430 is a computer-readable medium. In various embodiments, storage device 430 can be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In additional embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as memory 420, storage device 430, or memory on processor 410.

[0053] High-speed controller 440 manages bandwidth-intensive operations for computing device 400, while low-speed controller 460 manages lower bandwidth-intensive operations. Such allocation of functions is exemplary only. In some embodiments, high-speed controller 440 is coupled to memory 420, display 480 (e.g., through graphics processor or accelerator), and to high-speed expansion ports 450, which can accept various expansion cards (not shown). In some embodiments, low-speed controller 460 is coupled to storage device 430 and low-speed expansion port 490. The low-speed expansion port 490 can include various communication ports (e.g., USB, Bluetooth, Ethernet, and wireless Ethernet) that can be coupled to one or more input / output devices, such as keyboard and pointing devices (e.g., mouse), scanners, or network devices such as switches or routers.

[0054] As shown, computing device 400 can be implemented using a variety of forms. For example, computing device 800 can be implemented as a standard server 400a, or multiple times in a group of such servers 400a, as a laptop computer 400b, or as part of a rack server system 400c.

[0055] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, and / or optical circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0056] These computer programs (also known as programs, software, software applications or code) include machine instructions for the programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer readable medium, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0057] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical, or optical disks. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0058] To provide for interaction with a user, one or more aspects of the disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or a touch screen for displaying information to the user and optionally a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

[0059] A number of implementations have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.

Claims

1. A method for interpreting tabular data, characterized in that, include: The feature set is received at the deep tabular data learning network TabNet(200) executed on the data processing hardware (132); For each of the multiple sequential processing steps: The data processing hardware (132) uses the sparse mask (214) of the TabNet (200) to select a subset of relevant features of the feature set; The data processing hardware (132) uses the feature transformer (224) of the TabNet (200) to process the subset of relevant features to generate a decision step output (208) and information (226) for the next processing step in the plurality of sequential processing steps; as well as The information (226) is provided by the data processing hardware (132) to the next processing step; and The data processing hardware (132) determines the final decision output by aggregating the decision step outputs generated by the plurality of sequential processing steps.

2. The method (300) according to claim 1, characterized in that, Also includes: For each of the plurality of processing steps, the data processing hardware (132) uses the attention transformer (212) of the TabNet (200) to determine the total number of times each feature (206) in the feature set has been processed in each of the plurality of sequential processing steps in the preprocessing step.

3. The method (300) according to claim 2, characterized in that, The attention transformer (212) includes a fully connected layer (FC) and batch normalization (BN).

4. The method (300) according to any one of claims 1-3, characterized in that, Providing the information (226) to the next processing step includes providing the information (226) to the attention transformer (212) of the TabNet (200), which determines, based on the provided information (226), the total number of times each feature (206) in the feature set has been processed in each of the plurality of sequential processing steps in the preprocessing step.

5. The method (300) according to any one of claims 1-3, characterized in that, The feature set includes tabular data (12).

6. The method (300) according to any one of claims 1-3, characterized in that, The feature transformer (224) of the TabNet (200) includes multiple neural network layers, each of which includes a fully connected layer (FC), batch normalization (BN), and a generalized linear unit (GLU) nonlinearity.

7. The method (300) according to claim 6, characterized in that, The first portion of the plurality of neural network layers is shared across each of the plurality of sequential processing steps, and the remaining second portion of the plurality of neural network layers depends on the corresponding processing step in the plurality of sequential processing steps.

8. The method (300) according to any one of claims 1-3, characterized in that, The decision step output (208), generated by processing a subset of the relevant features, is passed through the modified linear unit (ReLU) (240) of the TabNet (200).

9. The method (300) according to any one of claims 1-3, characterized in that, The subset of features corresponds to the interdependent feature columns of the table data (12).

10. The method (300) according to any one of claims 1-3, characterized in that, Also includes: The data processing hardware (132) receives tabular data (12) including one or more missing features; and The data processing hardware (132) uses unsupervised pre-training of the TabNet (200) to predict one or more missing features.

11. A system (100) for interpreting tabular data, characterized in that, include: Data processing hardware (132); and Memory hardware (134) that communicates with and stores instructions on the data processing hardware (132), the instructions causing the data processing hardware (132) to perform the following operations when executed on the data processing hardware (132): The feature set is received at the deep tabular data learning network TabNet (200) executed on the data processing hardware (132); For each of the multiple sequential processing steps: The sparse mask (214) of the TabNet (200) is used to select a subset of relevant features of the feature set; The feature transformer (224) of the TabNet (200) is used to process the subset of relevant features to generate decision step output (208) and information (226) for the next processing step in the plurality of sequential processing steps; as well as Provide the information to the next processing step (226); and The final decision output is determined by aggregating the outputs of the decision steps generated by the multiple sequential processing steps.

12. The system (100) according to claim 11, characterized in that, The operation further includes, for each of the plurality of processing steps, using the attention transformer (212) of the TabNet (200) to determine the total number of times each feature (206) in the feature set has been processed in each of the plurality of sequential processing steps in the preprocessing step.

13. The system (100) according to claim 12, characterized in that, The attention transformer (212) includes a fully connected layer (FC) and batch normalization (BN).

14. The system (100) according to any one of claims 11-13, characterized in that, Providing the information (226) to the next processing step includes providing the information (226) to the attention transformer (212) of the TabNet (200), which determines, based on the provided information (226), the total number of times each feature (206) in the feature set has been processed in each of the plurality of sequential processing steps in the preprocessing step.

15. The system (100) according to any one of claims 11-13, characterized in that, The feature set includes tabular data (12).

16. The system (100) according to any one of claims 11-13, characterized in that, The feature transformer (224) of the TabNet (200) includes multiple neural network layers, each of which includes a fully connected layer (FC), batch normalization (BN), and a generalized linear unit (GLU) nonlinearity.

17. The system (100) according to claim 16, characterized in that, The first portion of the plurality of neural network layers is shared across each of the plurality of sequential processing steps, and the remaining second portion of the plurality of neural network layers depends on the corresponding processing step in the plurality of sequential processing steps.

18. The system (100) according to any one of claims 11-13, characterized in that, The decision step output (208), generated by processing a subset of the relevant features, is passed through the modified linear unit (ReLU) (240) of the TabNet (200).

19. The system (100) according to any one of claims 11-13, characterized in that, The subset of features corresponds to the interdependent feature columns of the table data (12).

20. The system (100) according to any one of claims 11-13, characterized in that, The operation also includes: Receive tabular data including one or more missing features (12); and Use unsupervised pre-training with the TabNet(200) to predict one or more missing features.

Images