System and method for generating trained neural network from pre-trained machine learning model

Abstract

A method, computer program product, and computing system for processing training data and prediction data as a plurality of markers using a classification-based machine learning model. A plurality of weighted features associated with the training data and the prediction data are defined by processing an output of the machine learning model with an attention layer. By processing the plurality of weighted features with the attention layer, the plurality of weighted features are remodeled to generate weights for the trained neural network.

Description

Background Technology

[0001] Generative artificial intelligence (AI) has seen tremendous growth in text and vision domains, bringing significant changes across various technological fields. However, these advancements have been somewhat limited when applied to tabular data. For example, while general-purpose large language models can handle tabular data, constraints on the number of input tokens severely limit the amount of data that can be ingested. The token constraint for many large language models is defined as a combination of input, output, and instructions. However, even with increasingly stringent token constraints, conventional generative machine learning models are trained to generate predictions based on input cues and require substantial resources to operate. Attached Figure Description

[0002] Figure 1 It is a flowchart of one implementation of the training neural network generation process;

[0003] Figure 2 yes Figure 1 A schematic diagram of the generation process of a trained neural network during inference;

[0004] Figure 3 yes Figure 1 A schematic diagram of the trained neural network generation process during training;

[0005] Figure 4 This is a schematic diagram illustrating how a trained neural network classifies tabular data; and

[0006] Figure 5 It is a schematic diagram of a computer system and the generation process of a trained neural network coupled to a distributed computing network.

[0007] In the various figures, the same reference numerals indicate the same elements. Detailed Implementation

[0008] The implementation of this disclosure provides a base machine learning model that processes both a training dataset and a prediction dataset of tabular data to generate a fully trained neural network. For example, instead of generating hyperparameters or candidate machine learning model architectures, the base machine learning model directly generates a trained neural network. As will be discussed in more detail below, the implementation of this disclosure trains a general machine learning model on millions of tabular classification tasks to generate trained neural networks with corresponding weights specific to a particular “target” classification task (as a hint is provided). Moving from direct inference to weight generation significantly improves inference cost and scalability, retaining the efficiency of smaller custom models while incorporating the generality and ease of use of the base machine learning model.

[0009] For example, the implementation of this disclosure uses a classification-based machine learning model to process training and prediction data as multiple labels. In one example, the classification-based machine learning model is a base machine learning model trained on tabular data. Specifically, the classification-based machine learning model is trained using multiple synthetic tabular datasets and backpropagation. The classification-based machine learning model defines multiple weighted features associated with the training and prediction data. By utilizing attention layers to process the output of the machine learning model, the multiple weighted features are reshaped to generate the weights of the trained neural network. The weights are used to directly generate a fully trained neural network. In this way, the classification-based machine learning model is able to generate a trained neural network in a single forward pass without subsequent fine-tuning.

[0010] Details of one or more implementations are set forth in the accompanying drawings and description below. Other features and advantages will become apparent from the description, drawings, and claims. The process of generating a trained neural network:

[0011] refer to Figures 1 to 4 The trained neural network generation process 10 uses a classification-based machine learning model to process the training and prediction data as multiple labels 100. Multiple weighted features associated with the training and prediction data are defined 102 by utilizing an attention layer to process the output of the machine learning model. By utilizing the attention layer to process the multiple weighted features, the multiple weighted features are reshaped 104 to generate weights for the trained neural network.

[0012] In some implementations, the trained neural network generation process 10 uses a classification-based machine learning model to process training and prediction data as multiple labels 100. Classification is a supervised machine learning method in which a machine learning model attempts to predict the correct label for given input data. In classification, the machine learning model is fully trained using training data and then evaluated on test or prediction data before being used to perform predictions on new, unseen data. The classification-based machine learning model (e.g., classification-based machine learning model 200) processes the input data in labeled form. In one example, the classification-based machine learning model is trained for binary classification (i.e., classifying input data (e.g., input table data 202, 204, 206) into a single class or not into that class). In another example, the classification-based machine learning model is trained for multi-class classification (i.e., classifying input data into one of several potential classes). In some implementations, the trained neural network generation process 10 converts the training and prediction data from the input tabular data 202, 204, 206 into multiple labels (e.g., training data labels 208, 210, 212 and prediction data labels 214, 216, 218, respectively) for processing by a classification-based machine learning model. For example, the trained neural network generation process 10 performs tokenization on the training and prediction data to divide them into multiple parts.

[0013] In some implementations, both training and prediction data are tabular data. For example, tabular data includes data in tables, spreadsheets, and other data structures. Tabular data is generally considered the most common data type in machine learning applications. While machine learning models (e.g., general language models) can be adapted to handle tables and tabular data, these models typically have strict limitations on the number of input labels they can ingest. For example, when comparing the memory usage of large language models (e.g., LLAMA) and numerical data transformers (e.g., TabPFN), the large language model has a vocabulary size of 32,000 and an embedding dimension of 4,096 (in its minimum version). Expressing a single feature (i.e., two decimal places) in the iris dataset requires four labels, hence approximately 4,096. 4 = 16,384 floating-point numbers or a length of 32,000 A sparse vector of 4 = 128,000. A transformer using a floating-point table requires a single floating-point number. Representing the entire Iris dataset in a way that large language models can handle (e.g., four features, three categories, and 150 data points) yields 5,000 to 15,000 tokens (depending on the tokenizer and JSON representation), which exceeds the capabilities of most large language models (e.g., LLAMA supports 2,048, the current largest model GPT-4 supports up to 32,000, while TabPFN shows expansion to 100 features and 1,000 data points (up to 5,000 in extrapolation experiments)). It is noteworthy that the tokenization constraints of large language models combine input, output, and instructions, while the constraints of numerical data transformers are specific to the training set, assuming the test / prediction sets are of the same size. Therefore, the trained neural network generation process 10 adapts the classification-based model to tabular data. Figure 2 As shown in the table, data 202, 204, and 206 define the training data 208, 210, and 212 and the corresponding prediction data 214, 216, and 218, respectively.

[0014] In some implementations, the classification-based model is a base machine learning model trained on tabular data. As discussed above, general machine learning models are often inefficient at handling tabular data due to limitations in the size of the labeled input. Therefore, the classification-based machine learning model is a base machine learning model trained on tabular data. As will be discussed in more detail below, the classification-based machine learning model 200 is a base machine learning model trained on a wide array or range of tabular data (e.g., a table with various rows and columns of data).

[0015] In some implementations, processing 100 training and prediction data sets involves using a classification-based machine learning model to process over 106 labels in a single forward pass. (See also...) Figure 2 The trained neural network generation process 10 can process 100 training data 208, 210, 212 and prediction data 214, 216, 218 in multiple linear embeddings (e.g., linear embeddings 220, 222, 224) and multiple transformers (e.g., transformers 226, 228, 230). Although Figure 2 Three sets of training data, prediction data, linear embeddings, and transformers are shown, but it should be understood that any number of sets of training data, prediction data, linear embeddings, and transformers can be used within the scope of this disclosure.

[0016] In some implementations, training data 208, 210, and 212 include multiple features (e.g., "r" features) and multiple categories (e.g., "c" categories) for classifying the data. In one example, training data 208 includes a single row of tabular data with "r" features and "c" categories corresponding to prediction data 214. Similarly, training data 210 includes a single row of tabular data with "r" features and "c" categories corresponding to prediction data 216, and training data 212 includes "r" features and "c" categories corresponding to prediction data 218. In some implementations, linear embeddings 220, 222, and 224 are layers that map training data 208, 210, and 212 and prediction data 214, 216, and 218 from a high-dimensional space to a low-dimensional space. This allows the classification-based machine learning model 200 to learn more about the relationships between the inputs and process the data more efficiently. Transformers 226, 228, and 230 are neural networks that learn context by tracking relationships in sequence data. For example, input training data (e.g., training data 208, 210, 212) is encoded as labels, and each label is converted into a vector via lookup from an embedding table (e.g., linear embeddings 220, 222, 224). At each linear embedding layer, each label is context-associated with other (unmasked) labels within a context window via a parallel multi-head attention mechanism (e.g., multi-head attention mechanism 232). PFN Overview

[0017] like Figure 2 As shown, the classification-based machine learning model 200 preserves the input encoding and transformer layer structure of TabPFN. TabPFN is a prior data fitting network (PFN) and is trained offline once to approximate Bayesian inference on multiple tabular synthetic datasets. The general principle of PFN is as follows: In a supervised Bayesian framework, the prior defines the hypothesis space Φ on the relationship between the input set x and the output label y. Each hypothesis... ∈Φ can all be viewed as mechanisms for generating data distributions, from which samples are extracted to form datasets. For example, given a prior based on a structural causal model, Φ is the space of the structural causal model, assuming It is a specific structural causal model (SCM), and the dataset includes samples generated through this SCM. Specifically, the dataset includes training data with observed labels and test data with missing or retained labels to evaluate predictive performance. Test sample x test The posterior predictive distribution (PPD) specifies its label p(·|x) test D train The distribution of the training sample set D train :={(x1, y1),…,(xn , y n Given the condition )}, PPD can be obtained by integrating over the space of the assumption Φ, where the assumption )} is... The weights of ∈Φ are determined by their prior probability p( ) and given The likelihood p(D|) of the data D at time The following formula is used to determine the result:

[0018] In some implementations, prior fitting involves training a PFN to approximate the PPD, thereby enabling Bayesian prediction. This is achieved using a prior of the form p(D) = E. φ~p(φ) [p(D| The prior sampling scheme is specified in the diagram, which first utilizes... ~p( The hypothesis (generation mechanism) is sampled, and then D~p(D| Synthetic dataset. Synthetic dataset D := (x i , y i )i∈{1,...,n} are repeatedly sampled, and the parameter θ of PFN is optimized for D test D is used for prediction, provided that the rest of the dataset D is... train =D\D test Therefore, the training loss of PFN is the cross-entropy of the retained examples of the synthetic dataset. For a single test point {(xtest, ytest)}=Dtest, the training loss can be expressed as Equation 2:

[0019] In some implementations, minimizing this loss approximates the true Bayesian posterior prediction distribution. As part of the algorithm development, a synthetic prior fitting phase is performed only once for a given prior p(D). During inference, the trained model is applied to an unseen real-world dataset. train and test feature x test The new dataset will (D) train , x test Feeding this into the trained model will produce PPD q in a single forward pass. θ (y|x test D train The PPD class probabilities are then used as predictions for our real-world task. Therefore, PFN performs training and prediction in a single step (similar to prediction in a Gaussian process) and does not use gradient-based learning on the data seen at inference time.

[0020] Regarding architecture, PFNs rely on a transformer that encodes each feature vector and label as a token, allowing the tokens to represent mutual interest. They accept a variable-length training set D of feature and label vectors. train (Treat as set-valued input to take advantage of permutation invariance) and a variable-length query set x of eigenvectors. test ={x (test, 1) ,…x (test, n)} and returns the PPD estimate for each query. Tabular data prior

[0021] Regarding the prior, tabular datasets contain a range of characteristics; for example, feature types can be numerical, ordinal, or categorical, and feature values ​​may be missing, leading to feature sparsity. These problems are typically addressed through preprocessing, feature correlation, generating irregular functions, and using categorical features. For instance, during prior fitting, the input data is normalized to zero mean and unit variance, and the same steps are applied when evaluating the real data. Because tabular data often contains exponentially scaled data that may not exist during prior fitting, power scaling is applied during inference. Therefore, during inference on the real tabular dataset, the features match more closely to those seen during prior fitting. In one example, the training samples are used only to compute z-statistics, power transformations, and all other preprocessing.

[0022] The correlation of features in tabular data varies from dataset to dataset, ranging from independent to highly correlated. This poses a problem for classic deep learning methods. When considering large-space SCMs, different levels of correlation naturally arise in the priors. Furthermore, in real-world tabular data, the ordering of features is often unstructured, but adjacent features are often more correlated than others. In one example, "block feature sampling" is used to reflect the correlation structure between ordered features. SCM generation methods naturally provide a way to achieve this. For example, the first step in generating our SCM is to generate a unidirectional hierarchical network structure, where nodes in a layer can only receive input from the previous layer. Therefore, features within the same layer tend to have higher correlation. This is done by sampling adjacent nodes in the hierarchical network structure in blocks and using these ordered blocks in our feature set.

[0023] In some implementations with real-world data, some features are consistently more important than others. While random network weight initialization can lead to slightly different feature importance, the average influence of the input features reverts to the mean as the hidden dimension increases. These differences can be amplified by sampling the weight parameters for each input feature and multiplying all outgoing weights by that factor. In the prior, the graph connections are randomly sparsified. Therefore, hidden variables and output nodes are influenced by fewer parameters, resulting in more irregular patterns (because more parameters revert to the mean). Sparsity is also extended to variable blocks, leading to stronger interactions between some groups of variables. The sampling method for noise variables is also sparsified. Instead of sampling Gaussian noise for each node from the same distribution, a separate noise mean and standard deviation are sampled for each node first, and then the noise is sampled from that distribution. Furthermore, as observed in real-world data, non-uniformly distributed input data x is generated. The input variable x (propagated through our network) is sampled from a mixture of distributions, namely Gaussian, Zipfian, and multivariate distributions.

[0024] In some implementations, tabular data includes not only numerical features but also discrete categorical features. While categorical features should technically not be ordered, in practice they are sometimes ordered (i.e., the categories represent the binning degree of some latent variables). Categorical features are generated by selecting a random portion p of the categorical features for each dataset. cat Defined by (hyperparameters). Similar to transforming numerical class labels into discrete multi-class labels, dense features are transformed into discrete features. Also similar to multi-class labels, in the case of reshuffling classes, the shuffled portion p of the class features is selected. scat .

[0025] The performance of TabPFN depends heavily on the specification of a suitable prior, as PFN approximates the PPD of that prior. For example, almost all prior hyperparameters are estimated using distributions rather than points. Fitting a model typically requires finding suitable hyperparameters (e.g., the embedding size, number of layers, and activation function of a neural network). Typically, resource-intensive searches are employed to find suitable hyperparameters. However, the results of these searches are merely point estimates of hyperparameter choices. Integrating multiple architectures and hyperparameter settings yields a coarse approximation of the distributions over these hyperparameters. However, this scales linearly in cost with the number of choices considered. In contrast, PFN allows us to perform a fully Bayesian approach on our prior hyperparameters. By defining a probability distribution over the hyperparameter space in the prior (such as a BNN architecture), the PPD approximated by TabPFN is jointly integrated over that space and the corresponding model weights. This approach is extended not only to mixtures of hyperparameters but also to mixtures of different priors. In one example, a Bayesian neural network (BNN) prior and a structural causal model (SCM) prior are mixed, where each prior requires a mixture of architecture and hyperparameters.

[0026] Causal knowledge can facilitate various machine learning tasks, including semi-supervised learning, transfer learning, and out-of-distribution generalization. Tabular data often exhibits causal relationships between columns, and causal mechanisms have been shown as strong priors in human reasoning. Therefore, the TabPFN prior is based on SCM, which models causal relationships. The SCM is a set Z:=({z1,…,z...) of structure-assigned (called mechanisms). k Composition: z i = f i (z PA G(i), ε i PA G(i) It is the parent set (direct cause) of node i in the underlying DAG G (causal graph), f i It is a (potentially nonlinear) deterministic function, and z i It is a noise variable. Causal relationships in G are represented by directed edges from cause to effect, and each mechanism z... i They were all assigned to nodes in G.

[0027] Previous work has applied causal inference to predict observations on unseen data, a method that attempts to identify causal relationships between system components using intervention and observation data. The predicted causal representation is then used to make observational predictions or provide explanations for new samples. Most existing work focuses on determining a single causal graph for downstream prediction, which can be problematic because most types of SCMs are unidentifiable without intervention data, and the number of compatible DAGs proliferates due to the combinatorial nature of DAG spaces. Recent methods use transformers to approximate causal graphs from observation and intervention data. In some implementations, explicit graph representations are skipped entirely in the inference step, and the PPD is directly approximated. In this example, causal inference is not performed; instead, the downstream prediction task is performed directly.

[0028] To create a PFN prior based on SCM, a sampling procedure for creating supervised learning tasks (i.e., datasets) is defined. Here, each dataset is based on a randomly sampled SCM (including the DAG structure and the deterministic function f). i Given an SCM, select a set of nodes z from the causal graph G to synthesize each feature in the dataset. X and a node z y These nodes are observation nodes: z X The values ​​from z are included in the feature set, while those from z y The value serves as the target. For each such SCM and node z... X and z y The list is generated by the following steps: sampling all noise variables in the SCM n times; propagating these noise variables through the graph; and obtaining all n samples at node z. X and z y The value at the point. By generating a DAG structure, the obtained features and target are associated. This leads to the formation of conditional dependencies between features through positive and negative causal relationships, that is, the target may be the cause or result of the feature.

[0029] As discussed above, and in some implementations, the BNN prior is mixed with the SCM prior described above by randomly sampling the dataset during PFN training with equal probability from either the BNN prior or the SCM prior described above. To sample the dataset from the BNN prior, the NN architecture and its weights are sampled. For each data point in the dataset to be generated, the input x is sampled, fed through the BNN with the sampled noise variables, and the output y is used as the target.

[0030] The prior returned scalar labels described above. To generate synthetic classification labels for imbalanced multi-class datasets, scalar labels... It is transformed into discrete category labels y. In one example... The values ​​are split into intervals mapped to category labels. For example, for the number of categories N... c ~p(N c Sampling is performed, where p(N) c () is a distribution over integers. From continuous targets Randomly sample N from the set c -1 category boundary B i Each scalar label i All are mapped to the index of the unique interval containing it, as shown in Equation 3 below: Where [·] is an indicator function.

[0031] For example, in N c In the case of 3 lists, the boundary B c ={-0.1, 0.5} defines three intervals: {(-∞, -0.1], (-0.1, 0.5], (0.5, ∞)}. If If less than -0.1, then any All are mapped to label 0, if it is in (-0.1, 0.5], then mapped to label 1, otherwise mapped to label 2. Finally, the category labels are shuffled (i.e., by removing the order of the category labels relative to the range). TabPFN Overview

[0032] TabPFN is a prior data fitting network that fits data to a new prior sampled from tabular data. TabPFN is formed by modifying the original PFN architecture in two ways: i) slightly modifying the attention mask to reduce inference time, and ii) using zero-padding to enable the model to handle datasets with varying numbers of features. For example, the original PFN architecture uses a single multi-head self-attention module to compute attention between all training examples and attention from validation examples to training examples. This multi-head self-attention module is replaced by two modules sharing weights: one module computes self-attention between training examples, and the other module computes only cross-attention from validation examples to training examples. Conceptually, this is equivalent to the original architecture, except that it uses a slightly different self-attention mask (the original architecture allows all examples to focus on themselves). For validation examples, attention to themselves is removed. However, information about the state at the current position is still passed through the residual branch.

[0033] In some implementations, the number of input dimensions (features) of the dataset varies, while PFN uses an encoder layer that accepts inputs with a fixed number of dimensions. Datasets with varying numbers of dimensions can be modeled using a single PFN by uniformly and randomly sampling the number of dimensions (up to 100) during training. The encoder is modified by zero-padding the dataset to accommodate training and inference with such varying numbers of features, where the number of features k is less than the maximum number of features K, and these features are scaled by K / k to keep the magnitude constant.

[0034] In one example, the final TabPFN model was trained for 18,000 steps on a batch size of 512 datasets. In this example, the modified TabPFN was trained on 9,216,000 synthetically generated datasets. This training took 20 hours on 8 GPUs (Nvidia® RTX 2080 Ti). Each dataset had a fixed size of 1,024 and was randomly and uniformly split into training and validation datasets. After approximately 10 million datasets, the learning curve flattened out, and overall noise increased. This is likely due to the diverse range of datasets generated in the prior. Regarding the prior hyperparameters, they are chosen based on the simplicity of the validation datasets and observations such as their class distribution or the strength of feature correlations. Furthermore, during algorithm development, the model is evaluated on this set of datasets to determine if the developed method is correct and effective. Because the prior hyperparameters specify distributions rather than exact values, they can be chosen over a wide range, similar to the interval chosen for a random hyperparameter search.

[0035] As described above, TabPFN trains a machine learning model that generalizes to supervised classification on real-world tabular datasets. In some implementations, the trained neural network generation process 10 extends the classification-based machine learning model 200 beyond TabPFN to generate dataset-specific trained neural networks. In this example, similar to TabPFN, the classification-based machine learning model 200 generates activations of size “m” for each pair of training data and corresponding prediction data (e.g., training data 208, 210, 212 and prediction data 214, 216, 218, respectively). In some implementations, the multi-head attention mechanism 232 includes another attention layer that reduces all activations to a single dataset embedding of size “n” (e.g., embedding 236) based on query 234. In some implementations, query 234 exists only in the multi-head attention mechanism at the top of the transformer, and not in each layer of the transformer. This allows the trained neural network generation process 10 to control the size "n" of the embedding 236 independently of the size of the embedded activation in the transformer (e.g., the activation size "m").

[0036] In some implementations, the trained neural network generation process 10 defines 102 multiple weighted features associated with the training and prediction data. For example, the trained neural network generation process 10 decodes the embedding 236 from the classification-based machine learning model 200 to generate multiple weighted features. For example, as Figure 2 As shown, the trained neural network generation process 10 uses an attention layer (e.g., attention layer 238) to decode (i.e., transform) the embedding 236 into multiple weighted features (e.g., weighted features 240, 242). In one example, attention layer 238 is part of a neural network (e.g., a two-hidden-layer feedforward neural network) that decodes the embedding 236 into a weighted feature vector (e.g., weighted features 240, 242).

[0037] In some implementations, the trained neural network generation process 10 reshapes 104 weighted features by processing multiple weighted features using an attention layer to generate the weights of the trained neural network. For example, weighted features 240 and 242 describe the biases of specific layers in the trained neural network. In one example, it is assumed that weighted features 240 and 242 describe a vector with 66,954 entries. In this example, the number 66,954 corresponds to the weights and biases of a neural network with an input embedding dimension of 512, a hidden layer size of 128, and an output layer size of 10. The trained neural network generation process 10 reshapes 104 of the weighted features 240 and 242 (e.g., using a reshaping module 244) to generate multiple weights and biases (e.g., weights 246 and 248) of the trained neural network (e.g., trained neural network 250). The reshaping module 244 is a hardware and / or software component that reshapes 104 weighted features 240, 242 by modifying the vectors of weighted features 240, 242 into multiple weight matrices and biases (e.g., weights 246, 248).

[0038] In some implementations, the trained neural network generation process 10 uses multiple weights to generate 108 trained neural networks. For example, the trained neural network generation process 10 uses multiple weights and biases (e.g., weights 246, 248) to generate a trained neural network 250. In one example, generating 108 trained neural networks 250 involves inserting a weight matrix and bias vector (e.g., weights 246, 248) reshaped from weighted features 240, 242 into layers of the trained neural network 250. Therefore, the trained neural network 250 is not trained until weights 246, 248 are applied to the layers of the trained neural network 250. Since the classification-based machine learning model 200 predefines weights 246, 248, the trained neural network is able to generate 110 predictions for input data without further training. In this way, the trained neural network 250 is trained in a single forward pass of the classification-based machine learning model 200 without any subsequent fine-tuning.

[0039] In some implementations, the trained neural network generation process 10 uses multiple synthetic tabular datasets and backpropagation to train 112 a classification-based machine learning model. For example, also refer to... Figure 3 Training data (e.g., multiple synthetic tabular datasets 300) is provided as input to a classification-based machine learning model 200. The trained neural network generation process 10 trains the classification-based machine learning model 200 by processing input rows and corresponding results from the synthetic tabular datasets 300. In one example, the trained neural network generation process 10 trains the classification-based machine learning model 200 by performing backpropagation in the classification-based machine learning model 200 using results from multiple synthetic tabular datasets and the output (e.g., output 302) of the classification-based machine learning model 200 for the corresponding input tabular data (e.g., input data 300). For example, the trained neural network generation process 10 trains the classification-based machine learning model 200 using backpropagation through the entire architecture of the classification-based machine learning model 200 (i.e., backpropagating from the output 302 of the trained neural network 250 to the weighted features 240, 242 generated by the classification-based machine learning model 200 in response to the synthetic tabular datasets 300).

[0040] In some implementations, training 112 classification-based machine learning model 200 is described as follows: Equation 4: Where θ is the parameter of the transformer of the classification machine learning model 200. This is the prediction part of the synthetic tabular dataset 300. It is a feedforward neural network, whose parameters are determined by... Given (i.e., for parameter θ, used for training and) The parameters of the transformer for the corresponding classification-based machine learning model 200. It is a classification-based machine learning model 200 with cross-entropy loss for dataset D.

[0041] During training, a synthetic tabular dataset 300 is used to learn and freeze the parameters θ. As discussed above, to apply the trained classification-based machine learning model 200 to new datasets (e.g., training datasets 208, 210, 212 and corresponding prediction datasets 214, 216, 218), the classification-based machine learning model 200 processes the training datasets 208, 210, 212 and the prediction datasets 214, 216, 218 to define 102 multiple weighted features (e.g., weighted features 240, 242), and reshapes the weighted features 240, 242 into multiple weights (e.g., weights 246, 248). Using the weights 246, 248, the trained neural network generation process 10 generates 108 trained neural networks 250. Therefore, the trained neural network generation process 10 allows the classification-based machine learning model 200 to generate trained neural networks from training and prediction data in a single forward pass. Using trained neural networks, neural networks trained on tabular datasets can directly perform various machine learning tasks based on training data, prediction data, and the base model defined by the classification-based machine learning model and attention layer described above, without the need to fine-tune the generated neural network.

[0042] Compared to conventional models trained using backpropagation and fine-tuning, the implementation of this disclosure develops or generates trained neural networks for tabular data processing with a speedup of two orders of magnitude (e.g., using a central processing unit (CPU)) or three orders of magnitude (e.g., using a graphics processing unit (GPU)). In this way, the inference time processing using the classification-based machine learning model and attention layers described above is consistent with the base model approach. Therefore, the trained neural network generation process 10 provides a base model that is continuously trained on synthetic tabular data to directly generate trained neural networks for processing tabular datasets, which are two to three orders of magnitude faster than conventional methods.

[0043] In some implementations, the trained neural network generation process 10 classifies the tabular data using a trained neural network, at least in part based on multiple features within the tabular data, to generate 114 predictions for subsequent portions of the tabular data. For example, also refer to... Figure 4Assume that the trained neural network generation process 10 receives tabular data 400 to be processed. In this example, assume that the tabular data 400 includes multiple sections organized into columns and rows (e.g., sections 402, 404, 406, 408, 410) (e.g., the first column has sections 402, 404, 406, and the second column has sections 408, 410; sections 402 and 408 are in the first row, sections 404 and 410 are in the second row, and section 406 is in the third row). Assume that the tabular data 400 is processed (e.g., by user selection or automatically) to generate predictions for subsequent sections. Therefore, the trained neural network 250 can classify each part (e.g., parts 402, 404, 406, 408, 410) to generate predictions for subsequent parts (e.g., subsequent parts 412, 414) based at least in part on multiple features within the tabular data 400 (e.g., columns, rows, numbers, names, or other characteristics used to identify relationships between different parts). As discussed above, the trained neural network 250 is generated in a single forward pass using a classification-based machine learning model 200 without the need for fine-tuning. In this way, the trained neural network generation process 10 is able to generate trained neural networks capable of classifying tabular data much faster than conventional methods (e.g., the neural network generation speed is improved by two to three orders of magnitude). System Overview:

[0044] refer to Figure 5 The trained neural network generation process 10 is shown residing on and executed by a storage system 500 connected to a network 502 (e.g., the Internet or a local area network). Examples of storage systems 500 include: network attached storage (NAS) systems, storage area networks (SANs), personal computers with storage systems, server computers with storage systems, and cloud-based devices with storage systems. SANs include one or more of the following: personal computers, server computers, a series of server computers, minicomputers, mainframes, RAID devices, and NAS systems.

[0045] The various components of the storage system 500 execute one or more operating systems, examples of which include: Microsoft® Windows®, Mac® OS X®, Red Hat® Linux®, Windows® Mobile, Chrome OS, Blackberry OS, Fire OS, or a custom operating system (Microsoft and Windows are registered trademarks of Microsoft Corporation in the U.S., other countries, or both; Mac and OS X are registered trademarks of Apple Inc. in the U.S., other countries, or both; Red Hat is a registered trademark of Red Hat Inc. in the U.S., other countries, or both; and Linux is a registered trademark of Linus Torvalds in the U.S., other countries, or both).

[0046] The instruction set and subroutines of the trained neural network generation process 10, stored on storage device 504 included within storage system 500, are executed by one or more processors (not shown) and one or more memory architectures (not shown) included within storage system 500. Storage device 504 may include: hard disk drives; optical disk drives; RAID devices; random access memory (RAM); read-only memory (ROM); and all forms of flash memory. Additionally or alternatively, portions of the instruction set and subroutines of the trained neural network generation process 10 may be stored on storage devices outside storage system 500 (and / or executed by external processors and memory architectures).

[0047] In some implementations, network 502 is connected to one or more secondary networks (e.g., network 506), examples of which include: local area network; wide area network; or intranet.

[0048] Various input / output (IO) requests (e.g., IO request 508) are sent from client applications 510, 512, 514, and 516 to storage system 500. Examples of IO requests 508 include data write requests (e.g., requests to write content to storage system 500) and data read requests (e.g., requests to read content from storage system 500).

[0049] Instruction sets and subroutines of client applications 510, 512, 514, 516, which can be stored on storage devices 518, 520, 522, 524 (correspondingly) coupled to client electronic devices 526, 528, 530, 532 (correspondingly), can be executed by one or more processors (not shown) and one or more memory architectures (not shown) incorporated into client electronic devices 526, 528, 530, 532 (correspondingly). Storage devices 518, 520, 522, 524 may include: hard disk drives; tape drives; optical disk drives; RAID devices; random access memory (RAM); read-only memory (ROM); and all forms of flash memory. Examples of client electronic devices 526, 528, 530, 532 include personal computers 526, laptop computers 528, smartphones 530, laptop computers 532, servers (not shown), data-enabled devices, and dedicated network devices (not shown). Each client electronic device 526, 528, 530, 532 executes an operating system.

[0050] Users 534, 536, 538, and 540 can access the storage system 500 directly through network 502 or through secondary network 506. Furthermore, the storage system 500 can be connected to network 502 through secondary network 506, as illustrated by link line 542.

[0051] Various client electronic devices can be directly or indirectly coupled to network 502 (or network 506). For example, personal computer 526 is shown as being directly coupled to network 502 via a hardwired network connection. Further, laptop computer 532 is shown as being directly coupled to network 506 via a hardwired network connection. Laptop computer 528 is shown as being wirelessly coupled to network 502 via a wireless communication channel 544 established between laptop computer 528 and wireless access point (e.g., WAP) 546, which is shown as being directly coupled to network 502. WAP 546 can be, for example, an IEEE 802.11a, 802.11b, 802.11g, 802.11n, Wi-Fi®, and / or Bluetooth® device capable of establishing a wireless communication channel 544 between laptop computer 528 and WAP 546. The smartphone 530 is shown to be wirelessly coupled to the network 502 via a wireless communication channel 548 established between the smartphone 530 and the cellular network / bridge 550, which is shown to be directly coupled to the network 502. Overview:

[0052] As those skilled in the art will appreciate, this disclosure can be implemented as a method, system, or computer program product. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which are generally referred to herein collectively as “circuit,” “module,” or “system.” Furthermore, this disclosure can take the form of a computer program product on a computer-usable storage medium containing computer-usable program code.

[0053] Any suitable computer-usable or computer-readable medium may be used. For example, a computer-usable or computer-readable medium may be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, devices, or propagation media. More specific examples (not an exhaustive list) of computer-readable media may include: electrical connections having one or more wires, portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, transmission media (such as those supporting the Internet or intranets), or magnetic storage devices. A computer-usable or computer-readable medium may also be paper for printing programs or another suitable medium, as the program may be electronically captured, for example, by optical scanning of paper or other media, and then compiled, interpreted, or otherwise processed as necessary, and then stored in computer memory. In the context of this document, a computer-usable or computer-readable medium may be any medium capable of containing, storing, transmitting, propagating, or transporting programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-usable media may include propagated data signals (in baseband or as part of a carrier wave) containing computer-usable program code. Any suitable medium may be used to transmit the computer-usable program code, including but not limited to the Internet, wired lines, fiber optic cables, RF, etc.

[0054] Computer program code used to perform the operations of this disclosure can be written in an object-oriented programming language. However, computer program code used to perform the operations of this disclosure can also be written in a conventional procedural programming language, such as the "C" programming language or a similar programming language. The program code can be executed entirely on the user's computer, partially on the user's computer (as a standalone software package), partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer via a local area network (LAN) / wide area network (WAN) / the Internet.

[0055] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It is to be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer / special-purpose computer / other programmable data processing apparatus, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0056] These computer program instructions may also be stored in a computer-readable storage medium that can instruct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of writing including instruction means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0057] Computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be executed on the computer or other programmable apparatus, thereby producing a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0058] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or code portion comprising one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a manner not shown in the drawings. For example, two blocks shown consecutively may actually be executed substantially concurrently, or these blocks may sometimes be executed in reverse order, not executed at all, or executed in any combination with any other flowchart depending on the function involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a system based on dedicated hardware that performs the specified function or action, or by a combination of dedicated hardware and computer instructions.

[0059] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that when the terms “comprises” and / or “comprising” are used in this specification, they specify the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0060] The corresponding structures, materials, actions, and equivalents of all means or step plus function elements in the claims are intended to include any structure, material, or action for performing a function in combination with other specifically claimed elements. The description of this disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the disclosure to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of this disclosure. The embodiments were chosen and described in order to best explain the principles and practical application of this disclosure and to enable others skilled in the art to understand the various embodiments of this disclosure, which have various modifications suitable for the particular intended use.

[0061] Many implementations have been described. Therefore, having described the disclosure of this application in detail with reference to embodiments thereof, it is apparent that modifications and variations are possible without departing from the scope of this disclosure as defined in the appended claims.

Claims

1. A computer-implemented method executed on a computing device, comprising: Use a classification-based machine learning model to process the training and prediction data as multiple labels; Define multiple weighted features associated with the training data and the prediction data; as well as By processing the multiple weighted features using an attention layer, the multiple weighted features are reshaped to generate weights for the trained neural network.

2. The computer-implemented method according to claim 1 further includes: The multiple weights are used to generate a trained neural network.

3. The computer-implemented method according to claim 2 further includes: Predictions for subsequent portions of the table data are generated by classifying the table data, at least in part, based on multiple features within the table data, using the trained neural network.

4. The computer-implemented method of claim 2, wherein generating the trained neural network comprises generating the trained neural network without subsequent fine-tuning.

5. The computer-implemented method according to claim 1, wherein the training data and the prediction data are tabular data.

6. The computer-implemented method of claim 1, wherein processing the training data and the prediction data includes processing the plurality of labels using the classification-based machine learning model in a single forward pass.

7. The computer-implemented method of claim 1, wherein the classification-based machine learning model is a base machine learning model trained on tabular data.

8. A computing system, comprising: Memory; as well as A processor configured to train a prior data fitting network classification-based machine learning model using multiple synthetic tabular datasets, to use the trained classification-based machine learning model to process training data and prediction data as multiple labels, to reshape the multiple weighted features to generate multiple weights for the trained neural network, and to generate the trained neural network.

9. The computing system of claim 8, wherein the processor is further configured to: The trained neural network is generated using the plurality of weighted features associated with the training data and the prediction data.

10. The computing system of claim 9, wherein using the plurality of weighted features to generate the trained neural network comprises generating the trained neural network without subsequent fine-tuning.

11. The computing system of claim 8, wherein the training data and the prediction data are tabular data.

12. The computing system of claim 8, wherein processing the training data and the prediction data includes using the classification-based machine learning model to process the plurality of labels in a single forward pass.

13. The computing system of claim 8, wherein reshaping the plurality of weighted features includes using an attention layer to process the plurality of weights.

14. The computing system of claim 8, wherein the classification-based machine learning model is a base machine learning model trained on tabular data.

15. A computer program product residing on a computer-readable medium having a plurality of instructions stored thereon, the plurality of instructions causing the processor to perform operations when executed by a processor, the operations including: In a single forward pass, prior data is used to fit the network to a classification-based machine learning model, treating the training and prediction data as multiple labels. Define multiple weighted features associated with the training data and the prediction data; The multiple weighted features are reshaped to generate multiple weights for the trained neural network; as well as The trained neural network is generated using the multiple weights.

Images