Tabular graph language models with multimodal learning

Abstract

This disclosure relates to a multimodal graph language model configured for tabular data. [Solution] The method may include the step of obtaining a tabular dataset. The tabular dataset can be converted into a first dataset having a first data type using a first converter. The tabular dataset can be converted into a second dataset having a second data type using a second converter. The method may further include the step of generating a first embedding set in a first-dimensional space based on the first dataset using a first encoder, and generating a second embedding set in a second-dimensional space based on the second dataset using a second encoder. The method may further include the step of training one or both of the first encoder and the second encoder based on the first and second embedding sets.

Description

Technical Field

[0001] The embodiments described in this disclosure relate to mulmodal graph language models configured for tabular data.

Background Art

[0002] Tabular data sets often contain features that include multiple types of data. The quality of feature understanding by an artificial intelligence model can be affected by how well the model understands the relationships between features and the semantic content of the features.

[0003] The subject matter claimed in this specification is not limited to embodiments that solve any disadvantages or that operate only in the environments described above. Rather, this background art is provided only to explain a technical area in which some of the embodiments described in this specification may be implemented.

Summary of the Invention

[0004] According to one aspect of an embodiment, a method may include obtaining a tabular data set. The tabular data set can be converted into a first data set having a first data type using a first converter. The tabular data set can be converted into a second data set having a second data type using a second converter. The method may further include generating, using a first encoder, a first set of embeddings in a first dimensional space based on the first data set, and generating, using a second encoder, a second set of embeddings in a second dimensional space based on the second data set. The method may further include training one or both of the first encoder and the second encoder based on the first set of embeddings and the second set of embeddings.

[0005] The objectives and advantages of the embodiments are realized and achieved at least by the elements, features, and combinations particularly indicated in the claims. It should be understood that both the above general description and the following detailed description are illustrative and descriptive, and not limitations of the claimed invention. [Brief explanation of the drawing]

[0006] Exemplary embodiments are described and illustrated with further specifics and details through the use of the attached drawings below.

[0007] [Figure 1A] This illustrates an example of a system configured to perform multimodal learning or training of an artificial intelligence (AI) model, according to at least one embodiment of the present disclosure. [Figure 1B] This illustrates an example of a system configured to perform multimodal learning or training of an artificial intelligence (AI) model, according to at least one embodiment of the present disclosure.

[0008] [Figure 2] A flowchart illustrating an exemplary method for multimodal training of an AI model, comprising at least one embodiment of the present disclosure, is shown.

[0009] [Figure 3] A block diagram of an exemplary computing system that may be used with a multimodal training system according to one or more embodiments of the present disclosure is shown. [Modes for carrying out the invention]

[0010] An artificial intelligence (AI) model is generally an algorithm or computing system configured to perform tasks that require human intervention. AI models can solve problems by learning patterns, making decisions or predictions, and processing data. The performance of an AI model can vary depending on the quality of the input data and how well the AI ​​model understands it. Input data may be presented or organized in various formats depending on the various implementations and / or application areas.

[0011] For example, input data and / or new data for generating predictions can be organized in a tabular format. A tabular dataset is a type of data structure in which data is organized into rows and columns. Each row can represent a data entry, and each column can represent a feature or attribute of that data entry. In general, tabular datasets can contain various types of data. For example, a tabular dataset may contain numerical data, categorical data, Boolean data, text data, etc. The various types of variables and formatting in a tabular dataset help to provide data in a structured format that reflects the relationships between features. Such a format allows tabular datasets to be used in various industries and / or fields.

[0012] AI models can be configured to learn various features of tabular datasets so that they can generate predictions for new data entries. Tabular datasets may be used by AI models for a variety of tasks such as classification, regression, and clustering. However, the performance of an AI model on tabular datasets may be limited by how effectively the AI ​​model understands the various types of data or features contained in the tabular dataset. For example, AI models typically learn data by converting it into numerical encodings. However, the various types of data contained in tabular datasets can present challenges in such conversions. For instance, if a tabular dataset contains data that may be a combination of numerical, categorical, and text data types, the relationships between features can confuse the AI ​​model, making it difficult to learn meaningful patterns. Furthermore, collaborative learning from various data types within a tabular dataset can be difficult. In addition, the AI ​​model may have difficulty identifying the most relevant features and converting raw data into useful inputs or embeddings for the AI ​​model to learn. AI models can also suffer from overfitting, which occurs when an AI model performs well on training data but fails to generalize to unknown data.

[0013] Some existing approaches may involve transforming tabular datasets into datasets with data types that are easier for AI models to understand. For example, a tabular dataset can be transformed into a dataset with data types such as images, text, or graphs. However, such approaches are limited or ineffective in understanding both the semantic content of features and the structural relationships between features within a tabular dataset. For instance, transforming a tabular dataset into text data may result in a loss of understanding of the structural relationships between features.

[0014] According to one or more embodiments of this disclosure, an AI model may be trained on a multimodal learning framework with respect to a tabular dataset. For example, a tabular dataset may be transformed into multiple types or modalities of data. In particular, as described in detail in this disclosure, the multimodal learning framework may include transforming and / or converting a tabular dataset into graph data and text data. Multimodel data (e.g., graph data and text data) may be aligned. Such alignment may help improve the quality of learning of the tabular dataset by the AI ​​model by leveraging complementary information from multiple modalities.

[0015] AI models trained using multimodal data can generate predictions with improved accuracy on newly presented data. Multimodal learning frameworks can enhance the ability of AI models to model feature heterogeneity across various tasks. For example, an AI model can better understand or learn different types of data present within a tabular dataset, enabling it to generate more accurate predictions.

[0016] Embodiments of this disclosure will be described with reference to the accompanying drawings.

[0017] Figure 1A shows an example of a system 100 configured to perform multimodal learning or training of an artificial intelligence (AI) model according to one or more embodiments of the present disclosure. In some embodiments, system 100 may include a first converter 104, a second converter 106, and a training module 112. Generally, system 100 can be configured to train an AI model based on a tabular dataset 102. The tabular dataset 102 may be formatted using rows and columns. Each row may correspond to a unique observation entry or instance within the tabular dataset 102. For example, each row may represent a unique data entry or data point. For example, each row may represent a transaction, customer, product, etc. Each column may represent a feature or attribute of the data. A feature may be a variable or characteristic that describes each data observation or entry. For example, if a row represents a customer, the column may include different variables that describe the customer, such as age, gender, medical history, etc. The columns or variables of the tabular dataset 102 may be of different data types, such as numeric, categorical, boolean, or text. The tabular dataset 102 may be used to train an AI model to generate predictions when new data entries are provided to the AI ​​model.

[0018] In some embodiments, the first converter 104 and the second converter 106 may be configured to convert or transform the tabular dataset 102 into data of a different type or modality. In some embodiments, the first converter 104 and the second converter 106 may include code and routines configured to enable a computing system to perform one or more operations corresponding to the first converter 104 and the second converter 106. Additionally or alternatively, the first converter 104 and the second converter 106 may be implemented using hardware including one or more processors, microprocessors, microcontrollers, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or any other digital or analog circuits configured to interpret and / or execute program instructions and / or process data. Furthermore, references to operations performed by the first converter 104 and the second converter 106 may include operations that the first converter 104 and the second converter 106 have other components perform.

[0019] In these and other embodiments, the first converter 104 may generate a first dataset 108, and the second converter 106 may generate a second dataset 110. In these and other embodiments, the first dataset 108 and the second dataset 110 may contain data of different modalities from a tabular dataset. Furthermore, the first dataset 108 and the second dataset 110 may contain different types of data from each other.

[0020] For example, in some embodiments, the first dataset 108 may include graph data, and the second dataset 110 may include text data. In these and other embodiments, the first converter 104 and the second converter 106 may include any suitable type of converter for converting the tabular dataset 102 into graph data and text data, respectively. While graph data and text data are described below, the first dataset 108 and the second dataset 110 may include other modalities.

[0021] In some embodiments, the first converter 104 can be configured to generate a first dataset 108 containing graph data corresponding to the tabular dataset 102. In some embodiments, the first converter 104 can divide the tabular dataset 102 into two discontinuous components based on the types of data present in different columns of the tabular dataset 102. For example, some columns may contain numerical columns and others may contain categorical columns.

[0022] In some embodiments, the first converter 104 may include a normalizer and a numeric encoder. The normalizer may be configured to normalize the numeric data of a numeric column. In these and other embodiments, the normalizer may include any suitable type of normalizer, such as a min-max normalizer. The numeric encoder may be configured to convert the categorical data of a categorical column into a numeric encode. In these and other embodiments, the numeric encoder may include any suitable type of numeric encoder, such as a OneHot encoder, an ordinal encoder, a binary encoder, a frequency encoder, or a target encoder. In these and other embodiments, the numeric encoded and normalized numeric data may be combined or combined to generate a first dataset 108 including graph data.

[0023] For example, in some embodiments, the first dataset 108 can include a set of graphs, and the individual graphs of the set of graphs correspond to the individual rows of the tabular dataset 102. Each graph of the set of graphs can include nodes and edges connecting the nodes. In these and other embodiments, the nodes can represent features or columns associated with the rows, and the edges can represent interactions between the nodes or between the columns corresponding to the nodes. In some embodiments, the edges can be weighted, and the weights represent the importance of the relationships between the nodes or between the rows.

[0024] In some embodiments, the second converter 106 can be configured to generate a second dataset 110 that includes text data corresponding to the tabular dataset 102. For example, in some embodiments, the second dataset 110 can include a set of serialized text data, and each serialized text of the set of serialized text data represents a row of the tabular dataset 102. In some embodiments, the second converter 106 can include a text tokenizer configured to break the serialized text into tokens. The tokens can include words, characters, subwords, or phrases from the serialized text data.

[0025] In some embodiments, the first converter 104 and the second converter 106 can be configured to transform or distort a portion or batch of the tabular dataset 102. In these and other embodiments, the first converter 104 and the second converter 106 can transform the same portion or batch of the tabular dataset 102. In some embodiments, the batch size can be defined by a user or operator of the system 100 based on various parameters such as memory availability. For example, the user can decrease the batch size when there is less memory available to the system 100 and increase the batch size when there is more memory available.

[0026] In some embodiments, the training module 112 may be configured to obtain a first data set 108 and a second data set 110. In some embodiments, the training module 112 may include code and routines configured to enable a computing system to perform one or more operations corresponding to the training module 112. Additionally or alternatively, the training module 112 may be implemented using hardware including one or more processors, microprocessors, microcontrollers, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or any other digital or analog circuitry configured to interpret and / or execute program instructions and / or process data. Further, references to operations performed by the training module 112 may include operations that the training module 112 causes to be performed by some other component.

[0027] The training module 112 can be configured to train an AI model based on the first data set 108 and the second data set 110. In these and other embodiments, the training module 112 can train an AI model based on multiple modalities of the first data set 108 and the second data set 110 (e.g., graph data and text data corresponding to the tabular data set 102) to learn patterns and content present in the tabular data set 102. In these and other embodiments, the multiple modalities enable the training module 112 to train an AI model to better understand the tabular data set 1 having both structural information and semantic information present in the tabular data set 102. In some embodiments, the training module 112 can be implemented as part of an AI model. The training module 112 can be described in further detail with respect to FIG. 1B.

[0028] Referring to Figure 1B, in some embodiments, the training module 112 may include a first encoder 114 and a second encoder 116. In some embodiments, the first encoder 114 and the second encoder 116 may be configured to acquire a first dataset 108 and a second dataset 110, respectively. The first encoder 114 and the second encoder 116 may be configured to generate a first embedding set 118 and a second embedding set 120, respectively, based on the first dataset 108 and the second dataset 110.

[0029] In these and other embodiments, the first encoder 114 and the second encoder 116 may be or include encoders of a type corresponding to the first dataset 108 and the second dataset 110, respectively. For example, if the first dataset 108 contains graph data corresponding to a tabular dataset 102, the first encoder 114 may be or include any type of encoder suitable for generating graph embeddings based on the graph data. For example, the first encoder 114 may be or include a graph neural network (GNN) model. For example, the GNN may be trained to generate graph embeddings from the graph data. In some embodiments, the first embedding set 118 may be generated in a first-dimensional space. The first-dimensional space may correspond to the dimension or size of the output vector generated from the first embedding set 118. In these and other embodiments, the first embedding set 118 may represent structural relationships between features or columns.

[0030] In some embodiments, if the second dataset 110 includes text data corresponding to the tabular dataset 102, the second encoder 116 may be any encoder suitable for generating text embeddings based on the text data, or may include such an encoder. For example, the second encoder 116 may include any encoder capable of converting text (e.g., words, sentences, paragraphs, etc.) into numerical representations or embeddings that can be processed by an AI model.

[0031] For example, in some embodiments, the second encoder 116 may be a large language model (LLM) encoder configured to encode the serialized text into a second set of embeddings 120 (e.g., text embeddings), or may include such an encoder. The LLM encoder may be trained to be used directly as a text encoder. For example, the LLM encoder may be trained to receive text data (e.g., serialized text) and generate embeddings that capture the meaning of the input text data. While the second encoder 116 is described in relation to an LLM, other suitable types of text encoders such as Word2Vec, GloVe, or transformer-based encoders may be used. In some embodiments, the second encoder 116 may generate a second set of embeddings 120 in a second-dimensional space. In some embodiments, the second-dimensional space may be the same as the first-dimensional space. In other embodiments, the second-dimensional space may be different from the first-dimensional space. The embeddings may include structured vector representations that capture the meaning and context of the text. In this disclosure, references to an LLM may include references to the encoder portion of an LLM.

[0032] In some embodiments, the training module 112 may include a consistency module 122 configured to determine a first loss 128. In some embodiments, the consistency module 122 may be configured to determine the first loss 128 based on a first embedding set 118 and a second embedding set 120. In these and other embodiments, the first loss 128 may represent the distance between the first embedding set 118 and the second embedding set 120. For example, individual embeddings in the first embedding set 118 and the second embedding set 120 that correspond to the same data entry or row in a tabular dataset 102 can be identified, and the distance between individual embeddings corresponding to the same data entry can be determined.

[0033] In some examples, the first-dimensional space of the first embedding set 118 may differ from the second-dimensional space of the second embedding set 120. In such examples, the first embedding set 118 or the second embedding set 120 can be projected into different dimensional spaces such that the first embedding set 118 and the second embedding set 120 are in the same dimensional space. For example, the first embedding set 118 can be projected into a second-dimensional space, or the second embedding set 120 can be projected into a first-dimensional space, so that the first embedding set 118 and the second embedding set 120 can be compared in the same dimensional space. In some examples, a modality gap may exist between the first embedding set 118 and the second embedding set. However, such a modality gap may be such that it is negligible.

[0034] In some embodiments, the training module 112 may include a classifier 124 or classifier head configured to map a first embedding set 118 from a first-dimensional space to a third-dimensional space. Generally, the third-dimensional space may be a lower-dimensional space compared to the first-dimensional space. In some embodiments, the third-dimensional space may correspond to the number of classes in the classification task or the dimensional space of the labeled dataset. The labeled dataset 126 may include a target class label set. The target class label set can represent the target that the AI ​​model is trying to predict. For example, with respect to a classification task, the labeled dataset 126 may include categorical variables that represent the category or class that the AI ​​model is trying to predict. The target class labels can represent the true category or actual category to which each data point belongs. In these and other embodiments, the labeled dataset 126 can function as ground truth.

[0035] In these and other embodiments, the classifier 124 can map the first embedding set 118 to a labeled dataset 126 to determine a second loss 130. In some embodiments, the second loss 130 can represent the difference between the first embedding set 118 and the true values ​​or the labeled dataset 126. For example, the classifier 124 can generate predictions from each embedding in the first embedding set 118, where each prediction corresponds to a class. The predicted classes can be compared to the actual classes to calculate a cross-entry loss (e.g., the second loss 130). In some embodiments, the classifier 124 may be or include a regression head configured to output numerical values ​​as predictions for each embedding in the first embedding set 118. In such cases, the second loss 130 can be determined using different techniques, such as calculating the mean squared error (MSE). The MSE can measure the squared difference between the prediction (e.g., numerical values) and the true output (e.g., the labeled dataset 126), averaged over all data points. The second loss 130 can be determined using any other technique, such as a specialized loss function like cross-entropy loss.

[0036] In some embodiments, the training module 112 may be configured to obtain a total loss based on a first loss 128 and a second loss 130. In some embodiments, the total loss may be the sum of the first loss 128 and the second loss 130. In some embodiments, the total loss may be a weighted sum of the first loss 128 and the second loss 130, with different weightings for each. For example, in some examples, the first loss 128 may be weighted more than the second loss 130 (e.g., a 7:3 ratio between the first loss 128 and the second loss 130). In other examples, the second loss 130 may be weighted more than the first loss 128. In some embodiments, the weighting between the first loss 128 and the second loss 130 may be defined by the user.

[0037] In some embodiments, the training module 112 may be configured to adjust one or more parameters of the first encoder 114 and / or the second encoder 116 to reduce the total loss. In some embodiments, the one or more parameters may include the weights of the neural network layers. For example, adjusting the parameters of the first encoder 114 (e.g., GNN) may include adjusting the weights of different layers of the GNN. In these and other embodiments, the training module 112 may perform gradient calculations to identify how much each parameter contributes to the total loss. For example, the training module 112 may use backpropagation to determine how much each parameter contributes to the total loss 130. The gradient may be used to adjust one or more parameters to reduce or minimize the loss. For example, the training module 112 may include an optimizer (e.g., stochastic gradient descent (SGD) Adam, etc.) configured to adjust one or more parameters based on the gradient.

[0038] In some embodiments, the parameters of only one of the first encoder 114 and the second encoder 116 may be adjusted to reduce the total loss. For example, in some embodiments, the parameters of the first encoder 114 may be adjusted. In some embodiments, the encoder whose parameters are adjusted between the first encoder 114 and the second encoder 116 can be determined based on the resources and / or cost associated with training or adjusting the encoder parameters. For example, parameter adjustment or training an LLM may require more computing resources compared to training a GNN. In such cases, only the GNN or the first encoder 114 may be trained (e.g., with parameters adjusted to reduce the total loss).

[0039] In some embodiments, the training module 112 can repeat a training process that determines the total loss and reduces the total loss by adjusting the parameters of one or more encoders. For example, the training process can be repeated until a threshold level of total loss is reached. In some embodiments, the user can define the threshold level of total loss. In these and other embodiments, the user can define the threshold level based on various parameters such as training cost, data complexity, and application field. Additionally or alternatively, the training process may include other stopping criteria, such as early termination based on the maximum number of iterations or validation performance.

[0040] In these and other embodiments, a training process that reduces total loss can reduce a first loss (between the first embedding set 118 and the second embedding set 120) and / or a second loss (between the first embedding set 118 and the labeled dataset 126). Such a training process improves the accuracy of the AI ​​model by reducing the differences between the first embedding set 118 and the second embedding set 120, and between the first embedding set 118 and the labeled dataset 126. Reducing the first loss between the first embedding set 118 and the second embedding set 120 can help the AI ​​model capture both structural relationships (from the first embedding set 118) and semantic information (from the second embedding set 120) present in the tabular dataset 102. Such an AI model can produce more accurate predictions than an AI model trained on a single modality (such as text, graphs, or images).

[0041] Modifications, additions, or omissions may be made to System 100 without departing from the scope of this disclosure. For example, in some embodiments, System 100 may include any number of other components, which may or may not be expressly described.

[0042] Figure 2 shows a flowchart of an exemplary method 200 for multimodal AI model training according to one or more embodiments of the present disclosure. Method 200 may be performed using any suitable system, equipment, or apparatus. For example, Method 200 can be performed using System 100 in Figures 1A-1B or Computing System 300 in Figure 3. Although shown by separate blocks, steps and operations related to one or more blocks of Method 200 may be separated into additional blocks, combined into fewer blocks, or deleted, depending on the specific implementation.

[0043] In some embodiments, method 200 may begin in block 202. In block 202, a tabular dataset may be obtained. The tabular dataset may include a set of data organized in a tabular format. In some embodiments, the tabular dataset may include data entries containing different types of data. For example, the tabular dataset may include numerical data, categorical data, text data, etc. In some embodiments, a user may provide the tabular dataset. In these and other embodiments, the tabular dataset may be a training dataset for training an AI model for a specific purpose defined by the user. For example, a particular AI model may be trained to detect a particular medical condition of a patient based on a set of patient-related variables. In such an example, the tabular dataset may include a set of patient records that include the presence of a medical condition.

[0044] In block 204, the tabular dataset may be converted to a first dataset having a first data type using a first converter. The first converter may include hardware and / or software configured to convert the tabular dataset to a first dataset having a first data type. In some embodiments, the first data type may be different from tabular data. For example, in some embodiments, the first dataset may include graph data. In these and other embodiments, the first converter may include any suitable hardware and / or software capable of converting tabular data to graph data.

[0045] In these and other embodiments, the first dataset may include a set of graphs generated based on a tabular dataset. For example, in some embodiments, the first dataset may include a graph for each row of the tabular dataset. Each graph may include nodes representing features contained in the columns associated with each row. Furthermore, each graph may include edges connecting the nodes, where the edges represent structural relationships between nodes and / or between features corresponding to the nodes. In some embodiments, the edges may be weighted to represent the importance or significance of the relationships between nodes. In some embodiments, the first converter can be described in more detail with respect to the first converter 104 in Figure A.

[0046] In block 206, the tabular dataset may be converted to a second dataset having a second data type using a second converter. The second converter may include hardware and / or software configured to convert the tabular dataset to a second dataset having a second data type. In some embodiments, the second data type may be different from the tabular data and the first data type. For example, in some embodiments, the second dataset may include text data. In these and other embodiments, the second converter may include any suitable hardware and / or software capable of converting tabular data to text data.

[0047] In these and other embodiments, the second dataset may include a set of serialized text sentences, where each sentence represents a row in the tabular dataset. In some embodiments, the second converter may include a text tokenizer configured to break down the serialized text into tokens. The tokens may include words, characters, subwords, or phases from the serialized text data. In some embodiments, the first and second converters may convert a portion or batch of the tabular dataset into the first and second datasets, respectively. In these and other embodiments, the first and second converters may convert the same portion or batch of the tabular dataset. In some embodiments, the batch of tabular datasets to be converted may be defined by the user. In some embodiments, the second converter can be described in more detail with respect to the second converter 106 in Figure 1A.

[0048] In block 208, a first encoder can be used to generate a first set of embeddings in a first-dimensional space based on a first dataset. The first encoder may include any software and / or hardware suitable for generating a first set of embeddings based on a first dataset. For example, if the first dataset includes graph data, the first encoder may be, or include, software and / or hardware configured to generate graph embeddings based on graph data. For example, in some embodiments, the first encoder may include a GNN model trained to generate graph embeddings based on graph data. In some embodiments, the first encoder can be described in more detail with respect to the first encoder 114 in Figure 1B.

[0049] In block 210, a second encoder can be used to generate a second set of embeddings in a second-dimensional space based on a second dataset. In some embodiments, the second-dimensional space may be the same dimension as the first-dimensional space. In other embodiments, the second-dimensional space may be different from the first-dimensional space. The second encoder may include any software and / or hardware suitable for generating a second set of embeddings based on the second dataset. For example, if the second dataset contains text data, the second encoder may be or include software and / or hardware configured to generate text embeddings based on the text data. For example, in some embodiments, the second encoder may include an LLM (e.g., the encoder portion of an LLM). In some embodiments, the second encoder can be described in more detail with respect to the second encoder 116 in Figure 1B.

[0050] In block 212, one or both of the first encoder and / or the second encoder may be trained based on the first and second embedding sets. In some embodiments, training one or both of the first and second encoders may include adjusting or updating one or more parameters of the first encoder and / or the second encoder.

[0051] In some embodiments, a labeled dataset containing a set of target labels may be obtained. Target labels can represent values ​​or targets that the AI ​​model is trying to predict. For example, in a classification task, the labeled dataset may include categorical targets that represent categories or classes that the AI ​​model is trying to predict. Target class labels can represent the true category or actual category to which each data point belongs. In these and other embodiments, the labeled dataset can function as ground truth.

[0052] In these and other embodiments, a first loss between the first and second embedding sets may be determined. In some embodiments, the first loss may represent the loss of consistency between the first and second embedding sets. In some embodiments, determining the first loss may include verifying that the first-dimensional space coincides with the second-dimensional space. If the first-dimensional space differs from the second-dimensional space, the first embedding set may be projected onto the second-dimensional space. In some embodiments, individual embeddings in the first and second embedding sets corresponding to the same data point or row in a tabular dataset may be determined, and the first loss may be determined by calculating the distance between the corresponding individual embeddings.

[0053] In some embodiments, a second loss may be determined between the first embedding set and the labeled dataset. In some embodiments, the second loss may represent the deviation of the first embedding set from the target values ​​or labels. In some embodiments, the first embedding set may be projected into a three-dimensional space corresponding to the labeled dataset so that the first embedding set can be compared with the labeled dataset. The first embedding set may be mapped to a target label set contained in the labeled dataset using a classifier. The second loss may be determined based on the mapping.

[0054] In some embodiments, the total loss may be determined based on the first loss and the second loss. In some embodiments, the total loss may be the sum of the first loss and the second loss. In some embodiments, the total loss may be a weighted sum of the first loss and the second loss. For example, the first loss and the second loss may be weighted differently. For example, the first loss may be weighted more than the second loss. In another example, the second loss may be weighted more than the first loss. In some embodiments, the weighting of the first loss and the second loss may be defined by the user.

[0055] In some embodiments, one or more parameters of either the first encoder or the second encoder can be updated based on total loss. For example, one or more parameters can be adjusted or updated so that the total loss is reduced. If the first encoder and / or the second encoder include a neural network such as a GNN, updating one or more parameters may include updating the weights and / or biases of different layers of the neural network.

[0056] In some embodiments, the parameters of only one of the first encoder and the second encoder can be adjusted based on the total loss. For example, in some embodiments, the parameters of only the first encoder may be adjusted. In some embodiments, one of the first encoder and the second encoder can be selected to be adjusted based on the complexity and / or cost associated with adjusting the parameters. For example, since the LLM may be more complex or larger, adjusting the parameters of the GNN may be more cost-effective than adjusting the parameters of the LLM. In such cases, the parameters of only the GNN can be adjusted.

[0057] In some embodiments, a first embedding set and / or a second embedding set can be used to train based on input data. For example, if the input data consists only of text data, a second encoder can be trained using a second embedding set corresponding to a text encoder (e.g., a second encoder). However, if the input data consists only of mixed-type data or only numerical data, both the first and second embedding sets can be used for training.

[0058] Modifications, additions, or omissions may be made to Method 200 without departing from the scope of this disclosure. For example, the outlined steps and operations are provided only as examples, and some of the steps and operations may be optional, combined with fewer steps and operations, or extended to additional steps and operations without impairing the essence of the disclosed embodiments.

[0059] For example, method 200 may further include obtaining unknown data entries. Unknown data entries may include feature sets, such as features or columns, contained within a tabular dataset. For example, if an AI model is trained to detect a patient's medical condition, unknown data entries may include information about a new patient. The AI ​​model may be configured to generate predictions (e.g., the presence of a medical condition) for unknown data entries (e.g., a new patient).

[0060] In some embodiments, predictions can be generated during inference using only one of the first and second encoders. For example, in some embodiments, predictions can be generated using only the first encoder (e.g., a GNN). In these and other embodiments, the first and / or second encoders are trained to reduce the total loss (e.g., reduce the loss of consistency between the first and second embedding sets), so that improved predictions can still be generated using only one of the first or second encoders. Using only one of the first or second encoders can help the AI ​​model reduce resources and generate predictions faster.

[0061] Figure 3 shows a block diagram of an exemplary computing system that may be used with a multimodal training system according to at least one embodiment of the present disclosure. For example, computing system 300 can be used to implement the multimodal learning framework described with respect to Figures 1A to 2.

[0062] The computing system 300 may include a processor 310, memory 312, data storage device 314, and user interface 316. The processor 310, memory 312, data storage device 314, and user interface may be communicatively coupled.

[0063] Typically, the processor 310 may include any suitable dedicated or general-purpose computer, computing entity, or processing unit, including various computer hardware or software modules, and may be configured to execute instructions stored in any suitable computer-readable storage medium. For example, the processor 310 may include a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or any other digital or analog circuitry configured to interpret and / or execute program instructions and / or process data. Although shown as a single processor in Figure 4, the processor 310 may include any number of processors configured to perform or direct any number of operations described herein, individually or collectively. Furthermore, one or more of the processors may reside on one or more different electronic devices, such as different servers.

[0064] In some embodiments, the processor 310 may be configured to interpret and / or execute program instructions and / or process data stored in memory 312, data storage device 314, or memory 312 and data storage device 314. In some embodiments, the processor 310 may fetch program instructions from data storage device 314 and load the program instructions into memory 312. After the program instructions are loaded into memory 312, the processor 310 may execute the program instructions.

[0065] The memory 312 and data storage device 314 may include computer-readable storage media that carry or have stored computer-executable instructions or data structures. Such computer-readable storage media may include any commercially available media that can be accessed by a general-purpose or dedicated computer, such as the processor 310. For example, but not limited to, such computer-readable storage media may include tangible or non-temporary computer-readable media, such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM), or other optical disk storage devices, magnetic disk storage devices, or other magnetic storage devices, flash memory elements (e.g., solid memory devices), or any other storage media that can be used to store specific program code in the form of computer-executable instructions or data structures and can be accessed by a general-purpose or dedicated computer. Combinations of the above may also be included in the scope of computer-readable storage media. Computer-executable instructions may include, for example, instructions and data configured to cause the processor 310 to perform a specific operation or a set of operations.

[0066] The user interface 316 may include any device that allows the user to interface with the computing system 300. For example, the user interface 316 may include, among other devices, a mouse, trackpad, keyboard, buttons, camera, and / or touchscreen. The user interface 316 can receive input from the user and provide input to the processor 310.

[0067] Modifications, additions, or omissions may be made to the computing system 300 without departing from the scope of this disclosure. For example, in some embodiments, the computing system 300 may include any number of other components, which may or may not be expressly described.

[0068] The terms used in this disclosure and in particular in the appended claims (e.g., the appended claim bodies) are generally intended to be “broad” terms (for example, the term “including” should be interpreted as “including, but not limited to,” and the term “having” should be interpreted as “having, but not limited to,” etc.).

[0069] Furthermore, where a specific number of introduced enumerations of claims is intended, such intention is explicitly indicated in the claim; where there is no such enumeration, such intention does not exist. For example, for the sake of understanding, the claims attached below may include the use of the introductory phrases “at least one” and “one or more” to introduce an enumeration of claims. However, the use of such phrases should not be interpreted as meaning that the introduction of an enumeration of claims by the indefinite article “a” or “an” limits any particular claim containing such an introduced enumeration of claims to embodiments containing only one such enumeration, even when the same claim includes the introductory phrase “one or more” or “at least one” and the indefinite article “a” or “an” (which should be interpreted as meaning, for example, “a” and / or “an at least one” or “one or more”). In other words, the same applies to the use of the definite article used to introduce an enumeration of claims.

[0070] Furthermore, where an explicit enumeration of claims introducing a specific number is stated, a person skilled in the art will understand that such an enumeration should be interpreted as meaning at least the number enumerated (for example, the statement “two enumerations” without other qualifications means at least two enumerations, or two or more enumerations). Furthermore, in examples where a statement similar to “at least one of A, B, and C etc.” or “one or more of A, B, and C etc.” is used, such a configuration is usually intended to include A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, etc. Furthermore, the use of the term “and / or” is intended to be interpreted in this manner.

[0071] Furthermore, any disjunct word or phrase representing two or more alternative terms should be understood to imply the possibility of including one of the terms, either of the terms, or both of the terms, whether in the description, claims, or drawings. For example, the phrase "A or B" should be understood to include the possibility of "A" or "B" or "A and B," even if the terms "and / or" are used elsewhere.

[0072] All examples and conditional language described herein are intended for teaching purposes to help readers understand the disclosure and the concepts that the disclosure contributes to the further development of the technology, and should be construed as not being limited to such specifically listed examples and conditions. While embodiments of the disclosure are described in detail, various modifications, substitutions, and choices can be made to them without departing from the spirit and scope of the disclosure.

[0073] In addition to the embodiments described above, the following additional information is disclosed. (Note 1) Steps to obtain a tabular dataset, The steps include: using a first converter to convert the tabular dataset into a first dataset having a first data type; The steps include using a second converter to convert the tabular dataset into a second dataset having a second data type different from the first data type, A step of generating a first embedding set in a first-dimensional space based on the first dataset using a first encoder, The steps include: using a second encoder to generate a second set of embeddings in a two-dimensional space based on the second dataset; A step of training one or both of the first encoder and the second encoder based on the first embedding set and the second embedding set, A method that includes this. (Note 2) The step of training one or both of the first encoder and the second encoder based on the first embedding set and the second embedding set is: Steps include obtaining a labeled dataset containing the target label set, A step of determining a first loss between the first embedding set and the second embedding set, The steps include determining a second loss between the first embedding set and the labeled dataset, A step of determining the total loss based on the first loss and the second loss, The steps include updating one or more parameters of either the first encoder or the second encoder based on the total loss, The method described in Appendix 1, including the method described in Appendix 1. (Note 3) The step of determining the second loss between the first embedding set and the labeled dataset is: The steps include projecting the first embedding set onto a third-dimensional space corresponding to the labeled dataset, A step of using a classifier to map the first embedding set to the target label set included in the labeled dataset, A step of determining the second loss based on the mapping, The method described in Appendix 2, including the method described in Appendix 2. (Note 4) The method according to Note 2, wherein the total loss is determined as the sum of the first loss and the second loss, and the first loss and the second loss are weighted differently based on the first weight and the second weight, respectively. (Appendix 5) The method according to Appendix 2, wherein one or more parameters of only the first encoder are updated. (Note 6) The step of determining the first loss between the first embedding set and the second embedding set is: A step to verify that the first-dimensional space coincides with the second-dimensional space, A step of determining each embedding in the first embedding set that corresponds to each embedding in the second embedding set, A step of calculating the distance between each embedding in the first embedding set and each embedding in the second embedding set, The method described in Appendix 2, including the method described in Appendix 2. (Note 7) The method according to Note 1, wherein the first data type is graph data and the second data type is text data. (Appendix 8) The method according to Appendix 7, wherein the first encoder is a graph neural network and the second encoder is the encoder portion of a large-scale language model. (Note 9) The method according to Note 7, wherein the first dataset includes one or more graphs representing the relationships between feature columns of the tabular dataset. (Note 10) The method according to Note 7, wherein the second dataset includes the text data representing the semantic information of the tabular dataset. (Note 11) The tabular dataset is the method described in Note 1, which includes numerical data and categorical data. (Note 12) The method described in Note 1, wherein the batch of the tabular dataset is converted into the first dataset and the second dataset. (Note 13) One or more non-temporary computer-readable media for storing instructions, wherein, when the instructions are executed by one or more processors, the system causes the system to perform an action, and the action is Steps to obtain a tabular dataset, The steps include: using a first converter to convert the tabular dataset into a first dataset having a first data type; The steps include using a second converter to convert the tabular dataset into a second dataset having a second data type different from the first data type, A step of generating a first embedding set in a first-dimensional space based on the first dataset using a first encoder, The steps include: using a second encoder to generate a second set of embeddings in a two-dimensional space based on the second dataset; A step of training one or both of the first encoder and the second encoder based on the first embedding set and the second embedding set, One or more non-temporary computer-readable media, including [the specified text]. (Note 14) The step of training one or both of the first encoder and the second encoder based on the first embedding set and the second embedding set is: Steps include obtaining a labeled dataset containing the target label set, A step of determining a first loss between the first embedding set and the second embedding set, The steps include determining a second loss between the first embedding set and the labeled dataset, A step of determining the total loss based on the first loss and the second loss, The steps include updating one or more parameters of either the first encoder or the second encoder based on the total loss, One or more non-temporary computer-readable media as described in Appendix 13, including the following: (Note 15) The step of determining the second loss between the first embedding set and the labeled dataset is: The steps include projecting the first embedding set onto a third-dimensional space corresponding to the labeled dataset, A step of using a classifier to map the first embedding set to the target label set included in the labeled dataset, A step of determining the second loss based on the mapping, One or more non-temporary computer-readable media as described in Appendix 14, including the following: (Note 16) The total loss is determined as the sum of the first loss and the second loss, and the first loss and the second loss are weighted differently based on the first weight and the second weight, respectively, in one or more non-temporary computer-readable media as described in Note 14. (Note 17) One or more non-temporary computer-readable media as described in Note 14, on which one or more parameters of only the first encoder are updated. (Note 18) The step of determining the first loss between the first embedding set and the second embedding set is: A step to verify that the first-dimensional space coincides with the second-dimensional space, A step of determining each embedding in the first embedding set that corresponds to each embedding in the second embedding set, A step of calculating the distance between each embedding in the first embedding set and each embedding in the second embedding set, One or more non-temporary computer-readable media as described in Appendix 14, including the following: (Note 19) One or more non-temporary computer-readable media as described in Note 13, wherein the first data type is graph data and the second data type is text data. (Note 20) A system, One or more processors, One or more non-temporary computer-readable media for storing instructions, wherein, when an instruction is executed, it causes the system to perform an action, and the action is Steps to obtain a tabular dataset, The steps include: using a first converter to convert the tabular dataset into a first dataset having a first data type; The steps include using a second converter to convert the tabular dataset into a second dataset having a second data type different from the first data type, A step of generating a first embedding set in a first-dimensional space based on the first dataset using a first encoder, The steps include: using a second encoder to generate a second set of embeddings in a two-dimensional space based on the second dataset; A step of training one or both of the first encoder and the second encoder based on the first embedding set and the second embedding set, A system that includes this. [Explanation of Symbols]

[0074] 102 Tabular Datasets 104 First Converter 106 Second Converter 108. Dataset 1 110 Second Dataset 112 Training Modules

Claims

1. Steps to obtain a tabular dataset, The steps include: using a first converter to convert the tabular dataset into a first dataset having a first data type; The steps include using a second converter to convert the tabular dataset into a second dataset having a second data type different from the first data type, A step of generating a first embedding set in a first-dimensional space based on the first dataset using a first encoder, The steps include using a second encoder to generate a second embedding set in a two-dimensional space based on the second dataset, A step of training one or both of the first encoder and the second encoder based on the first embedding set and the second embedding set, A method that includes this.

2. The step of training one or both of the first encoder and the second encoder based on the first embedding set and the second embedding set is: Steps include obtaining a labeled dataset containing the target label set, A step of determining a first loss between the first embedding set and the second embedding set, The steps include determining a second loss between the first embedding set and the labeled dataset, A step of determining the total loss based on the first loss and the second loss, The steps include updating one or more parameters of either the first encoder or the second encoder based on the total loss, The method according to claim 1, including the method described in claim 1.

3. The step of determining the second loss between the first embedding set and the labeled dataset is: The steps include projecting the first embedding set onto a third-dimensional space corresponding to the labeled dataset, A step of using a classifier to map the first embedding set to the target label set included in the labeled dataset, A step of determining the second loss based on the mapping, The method according to claim 2, including the method described in claim 2.

4. The method according to claim 2, wherein the total loss is determined as the sum of the first loss and the second loss, and the first loss and the second loss are weighted differently based on a first weight and a second weight, respectively.

5. The method according to claim 2, wherein one or more parameters of only the first encoder are updated.

6. The step of determining the first loss between the first embedding set and the second embedding set is: A step to verify that the first-dimensional space coincides with the second-dimensional space, The steps include determining the individual embeddings of the first embedding set that correspond to the individual embeddings of the second embedding set, A step of calculating the distance between each embedding in the first embedding set and each embedding in the second embedding set, The method according to claim 2, including the method described in claim 2.

7. The method according to claim 1, wherein the first data type is graph data and the second data type is text data.

8. The method according to claim 7, wherein the first encoder is a graph neural network and the second encoder is the encoder portion of a large-scale language model.

9. The method according to claim 7, wherein the first dataset includes one or more graphs representing the relationships between feature columns of the tabular dataset.

10. The method according to claim 7, wherein the second dataset includes the text data representing the semantic information of the tabular dataset.

11. The method according to claim 1, wherein the tabular dataset includes numerical data and categorical data.

12. The method according to claim 1, wherein a batch of the tabular dataset is converted into the first dataset and the second dataset.

13. One or more non-temporary computer-readable media for storing instructions, wherein, when the instructions are executed by one or more processors, they cause the system to perform an action, and the action is Steps to obtain a tabular dataset, The steps include: using a first converter to convert the tabular dataset into a first dataset having a first data type; The steps include using a second converter to convert the tabular dataset into a second dataset having a second data type different from the first data type, A step of generating a first embedding set in a first-dimensional space based on the first dataset using a first encoder, The steps include using a second encoder to generate a second embedding set in a two-dimensional space based on the second dataset, A step of training one or both of the first encoder and the second encoder based on the first embedding set and the second embedding set, One or more non-temporary computer-readable media, including [the specified text].

14. The step of training one or both of the first encoder and the second encoder based on the first embedding set and the second embedding set is: Steps include obtaining a labeled dataset containing the target label set, A step of determining a first loss between the first embedding set and the second embedding set, The steps include determining a second loss between the first embedding set and the labeled dataset, A step of determining the total loss based on the first loss and the second loss, The steps include updating one or more parameters of either the first encoder or the second encoder based on the total loss, One or more non-temporary computer-readable media according to claim 13, including the following:

15. The step of determining the second loss between the first embedding set and the labeled dataset is: The steps include projecting the first embedding set onto a third-dimensional space corresponding to the labeled dataset, A step of using a classifier to map the first embedding set to the target label set included in the labeled dataset, A step of determining the second loss based on the mapping, One or more non-temporary computer-readable media according to claim 14, including the following:

16. The total loss is determined as the sum of the first loss and the second loss, and the first loss and the second loss are weighted differently based on a first weight and a second weight, respectively, one or more non-temporary computer-readable media according to claim 14.

17. One or more non-temporary computer-readable media according to claim 14, wherein one or more parameters of only the first encoder are updated.

18. The step of determining the first loss between the first embedding set and the second embedding set is: A step to verify that the first-dimensional space coincides with the second-dimensional space, The steps include determining the individual embeddings of the first embedding set that correspond to the individual embeddings of the second embedding set, A step of calculating the distance between each embedding in the first embedding set and each embedding in the second embedding set, One or more non-temporary computer-readable media according to claim 14, including the following:

19. The one or more non-temporary computer-readable media according to claim 13, wherein the first data type is graph data and the second data type is text data.

20. It is a system, One or more processors, One or more non-temporary computer-readable media for storing instructions, wherein, when an instruction is executed, it causes the system to perform an action, and the action is Steps to obtain a tabular dataset, The steps include: using a first converter to convert the tabular dataset into a first dataset having a first data type; The steps include using a second converter to convert the tabular dataset into a second dataset having a second data type different from the first data type, A step of generating a first embedding set in a first-dimensional space based on the first dataset using a first encoder, The steps include using a second encoder to generate a second embedding set in a two-dimensional space based on the second dataset, A step of training one or both of the first encoder and the second encoder based on the first embedding set and the second embedding set, A system that includes this.

Images