A Design and Training Method for a Multimodal Model of Two-Dimensional Tabular Data

Abstract

The present invention discloses a design and training method for a two-dimensional tabular data multi-modal model. First, the present invention designs a two-dimensional tabular data multi-modal model, which includes a table representation module composed of multiple serial bidirectional attention modules; through the processing of the bidirectional attention modules, the table representation word embedding set can interact with other relevant cells to capture the global structured table semantics; then pre-train the table representation module, using a large amount of original table data to train the table representation module so that it can output accurate table representation information; finally, perform alignment training on the table representation module and the large language model decoder, and adopt joint supervised fine-tuning of the table representation module and the large language model decoder. The present invention can realize the co-input of the table modality and the text modality into the large model for generative tasks such as table question answering and data analysis, effectively improving the effect and performance of tasks such as two-dimensional table question answering and code generation.

Description

Technical Field

[0001] The present invention belongs to the fields of big data analysis and artificial intelligence, and relates to a design and training method for a multi-modal model of two-dimensional tabular data. Background Art

[0002] The application of large language models (LLMs) in tabular tasks is becoming increasingly widespread, especially in the use of in-context learning. Converting a table into serialized information that an LLM can understand (usually in formats such as markdown, html, formatted string, json, or SQL schema), and combining it with the context information in the overall tabular task for various common tabular tasks such as code generation, table question answering, table understanding, cell extraction, etc., is the most common research and application mode at present. The prior art has evaluated the performance of LLMs in basic structure understanding tasks (such as data lookup and table conversion) for different table formats (such as JSON, DFLoader, Markdown, etc.) and eight noise operations (such as column name renaming, row and column randomization, etc.) through self-supervised table structure understanding tasks.

[0003] However, due to the uncertainty and complexity of the fields and scales of two-dimensional tabular data, in actual use, there are usually the following two deficiencies: 1) The serialized input form of the table has a great impact on the final performance of the task and it is difficult to achieve stable results in different models and different task scenarios; 2) In the case of wide tables (referring to tables with more than 100 fields) and multi-table scenarios, after converting to serialized information, the context length is too large, and it is often necessary to simplify the table information (such as input truncation, row and column extraction, etc.), which further affects the accuracy and effect of the task. Summary of the Invention

[0004] The purpose of the present invention is to provide a design and training method for a multi-modal model of two-dimensional tabular data in view of the deficiencies of the prior art.

[0005] The present invention includes the following steps:

[0006] Step 1: Design a multi-modal model for two-dimensional tabular data

[0007] The multi-modal model for two-dimensional tabular data includes a table representation module, and the table representation module is composed of a plurality of serial bidirectional attention modules;

[0008] The input of the table representation module comes from an external table, and the output of the table representation module is connected to the table representation word embedding set;

[0009] Through the processing of the bidirectional attention modules in the table representation module, the table representation word embedding set can interact with other relevant cells to capture the global structured table semantics;

[0010] Step 2: Pretrain the table representation module

[0011] Use a large amount of original table data to train the table representation module so that it can output accurate table representation information;

[0012] Step 3: Align and train the table representation module and the large language model decoder

[0013] Perform joint supervised fine-tuning using the table representation module and the large language model decoder. The supervised fine-tuning uses pure table QA data, and the training loss function uses the supervised fine-tuning loss.

[0014] The beneficial effects of the present invention are as follows:

[0015] The present invention designs a two-dimensional table data representation network module based on the Transformer architecture. This module can output the global information representation of the two-dimensional table, unify the input paradigm of the table serialization information, and make the large language model no longer overly sensitive to the input form of the table serialization information; since it directly outputs unified table representation information to the downstream large language model decoder (LLM-decoder), it can effectively alleviate the long text problem of the large language model input in the scenarios of wide tables and multiple tables.

[0016] In addition, the present invention also provides a two-stage training method combining self-supervised basic table pre-training of the table semantic representation module and multi-module joint supervised fine-tuning. Compared with a single large language decoder model (LLM-decoder), it can obtain more stable and excellent two-dimensional table task performance, and significantly improve the feasibility and accuracy of the large language model in the practical application of two-dimensional tables. Description of the Drawings

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0018] Figure 1 It is the overall process steps of the design and training method of the two-dimensional table data multi-modal model provided by the embodiment of the present invention;

[0019] Figure 2 It is the overall design structure diagram of the model provided by the embodiment of the present invention. Detailed Embodiments

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0021] This application designs a neural network model module based on the Transformer architecture. This module can take the table itself as the input and output the overall representation information of the two-dimensional table. In addition, this application also provides a pre-training method for this network module and an alignment training method with the downstream large language model decoder (LLM-Decoder) module, realizing a multi-modal input large model for the table modality and the natural language modality. This model can solve the standardization and normalization of table information input in table tasks, greatly reduce the context length in table task scenarios, solve the long text input problem in table tasks, and at the same time improve the global understanding ability of the large model for table information, thereby improving the accuracy and practicality in table task scenarios.

[0022] As Figure 1 shown, a design and training method for a two-dimensional table data multi-modal model provided by an embodiment of this application is as follows:

[0023] Step 1: Design of the two-dimensional table data multi-modal model.

[0024] Considering the unique structural characteristics of table data, for any table T with m rows and n columns, it can be expressed as T = [c 11 , c 12 , …, c mn , where c ij represents the content of the cell in the i-th row and j-th column.

[0025] Apply a text transformer (cell content) Φ to obtain a compact table representation word embedding set E(T) = [Φ(C 11 ), …, Φ(C mn )] ∈ R m×n×d , where d is the dimension of each cell embedding. These embeddings are output after being processed by a series of serial bidirectional attention modules (2D-Attn) so that they can interact with other relevant cells and capture the global structured table semantics: E′(T) = 2D-Attn(E(T)) ∈ R m×n×d . Since the dimension d itself is controllable and generally set to a relatively small value, for scenarios such as wide tables and multiple tables, the table semantic embedding dimension will be significantly lower than common table serialization formats such as markdown and html.

[0026] Furthermore, in each bidirectional attention module, cross-attention is calculated along rows and columns alternately using Qformer-style bidirectional attention to capture the distribution attributes and relationships of table rows and columns. To maintain the permutation invariance of table rows and columns, position-embedding information is excluded in the bidirectional dimension attention module.

[0027] In addition, to generate a more complete table information representation, the bidirectional attention unit is connected to an adapter that can aggregate cell content information at the table column level, align it with text features, convert a table with m rows into a fixed-length column representation of size k, which matches the embedding dimension of the downstream decoder (Decoder-LLM). In this way, a complete table representation can be output and jointly process the text + table modality input task with the downstream Decoder-LLM.

[0028] Furthermore, the bidirectional attention module (2D-Attn) is a mechanism commonly used in deep learning models to enhance information flow and feature interaction. Its core idea is to optimize the expression of features by introducing a bidirectional attention mechanism, which can better capture the dependency relationships between different parts of the input data.

[0029] Bidirectional attention means that the model can not only focus on the forward information flow from input to output (usually forward attention), but also capture the reverse information flow from output to input (i.e., backward attention). This is different from unidirectional attention (such as the traditional self-attention mechanism). Bidirectional attention allows information to interact in both directions, thus enhancing the ability to capture context information.

[0030] The bidirectional attention module generally consists of the following two parts:

[0031] 1) Forward Attention mechanism: It generates a weighted feature representation based on the input features to enhance important features.

[0032] 2) Backward Attention mechanism: It affects the representation of input features by considering the dependencies of output information, thus providing additional context information.

[0033] This bidirectional interaction method can better capture the potential connections between rows, columns, and cells in table data, improving the feature extraction effect of the model.

[0034] The Qformer is a special type of Transformer model. Besides being used in the scenario of the form modality in this application, it is also widely used in information fusion tasks such as vision-language (VL) modalities. Its core idea is to guide the model to focus on important features through the query mechanism, thereby effectively enabling information interaction and fusion between multiple modalities (such as images and text).

[0035] Furthermore, Qformer introduces a concept of queries, that is, using "query vectors" to selectively focus on input features. These query vectors are used to guide the model to interact between different parts of the input data. In multi-modal tasks, Qformer uses these query vectors to guide the fusion of image and text information, enabling the model to effectively focus on important relationships between different modalities. Compared with traditional Transformer structures, Qformer embeds query vectors in the input sequence, only focuses on information related to the query vectors, reduces redundancy in model calculations, and improves computational efficiency.

[0036] The cross-attention mechanism is extended based on the standard self-attention mechanism. Its main idea is to use the query of one modality to focus on the key and value of another modality, thereby achieving feature interaction between two different input sequences. Its calculation formula is as follows:

[0037] Cross-Attn=softmax((W Q S2)(W k S1) T )W V S1

[0038] where S1 and S2 respectively represent two input sequences. In the scenario of this embodiment, they are respectively the sequence information composed of the rows and columns of the table; W Q represents the query linear transformation matrix corresponding to S1, and W k and W v respectively represent the linear transformation matrices corresponding to the key and value of sequence S2.

[0039] Step 2: Pre-train the table representation module.

[0040] Similar to the pre-training of other multi-modal models, the goal of the pre-training stage is to use a large amount of original table data to train the table encoder so that it can output accurate table representation information.

[0041] Further, the training task is based on the collected open-source data tables. For each table in the batch, random sampling is performed to create two table slices S1 and S2. These two table slices share the same dimension (such as m×n), but use different table cells. Then, a table representation module is used to generate an embedding pool P (embedding-pool) containing the columns in each slice.

[0042] Further, contrastive learning loss is used in the training, where the positive sample pairs are composed of the word embeddings corresponding to the same columns in two snapshots. The definition of the contrastive loss is as follows:

[0043]

[0044] In the formula, represents the loss function, represents the generated temperature parameter, is the embedding pool (embedding-pool or called the embedding set), e T and e + are the word embeddings corresponding to the same columns as the set {e} but from different slices of different tables.

[0045] Step 3: Aligning training of the table representation module and the downstream large language model decoder (LLM-decoder).

[0046] To further strengthen the model's ability to follow instructions and generate in downstream table tasks, joint supervised fine-tuning is performed on the table representation module, the adapter, and the downstream large language model decoder.

[0047] Further, in this stage of supervised fine-tuning, pure table QA data is used, and the training loss function uses the supervised fine-tuning loss (Superwised-finetuning loss).

[0048] The input of the table QA data includes three parts: the representation information of the table, the code generation prompt words, and the human questions; the output is divided into two types: python code and SQL code according to the requirements of the prompt words, and a question summary statement will also be output according to the requirements of the input prompt words.

[0049] The expression of the supervised fine-tuning loss (Superwised-finetuning loss) is as follows:

[0050]

[0051] In the above formula, m represents the number of tokens of the generated content, and |V| represents the vocabulary set, Is the true encoding of each token position in the generated content And the corresponding log probability of the model output Of the product.

[0052] Example:

[0053] This example takes the design and training process of a multimodal model for two-dimensional table data of a certain bank as an example, and its overall implementation process is as follows:

[0054] 1) According to the model structure design as shown Figure 2 Write the model code. In the example, a bidirectional attention unit with a depth of 10 is used, that is, each time the table information is passed into the representation module, the attention in the row direction and column direction is always calculated 10 times respectively, and the embedding dimension d of a single cell is set to 3.

[0055] The adapter uses a fully connected neural network, and its main purpose is to project the table embedding information output by the table information representation network module to the same dimension as the embedding layer of the downstream large model decoder. Specifically, the projection dimension is set to 3584.

[0056] 2) Batch screen and process the basic table data through the following rules:

[0057] 1. Eliminate tables with less than 5 rows and less than 2 columns;

[0058] 2. Delete the anonymous numerical columns in the table, where the anonymous numerical columns refer to the columns without clear business meanings in the fields. Such as columns named a, b, c,.. single-letter columns;

[0059] 3. Eliminate columns with no specific business meanings in the column values, such as columns with pure hash values or base64 strings;

[0060] 4. Delete columns with more than 30% nan values; 5. If the proportion of nan values in the column is less than 30%, use the column mean to fill the missing values in the column.

[0061] At the same time, use the following three methods to enhance or add noise to the original data table:

[0062] 1. Randomly select columns from tables outside this table and splice them into this table;

[0063] 2. Randomly eliminate columns in the table;

[0064] 3. Randomly select cell values in the category columns for contamination processing (such as deleting characters, mean replacement, etc.).

[0065] Finally, keep about 200,000 sheets of basic table training data.

[0066] Pre-training the table representation module model using contrastive learning loss: For each table, obtain two table slices S1 and S2 with the same dimensions respectively. For the cell word embedding sets {e} corresponding to the two slices, if the cell word embedding e from S1 T and the cell word embedding e from S2 + are from the same column in the original table, they form a positive sample pair; otherwise, they form a negative sample pair.

[0067] 3) Use the pre-generated pure table QA data. In this embodiment, 500k table QA data are retained, and the ratio of Python to SQL data is 4:1. Use the supervised fine-tuning loss (cross-entropy loss) to jointly supervise and fine-tune the table representation module model, the adapter, and the downstream large model decoder module.

[0068] In this embodiment, the downstream large model decoder (LLM-decoder) uses Qwen2-instruct-7b as the base model. During the joint supervised fine-tuning process, freeze the weights of the downstream large model decoder and only update the parameters of the adapter and the table representation unit. Additionally, the learning rate lr = 1e-6 is used for training.

[0069] Note that the above is only a preferred embodiment of the present invention and the technical principles applied. Those skilled in the art will understand that the present invention is not limited to the specific embodiments here. Various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, more other equivalent embodiments can be included, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A design and training method for a multi-modal model of two-dimensional tabular data, characterized in that It includes the following steps: Step 1: Design a multi-modal model for two-dimensional table data The multi-modal model for two-dimensional table data includes a table representation module, which is composed of multiple serial bidirectional attention modules; The input of the table representation module comes from an external table, and the output of the table representation module is connected to the table representation word embedding set; Through the processing of the bidirectional attention module in the table representation module, the table representation word embedding set can interact with other relevant cells to capture the global structured table semantics; Step 2: Pre-train the table representation module Use a large amount of original table data to train the table representation module so that it can output accurate table representation information; Step 3: Align and train the table representation module and the large language model decoder Adopt joint supervised fine-tuning of the table representation module and the large language model decoder. The supervised fine-tuning uses pure table QA data, and the training loss function uses the supervised fine-tuning loss; In Step 1, each bidirectional attention module alternately uses Qformer-style bidirectional attention to calculate cross-attention along rows and columns respectively to capture the row and column distribution attributes and row-column relationships of the table. In order to maintain the permutation invariance of the table rows and columns, position embedding information is excluded in the bidirectional attention; Each bidirectional attention module is also connected to an adapter, which is used to generate a more complete table information representation; the adapter aggregates cell content information at the table column level, aligns it with text features, and converts a table with m rows into a fixed-length column representation of size k, which matches the embedding dimension of the large language model decoder; It further includes a text transformer for obtaining a compact set of tabular representation word embeddings E(T)=[Φ(C 11 ),…,Φ(C mn )]∈R m×n×d , where d is the dimension of each cell embedding, and c ij represents the cell content of the i-th row and j-th column, and Φ is the text transformer; In Step 2, the training task is based on the collected open-source data tables. Random sampling is performed on each open-source data table in the batch to create two table slices. These two table slices share the same dimension but use different table cells. Use the table representation module to generate an embedding pool containing the columns in each slice; The training uses contrastive learning loss, where the positive sample pairs are composed of word embeddings corresponding to the same columns in two snapshots; In Step 3, the input of the table QA data includes three parts: the representation information of the table, code generation prompt words, and human questions. The output of the table QA data is divided into two types: python code and SQL code according to the requirements of the prompt words.

2. The design and training method of a two-dimensional table data multi-modal model according to claim 1, characterized in that: It also includes outputting a problem summary statement according to the requirements of the prompt words.

Images