Method for designing and training two-dimensional table data multi-modal model

Abstract

The invention discloses a method for designing and training a two-dimensional table data multi-modal model. According to the method, firstly, a two-dimensional table data multi-modal model is designed, the two-dimensional table data multi-modal model comprises a table representation module, and the table representation module is composed of a plurality of serial bidirectional attention modules; the table representation word embedding set can interact with other related cells through processing of the bidirectional attention module, and global structured table semantics are captured; pre-training a table representation module, and training the table representation module by using a large amount of original table data to enable the table representation module to output accurate table representation information; and finally, carrying out alignment training on the table representation module and the large language model decoder, and carrying out joint supervision fine tuning by adopting the table representation module and the large language model decoder. According to the method, the table modality and the text modality can be jointly input into the large model to carry out table question-answering, data analysis and other generative tasks, and the effect and performance of two-dimensional table question-answering, code generation and other tasks are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of big data analysis and artificial intelligence, and relates to a design and training method of a two-dimensional table data multimodal model. Background Art

[0002] Large language models (LLMs) are increasingly used in table tasks, especially using contextual learning. Converting tables into serialized information that LLMs can understand (usually in formats such as markdown, html, formatted string, json, or SQLschema), combining it with the contextual information of the overall table task, and using it in various common table tasks such as code generation, table question answering, table understanding, cell extraction, etc., is currently the most common research and application mode. The existing technology generates self-supervised table structure understanding tasks and evaluates the performance of LLMs in basic structure understanding tasks (such as data search and table conversion) with different table formats (such as JSON, DFLoader, Markdown, etc.) and eight noise operations (such as column name renaming, row and column randomization, etc.).

[0003] However, due to the uncertainty and complexity of the data fields and scale of two-dimensional tables, there are often the following two shortcomings in actual use: 1) In different models and different task scenarios, the serialization input form of the table has a great impact on the final performance of the task, and the effect is difficult to stabilize; 2) In wide tables (referring to the number of fields exceeding 100) and multi-table scenarios, after conversion into serialized information, the context length is too large, and it is often necessary to simplify the table information (such as input truncation, row and column extraction, etc.), which further affects the accuracy and effect of the task. Summary of the invention

[0004] The purpose of the present invention is to provide a design and training method for a two-dimensional tabular data multimodal model in view of the deficiencies in the prior art.

[0005] The present invention comprises the following steps:

[0006] Step 1: Design a multimodal model for two-dimensional tabular data

[0007] The two-dimensional table data multimodal model includes a table representation module, which is composed of a plurality of serial bidirectional attention modules;

[0008] The input of the table representation module comes from an external table, and the output of the table representation module is connected to the table representation word embedding set;

[0009] The bidirectional attention module in the table representation module enables the table representation word embedding set to interact with other related cells to capture the global structured table semantics;

[0010] Step 2: Pre-training table representation module

[0011] Use a large amount of original table data to train the table representation module so that it can output accurate table representation information;

[0012] Step 3: Align the table representation module and the large language model decoder

[0013] A tabular representation module and a large language model decoder are used for joint supervised fine-tuning. The supervised fine-tuning uses pure tabular QA data, and the training loss function uses supervised fine-tuning loss.

[0014] The beneficial effects of the present invention are:

[0015] The present invention designs a two-dimensional table data representation network module based on the Transformer architecture. The module can output the global information representation of the two-dimensional table, unify the input paradigm of the table serialization information, and make the large language model no longer overly sensitive to the input form of the table serialization information. Since the unified table representation information is directly output to the downstream large language model decoder (LLM-decoder), it can effectively alleviate the long text problem of the large language model input in wide table and multi-table scenarios.

[0016] In addition, the present invention also provides a two-stage training method that combines self-supervised basic table pre-training of the table semantic representation module and multi-module joint supervised fine-tuning. Compared with a single large language decoder model (LLM-decoder), it can obtain more stable and excellent two-dimensional table task performance, and significantly improves the feasibility and accuracy of the large language model in the practical application of two-dimensional tables. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0018] Figure 1 The overall process steps of the design and training method of the two-dimensional table data multimodal model provided by the embodiment of the present invention;

[0019] Figure 2 This is a diagram of the overall design structure of the model provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0021] This application designs a neural network model module based on the Transformer architecture, which can take the table itself as input and output the overall representation information of the two-dimensional table. In addition, this application also provides a pre-training method for the network module and an alignment training method with the downstream large language model decoder (LLM-Decoder) module, realizing a multimodal input large model of table modality and natural language modality. This model can solve the standardization and normalization of table information input under table tasks, greatly reduce the context length in table task scenarios, solve the problem of long text input in table tasks, and at the same time improve the global understanding ability of the large model of table information, thereby improving the accuracy and practicality in table task scenarios.

[0022] like Figure 1 As shown, a design and training method of a two-dimensional table data multimodal model provided in an embodiment of the present application is as follows:

[0023] Step 1: Design of multimodal model for two-dimensional tabular data.

[0024] Considering the unique structural characteristics of tabular data, for any table T with m rows and n columns, it can be expressed as T=[c 11 ,c 12 ,…,c mn ], where c ij Represents the cell content at row i and column j.

[0025] Apply a text transformer (cell content) Φ to obtain a compact table representation word embedding set E(T) = [Φ(C 11 ),…,Φ(C mn )]∈R m×n×d , where d is the dimension of each cell-embedding. These embeddings are processed through a series of bidirectional attention modules (2D-Attn) so that they can interact with other related cells and capture the global structured table semantics: E′(T)=2D-Attn(E(T))∈R m×n×d Since the dimension d itself is controllable and is generally set to a smaller value, for scenarios such as wide tables and multiple tables, the table semantic embedding dimension will be significantly lower than that of common table serialization formats such as markdown and html.

[0026] Furthermore, in each bidirectional attention module, Qformer-style bidirectional attention is used alternately to calculate cross-attention along rows and columns to capture the row-column distribution properties and row-column relationships of the table. In order to maintain the permutation invariance of table rows and columns, position embedding information is excluded in the bidirectional dimension attention module.

[0027] In addition, in order to generate a more complete representation of table information, the bidirectional attention unit of this application is connected to an adapter, which can aggregate cell content information at the table column level and align it with text features, converting a table with m rows into a fixed-length column representation of size k, matching the embedding dimension of the downstream decoder (Decoder-LLM). In this way, a complete table representation can be output and jointly processed with the downstream Decoder-LLM for text + table modality input tasks.

[0028] Furthermore, the bidirectional attention module (2D-Attn) is a mechanism commonly used in deep learning models to enhance information flow and feature interaction. The core idea is to optimize the expression of features by introducing a bidirectional attention mechanism, which can better capture the dependencies between different parts of the input data.

[0029] Bidirectional attention means that the model can not only pay attention to the forward information flow from input to output (usually forward attention), but also capture the reverse information flow from output to input (i.e. reverse attention). This is different from unidirectional attention (such as the traditional self-attention mechanism), which allows information to interact in two directions, thereby enhancing the ability to capture contextual information.

[0030] The bidirectional attention module usually consists of the following two parts:

[0031] 1) Forward Attention: It generates weighted feature representations based on the input features to enhance important features.

[0032] 2) Backward Attention: It affects the representation of input features by considering the dependency of output information, thereby providing additional contextual information.

[0033] This two-way interaction method can better capture the potential connections between rows, columns and cells in tabular data and improve the feature extraction effect of the model.

[0034] The Qformer is a special type of Transformer model. In addition to being used in the scenario of the form modality of this application, it is also widely used in information fusion tasks of modalities such as vision-language (VL). Its core idea is to guide the model to focus on important features through the query mechanism, so as to effectively interact and fuse information between multiple modalities (such as images and texts).

[0035] Furthermore, QFormer introduces the concept of query, which uses "query vectors" to selectively focus on input features. These query vectors are used to guide the model to interact between different parts of the input data. In multimodal tasks, QFormer uses these query vectors to guide the fusion of image and text information, allowing the model to effectively focus on important relationships between different modalities. Compared with the traditional Transformer structure, Qformer embeds query vectors in the input sequence and only focuses on information related to the query vector, reducing the redundancy of the model during calculation and improving computational efficiency.

[0036] The Cross-Attention mechanism is an extension of the standard Self-Attention mechanism. Its main idea is to focus on the key and value of another modality through the query of one modality, thereby realizing the feature interaction between two different input sequences. Its calculation formula is as follows:

[0037] Cross-Attn = softmax((W Q S2)(W k S1) T )W V S1

[0038] Wherein, S1 and S2 represent two input sequences respectively, and in the scenario of this embodiment, they are sequence information composed of rows and columns of a table respectively; Q It represents the query linear transformation matrix corresponding to S1, W k and W v They represent the linear transformation matrices corresponding to the key and value of sequence S2 respectively.

[0039] Step 2: Pre-training of table representation module.

[0040] Similar to the pre-training of other multimodal models, the goal of the pre-training phase is to use a large amount of original tabular data to train the table encoder so that it can output accurate table representation information.

[0041] Furthermore, the training task is based on the collected open source data tables, randomly sampling each table in the batch to create two table slices S1 and S2, which share the same dimensions (such as m×n) but use different table cells. Then, the table representation module is used to generate an embedding pool P (embedding-pool) containing the columns in each slice.

[0042] Furthermore, the training uses contrastive learning loss, where the positive sample pairs consist of word embeddings corresponding to the same columns in two snapshots. The definition of the contrastive loss is as follows:

[0043]

[0044] In the formula, represents the loss function, represents the generated temperature parameter, is the embedding pool (embedding-pool or embedding set), e T and e + are word embeddings of the same columns as set {e} but from different slices of different tables.

[0045] Step 3: Alignment training of the table representation module and the downstream large language model decoder (LLM-decoder).

[0046] In order to further enhance the model’s instruction-following and generation capabilities in downstream table tasks, the table representation module, adapter, and downstream large language model decoder are jointly supervised and fine-tuned.

[0047] Furthermore, in this stage, the supervised fine-tuning uses pure tabular QA data, and the training loss function uses the supervised-finetuning loss (Superwised-finetuning loss).

[0048] The input of the table QA data includes three parts: table representation information, code generation prompt words and human questions; the output is divided into two types: python code and SQL code according to the requirements of the prompt words, and a problem summary statement is also output according to the requirements of the input prompt words.

[0049] The expression of the supervised-finetuning loss is as follows:

[0050]

[0051] In the above formula, m represents the number of tokens of generated content, |V| represents the vocabulary set, Is the actual encoding of each token position in the generated content And its corresponding model output log probability The product of .

[0052] Example:

[0053] This embodiment takes the design and training process of a multimodal model of two-dimensional table data of a bank as an example, and its overall implementation process is as follows:

[0054] 1) According to Figure 2 The model structure design shown in the figure is used to write the model code. In the embodiment, a bidirectional attention unit with a depth of 10 is used, that is, each time the table information is passed to the representation module, the attention in the row direction and the column direction is always calculated 10 times respectively, and the embedding dimension d=3 of a single cell is set.

[0055] The adapter uses a fully connected neural network, whose main purpose is to project the tabular embedding information output by the tabular information representation network module to the same dimension as the downstream large model decoder embedding layer. Specifically, the projection dimension is set to 3584.

[0056] 2) Batch filter and process the basic table data according to the following rules:

[0057] 1. Eliminate tables with less than 5 rows and less than 2 columns;

[0058] 2. Delete anonymous numeric columns in the table. Anonymous numeric columns refer to columns whose fields have no clear business meaning, such as single letter columns named a, b, c, etc.;

[0059] 3. Eliminate columns whose values ​​have no specific business meaning, such as columns that are pure hash values ​​or base64 strings;

[0060] 4. Delete columns with more than 30% nan values; 5. If the proportion of nan values ​​in a column is less than 30%, use the column mean to fill in the missing values ​​in the column.

[0061] At the same time, the following three methods are used to enhance or add noise to the original data table:

[0062] 1. Randomly select columns from tables outside this table and splice them into this table;

[0063] 2. Randomly remove columns from the table;

[0064] 3. Randomly select cell values ​​in the category column for pollution processing (such as deleting characters, replacing the mean, etc.).

[0065] Finally, the basic table training data is kept at about 200,000.

[0066] Use contrastive learning loss to pre-train the table representation module model: For each table, obtain two table slices S1 and S2 of the same dimension. For the cell word embedding set {e} corresponding to the two slices, if the cell word embedding e from S1 T and the cell word embedding e from S2 + If they are from the same column in the original table, then the two constitute a positive sample pair, otherwise they are a positive and negative sample pair.

[0067] 3) Use the pre-generated pure table QA data. In this example, 500k table QA data are retained, of which the ratio of Python to SQL data is 4:1. Use supervised fine-tuning loss (cross entropy loss) to jointly supervise fine-tune the table representation module model, adapter, and downstream large model decoder module.

[0068] In this embodiment, the downstream large model decoder (LLM-decoder) uses Qwen2-instruct-7b as the base model. During the joint supervised fine-tuning process, the weights of the downstream large model decoder are frozen, and only the parameters of the adapter and table representation unit are updated. In addition, the training adopts the learning rate lr=1e-6.

[0069] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A design and training method for a two-dimensional tabular data multimodal model, characterized in that The steps include: Step 1: Design a multimodal model for two-dimensional tabular data The two-dimensional table data multimodal model includes a table representation module, which is composed of a plurality of serial bidirectional attention modules; The input of the table representation module comes from an external table, and the output of the table representation module is connected to the table representation word embedding set; The bidirectional attention module in the table representation module enables the table representation word embedding set to interact with other related cells to capture the global structured table semantics; Step 2: Pre-training table representation module Use a large amount of original table data to train the table representation module so that it can output accurate table representation information; Step 3: Align the table representation module and the large language model decoder A tabular representation module and a large language model decoder are used for joint supervised fine-tuning. The supervised fine-tuning uses pure tabular QA data, and the training loss function uses supervised fine-tuning loss.

2. The method for designing and training a two-dimensional table data multimodal model according to claim 1, characterized in that: A text transformer is also included to obtain a compact set of table representation word embeddings for each cell.

3. A method for designing and training a two-dimensional table data multimodal model according to claim 1 or 2, characterized in that: Also included are adapters for generating more complete representations of tabular information.

4. The method for designing and training a two-dimensional table data multimodal model according to claim 3, characterized in that: In step 1, each bidirectional attention module alternately uses Qformer-style bidirectional attention to calculate cross attention along rows and columns respectively to capture the row and column distribution properties and row-column relationships of the table.

5. The method for designing and training a two-dimensional table data multimodal model according to claim 3, characterized in that: The adapter aggregates cell content information at the table column level and aligns it with text features, converting a table with m rows into a fixed-length column representation of size k, matching the embedding dimension of the large language model decoder.

6. The method for designing and training a two-dimensional table data multimodal model according to claim 1, characterized in that: In step 2, the training task is based on the collected open source data tables. Each open source data table in the batch is randomly sampled to create two table slices. The two table slices share the same dimensions but use different table cells.

7. The method for designing and training a two-dimensional table data multimodal model according to claim 6, characterized in that: During training, a contrastive learning loss is used, where positive pairs consist of word embeddings corresponding to the same columns in two snapshots.

8. The method for designing and training a two-dimensional table data multimodal model according to claim 1, characterized in that: In step three, the input of the table QA data includes three parts: table representation information, code generation prompt words and human questions. The output of the table QA data is divided into two types: python code and SQL code according to the requirements of the prompt words.

9. The method for designing and training a two-dimensional table data multimodal model according to claim 8, characterized in that: The method also includes outputting a problem summary statement according to the requirements of the prompt word.

Images